Green metadata signaling

Abstract

Systems, methods, apparatuses, and computer-readable media for processing video data are disclosed. For example, an apparatus for processing video data may include: at least one memory; and at least one processor coupled to the at least one memory, the at least one processor being configured to: acquire a bitstream; retrieve a subtlety type syntax element associated with the bitstream, the subtlety type syntax element specifying the subtlety type of one or more pictures to which a complexity metric (CM) associated with the bitstream applies; retrieve a time period type syntax element associated with the bitstream, the time period type syntax element indicating an upcoming time period or a set of pictures to which the CM applies; and decode a portion of the bitstream based on the subtlety type syntax element and the time period type syntax element.

Description

Technical Field

[0001] This patent application claims the benefit of U.S. Provisional Application No. 63 / 231,015, filed August 9, 2021, the entire contents of which are incorporated herein by reference and used for all purposes.

[0002] This case typically relates to video processing. For example, the nature of this case concerns improving video decoding technology (e.g., encoding and / or decoding video) for green relay data. Prior Technology

[0003] Digital video capabilities can be integrated into a wide variety of devices, including digital televisions, digital live broadcasting systems, wireless broadcasting systems, personal digital assistants (PDAs), laptops or desktop computers, tablets, e-book readers, digital cameras, digital recording devices, digital media players, video gaming devices, video game consoles, cellular or satellite wireless phones, so-called "smartphones," video conferencing equipment, and video streaming devices. These devices allow for the processing and output of video data for consumption. Digital video data comprises a large amount of data to meet the needs of consumers and video providers. For example, consumers of video data expect the highest quality video with high fidelity, resolution, and playback speed. Therefore, the large amount of video data required to meet these needs places a burden on the communication networks and equipment that process and store the video data. []

[0004] Digital video equipment can implement video decoding technology to compress video data. Video decoding is performed according to one or more video decoding standards or formats. Examples of video decoding standards or formats include Universal Video Decoding (VVC), High-Efficiency Video Decoding (HEVC), Advanced Video Decoding (AVC), MPEG-2 Part 2 Decoding (MPEG stands for Moving Picture Experts Group), and proprietary video codec / formats, such as AOMedia Video 1 (AV1) developed by the Open Media Alliance. Video decoding typically employs prediction methods that utilize redundancy present in video images or sequences (e.g., inter-frame prediction, intra-frame prediction, etc.). One goal of video decoding technology is to compress video data into a form using a lower bit rate while avoiding or minimizing video quality degradation. With the emergence of increasingly sophisticated video services, there is a need for decoding technologies with higher decoding efficiency. [] Summary of the Invention

[0005] This document describes systems and techniques for processing video data. According to at least one example, a method for processing video is provided, comprising: obtaining a bitstream; and retrieving a subtlety type syntax element associated with the bitstream, the subtlety type syntax element specifying the subtlety type to which a complexity metric (CM) associated with the bitstream applies.

[0006] Systems, methods, apparatuses, and computer-readable media for processing video data are disclosed. In one illustrative example, an apparatus for processing video data is provided. The apparatus includes: at least one memory; and at least one processor (e.g., implemented in a circuit) coupled to the at least one memory, the at least one processor being configured to: obtain a bitstream; retrieve a subtlety type syntax element associated with the bitstream, the subtlety type syntax element specifying the subtlety type of one or more pictures to which a complexity metric (CM) associated with the bitstream applies; retrieve a time period type syntax element associated with the bitstream, the time period type syntax element indicating an upcoming time period or a set of pictures to which the CM applies; and decode a portion of the bitstream based on the subtlety type syntax element and the time period type syntax element.

[0007] In another example, a method for processing video data is provided. The method includes: obtaining a bitstream; retrieving a subtlety type syntax element associated with the bitstream, the subtlety type syntax element specifying the subtlety type of one or more pictures to which a complexity metric (CM) associated with the bitstream applies; retrieving a time period type syntax element associated with the bitstream, the time period type syntax element indicating an upcoming time period or a set of pictures to which the CM applies; and decoding a portion of the bitstream based on the subtlety type syntax element and the time period type syntax element.

[0008] In another instance, a non-transitory computer-readable medium is provided with instructions that, when executed by one or more processors, cause one or more processors to: obtain a bitstream; retrieve a subtlety type syntax element associated with the bitstream, the subtlety type syntax element specifying the subtlety type of one or more pictures to which a complexity metric (CM) associated with the bitstream applies; retrieve a time period type syntax element associated with the bitstream, the time period type syntax element indicating an upcoming time period or a set of pictures to which the CM applies; and decode a portion of the bitstream based on the subtlety type syntax element and the time period type syntax element.

[0009] In another example, an apparatus for processing video data is provided. The apparatus includes: components for acquiring a bitstream; components for retrieving a subtlety type syntax element associated with the bitstream, the subtlety type syntax element specifying the subtlety type of one or more pictures to which a complexity metric (CM) associated with the bitstream applies; components for retrieving a time period type syntax element associated with the bitstream, the time period type syntax element indicating an upcoming time period or a set of pictures to which the CM applies; and components for decoding a portion of the bitstream based on the subtlety type syntax element and the time period type syntax element.

[0010] In another example, an apparatus for processing video data is provided. The apparatus includes: at least one memory; and at least one processor (e.g., implemented in a circuit) coupled to the at least one memory, the at least one processor being configured to: acquire video data; generate a subtlety type syntax element for a bitstream, the subtlety type syntax element specifying the subtlety type of one or more pictures to which a complexity metric (CM) associated with the bitstream applies; generate a time period type syntax element associated with the bitstream, the time period type syntax element indicating an upcoming time period or a set of pictures to which the CM applies; generate a bitstream associated with the video data, the bitstream including the subtlety type syntax element and the time period type syntax element; and output the generated bitstream.

[0011] In another example, a method for processing video data is provided. The method includes: acquiring video data; generating a subtlety type syntax element for a bitstream, the subtlety type syntax element specifying the subtlety type of one or more pictures to which a complexity metric (CM) associated with the bitstream applies; generating a time period type syntax element associated with the bitstream, the time period type syntax element indicating an upcoming time period or a set of pictures to which the CM applies; generating a bitstream associated with the video data, the bitstream including the subtlety type syntax element and the time period type syntax element; and outputting the generated bitstream.

[0012] In another instance, a non-transitory computer-readable medium with instructions that, when executed by one or more processors, cause one or more processors to: acquire video data; generate a nuanced type syntax element for a bitstream, the nuanced type syntax element specifying the nuanced type of one or more pictures to which a complexity metric (CM) associated with the bitstream applies; generate a time period type syntax element associated with the bitstream, the time period type syntax element indicating an upcoming time period or a set of pictures to which the CM applies; generate a bitstream associated with the video data, the bitstream including the nuanced type syntax element and the time period type syntax element; and output the generated bitstream.

[0013] In another example, an apparatus for processing video data is provided. The apparatus includes: components for acquiring video data; components for generating a subtlety type syntax element for a bitstream, the subtlety type syntax element specifying the subtlety type of one or more pictures to which a complexity metric (CM) associated with the bitstream applies; components for generating a time period type syntax element associated with the bitstream, the time period type syntax element indicating an upcoming time period or a set of pictures to which the CM applies; components for generating a bitstream associated with the video data, the bitstream including the subtlety type syntax element and the time period type syntax element; and components for outputting the generated bitstream.

[0014] According to at least one other example, an apparatus for processing video data is provided, including at least one memory and one or more processors (e.g., implemented in a circuit) coupled to the memory. The one or more processors are configured to: obtain a bitstream; and retrieve a subtlety type syntax element associated with the bitstream, the subtlety type syntax element specifying the subtlety type to which a complexity metric (CM) associated with the bitstream applies. []

[0015] According to at least one other instance, a non-transitory computer-readable medium is provided that includes instructions which, when executed by one or more processors, cause one or more processors to: obtain a bitstream; and retrieve a subtlety type syntax element associated with the bitstream, the subtlety type syntax element specifying the subtlety type to which a complexity metric (CM) associated with the bitstream applies. []

[0016] According to at least one other example, an apparatus for processing video data is provided, comprising: a component for acquiring a bitstream; and a component for retrieving a subtlety type syntax element associated with the bitstream, the subtlety type syntax element specifying the subtlety type to which a complexity metric (CM) associated with the bitstream applies. []

[0017] In some cases, the value of the subtlety type syntax element specifies that the CM applies to images in bitstreams. []

[0018] In some cases, the value of the subtlety type syntax element specifies that the CM applies to slices of bitstreams. []

[0019] In some cases, the value of the subtlety type syntax element specifies that the CM applies to the tiles of the bitstream. []

[0020] In some cases, the value of the subtlety type syntax element specifies that the CM applies to the sub-image of the bitstream. []

[0021] In some cases, the value of the subtlety type syntax element specifies the CM for use as a scalable layer in a bitstream. []

[0022] In some styles, the value of the subtlety type syntax element specifies the line of the coding tree unit (CTU) to which the CM applies to the bitstream. []

[0023] In some cases, the methods, apparatus, and nontransitory computer-readable media described above may include retrieving a time-type syntax element associated with a bitstream, which specifies the type of the upcoming time-stream to which the CM applies. []

[0024] In some cases, the methods, apparatuses and nontransitory computer-readable media described above may include retrieving a picture-level CM syntax structure associated with a bitstream, the picture-level CM syntax structure specifying a complexity measure of one or more pictures within a time period. []

[0025] In some forms, the methods, apparatus, and non-transitory computer-readable media described above may include retrieving a fine-grained CM syntax structure associated with a bitstream, the fine-grained CM syntax structure specifying a fine-grained complexity measure of one or more entities over a period of time. In some forms, the one or more entities include at least one of slices, tiles, sub-pictures, and layers. []

[0026] In some cases, the methods, apparatus, and nontransitory computer-readable media described above may include retrieving sub-picture syntax elements associated with a bitstream, and when the time spans multiple pictures, signaling in the CM to notify the sub-picture syntax element of the sub-picture identifier (ID). []

[0027] In some cases, the methods, apparatus, and nontransitory computer-readable media described above may include retrieving the CTB number syntax element associated with the bitstream, and when the detail type is equal to a slice or tile and the time span spans multiple images, the CTB number syntax element may be signaled in the CM to indicate the total number of CTB number blocks in a time period. []

[0028] In some forms, the methods, apparatuses and nontransitory computer-readable media described above may include retrieving an average decode tree block (CTB) number syntax element associated with a bitstream, the average CTB number syntax element indicating the average number of CTBs or 4×4 blocks per image per minute. []

[0029] In some cases, when an in-frame decoded block is available in at least a portion of the bitstream, a signal delivery associated with at least a portion of the bitstream notifies the in-frame decoded block statistics. []

[0030] In some cases, when an inter-frame decoder block is available in at least a portion of the bitstream, signal transmission associated with at least a portion of the bitstream notifies the inter-frame decoder block statistics. []

[0031] In some embodiments, the methods, apparatus, and non-transitory computer-readable media described above may include retrieving one or more quality recovery metrics associated with one or more segments of a bitstream. In some embodiments, the one or more segments of the bitstream include at least one of slices, tiles, and sub-pictures. []

[0032] In some cases, the methods, apparatus, and nontransitory computer-readable media described above may include: receiving supplemental enhancement information (SEI) messages; and retrieving subtle type syntax elements from the SEI messages. []

[0033] In some cases, the methods, apparatuses, and non-transitory computer-readable media described above may include determining the operating frequency of the apparatus based on a CM associated with the bitstream. []

[0034] In some cases, the device includes a decoder. []

[0035] According to at least one other instance, a method for processing video is provided, comprising: acquiring video data; generating a bitstream associated with the video data; and generating a subtle type syntax element for the bitstream, the subtle type syntax element specifying the subtle type to which a complexity metric (CM) associated with the bitstream applies. []

[0036] According to at least one other example, an apparatus for processing video data is provided, including at least one memory and one or more processors (e.g., implemented in a circuit) coupled to the memory. The one or more processors are configured to: acquire video data; generate a bitstream associated with the video data; and generate a subtle type syntax element for the bitstream, the subtle type syntax element specifying the subtle type to which a complexity metric (CM) associated with the bitstream applies. []

[0037] According to at least one other instance, a non-transitory computer-readable medium is provided that includes instructions which, when executed by one or more processors, cause one or more processors to: acquire video data; generate a bitstream associated with the video data; and generate a subtle type syntax element for the bitstream, the subtle type syntax element specifying the subtle type to which a complexity metric (CM) associated with the bitstream applies. []

[0038] According to at least one other example, an apparatus for processing video data is provided, comprising: a component for acquiring video data; a component for generating a bitstream associated with the video data; and a component for generating a subtle type syntax element for the bitstream, the subtle type syntax element specifying the subtle type to which a complexity metric (CM) associated with the bitstream applies. []

[0039] In some cases, the value of the subtlety type syntax element specifies that the CM applies to images in bitstreams. []

[0040] In some cases, the value of the subtlety type syntax element specifies that the CM applies to slices of bitstream streams. []

[0041] In some cases, the value of the subtlety type syntax element specifies the tiles that the CM applies to in a bitstream. []

[0042] In some cases, the value of the subtlety type syntax element specifies that the CM applies to the sub-image of the bitstream. []

[0043] In some cases, the value of the subtlety type syntax element specifies the CM applicable to the scalable layer of the bitstream. []

[0044] In some styles, the value of the subtlety type syntax element specifies the line of the decode tree unit (CTU) that the CM applies to in the bitstream. []

[0045] In some cases, the methods, apparatuses and nontransitory computer-readable media described above may include a time-type syntax element that specifies the type of upcoming time period to which a CM applies for generating a bitstream. []

[0046] In some cases, the methods, apparatus, and nontransitory computer-readable media described above may include a picture-level CM syntax structure that generates a complexity measure of one or more images over a specified time period for a bitstream. []

[0047] In some forms, the methods, apparatus, and non-transitory computer-readable media described above may include a fine-level CM syntax structure that generates a fine-level complexity measure for one or more entities over a specified time period for a bitstream. In some forms, the one or more entities include at least one of slices, tiles, sub-pictures, and layers.

[0048] In some cases, the methods, apparatus, and nontransitory computer-readable media described above may include generating sub-image syntax elements for a bitstream, and when the time spans multiple images, signaling notifications in the CM to indicate sub-image identifiers (IDs) for the sub-image syntax elements.

[0049] In some forms, the methods, apparatus, and nontransitory computer-readable media described above may include a decode tree block (CTB) number syntax element that generates a bit stream, and when the detail type is equal to a slice or tile and the time span spans multiple images, the decode tree block (CTB) number syntax element can be signaled in the CM to indicate the total number of decode tree blocks within a time period.

[0050] In some forms, the methods, apparatuses and nontransitory computer-readable media described above may include generating an average decoded tree block (CTB) number syntax element for a bitstream, the average decoded tree block (CTB) number syntax element indicating the average number of CTBs or 4×4 blocks per image per minute.

[0051] In some cases, when an in-frame decoded block is available in at least a portion of the bitstream, a signal delivery associated with at least a portion of the bitstream notifies the in-frame decoded block statistics.

[0052] In some cases, when an inter-frame decoder block is available in at least a portion of the bitstream, signal transmission associated with at least a portion of the bitstream notifies the inter-frame decoder block statistics.

[0053] In some cases, the methods, apparatus, and nontransitory computer-readable media described above may include generating one or more quality recovery measures associated with one or more segments of the bitstream.

[0054] In some formats, one or more micro-segments of a bitstream include at least one of slices, tiles, and sub-pictures.

[0055] In some cases, the methods, apparatus and nontransitory computer-readable media described above may include: generating supplemental enhancement information (SEI) messages; and including subtle type syntax elements in the SEI messages.

[0056] In some cases, the methods, apparatus, and non-transitory computer-readable media described above may include storage bitstreams.

[0057] In some cases, the methods, apparatus, and non-transitory computer-readable media described above may include transmitting bit streams.

[0058] In some cases, the device includes an encoder.

[0059] In some embodiments, the device is, is part of, and / or includes mobile devices (e.g., mobile phones or so-called "smartphones" or other mobile devices), wearable devices, extended reality devices (e.g., virtual reality (VR) devices, augmented reality (AR) devices, or mixed reality (MR) devices), cameras, personal computers, laptops, server computers, vehicles or vehicle computing devices or components, robotic devices or systems, televisions, or other devices. In some embodiments, the device includes a camera or multiple cameras for capturing one or more images. In some embodiments, the device includes a display for displaying one or more images, notifications, and / or other displayable data. In some embodiments, the device may include one or more sensors (e.g., one or more inertial measurement units (IMUs), such as one or more gyroscopes, one or more accelerometers, any combination thereof, and / or other sensors). []

[0060] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used alone to determine the scope of the claimed subject matter. The subject matter should be understood by referring to the appropriate portions of the entire specification of this patent, any or all of the accompanying drawings, and each claim. []

[0061] The above and other features and embodiments will become more apparent when the following description, claims, and drawings are consulted. [] Simple Explanation of the Diagram

[0062] The following description, with reference to the accompanying drawings, details an illustrative example of this case:

[0063] Figure 1 is a block diagram illustrating examples of encoding and decoding devices according to some examples of this case;

[0064] Figure 2 is an example of a graph used to illustrate the level of detail complexity measurement of video images;

[0065] Figure 3 is a flowchart illustrating the technology used for decoding encoded video according to the present case.

[0066] Figure 4 is a flowchart illustrating the technology used for encoding video according to the present case.

[0067] Figure 5 is a block diagram illustrating an example video decoding device based on some examples of this case; and

[0068] Figure 6 is a block diagram illustrating an example video encoding device based on some examples of this case. Implementation

[0069] Certain aspects and embodiments of this invention are provided below. Some of these aspects and embodiments can be applied independently, and some can be combined, as will be apparent to those skilled in the art to which this invention pertains. In the following description, specific details are set forth for purposes of explanation in order to provide a thorough understanding of the embodiments of this invention. However, it will be apparent that various embodiments can be practiced without these specific details. The accompanying drawings and descriptions are not intended to be limiting.

[0070] The following description provides exemplary embodiments only and is not intended to limit the scope, applicability, or configuration of this invention. Rather, the following description of exemplary embodiments will provide an enabling description for those skilled in the art to which this invention pertains to implement the exemplary embodiments. It should be understood that various changes can be made to the function and arrangement of the elements without departing from the spirit and scope of the application as described in the appended claims.

[0071] Video decoding devices implement video compression techniques to efficiently encode and decode video data. Video compression techniques may include applying different prediction modes, including spatial prediction (e.g., intra-frame prediction or intra-frame prediction), temporal prediction (e.g., inter-frame prediction or inter-frame prediction), inter-layer prediction (across different layers of video data), and / or other prediction techniques to reduce or remove inherent redundancy in the video sequence. A video transcoder can segment each frame of the original video sequence into rectangular regions called video blocks or decoding units (described in more detail below). These video blocks can be encoded using specific prediction modes.

[0072] Video blocks can be divided into one or more smaller blocks in one or more ways. Blocks may include decode tree blocks, prediction blocks, transform blocks, or other suitable blocks. Unless otherwise specified, the reference to "block" generally refers to such video blocks (e.g., decode tree blocks, decode blocks, prediction blocks, transform blocks, or other suitable blocks or sub-blocks, as will be understood by one of ordinary skill). Furthermore, each of these blocks may be interchangeably referred to herein as a "unit" (e.g., decode tree unit (CTU), decode unit, prediction unit (PU), transform unit (TU), etc.). In some cases, a unit may refer to a decoded logic unit encoded in a bitstream, while a block may refer to a portion of the target video frame buffer that the program is targeting.

[0073] For inter-frame prediction modes, the video transcoder searches for blocks similar to those encoded in a frame (or image) located at another time position; this frame (or image) is called a reference frame or reference image. The video transcoder can limit the search to a specific spatial displacement from the block to be encoded. A two-dimensional (2D) motion vector, including horizontal and vertical displacement components, can be used to locate the best match. For intra-frame prediction modes, the video transcoder can form prediction blocks using spatial prediction techniques based on data from previously encoded adjacent blocks within the same image.

[0074] Video transcoders can determine prediction error. For example, prediction can be determined as the difference between primitive values ​​in the encoded block and primitive values ​​in the predicted block. Prediction error can also be called residual. Video transcoders can also apply a transform (e.g., Discrete Cosine Transform (DCT) or other suitable transform) to the prediction error to produce transform coefficients. After the transform, the video transcoder can quantize the transform coefficients. The quantized transform coefficients and motion vectors can be represented using syntax elements and, together with control information, form a decoded representation of the video sequence. In some instances, the video transcoder can entropy decode the syntax elements, further reducing the number of bits required for its representation.

[0075] A video decoder can use the syntax elements and control information discussed above to construct prediction data (e.g., prediction blocks) for decoding the current frame. For example, a video decoder can add prediction blocks and compress prediction errors. The video decoder can determine the compressed prediction error by using a weighted transform basis function with quantization coefficients. The difference between the reconstructed frame and the original frame is called the reconstruction error.

[0076] The Energy-Saving Media Consumption (Green Relay Data or Green MPEG) standard, with international standard number ISO / IEC 23001-11 (the entire contents of which are incorporated herein by reference and used for all purposes), specifies green relay data to promote reduced energy use during media consumption. Green relay data for energy-saving decoding specifies two sets of information: Complexity Metric (CM) relay data and Decoding Operation Reduction Request (DOR-Req) relay data. For example, a decoder can use CM relay data to change the processor's operating frequency, thereby reducing decoder power consumption. In an illustrative example, in a peer-to-peer video conferencing application, a remote encoder (which can generate a bitstream of encoded data) can receive DOR-Req relay data and use it to modify the decoding complexity of the bitstream, thereby reducing the power consumption of the local decoder. By signaling the decoding complexity of the bitstream, the local decoder can estimate the amount of power required to decode the bitstream and potentially adjust the bitstream based on, for example, the amount of battery power remaining after requesting a bitstream with less (or more) complexity. In some cases, Supplemental Enhancement Information (SEI) messages can be used to signal-notify green relay data in bitstreams (e.g., AVC, HEVC, VVC, AV1, or other streams).

[0077] The second edition of ISO / IEC 23001-11 specifies green relay data for AVC and HEVC. The third edition of the green MPEG (MPEG MDS20584_WG03_N00330) working draft proposes new green relay data supporting VVC encoder-decoder (codec) and specifies CMs at various nuances. The syntax structure can be improved to support more nuance types across various time periods. Furthermore, using a single type of signal to transmit CMs indicating slice, tile, sub-picture, or layer nuances may present problems. In some cases, such as for VVC, the encoder can divide the encoded video's picture (e.g., frame) into one or more parts, such as slices, tiles, sub-pictures, layers, etc. For example, a picture can be divided into one or more tiles, and each tile can be divided into one or more blocks. A slice can include multiple tiles or multiple blocks within a tile. A sub-picture can be one or more complete rectangular slices, each rectangular slice covering a rectangular area of ​​the picture. Sub-pictures can be decoded, or may be decoded independently of other sub-pictures of the same picture.

[0078] Currently, decoders such as those for AVC / HEVC can use multiple slices and tiles to identify whether computational detail (CM) is calculated on slices or tiles. For example, identifying CM granularity becomes complex when the number of slices equals the number of tiles. Furthermore, AVC and HEVC do not support sub-image granularity. Defining different types of slice granularity and tile granularity, as well as defining sub-image and layer granularity, would be beneficial.

[0079] VVC allows subpicks to be replaced with different subpicks within a Layer Decoded Video Sequence (CLVS). A CLVS can be a hierarchical set of CLVSs. In some cases, it is necessary to use subpick identifiers (IDs) to map CMs to signal delivery for specific subpicks.

[0080] VVC also allows changing the resolution within CLVS. In some cases, parsing each slice header to export the total number of decoded blocks over a period of time to decode the normalized coding statistics for each slice or tile can be complex. Slice headers can be included in slices, and slice headers can convey information about associated slices. Information applicable to all slices of a picture can be transmitted in the picture header. Syntax elements indicating the total CTB will help simplify derivation.

[0081] Currently, when all blocks are decoded within a frame, CM provides in-frame decoded block statistics. P and B slices may have more in-frame decoded blocks than inter-frame decoded blocks, or P or B images may have more in-frame decoded blocks than inter-frame decoded blocks. Therefore, CM may not accurately represent complexity. In-frame decoded blocks refer to blocks predicted based on another block within the same image, while inter-frame decoded blocks refer to blocks predicted based on another block from a different image. I slices refer to slices that include in-frame decoded blocks but not inter-frame decoded blocks. P and B slices can include both in-frame decoded blocks and inter-frame decoded blocks.

[0082] Furthermore, in VVC, quality metrics can be applied to individual sub-images instead of the entire image.

[0083] This document describes systems, apparatus, methods, and computer-readable media (collectively, the “Systems and Technologies”) for providing enhanced green relay data transmission (such as for improving complexity metric (CM) signal transmission). For example, in some cases, granularity type indicators (e.g., granularity type syntax elements, such as granularity_type) are provided to support various granularities, such as slices, tiles, subpictures, expandable layers, and / or other granularities. In some instances, the semantics of period type syntax elements (e.g., period_type) are modified.

[0084] In some cases, systems and techniques provide improved complexity metric (CM) signaling. For example, the systems and techniques described herein provide video decoders (e.g., video transcoders, video transcoders, or combined video transcoder-decoders) with the ability to specify CM values ​​applicable to a portion of a video (such as slices, tiles, sub-pictures, and / or layers) for multiple pictures of a video. For example, as previously mentioned, sub-pictures can be defined for an encoded video. A sub-picture includes a portion of an image, such as the upper right corner of the image. A CM can be specified for a sub-picture, where the CM is different from at least one other CM specified for a slice (or other portion) of the image. The CM value associated with a sub-picture can be defined once for multiple pictures (e.g., for 30 pictures from a first picture). The CM can be provided as part of the relay data included in the encoded video. Allowing a single CM value to be specified for a sub-picture spanning multiple frames helps reduce the size of the relay data in the encoded video, while allowing for increased flexibility and granularity in defining the CM for a portion of an image.

[0085] In some cases, changes in CM signal delivery associated with resolution changes are provided. In others, changes in CM signal delivery with respect to in-frame decoded block statistics are provided. In still others, sub-image quality metrics are provided.

[0086] The systems and techniques described herein can be applied to any existing video decoder, such as Universal Video Decoder (VVC), High-Efficiency Video Decoder (HEVC), Advanced Video Decoder (AVC), VP9, ​​AV1 format / decoder and / or other video decoding standards, decoders, formats, etc. that are under development or will be developed.

[0087] Figure 1 is a block diagram illustrating an example of a system 100 including an encoding device 104 and a decoding device 112. The encoding device 104 may be part of a source device, and the decoding device 112 may be part of a receiving device. The source device and / or receiving device may include electronic devices such as mobile or landline handsets (e.g., smartphones, cellular phones, etc.), desktop computers, laptops or notebook computers, tablets, set-top boxes, televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, Internet Protocol (IP) cameras, or any other suitable electronic devices. In some instances, the source device and receiving device may include one or more wireless transceivers for wireless communication. The decoding techniques described herein are applicable to video decoding in a variety of multimedia applications, including streaming video transmission (e.g., via the Internet), television broadcasting or transmission, encoding of digital video stored on data storage media, decoding of digital video stored on data storage media, or other applications. As used herein, the term decoding may refer to encoding and / or decoding. In some instances, system 100 may support one-way or two-way video transmission to support applications such as video conferencing, video streaming, video replay, video broadcasting, gaming, and / or video telephony.

[0088] Encoding device 104 (or encoder) can be used to encode video data using video decoding standards, formats, decoders, or protocols to produce an encoded video bitstream. Examples of video decoding standards and formats / decoders include ITU-T H.261, ISO / IEC MPEG-1 Visual, ITU-T H.262 or ISO / IEC MPEG-2 Visual, ITU-T H.263, ISO / IEC MPEG-4 Visual, ITU-T H.264 (also known as ISO / IEC MPEG-4 AVC), including its Extensible Video Decoding (SVC) and Multi-View Video Decoding (MVC) extensions, High-Efficiency Video Decoding (HEVC) or ITU-T H.265, and Universal Video Decoding (VVC) or ITU-T H.266. Various extensions of HEVC exist for multi-layer video decoding, including range and screen content decoding extensions, 3D video decoding (3D-HEVC), multi-view extensions (MV-HEVC), and scalable extensions (SHVC). HEVC and its extensions were developed by the Joint Coordinating Team for Video Decoding (JCT-VC), the Joint Coordinating Team for the Development of 3D Video Decoding Extensions of the ITU-T Video Decoding Experts Group (VCEG) (JCT-3V), and the ISO / IEC Cinema Experts Group (MPEG). VP9, ​​AOMedia Video 1 (AV1) developed by the Open Media Consortium (AOMedia), and Basic Video Decoding (EVC) are other video decoding standards to which the technologies described herein can be applied.

[0089] The techniques described herein can be applied to any existing video decoder (e.g., High Efficiency Video Decoder (HEVC), Advanced Video Decoder (AVC), or other suitable existing video decoders), and / or can be efficient decoding tools for any video decoding standard under development and / or future video decoding standards (e.g., VVC and / or other video decoding standards under development or to be developed). For example, the examples described herein can be performed using video decoders such as VVC, HEVC, AVC, and / or their extensions. However, the techniques and systems described herein can also be applied to other decoding standards, decoders, or formats, such as MPEG, JPEG (or other decoding standards for still images), VP9, ​​AV1, their extensions, or other suitable decoding standards that are already available or not yet available or under development. For example, in some instances, encoding device 104 and / or decoding device 112 can operate according to proprietary video decoders / formats, such as AV1, AVI extensions and / or subsequent versions of AV1 (e.g., AV2), or other proprietary formats or industry standards. Therefore, although the techniques and systems described herein may be described with reference to a specific video decoding standard, those skilled in the art to which this invention pertains will understand that the description should not be construed as applicable only to that particular standard.

[0090] Referring to FIG1, video source 102 can provide video data to encoding device 104. Video source 102 may be part of a source device or part of a device other than a source device. Video source 102 may include video capturing devices (e.g., cameras, video phones, videophones, etc.), video archives containing stored video, video servers or content providers providing video data, video feed interfaces receiving video from video servers or content providers, computer graphics systems for generating computer graphics video data, combinations of these sources, or any other suitable video source.

[0091] Video data from video source 102 may include one or more input pictures or frames. A picture or frame is a still image and, in some cases, is part of the video. In some instances, the data from video source 102 may be a still image that is not part of the video. In HEVC, VVC, and other video decoding specifications, a video sequence may include a series of pictures. A picture may include three sample arrays, denoted as SL, SCb, and SCr. SL is a two-dimensional array of luminance samples, SCb is a two-dimensional array of Cb chrominance samples, and SCr is a two-dimensional array of Cr chrominance samples. Chrominance samples may also be referred to herein as "chroma" samples. A primitive may reference all three components (luminance and chrominance samples) at a given location in the picture array. In other instances, the picture may be monochrome and may include only an array of luminance samples; in this case, the terms primitive and sample may be used interchangeably. Regarding the example technique described herein for illustrative purposes of referencing a single sample, the same technique can be applied to primitives (e.g., all three sample components at a given location in the picture array). Regarding the example technique described herein for illustrative purposes, which is based on reference primitives (e.g., all three sample components at a given location in an image array), the same technique can be applied to a single sample.

[0092] The encoder engine 106 (or encoder) of the encoding device 104 encodes the video data to produce an encoded video bitstream. In some instances, the encoded video bitstream (or "video bitstream" or "bitstream") is a series of one or more decoded video sequences. The decoded video sequence (CVS) comprises a series of access units (AUs), starting with an AU that has a random access point picture in the base layer and has specific attributes, up to and not including the next AU that has a random access point picture in the base layer and has specific attributes. For example, a specific attribute of the random access point picture that initiates the CVS may include a RASL flag equal to 1 (e.g., NoRaslOutputFlag). Otherwise, a random access point picture (RASL flag equal to 0) will not initiate the CVS. An access unit (AU) includes one or more decoded pictures and control information corresponding to the decoded pictures that share the same output time. Decoded slices of pictures are encapsulated at the bitstream level into data units called Network Abstraction Layer (NAL) units. For example, an HEVC video bitstream may include one or more CVSs that include NAL units. Each NAL unit has a NAL unit header. In one instance, the header is a byte for H.264 / AVC (except for multi-layer extensions) and two bytes for HEVC. The syntax elements in the NAL unit header use specified bits, making them visible to all types of systems and transport layers, such as transport streams, Real-Time Transport (RTP) protocols, file formats, etc.

[0093] The HEVC standard contains two types of NAL units: Video Decoding Layer (VCL) NAL units and non-VCL NAL units. VCL NAL units contain decoded image data that forms the decoded video bitstream. For example, a VCL NAL unit contains a sequence of bits that form the decoded video bitstream. A VCL NAL unit may include a slice or fragment of the decoded image data (as described below), while non-VCL NAL units contain control information associated with one or more decoded images. In some cases, NAL units may be referred to as packets. A HEVC AU includes VCL NAL units containing decoded image data and non-VCL NAL units (if any) corresponding to the decoded image data. Among other information, non-VCL NAL units may also contain parameter sets with high-level information associated with the decoded video bitstream. For example, parameter sets may include Video Parameter Sets (VPS), Sequence Parameter Sets (SPS), and Picture Parameter Sets (PPS). In some cases, each slice or other portion of the bitstream can reference a single active PPS, SPS, and / or VPS to allow the decoding device 112 to access information that can be used to decode the slice or other portion of the bitstream.

[0094] NAL units may contain bit sequences that form a decoded representation of video data (e.g., an encoded video bitstream, a CVS of a bitstream, etc.), such as a decoded representation of an image in a video. Encoder engine 106 generates a decoded representation of an image by dividing each image into multiple slices. Slices are independent of other slices, so information in a slice does not depend on data from other slices in the same image during decoding. A slice includes one or more slice fragments, including independent slice fragments and one or more dependent slice fragments (if any) that depend on previous slice fragments.

[0095] In HEVC, slices are divided into code tree blocks (CTBs) of luma and chroma samples. The CTB of luma samples and one or more CTBs of chroma samples, along with the syntax for those samples, are called code tree units (CTUs). CTUs can also be called "tree blocks" or "maximum decoding units" (LCUs). The CTU is the basic processing unit in HEVC encoding. A CTU can be divided into multiple code tree units (CUs) of different sizes. Each CU contains an array of luma and chroma samples called a code tree block (CB).

[0096] Luminance and chrominance CBs can be further subdivided into prediction blocks (PBs). A PB is a block of samples of the luminance or chrominance components that uses the same motion parameters for inter-frame prediction or intra-block copy (IBC) prediction (when available or enabled). The luminance PB and one or more chrominance PBs, along with their associated syntax, form a prediction unit (PU). For inter-frame prediction, a set of motion parameters (e.g., one or more motion vectors, reference indices, etc.) is passed in the bitstream for each PU signal and used for inter-frame prediction of the luminance PB and one or more chrominance PBs. Motion parameters can also be referred to as motion information. CBs can also be subdivided into one or more transform blocks (TBs). A TB represents a square block of samples of the color components that has been decoded by applying a residual transform (e.g., in some cases, the same two-dimensional transform). A transform unit (TU) represents the TB of the luminance and chrominance samples and the corresponding syntax elements. Transform decoding is described in more detail below.

[0097] The size of the CU corresponds to the size of the decoding mode and can be square. For example, the size of the CU can be 8×8 samples, 16×16 samples, 32×32 samples, 64×64 samples, or any other suitable size up to the size of the corresponding CTU. The phrase "N×N" is used herein to represent the primitive size of a video block in both vertical and horizontal dimensions (e.g., 8 primitives × 8 primitives). Primitives in a block can be arranged in rows and columns. In some implementations, a block may not have the same number of primitives in the horizontal direction as it does in the vertical direction. For example, the syntax data associated with the CU may describe dividing the CU into one or more PUs. The partitioning mode can differ between CUs that are intra-frame predictive mode coded and inter-frame predictive mode coded. PUs can be divided into non-squares. For example, the syntax data associated with the CU may also describe dividing the CU into one or more TUs according to the CTU. The shape of the TU can be square or non-square.

[0098] According to the HEVC standard, transformations can be performed using Transform Units (TUs). TUs may vary depending on the CU. The size of the TU can be determined based on the size of the PU within a given CU. The TU may be the same size as or smaller than the PU. In some instances, a quadtree structure called a Residual Quadtree (RQT) can be used to subdivide the residual samples corresponding to the CU into smaller units. Leaf nodes of the RQT can correspond to TUs. The primitive differences associated with the TU can be transformed to produce transform coefficients. These transform coefficients can then be quantized by the encoder engine 106.

[0099] Once the video data is segmented into Units (CUs), the encoder engine 106 predicts each Unit (PU) using a prediction mode. The prediction unit or block is then subtracted from the original video data to obtain a residual (described below). For each CU, the prediction mode can be communicated via syntax data within the bitstream signaling. Prediction modes can include intra-frame prediction (or intra-picture prediction) or inter-frame prediction (or inter-picture prediction). Intra-frame prediction utilizes the correlation between spatially adjacent samples within a picture. For example, using intra-frame prediction, each PU is predicted from adjacent image data in the same picture, such as using DC prediction to find the average value of the PU, using planar prediction to fit a planar surface to the PU, using orientation prediction to extrapolate from adjacent data, or using any other suitable type of prediction. Inter-frame prediction uses the temporal correlation between pictures to derive motion-compensated predictions for image sample blocks. For example, using inter-frame prediction, each PU is predicted using motion-compensated predictions from image data from one or more reference pictures (in output order before or after the current picture). For example, a decision can be made at the CU level whether to use inter-picture prediction or intra-picture prediction to decode a picture region.

[0100] Encoder engine 106 and decoder engine 116 (described in more detail below) can be configured to operate according to VVC. According to VVC, the video decoder (e.g., encoder engine 106 and / or decoder engine 116) segments the image into a plurality of decoder tree units (CTUs) (where the CTB of luminance samples and one or more CTBs of chrominance samples, along with the syntax used for the samples, are referred to as CTUs). The video decoder can segment CTUs according to a tree structure (e.g., a quadtree-binary tree (QTBT) structure or a multi-type tree (MTT) structure). The QTBT structure eliminates the concept of multiple partition types, such as the separation of CU, PU, ​​and TU in HEVC. The QTBT structure comprises two levels: a first level comprising segmentation according to a quadtree and a second level comprising segmentation according to a binary tree. The root node of the QTBT structure corresponds to a CTU. The leaf nodes of the binary tree correspond to decoder units (CUs).

[0101] In the MTT partitioning structure, blocks can be partitioned using quadtree partitioning, binary tree partitioning, and one or more types of ternary tree partitioning. A ternary tree partition is a partition in which a block is divided into three sub-blocks. In some instances, a ternary tree partition divides a block into three sub-blocks without partitioning the initial block via a center. The partitioning types in MTT (such as ternary, binary, and ternary trees) can be symmetric or asymmetric.

[0102] When operating according to the AV1 decoder, the video transcoder 200 and video decoder 300 can be configured to decode video data in blocks. In AV1, the largest decoded block that can be processed is called a superblock. In AV1, a superblock can be 128×128 luminance samples or 64×64 luminance samples. However, in subsequent video decoding formats (e.g., AV2), superblocks can be defined by different (e.g., larger) luminance sample sizes. In some instances, the superblock is the top level of a block quadtree. The video transcoder 200 can further divide the superblock into smaller decoded blocks. The video transcoder 200 can divide the superblock and other decoded blocks into smaller blocks using square or non-square partitioning. Non-square blocks can include N / 2×N, N×N / 2, N / 4×N, and N×N / 4 blocks. The video transcoder 200 and video decoder 300 can perform separate prediction and transform processing on each decoded block.

[0103] AV1 also defines video data tiles. A tile is a rectangular array of superblocks that can be decoded independently of other tiles. That is, the video transcoder 200 and the video decoder 300 can encode and decode the decoding blocks within a tile separately, without using video data from other tiles. However, the video transcoder 200 and the video decoder 300 can perform filtering across tile boundaries. The size of the tiles can be uniform or non-uniform. Tile-based decoding enables parallel processing and / or multithreading of the encoder and decoder implementations.

[0104] In some instances, a video decoder may use a single QTBT or MTT structure to represent each of the luminance and chrominance components, while in other instances, a video decoder may use two or more QTBT or MTT structures, such as one QTBT or MTT structure for the luminance component and another QTBT or MTT structure for the two chrominance components (or two QTBT and / or MTT structures for their respective chrominance components).

[0105] The video decoder can be configured to use quadtree partitioning, QTBT partitioning, MTT partitioning, superblock partitioning, or other partitioning structures.

[0106] In some instances, one or more slices of an image are assigned slice types. Slice types include intra-frame decoded slices (I-slices), inter-frame decoded P-slices, and inter-frame decoded B-slices. An I-slice (intra-frame decoded frame, independently decodable) is a slice of an image that is decoded solely by intra-frame prediction and is therefore independently decodable because an I-slice only requires intra-frame data to predict any prediction unit or prediction block of that slice. A P-slice (one-way prediction frame) is a slice of an image that can be decoded using both intra-frame prediction and one-way inter-frame prediction. Each prediction unit or prediction block within a P-slice is decoded using either intra-frame prediction or inter-frame prediction. When inter-frame prediction is applied, the prediction unit or prediction block is predicted from only one reference image, and therefore the reference sample comes from only one reference region of a frame. A B-slice (bidirectional prediction frame) is a slice of an image that can be decoded using intra-frame prediction and inter-frame prediction (e.g., bidirectional or unidirectional prediction). A prediction unit or block of a B-slice can be bidirectionally predicted from two reference images, where each image contributes a reference region, and the sample sets of the two reference regions are weighted (e.g., with equal weights or different weights) to produce the prediction signal for the bidirectional prediction block. As mentioned above, slices of an image are decoded independently. In some cases, an image can be decoded into only one slice.

[0107] As mentioned earlier, intra-image prediction utilizes the correlation between spatially adjacent samples within the image. There are multiple intra-frame prediction modes (also called "intra-frame patterns"). In some instances, intra-frame prediction for a luma patch includes 35 modes, including planar modes, DC modes, and 33 angular modes (e.g., diagonal intra-frame prediction modes and adjacent angular modes). The indices of the 35 intra-frame prediction modes are shown in Table 1 below. In other instances, more intra-frame patterns can be defined, including prediction angles that may not yet be represented by the 33 angular modes. In other instances, the prediction angles associated with angular modes may differ from the prediction angles used in HEVC. [In-frame prediction mode] [Related Names] 0 Frame_Planar 1 _DC within the frame 2..34 Angle 2 within the frame.. Angle 3 within the frame Table 1 – Specifications of In-Frame Prediction Modes and Related Names

[0108] Inter-image prediction uses the temporal correlation between images to derive motion-compensated predictions for image sample patches. Using a translational motion model, the position of the patch within the previously decoded image (reference image) is determined by the motion vector (…). ) indicates, among which Specify the horizontal displacement of the reference block relative to the position of the current block. Specifies the vertical displacement of the reference block relative to the position of the current block. In some cases, the motion vector ( The motion vector can be integer sample precision (also known as integer precision), in which case the motion vector points to the integer primitive grid (or integer primitive sample grid) of the reference frame. In some cases, the motion vector ( The motion vector precision can be fractional sample precision (also known as fractional primitive precision or non-integer precision) to more accurately capture the motion of the underlying object, and is not limited to the integer primitive grid of the reference frame. The precision of the motion vector can be represented by the quantization level of the motion vector. For example, the quantization level can be integer precision (e.g., 1 primitive) or fractional primitive precision (e.g., ¼ primitive, ½ primitive, or other sub-primitive values). When the corresponding motion vector has fractional sample precision, interpolation is applied to the reference image to derive the predicted signal. For example, samples available at integer positions can be filtered (e.g., using one or more interpolation filters) to estimate values ​​at fractional positions. The previously decoded reference image is indicated by a reference index (refIdx) in the reference image list. The motion vector and reference index can be referred to as motion parameters. Two types of inter-image prediction can be performed, including single prediction and double prediction.

[0109] In inter-frame prediction using dual prediction (also known as bidirectional inter-frame prediction), two sets of motion parameters are used ( and Two motion-compensated predictions (from the same reference image or possibly from different reference images) are generated. For example, in dual prediction, each prediction block uses two motion-compensated prediction signals and produces B prediction units. These two motion-compensated predictions are then combined to obtain the final motion-compensated prediction. For example, the two motion-compensated predictions can be combined by averaging. In another instance, weighted prediction can be used, in which case different weights can be applied to each motion-compensated prediction. The reference images available for dual prediction are stored in two separate lists, denoted as list 0 and list 1. Motion parameters can be derived at the encoder using a motion estimation procedure.

[0110] In inter-frame prediction using single prediction (also known as one-way inter-frame prediction), a set of motion parameters is used ( Motion-compensated predictions are generated from a reference image. For example, for a single prediction, each prediction block uses at most one motion-compensated prediction signal and generates P prediction units.

[0111] The prediction unit (PU) may include data related to the prediction process (e.g., motion parameters or other suitable data). For example, when encoding the PU using intra-frame prediction, the PU may include data describing the intra-frame prediction mode used for the PU. As another example, when encoding the PU using inter-frame prediction, the PU may include data defining the motion vectors used for the PU. The data defining the motion vectors used for the PU may describe, for example, the horizontal components of the motion vectors (…). ), the vertical component of the motion vector ( The resolution of the motion vector (e.g., integer precision, quarter-key precision, or eighth-key precision), the reference picture to which the motion vector points, the reference index of the motion vector, a list of reference pictures (e.g., list 0, list 1, or list C), or any combination thereof.

[0112] AV1 includes two common techniques for encoding and decoding blocks of video data. The two common techniques are intra-frame prediction (e.g., intra-frame prediction or spatial prediction) and inter-frame prediction (e.g., inter-frame prediction or temporal prediction). In the context of AV1, when predicting blocks of the current video data frame using an intra-frame prediction mode, the video transcoder 200 and the video decoder 300 do not use video data from other video data frames. For most intra-frame prediction modes, the video encoding device 104 encodes the block of the current frame based on the difference between the sample values ​​in the current block and the predicted values ​​generated from reference samples in the same frame. The video encoding device 104 determines the predicted values ​​generated from the reference samples based on the intra-frame prediction mode.

[0113] After performing prediction using intra-frame and / or inter-frame prediction, encoding device 104 can perform transformation and quantization. For example, after prediction, encoder engine 106 can calculate a residual value corresponding to the PU. The residual value can include primitive differences between the current block (PU) of the decoded primitives and the prediction block used to predict the current block (e.g., a prediction version of the current block). For example, after generating a prediction block (e.g., issuing inter-frame or intra-frame prediction), encoder engine 106 can generate a residual block by subtracting the prediction block generated by the prediction unit from the current block. The residual block includes a set of primitive differences that quantize the differences between the primitive values ​​of the current block and the primitive values ​​of the prediction block. In some instances, the residual block can be represented in a two-dimensional block format (e.g., a two-dimensional matrix or array of primitive values). In such instances, the residual block is a two-dimensional representation of the primitive values.

[0114] Any residual data that may remain after prediction is performed is transformed using block transforms, which can be based on discrete cosine transform, discrete sine transform, integer transform, wavelet transform, other suitable transform functions, or any combination thereof. In some cases, one or more block transforms (e.g., of size 32×32, 16×16, 8×8, 4×4, or other suitable sizes) may be applied to the residual data in each CU. In some embodiments, TUs may be used in the transform and quantization procedures implemented by encoder engine 106. A given CU having one or more PUs may also include one or more TUs. As described in further detail below, the residuals can be transformed into transform coefficients using block transforms, which can then be quantized and scanned using TUs to produce serialized transform coefficients for entropy decoding.

[0115] In some embodiments, after decoding using the PU (Programmable Component) of the CU (Crystal Array) for intra-frame or inter-frame prediction, the encoder engine 106 can calculate residual data for the TU (Transform Array) of the CU. The PU may include primitive data in the spatial domain (or primitive domain). After applying block transform, the TU may include coefficients in the transform domain. As mentioned above, the residual data may correspond to primitive differences between primitives in the uncoded image and the predicted values ​​corresponding to the PU. The encoder engine 106 can form a TU including the residual data for the CU, and then transform the TU to produce transform coefficients for the CU.

[0116] The encoder engine 106 can perform quantization of the transform coefficients. Quantization provides further compression by reducing the amount of data used to represent the coefficients. For example, quantization can reduce the bit depth associated with some or all of the coefficients. In one instance, a coefficient with an n-bit value can be rounded down to an m-bit value during quantization, where n is greater than m.

[0117] Once quantization is performed, the decoded video bitstream includes quantized transform coefficients, prediction information (e.g., prediction patterns, motion vectors, block vectors, etc.), segmentation information, and any other suitable data, such as other grammatical data. The different elements of the decoded video bitstream can then be entropy-encoded by encoder engine 106. In some instances, encoder engine 106 may use a predefined scanning order to scan the quantized transform coefficients to produce a serialized vector that can be entropy-encoded. In some instances, encoder engine 106 may perform a self-adjusting scan. After scanning the quantized transform coefficients to form a vector (e.g., a one-dimensional vector), encoder engine 106 may entropy-encode the vector. For example, encoder engine 106 may use context-adjusting variable-length decoding, context-adjusting binary arithmetic decoding, grammar-based context-adjusting binary arithmetic decoding, probabilistic interval partitioning entropy decoding, or another suitable entropy coding technique.

[0118] The output 110 of the encoding device 104 can transmit NAL units constituting the encoded video bitstream data to the decoding device 112 of the receiving device via the communication link 120. The input 114 of the decoding device 112 can receive the NAL units. The communication link 120 may include a channel provided by a wireless network, a wired network, or a combination of wired and wireless networks. The wireless network may include any wireless interface or combination of wireless interfaces, and may include any suitable wireless network (e.g., the Internet or other wide area networks, packet-based networks, WiFi™, radio frequency (RF), ultra-wideband (UWB), WiFi-Direct, cellular, 5G New Radio (NR), Long Term Evolution (LTE), WiMax™, etc.). The wired network may include any wired interface (e.g., fiber optic, Ethernet, powerline Ethernet, coaxial cable Ethernet, digital signal line (DSL), etc.). Wired and / or wireless networks can be implemented using various equipment, such as base stations, routers, access points, bridges, gateways, and switches. Encoded video bitstream data can be modulated according to communication standards such as wireless communication protocols and transmitted to the receiving device.

[0119] In some instances, encoding device 104 may store encoded video bitstream data in memory 108. Output 110 may retrieve the encoded video bitstream data from encoder engine 106 or from memory 108. Memory 108 may include any of a variety of distributed or locally accessed data storage media. For example, memory 108 may include a hard disk, storage disk, flash memory, volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded video data. Memory 108 may also include a decoded picture buffer (DPB) for storing reference pictures used for inter-frame prediction. In another instance, memory 108 may correspond to a file server or another intermediate storage device that may store encoded video generated by a source device. In this case, receiving device including decoding device 112 may access the stored video data from the storage device via data streaming or download. The file server may be any type of server capable of storing encoded video data and sending it to the receiving device. The instance file server includes a web server (e.g., for a website), an FTP server, a network attached storage (NAS) device, or a local disk drive. The receiving device can access the encoded video data via any standard data connection (including an internet connection). This can include wireless channels (e.g., Wi-Fi connections), wired connections (e.g., DSL, cable modems, etc.), or a combination of both, suitable for accessing encoded video data stored on the file server. Sending the encoded video data from memory 108 can be streaming, downloading, or a combination thereof.

[0120] The input 114 of the decoding device 112 receives encoded video bitstream data and can provide the video bitstream data to the decoder engine 116 or to memory 118 for later use by the decoder engine 116. For example, memory 118 may include a DPB for storing reference pictures used for inter-frame prediction. A receiving device including the decoding device 112 can receive the encoded video data to be decoded via memory 108. The encoded video data can be modulated according to a communication standard such as a wireless communication protocol and transmitted to the receiving device. The communication medium used to transmit the encoded video data can include any wireless or wired communication medium, such as radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium can form part of a packet-based network, such as a local area network, a wide area network, or a global network such as the Internet. The communication medium can include a router, a switch, a base station, or any other equipment that can be used to facilitate communication from the source device to the receiving device.

[0121] Decoder engine 116 can decode encoded video bitstream data via entropy decoding (e.g., using an entropy decoder) and by extracting elements constituting one or more decoded video sequences. Decoder engine 116 can then rescale the encoded video bitstream data and perform an inverse transform. The residual data is then passed to the prediction stage of decoder engine 116. Decoder engine 116 then predicts blocks (e.g., PUs). In some instances, the prediction is added to the output of the inverse transform (the residual data).

[0122] The video decoding device 112 can output the decoded video to the video target device 122, which may include a display or other output device for displaying the decoded video data to a consumer of the content. In some embodiments, the video target device 122 may be part of a receiving device that includes the decoding device 112. In some embodiments, the video target device 122 may be part of a separate device other than the receiving device.

[0123] In some embodiments, video encoding device 104 and / or video decoding device 112 may be integrated with audio encoding device and audio decoding device, respectively. Video encoding device 104 and / or video decoding device 112 may also include other hardware or software necessary for implementing the above-described decoding technology, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combination thereof. Video encoding device 104 and video decoding device 112 may be integrated as part of a combined encoder / decoder (decoder) in each device.

[0124] The example system shown in Figure 1 is an illustrative example that can be used herein. The techniques used to process video data using the methods described herein can be implemented by any digital video encoding and / or decoding device. Although the techniques described herein are typically implemented by video encoding or video decoding devices, these techniques can also be implemented by a combined video transcoder-decoder (commonly referred to as a "decoder"). Furthermore, the techniques described herein can also be implemented by a video preprocessor. The source device and receiving device are merely examples of such decoding devices, where the source device generates decoded video data for transmission to the receiving device. In some instances, the source device and receiving device can operate in a substantially symmetrical manner, such that each device includes video encoding and decoding components. Therefore, the example system can support one-way or two-way video transmission between video devices, for example, for video streaming, video replay, video broadcasting, or video telephony.

[0125] Extensions to the HEVC standard include the Multi-View Video Decoding Extension (MV-HEVC) and the Scalable Video Decoding Extension (SHVC). MV-HEVC and SHVC extensions share the concept of layered decoding, comprising different layers within the encoded video bitstream. Each layer in the decoded video sequence is addressed by a unique layer identifier (ID). The layer ID may reside in the header of a NAL unit to identify the layer associated with that NAL unit. In MV-HEVC, different layers typically represent different views of the same scene within the video bitstream. In SHVC, different scalable layers are provided to represent the video bitstream at different spatial resolutions (or picture resolutions) or different reconstruction fidelities. Scalable layers may include a base layer (layer ID=0) and one or more enhancement layers (layer IDs=1, 2, ..., n). The base layer may conform to the HEVC first-version profile and represents the lowest available layer in the bitstream. Compared to the base layer, enhancement layers offer increased spatial resolution, temporal resolution, or playback rate and / or reconstruction fidelity (or quality). Enhancement layers are hierarchically organized and may (or may not) depend on lower layers. In some instances, a single standard decoder can be used to decode different layers (e.g., encoding all layers using HEVC, SHVC, or other decoding standards). In other instances, multiple standard decoders can be used to decode different layers. For example, AVC can be used to decode the base layer, while SHVC and / or MV-HEVC extensions of the HEVC standard can be used to decode one or more enhancement layers.

[0126] Typically, a layer consists of a set of VCL NAL units and a corresponding set of non-VCL NAL units. NAL units are assigned specific layer ID values. Layers can be hierarchical in the sense that they can depend on lower layers. A layer set is a self-contained set of layers represented in a bitstream, meaning that layers in a layer set can depend on other layers in the layer set in the decoding process, but not on any other layers for decoding. Therefore, layers in a layer set can form independent bitstreams capable of representing video content. The layer set in a layer set can be obtained from another bitstream via operations of a sub-bitstream extractor. A layer set can correspond to the set of layers that the decoder wants to decode when it wants to operate based on certain parameters.

[0127] As mentioned earlier, the HEVC bitstream includes a group of NAL units, including VCL NAL units and non-VCL NAL units. VCL NAL units contain decoded picture data that forms the decoded video bitstream. For example, a sequence of bits forming the decoded video bitstream exists within a VCL NAL unit. Non-VCL NAL units may also contain parameter sets with high-level information related to the decoded video bitstream, among other information. For example, parameter sets may include a Video Parameter Set (VPS), a Sequence Parameter Set (SPS), and a Picture Parameter Set (PPS). Examples of objectives for parameter sets include bit rate efficiency, error recovery capability, and providing a system-level interface. Each slice references a single active PPS, SPS, and VPS to access information available to the decoding device 112 for decoding the slice. A decoding identifier (ID) may be provided for each parameter set, including a VPS ID, an SPS ID, and a PPS ID. An SPS includes an SPS ID and a VPS ID. A PPS includes a PPS ID and an SPS ID. Each slice header includes a PPS ID. Using IDs, the set of active parameters can be identified for a given slice.

[0128] The PPS includes information applicable to all slices in a given image. Therefore, all slices in an image reference the same PPS. Slices in different images can also reference the same PPS. The SPS includes information applicable to all images in the same decoded video sequence (CVS) or bitstream. As mentioned earlier, a decoded video sequence is a series of access units (AUs) that begin with a random access point image in the base layer (e.g., a transient decoding reference (IDR) image or a broken link access (BLA) image, or other suitable random access point image) and have certain properties (as mentioned earlier), until and not including the next AU with a random access point image in the base layer and certain properties (or ending with the bitstream). Within a decoded video sequence, the information in the SPS cannot be changed between images. Images in a decoded video sequence can use the same SPS. The VPS includes information applicable to all layers within the decoded video sequence or bitstream. The VPS includes a syntax structure with syntax elements applicable to the entire decoded video sequence. In some embodiments, the VPS, SPS, or PPS may be transmitted in-band along with the encoded bitstream. In some embodiments, the VPS, SPS, or PPS may be transmitted out-of-band in a separate transmission, distinct from the NAL unit containing the decoded video data.

[0129] This term typically refers to "signal delivery notification" of certain information, such as syntax elements. The term "signal delivery notification" generally refers to communication of values ​​for syntax elements and / or other data used to decode encoded video data. For example, video encoding device 104 may signal delivery notification of the value of a syntax element in a bitstream. Generally, signal delivery notification refers to the generation of a value in a bitstream. As mentioned above, video source 102 may deliver the bitstream to video target device 122 substantially instantaneously or non-instantaneously, such as when storing syntax elements in storage 108 for later retrieval by video target device 122.

[0130] Video bitstreams may also include Supplemental Enhancement Information (SEI) messages. For example, an SEI NAL unit can be part of the video bitstream. In some cases, SEI messages may contain information that is not required by the decoding process. For example, the information in the SEI message may not be necessary for the decoder to decode the video picture of the bitstream, but the decoder can use the information to improve the display or processing of the picture (e.g., decoding output). The information in the SEI message can be embedded in relay data. In one illustrative example, the information in the SEI message can be used by the decoder-side entity to improve the visibility of the content. In some instances, certain application standards may mandate the presence of such SEI messages in the bitstream so that the quality improvements can be brought to all devices that comply with the application standard (e.g., frame-packing SEI messages for frame-compatible planar stereoscopic 3DTV video formats, handling restoring point SEI messages when SEI messages are carried in every frame of the video, using panscanning scan rectangle SEI messages in DVB, and many other examples).

[0131] As mentioned earlier, the Energy Conservation Media Consumption Standard (ISO / IEC 23001-11) specifies green relay data to promote reduced energy use during media consumption. Green relay data includes Complexity Metric (CM) relay data and Decoding Operation Reduction Request (DOR-Req) relay data. Decoders can use CM relay data to help adjust the operating frequency of the processor performing decoding to account for reduced power consumption. As previously mentioned, this paper describes systems and techniques for improving green relay data such as CM signaling. For example, using a single type of signaling over multiple images to inform CMs of slice, tile, and / or subframe nuances can be problematic. In some cases, the systems and techniques described herein improve the syntax of green relay data to support more nuance types (e.g., slice nuances, tile nuances, etc.) and, in some cases, nuance types across various time periods. In some cases, the systems and techniques described herein provide signaling to map one or more CMs to a specific sub-image using a sub-image identifier (ID). In some instances, the systems and techniques described herein provide signaling (e.g., syntax elements) to indicate the total number of blocks (e.g., CTBs or other blocks). In some cases, such signaling (indicating the total number of blocks) can simplify the derivation of the total number of decoded blocks over a period of time to account for the normalized coding statistics of each slice or tile. Signaling can be included in the CM relay data included in the encoded video. In some instances, the systems and techniques described herein provide signaling for intra-frame decoded block statistics. In some cases, such intra-frame decoded block statistics signaling can address the problem that arises when the CM provides intra-frame decoded block statistics when all blocks are intra-frame decoded (e.g., when P and B slices have more intra-frame decoded blocks than inter-frame decoded blocks, when P or B pictures have more intra-frame decoded blocks than inter-frame decoded blocks, etc.). In some cases, the systems and techniques described herein provide mechanisms for applying quality metrics to individual parts of an image (e.g., to individual sub-images), such as in VVC, rather than applying the quality metrics to the entire image.

[0132] The various states of the Complexity Metric (CM) signaling mentioned above will now be described. For example, in some states, a granularity type indicator (e.g., a granularity type syntax element, such as granularity_type) is provided to support various granularities (e.g., granular segments), such as slices, tiles, sub-pictures, expandable layers, and / or other granularities. For example, encoding device 104 may signal notifications of the granularity type indicator in or with a bitstream. The granularity type indicator may be used in combination with a period type syntax element to support granular CM signaling applicable to multiple pictures. In some instances, the semantics of the period type syntax element (e.g., period_type) are modified. In an illustrative example, Table 2 below provides CM signaling for VVC green relay data (in <> (e.g.<added language> (The diagram between the symbols is a supplement to ISO / IEC 23001-11) Table 2 – Syntax of VVC CM [] [Size (bits)] [Descriptor] [period_type] 8 unsigned integers < [granularity_type]> 8 unsigned integers if (period_type == 2) { [num_seconds] 16 unsigned integers } else if ( period_type == 3 ) { [num_pictures] 16 unsigned integers <} if ( []granularity_type [==0]) { <picture_level_CMs()> } [] else { granularity_level_CMs() } } > …

[0133] The period_type syntax element (e.g., variable) specifies the type of the upcoming period to which the complexity metric applies, and the values ​​of the period_type syntax element can be defined in Table 3 below (as illustrative examples): Table 3 – VVC period_type specification [value] [describe] 0x00 Complexity metrics are applicable to a single image 0x01 The complexity metric applies to all images in the decoding order, up to (but not including) the image containing the next I-slice. 0x02 The complexity metric applies to all images within a specified time interval, measured in seconds. 0x03 The complexity metric applies to a specified number of images, counted in decoding order. 0x04-0xFF reserve

[0134] The `granularity_type` syntax element specifies the granularity type to which the complexity metric applies, and the values ​​for the `granularity_type` syntax element can be defined in Table 4 below (as illustrative examples): Table 4 – VVC Granularity_Type Specification [value] [describe] 0x00 CM is suitable for the fine details of images. 0x01 CM is suitable for the fineness of sectioning. 0x02 CM applies to the tile detail of a tile. 0x03 CM is applicable to the finer details of sub-images. 0x04 CM is suitable for the scalable layer's granularity. 0x05 CM is applicable to the CTU line's subtlety. 0x07-0xFF reserve

[0135] The `picture_level_CMs` syntax structure specifies a complexity metric for a particular image over a given time period. In this paper, the `picture_level_CMs` syntax structure may be referred to as the image-level CM syntax structure.

[0136] The granularity_level_CMs syntax structure specifies the granularity level complexity measure for each entity (such as a slice, tile, sub-image, or layer) over a period of time. In this paper, the granularity_level_CMs syntax structure may be referred to as the granularity level CM syntax structure.

[0137] Figure 2 is a diagram used to illustrate an example of a detail level CM of a video image 200 (also referred to as a frame or image) according to the present case. The video image 200 includes a cyclist 202 in motion, a view of the cyclist riding through the video image 200, and the cyclist 202 appears in a set of images 204 of the video image 200. Each image of the video image 200 can be divided into one or more parts (e.g., via an encoder such as encoding device 104), such as slices, tiles, sub-images, layers, etc. Image 206 in the set of images 204 is shown divided into sixteen slices 208, four tiles 210, and one sub-image 212, where each tile 210 includes four slices 208, and the sub-image 212 includes two tiles 210 at the bottom of the image.

[0138] In some cases, the ability to specify time-segment type and granularity type to a decoding device (e.g., decoding device 112) allows for greater flexibility and reduced signal transmission. This is achieved by allowing a single granularity level CM to be defined for slices, tiles, subpictures, or layers of multiple pictures. For example, an encoding device (e.g., encoding device 104) can apply a single granularity level CM to subpictures of all pictures within a specified time interval, instead of having to define a granularity level CM for each subpicture of a picture. In video picture 200, the area where cyclist 202 appears may be more complex to encode / decode than other areas of picture set 204 (which have less or no motion), and different CMs can be specified for these areas using granularity level CMs. For example, a single granularity level CM can be specified for a subpicture 212 region (e.g., granularity_type=3) of multiple pictures (such as six pictures in picture set 204 (e.g., num_pictures=6)). By allowing the setting of a granularity level CM for a specific time interval (e.g., the number of images in a set, a time period, etc.), a single granularity level CM can be used in the relay data corresponding to the first image of image set 204, and based on the specified time interval, this granularity level CM can be applied to all images in image set 204. After image set 204, the granularity level CM of sub-image 212 can be adjusted because cyclist 202 is no longer in the area covered by sub-image 212, and that area can now be encoded / decoded with less complexity. Similarly, multiple, possibly different, granularity level CMs can be specified for any number of slices, tiles, sub-images, or layers in an image, where each granularity level CM can be applied to different upcoming time periods (e.g., a single image, all images within a specified time interval, multiple images, all images up to the image containing the next slice, etc.).

[0139] In some formats, the encoding device (e.g., encoding device 104) can specify CM values ​​for portions of the video (such as slices, tiles, subpictures, and / or layers) to be used. For example, according to some formats, the encoding device can generate and signal-transmit subpicture CM signal transmissions. Subpicture CM signal transmissions indicate which subpictures of one or more images the CM applies to. In one instance, for subpicture detail, a syntax element (e.g., called a subpicture syntax element) indicates that when the time spans multiple images (e.g., where detail level CM applies to multiple images), the signal transmission notification subpicture ID is included in the CM relay data. Table 5 below illustrates an example: Table 5 Sub-image CM [] [size] [Size] [big] [(bits)] [Descriptor] [...] if ( granularity_type == 0x03 && period_type > 0 ) { num_subpics_minus1 16 unsigned integers for (i = 0; i <= num_subpics_minus1; i++) subpic_id[i] 16 unsigned integers subpic_CM( i ) } [] … }

[0140] subpic_id[I] specifies the subpic ID of the associated complexity metric (CM).

[0141] subpic_CM is the complexity metric structure for the i-th subpic.

[0142] In some cases, `subpic_id` and / or `subpic_CM(i)` can be replaced by one or more syntax elements referencing a segment address. In some cases, a segment can be a slice, tile, or subpicture, and the segment address can identify, for example, a specific slice, tile, and / or subpicture of a picture. As an example, the segment address [t] can indicate the address of the t-th segment. Therefore, in cases where the subpicture detail is specified by the detail type, the segment address [t] can indicate the subpicture ID of the t-th subpicture.

[0143] In some cases, the status pattern is associated with resolution variation. For example, in VVC, resolution variation within a Layer-of-Decoding Video Sequence (CLVS) applies to picture, slice, and tile nuances, but not to sub-picture nuances. Depending on some status patterns, such as when the nuance type equals a slice (e.g., 0x01 from Table 4) or a tile (e.g., 0x02 from Table 4), and the time period type spans multiple pictures, a signal delivery notification can be included in the green relay data (e.g., as one or more syntax elements in the CM syntax table, such as num_ctbs_minus1 in Table 6 below) indicating the total number of decoder tree luminance blocks within a time period (e.g., a syntax element called the decoder tree block (CTB) number). Table 6 below illustrates an example: Table 6 Complexity Measurement Syntax [] [Size (bits)] [Descriptor] [...] if ( (granularity_type == 0x01 || granularity_type == 0x02) && period_type > 0 ) { num_ctbs_minus1 16 unsigned integers granularity_level_CMs() } [] … }

[0144] num_ctbs_minus1 specifies the total number of decoder blocks with associated complexity metrics within a given time period.

[0145] In some formats, alternative syntax elements (e.g., avg_number_ctbs_minus1) can indicate the average number of CTBs per minute per image or per 4×4 block (or other block size) instead of the total number of CTBs over a period of time, to reduce administrative burden. Such syntax elements can be called average CTB number syntax elements.

[0146] In some cases, the status is associated with in-frame decoded block statistics. For example, the current Green Relay data CM syntax only notifies the in-frame decoded block statistics when all blocks are in-frame decoded blocks (e.g., portion_intra_predicted_blocks_area==255). Table 7 below illustrates the proposed CM signaling changes, where <> (e.g., ...<added language> Adding text using icons between () and deleting text using icons (e.g., deleted language). Signal delivery notifies the in-frame decoder statistics when an in-frame decoder is available. Signal delivery notifies the inter-frame decoder statistics when an inter-frame decoder is available. Table 7 Suggested CM syntax structure [] [Size (bits)] [Descriptor] [...] [portion_intra_predicted_blocks_area] [] [] 8 unsigned integers if ( portion_intra_predicted_blocks_area<!= 0> = = 255) { [] [portion_planar_blocks_in_intra_area] 8 unsigned integers [portion_dc_blocks_in_intra_area] 8 unsigned integers [portion_angular_hv_blocks_in_intra_area] 8 unsigned integers [portion_mip_blocks_in_intra_area] 8 unsigned integers } else <if ( portion_intra_predicted_blocks_area !="255 " )>{ [portion_bi_and_gpm_predicted_blocks_area] [] [] 8 unsigned integers } [portion_deblocking_instances] 8 unsigned integers [portion_sao_filtered_blocks] 8 unsigned integers [portion_alf_filtered_blocks] 8 unsigned integers … [] }

[0147] The following provides definition examples of the various syntax elements in Table 7 for VVC:

[0148] [portion_intra_predicted_blocks_area] uses 4-sample nuance to indicate the area covered by the predicted blocks within a frame in an image for a specified time period, as defined below: (5-82)

[0149] NumIntraPredictedBlocks is the number of predicted blocks within a frame over a specified time period using 4-sample nuance. On the encoder side, it is calculated as follows: (5-83)

[0150] For the specified time period, X=4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096 sample points, where NumIntraPredictedBlocks_X is the number of blocks predicted using in-frame prediction.

[0151] NumIntraPredictedBlocks is exported from portion_intra_predicted_blocks_area and TotalNum4BlocksInPeriod in the decoder.

[0152] [portion_planar_blocks_in_intra_area] indicates the portion of the in-frame planar prediction block region within the specified time period, defined as follows: (5-84)

[0153] When it does not exist, it equals 0.

[0154] NumPlanarPredictedBlocks is the number of planar predicted blocks within a frame over a specified time period using 4-sample nuance. On the encoder side, it is calculated as follows: (5-85)

[0155] For the specified time period, X=4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096, where NumIntraPlanarBlocks_X is the number of blocks predicted using in-frame planar prediction.

[0156] NumPlanarPredictedBlocks is exported from portion_planar_blocks_in_intra_area and NumIntraPredictedBlocks in the decoder.

[0157] [portion_dc_blocks_in_intra_area] indicates the portion of the DC prediction block region within the frame in the specified time period, defined as follows: (5-86)

[0158] When it does not exist, it equals 0.

[0159] NumDcPredictedBlocks is the number of DC prediction blocks within a frame over a specified time period using 4-sample nuance. On the encoder side, it is calculated as follows: (5-87)

[0160] For the specified time period, X=4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096 sample points, where NumIntraDcBlocks_X is the number of blocks predicted using DC within the frame.

[0161] NumDcPredictedBlocks is exported from portion_dc_blocks_in_intra_area and NumIntraPredictedBlocks in the decoder.

[0162] `portion_angular_hv_blocks_in_intra_area` (also known as `portion_hv_blocks_in_intra_area`) indicates the portion of the horizontal and vertical prediction block regions within a frame in a specified time period, defined as follows: (5-88)

[0163] When it does not exist, it equals 0.

[0164] NumHvPredictedBlocks is the number of horizontal and vertical predicted blocks within a frame over a specified time period using 4-sample nuance. On the encoder side, it is calculated as follows: (5-89)

[0165] For the specified time period, X=4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096 sample points, where NumIntraHvBlocks_X is the number of blocks predicted using the horizontal and vertical directions within the frame.

[0166] NumHvPredictedBlocks is exported from portion_hv_blocks_in_intra_area and NumIntraPredictedBlocks in the decoder.

[0167] [portion_mip_blocks_in_intra_area] indicates the portion of the MIP prediction block region within the frame in the specified time period, defined as follows: (5-90)

[0168] When it does not exist, it equals 0.

[0169] NumMipPredictedBlocks is the number of MIP prediction blocks within a frame over a specified time period using 4-sample nuance. On the encoder side, it is calculated as follows: (5-91)

[0170] For the specified time period, X=4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096 sample points, where NumIntraMipBlocks_X is the number of blocks predicted using in-frame MIP.

[0171] NumMipPredictedBlocks is exported from portion_mip_blocks_in_intra_area and NumIntraPredictedBlocks in the decoder.

[0172] [portion_bi_and_gpm_predicted_blocks_area] uses 4-sample nuance to indicate the area covered by inter-frame bi-predicted or GPM-predicted blocks in an image for a specified time period, defined as follows: (5-92)

[0173] NumBiAndGpmPredictedBlocks is the number of inter-frame double prediction or GPM prediction blocks within a specified time period using 4-sample nuance. On the encoder side, it is calculated as follows: (5-93)

[0174] For the specified time period, the number of sample points X=4, 4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096, where NumBiPredictedXBlocks is the number of blocks predicted using inter-frame double prediction or GPM prediction.

[0175] NumBiPredictedXBlocks is exported from the decoder portion_bi_and_gpm_predicted_blocks_area and TotalNum4BlocksInPeriod.

[0176] The [portion_deblocking_instances] directive indicates the instances of deblocking filtering within a specified time period, as defined in the Terminology and Definitions section of this document, as follows: (5-94)

[0177] NumDeblockingInstances is the number of deblocking filter instances in a specified time period. NumDeblockingInstances is exported from portion_deblocking_instances and MaxNumDeblockingInstances in the decoder.

[0178] [portion_sao_filtered_blocks] indicates the portion of the blocks filtered using SAO (Simplified Arithmetic Aspect-Oriented Filtering) within a specified time period using 4-sample nuance. On the encoder side, it is calculated as follows: (5-95)

[0179] NumSaoFilteredBlocks is the number of blocks that use SAO filtering within a specified time period with 4-sample nuance. NumSaoFilteredBlocks is exported from pportion_sao_filtered_blocks and TotalNum4BlocksInPeriod in the decoder.

[0180] [portion_alf_filtered_blocks] indicates the portion of the blocks filtered using ALF filtering within a specified time period using 4-sample nuance. On the encoder side, it is calculated as follows: (5-96)

[0181] NumAlfFilteredBlocks is the number of blocks filtered by ALF in a specified time period using 4-sample nuance. NumAlfFilteredBlocks is exported from portion_alf_filtered_blocks and TotalNum4BlocksInPeriod in the decoder.

[0182] In some cases, the state is associated with a sub-picture quality metric. For example, a quality restoration metric can be applied to each detail segment. In some cases, a segment can be a slice, tile, or sub-picture. Table 8 provides examples of sub-picture-based quality restoration metrics proposed for Green MPEG, where in <> (e.g.<added language> Add the illustration between the dots. Table 8 Quality Restoration Metrics for Green Relay Data green_metadata(payload_size) [] [Descriptor] [green_metadata_type] u(8) switch ( [green_metadata_type]) { case 0: … break; case 1: [xsd_metric_number_minus1] u(4) for ( i=0; i<= xsd_metric_number_minus1; i++ ) { [xsd_metric_type[ i ]] u(8) < [xsd_subpic_number_minus1>] ue(v) for ( j=0; i<= xsd_subpic_number_minus1; j++ ) { [xsd_metric_value[ i ]<[ j ]>] u(16) } } break; default: }

[0183] [xsd_subpic_number_minus1] specifies the number of subpicks available in the associated image. When xsd_subpic_number_minus1 equals 0, the quality restoration metric is applied to the entire image.

[0184] [xsd_metric_type[i]] indicates the type of the target quality metric for the i-th target quality metric.

[0185] [xsd_metric_value[i][j]] contains the value of the i-th target quality metric associated with the j-th sub-image.

[0186] The current quality metric describes the quality of the last image in each segment. The pattern described herein allows the SEI message to carry a quality metric describing the quality of the associated images. For example, an encoding device (e.g., encoding device 104 of Figures 1 and 4) can add a quality metric to the SEI message.

[0187] Figure 3 is a flowchart illustrating a procedure for decoding encoded video 300 according to the present case. In operation 302, procedure 300 may include obtaining a bitstream. In operation 304, procedure 300 may include retrieving a subtlety type syntax element associated with the bitstream, which specifies the subtlety type of one or more pictures to which a complexity metric (CM) associated with the bitstream applies. In some cases, the value of the subtlety type syntax element specifies that the CM applies to a picture or a portion of a picture in the bitstream, the portion of which is smaller than the entire picture. In some cases, the value of the subtlety type syntax element specifies that the CM applies to at least one of a slice, tile, subpicture, extensible layer, or decoder tree unit (CTU) row of one or more pictures in the bitstream.

[0188] In operation 306, procedure 300 may include retrieving a time-segment type syntax element associated with the bitstream, which indicates an upcoming time segment or the set of pictures to which the CM applies. In some cases, the time-segment type syntax element indicates a specified time interval of the upcoming time segment, the number of pictures in the upcoming time segment, that the upcoming time segment includes all pictures up to including the next slice, or that the upcoming time segment includes at least one of a single picture. In some cases, procedure 300 may also include retrieving a subtlety level CM syntax structure associated with the bitstream, which specifies a subtlety level complexity measure for one or more subtle segments of the bitstream within the upcoming time segment. In some cases, procedure 300 may also include retrieving an additional time-segment type syntax element associated with the bitstream, which is associated with the subtlety type syntax element, wherein the additional time-segment type syntax element is different from the time-segment type syntax element, and decoding a portion of the bitstream based on the subtlety type syntax element and the additional time-segment type syntax element.

[0189] In some cases, program 300 may also include retrieving at least one of the following: a subpicture syntax element associated with the bitstream, which indicates a signaling notification subpicture identifier (ID) when the CM is applied to multiple pictures; a decode tree block (CTB) count syntax element associated with the bitstream, which, when the detail type is equal to a slice or tile and the upcoming time spans multiple pictures, can be a signaling notification in the CM indicating the total number of decode tree detail blocks in the upcoming time span; or an average decode tree block (CTB) count syntax element associated with the bitstream, which indicates the average number of CTBs per detail or 4×4 blocks per picture.

[0190] In some cases, for program 300, when an available in-frame decoding block exists in at least a portion of the bit stream, a signal transmission associated with at least a portion of the bit stream notifies the in-frame decoding block statistics. In some cases, when an available inter-frame decoding block exists in at least a portion of the bit stream, a signal transmission associated with at least a portion of the bit stream notifies the inter-frame decoding block statistics. In some cases, program 300 may also include displaying at least a portion of the bit stream on a display. In some cases, program 300 may also include determining the operating frequency of the device based on the CM associated with the bit stream.

[0191] In operation 308, program 300 may include a portion of a bitstream decoded based on subtlety type syntax elements and time-based type syntax elements. In some cases, program 300 may be executed by one of a mobile device, wearable device, extended reality device, camera, personal computer, vehicle, robotic device, television, or computing device.

[0192] Figure 4 is a flowchart 400 illustrating a technique for encoding video according to the present invention. In operation 402, procedure 400 may include acquiring video data. In operation 404, procedure 400 may include generating a subtlety type syntax element for a bitstream, the subtlety type syntax element specifying the subtlety type of one or more pictures to which a complexity metric (CM) associated with the bitstream applies. In some cases, the value of the subtlety type syntax element specifies that the CM applies to a picture or a portion of a picture in the bitstream, the portion of which is smaller than the entire picture. In some cases, the value of the subtlety type syntax element specifies that the CM applies to at least one of a slice, tile, subpicture, extensible layer, or decoder tree unit (CTU) row of one or more pictures in the bitstream.

[0193] In operation 406, procedure 400 may include generating a time-segment type syntax element associated with the bitstream, which indicates the upcoming time segment or the set of pictures to which the CM applies. In some cases, the time-segment type syntax element indicates a specified time interval of the upcoming time segment, the number of pictures in the upcoming time segment, that the upcoming time segment includes all pictures up to the point where the next slice is included, or that the upcoming time segment includes at least one of a single picture.

[0194] In some cases, procedure 400 may also include generating a subtlety-level CM syntax structure for the bitstream, the subtlety-level CM syntax structure specifying a subtlety-level complexity measure for one or more entities in an upcoming time period. In some cases, procedure 400 may also include generating an additional time period type syntax element associated with the subtlety type syntax element for the bitstream, wherein the additional time period type syntax element is different from the time period type syntax element, and wherein the additional time period type syntax element is used to decode a portion of the bitstream having the subtlety type syntax element. In some cases, program 400 may also include generating at least one of the following for the bitstream: a subpicture syntax element associated with the bitstream, which indicates a subpicture identifier (ID) when the CM is applied to multiple pictures; a decode tree block (CTB) number syntax element associated with the bitstream, which, when the detail type is equal to a slice or tile and the upcoming time spans multiple pictures, can be used in the CM to indicate the total number of decode tree detail blocks in the upcoming time span; or an average decode tree block (CTB) number syntax element associated with the bitstream, which indicates the average number of CTBs per detail or 4×4 blocks per picture.

[0195] In some cases, for program 400, when an available in-frame decoding block exists in at least a portion of the bitstream, signal delivery associated with at least a portion of the bitstream notifies in-frame decoding block statistics. In some cases, when an available inter-frame decoding block exists in at least a portion of the bitstream, signal delivery associated with at least a portion of the bitstream notifies inter-frame decoding block statistics. In some cases, program 400 may be executed by one of a mobile device, wearable device, extended reality device, camera, personal computer, vehicle, robotic device, television, or computing device.

[0196] In some implementations, the programs (or methods) described herein may be executed by a computing device or apparatus (e.g., system 100 shown in FIG. 1). For example, the program may be executed by the encoding device 104 shown in FIG. 1 and FIG. 5, another video source-side device or video transmission device, the decoding device 112 shown in FIG. 1 and FIG. 6, and / or another client device (e.g., a player device, a display, or any other client device). In some cases, the computing device or apparatus may include one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, and / or other components (multiple) configured to perform the steps of one or more programs described herein.

[0197] In some instances, a computing device may include a mobile device, a desktop computer, a server computer and / or a server system, or other types of computing devices. Components of the computing device (e.g., one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers and / or other components) may be implemented in circuitry. For example, components may include and / or may be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, graphics processing units (GPUs), digital signal processors (DSPs), central processing units (CPUs), and / or other suitable electronic circuits), and / or may include and / or be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein. In some instances, a computing device or apparatus may include a camera configured to capture video data (e.g., a video sequence) including video frames. In some instances, the camera or other capturing device for capturing video data is separate from the computing device, in which case the computing device receives or acquires the captured video data. Computing devices may include a network interface configured to transmit video data. The network interface may be configured to transmit Internet Protocol (IP) based data or other types of data. In some instances, the computing device or apparatus may include a display for displaying output video content (e.g., samples of images from a video bitstream).

[0198] The program can be described using a logic flowchart, where actions represent sequences of operations that can be performed in hardware, computer instructions, or combinations thereof. In the context of computer instructions, actions represent computer-executable instructions stored on one or more computer-readable storage media that perform operations when executed by one or more processors. Typically, computer-executable instructions include routines, programs, objects, components, data structures, etc., that perform a specific function or implement a specific data type. The order in which operations are described is not intended to be construed as restrictive, and any number of described operations can be combined in any order and / or in parallel to implement the program.

[0199] Furthermore, the program can be executed under the control of one or more computer systems configured with executable instructions, and can be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that can be executed jointly on one or more processors via hardware or a combination thereof. As mentioned above, the code can be stored, for example, on a computer-readable or machine-readable storage medium in the form of a computer program comprising a plurality of instructions executable by one or more processors. The computer-readable or machine-readable storage medium can be non-transitory.

[0200] The decoding techniques discussed herein can be implemented in an example video encoding and decoding system (e.g., system 100). In some instances, the system includes a source device that provides encoded video data to the target device, which will later be decoded by the target device. Specifically, the source device provides the video data to the target device via computer-readable media. The source and target devices can include any of a variety of devices, including desktop computers, laptops, tablets, set-top boxes, handheld telephone devices such as so-called "smart" phones, so-called "smart" mats, televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, etc. In some cases, the source and target devices may be equipped for wireless communication.

[0201] The target device can receive encoded video data to be decoded via computer-readable media. Computer-readable media can include any type of media or device capable of moving encoded video data from a source device to the target device. In one example, computer-readable media can include communication media that enables the source device to send encoded video data directly to the target device in real time. The encoded video data can be modulated according to communication standards such as wireless communication protocols and sent to the target device. Communication media can include any wireless or wired communication media, such as radio frequency (RF) spectrum or one or more physical transmission lines. Communication media can form part of a packet-based network, such as a local area network, a wide area network, or a global network such as the Internet. Communication media can include routers, switches, base stations, or any other equipment that can be used to facilitate communication from the source device to the target device.

[0202] In some instances, encoded data can be output from an output interface to a storage device. Similarly, encoded data can be accessed from a storage device via an input interface. The storage device can include any of a variety of distributed or locally accessed data storage media, such as hard drives, Blu-ray discs, DVDs, CD-ROMs, flash memory, volatile or non-volatile memory, or any other suitable digital storage media for storing encoded video data. In another instance, the storage device can correspond to a file server or another intermediate storage device that can store encoded video generated by the source device. The target device can access the stored video data from the storage device via data streaming or downloading. The file server can be any type of server capable of storing and sending encoded video data to the target device. Examples of file servers include web servers (e.g., for websites), FTP servers, network attached storage (NAS) devices, or local disk drives. The target device can access the encoded video data via any standard data connection, including an internet connection. This can include wireless channels (e.g., Wi-Fi connections), wired connections (e.g., DSL, cable modems, etc.), or a combination of both, suitable for accessing encoded video data stored on a file server. Sending encoded video data from the storage device can be streaming, downloading, or a combination thereof.

[0203] The technology in this case is not necessarily limited to wireless applications or setups. It can be applied to video decoding supporting any of a variety of multimedia applications, such as over-the-air television broadcasting, cable television transmission, satellite television transmission, internet streaming video transmission, such as Dynamic Self-Adjusting Data Streaming (DASH) over HTTP, digital video encoded onto data storage media, decoding digital video stored on data storage media, or other applications. In some instances, the system can be configured to support one-way or two-way video transmission to support applications such as video data streaming, video replay, video broadcasting, and / or video telephony.

[0204] In one example, the source device includes a video source, a video transcoder, and an output interface. The target device may include an input interface, a video decoder, and a display device. The video transcoder of the source device may be configured to apply the techniques disclosed herein. In other examples, the source and target devices may include other components or arrangements. For example, the source device may receive video data from an external video source (e.g., an external camera). Similarly, the target device may interface with an external display device, rather than including an integrated display device.

[0205] The above example system is merely an example. Techniques for parallel processing of video data can be implemented by any digital video encoding and / or decoding device. Although the techniques described herein are typically implemented by video encoding devices, they can also be implemented by video transcoders / decoders (often referred to as "decoders"). Furthermore, the techniques described herein can also be implemented by a video preprocessor. The source and target devices are merely examples of such decoding devices, where the source device generates decoded video data for transmission to the target device. In some instances, the source and target devices can operate in a substantially symmetrical manner, such that each device includes video encoding and decoding components. Therefore, the example system can support one-way or two-way video transmission between video devices, for example, for video streaming, video replay, video broadcasting, or video telephony.

[0206] Video sources may include video capturing devices, such as cameras, video archives containing previously captured video, and / or video feed interfaces that receive video from video content providers. Alternatively, a video source may generate computer-graphics-based data as source video, or a combination of live video, archived video, and computer-generated video. In some cases, if the video source is a camera, the source and target devices may form a so-called camera phone or video phone. However, as mentioned above, the techniques described in this application are generally applicable to video decoding and can be applied to wireless and / or wired applications. In each case, captured, pre-captured, or computer-generated video may be encoded by a video transcoder. The encoded video information can then be output to a computer-readable medium via an output interface.

[0207] As mentioned above, computer-readable media can include transient media such as those transmitted via wireless broadcasting or wired networks, or storage media such as hard disks, flash memory drives, optical discs, digital video discs, Blu-ray discs, or other computer-readable media (i.e., non-transitory storage media). In some instances, a network server (not shown) can, for example, transmit encoded video data received from a source device via a network and provide the encoded video data to a target device. Similarly, a computing device in a media production facility, such as a disk stamping machine, can receive encoded video data from a source device and produce a disk containing the encoded video data. Therefore, in various instances, computer-readable media can be understood to include one or more forms of computer-readable media.

[0208] The target device's input interface receives information from computer-readable media. The information in the computer-readable media may include syntax information defined by a video transcoder, which is also used by the video decoder. This syntax information includes the characteristics of description blocks and other decoding units (e.g., groups of pictures (GOPs)) and / or processed syntax elements. The display device displays the decoded video data to the user and may include any of a variety of display devices, such as cathode ray tube (CRT), liquid crystal display (LCD), plasma display, organic light-emitting diode (OLED) display, or another type of display device. Various embodiments of this invention have been described.

[0209] Specific details of the encoding device 104 and the decoding device 112 are shown in Figures 5 and 6, respectively. Figure 5 is a block diagram illustrating one or more examples of the encoding device 104 that may implement the techniques described herein. For example, the encoding device 104 may generate the syntax elements and / or structures described herein (e.g., syntax elements and / or structures of green relay data, such as complexity metrics (CM), or other syntax elements and / or structures). The encoding device 104 may perform intra-frame prediction and inter-frame prediction decoding on video blocks within video slices, tiles, sub-pictures, etc. As mentioned above, intra-frame decoding relies at least in part on spatial prediction to reduce or remove spatial redundancy within a given video frame or picture. Inter-frame decoding relies at least in part on temporal prediction to reduce or eliminate temporal redundancy within neighboring or surrounding frames of a video sequence. Intra-frame mode (I mode) may refer to any of several spatial-based compression modes. Inter-frame modes such as one-way prediction (P mode) or two-way prediction (B mode) may refer to any of several time-based compression modes.

[0210] Encoding device 104 includes a segmentation unit 35, a prediction processing unit 41, a filter unit 63, an image memory 64, an adder 50, a transform processing unit 52, a quantization unit 54, and an entropy coding unit 56. Prediction processing unit 41 includes a motion estimation unit 42, a motion compensation unit 44, and an in-frame prediction processing unit 46. For video block reconstruction, encoding device 104 also includes an inverse quantization unit 58, an inverse transform processing unit 60, and an adder 62. Filter unit 63 is intended to represent one or more loop filters, such as a deblocking filter, an auto-adjusting loop filter (ALF), and a sample-adjusted offset (SAO) filter. Although filter unit 63 is shown as an in-loop filter in FIG. 5, in other configurations, filter unit 63 can be implemented as a post-loop filter. Post-processing device 57 can perform additional processing on the encoded video data generated by encoding device 104. The techniques of this invention can be implemented by encoding device 104 in some cases. However, in other instances, one or more techniques of this invention can be implemented by post-processing device 57.

[0211] As shown in Figure 5, encoding device 104 receives video data, and segmentation unit 35 segments the data into video blocks. Segmentation may also include segmentation into slices, slice fragments, tiles, or other larger units, as well as video block segmentation, for example, according to a quadtree structure of LCUs and CUs. Encoding device 104 typically illustrates the components that encode video blocks within a video slice to be encoded. The slice may be divided into multiple video blocks (and may be divided into a set of video blocks called tiles). Prediction processing unit 41 may select one of a plurality of possible decoding modes for the current video block based on error results (e.g., decoding rate and distortion level, etc.), such as one of a plurality of intra-frame prediction decoding modes or one of a plurality of inter-frame prediction decoding modes. Prediction processing unit 41 may provide the obtained intra-frame or inter-frame decoded blocks to adder 50 to generate residual block data, and to adder 62 to reconstruct the encoded blocks used as reference pictures.

[0212] The intra-frame prediction processing unit 46 within the prediction processing unit 41 can perform intra-frame prediction decoding of the current video block relative to one or more adjacent blocks in the same frame or slice as the current block to be decoded, to provide spatial compression. The motion estimation unit 42 and motion compensation unit 44 within the prediction processing unit 41 perform inter-frame prediction decoding of the current video block relative to one or more prediction blocks in one or more reference images, to provide temporal compression.

[0213] Motion estimation unit 42 can be configured to determine the inter-frame prediction mode of video slices based on a predetermined mode of the video sequence. The predetermined mode can designate video slices in the sequence as P-slices, B-slices, or GPB-slices. Motion estimation unit 42 and motion compensation unit 44 can be highly integrated, but are shown separately for conceptual purposes. Motion estimation performed by motion estimation unit 42 is a procedure that generates motion vectors that estimate the motion of video blocks. For example, the motion vectors can indicate the displacement of a prediction unit (PU) of a video block within the current video frame or picture relative to a prediction block within a reference picture.

[0214] A predicted block is a block whose primitive difference pattern closely matches the PU of the video block to be decoded. The primitive difference can be determined by the sum of absolute differences (SAD), sum of squared differences (SSD), or other difference metrics. In some instances, the encoding device 104 can compute the values ​​of sub-integer primitive positions of a reference image stored in the image memory 64. For example, the encoding device 104 can interpolate the values ​​of quarter-priority, eighth-priority, or other fractional primitive positions of the reference image. Therefore, the motion estimation unit 42 can perform motion search relative to full primitive positions and fractional primitive positions and output a motion vector with fractional primitive precision.

[0215] The motion estimation unit 42 calculates the motion vector of the PU in the video block of the inter-frame decoding slice by comparing the position of the PU with the position of the predicted block of the reference image. Reference images can be selected from a first reference image list (list 0) or a second reference image list (list 1), each list identifying one or more reference images stored in the image memory 64. The motion estimation unit 42 sends the calculated motion vector to the entropy coding unit 56 and the motion compensation unit 44.

[0216] Motion compensation performed by motion compensation unit 44 may involve acquiring or generating prediction blocks based on motion vectors determined by motion estimation, possibly performing interpolation of sub-primitive precision. Upon receiving the motion vector of the PU for the current video block, motion compensation unit 44 can locate the prediction block pointed to by the motion vector in a list of reference images. Encoding device 104 forms a residual video block by subtracting the primitive values ​​of the prediction block from the primitive values ​​of the decoded current video block, forming a primitive difference. The primitive difference forms the residual data of the block and may include luminance and chrominance difference components. Adder 50 represents one or more components performing the subtraction operation. Motion compensation unit 44 may also generate syntax elements associated with video blocks and video slices for use by decoding device 112 when decoding video blocks of video slices.

[0217] Intra-frame prediction processing unit 46 can predict the current block intra-frame as an alternative to inter-frame prediction performed by motion estimation unit 42 and motion compensation unit 44, as described above. Specifically, intra-frame prediction processing unit 46 can determine the intra-frame prediction mode used to encode the current block. In some instances, intra-frame prediction processing unit 46 can encode the current block using various intra-frame prediction modes (e.g., during individual coding passes), and intra-frame prediction processing unit 46 can select a suitable intra-frame prediction mode from the tested modes. For example, intra-frame prediction processing unit 46 can use rate-distortion analysis of intra-frame prediction modes for various tests to calculate rate-distortion values, and can select the intra-frame prediction mode with the best rate-distortion characteristics from the tested modes. Rate-distortion analysis typically determines the amount of distortion (or error) between the coded block and the original uncoded block encoded to produce the coded block, as well as the bit rate (i.e., number of bits) used to generate the coded block. The in-frame prediction processing unit 46 can calculate the ratio based on the distortion and rate of various coded blocks to determine which in-frame prediction mode exhibits the best rate-distortion value for that block.

[0218] In any case, after selecting an intra-frame prediction mode for a block, the intra-frame prediction processing unit 46 can provide information indicating the selected intra-frame prediction mode for the block to the entropy coding unit 56. The entropy coding unit 56 can encode the information indicating the selected intra-frame prediction mode. The encoding device 104 can include in the transmitted bitstream configuration data definitions for various blocks, as well as indications of the most probable intra-frame prediction mode for each context, an intra-frame prediction mode index table, and modified intra-frame prediction mode index tables. The bitstream configuration data can include a plurality of intra-frame prediction mode index tables and a plurality of modified intra-frame prediction mode index tables (also called coded character mapping tables).

[0219] After prediction processing unit 41 generates a prediction block for the current video block via inter-frame prediction or intra-frame prediction, encoding device 104 forms a residual video block by subtracting the prediction block from the current video block. The residual video data in the residual block can be included in one or more TUs and applied to transform processing unit 52. Transform processing unit 52 uses transformations such as discrete cosine transform (DCT) or conceptually similar transformations to transform the residual video data into residual transform coefficients. Transform processing unit 52 can convert the residual video data from the primitive domain to the transform domain, such as the frequency domain.

[0220] The transform processing unit 52 can send the obtained transform coefficients to the quantization unit 54. The quantization unit 54 quantizes the transform coefficients to further reduce the bit rate. Quantization processing can reduce the bit depth associated with some or all of the coefficients. The degree of quantization can be modified by adjusting the quantization parameters. In some instances, the quantization unit 54 can then perform a scan of a matrix including the quantized transform coefficients. Alternatively, the entropy coding unit 56 can perform the scan.

[0221] After quantization, entropy coding unit 56 performs entropy coding on the quantized transform coefficients. For example, entropy coding unit 56 can perform context-adjusted variable-length decoding (CAVLC), context-adjusted binary arithmetic decoding (CABAC), syntax-based context-adjusted binary arithmetic decoding (SBAC), probabilistic interval partitioning entropy (PIPE) decoding, or another entropy coding technique. After entropy coding by entropy coding unit 56, the encoded bitstream can be sent to decoding device 112 or archived for later transmission or retrieval by decoding device 112. Entropy coding unit 56 can also perform entropy decoding on the motion vectors and other syntax elements of the decoded current video slice.

[0222] Inverse quantization unit 58 and inverse transform processing unit 60 apply inverse quantization and inverse transform, respectively, to reconstruct the residual blocks in the primitive domain for later use as reference blocks in a reference image. Motion compensation unit 44 can compute the reference block by adding the residual blocks to a prediction block of one of the reference images in the reference image list. Motion compensation unit 44 can also apply one or more interpolation filters to the reconstructed residual blocks to compute sub-integer primitive values ​​for motion estimation. Adder 62 adds the reconstructed residual blocks to the motion-compensated prediction blocks generated by motion compensation unit 44 to produce a reference block for storage in image memory 64. This reference block can be used by motion estimation unit 42 and motion compensation unit 44 as a reference block for inter-frame prediction of blocks in subsequent video frames or images.

[0223] In this way, the encoding device 104 of Figure 5 represents an instance of a video transcoder configured to perform any of the techniques described herein. In some cases, some of the techniques described herein may also be implemented by the post-processing device 57.

[0224] Figure 6 is a block diagram of the illustrated example decoding device 112. The decoding device 112 includes an entropy decoding unit 80, a prediction processing unit 81, an inverse quantization unit 86, an inverse transform processing unit 88, an adder 90, a filter unit 91, and an image memory 92. The prediction processing unit 81 includes a motion compensation unit 82 and an in-frame prediction processing unit 84. In some instances, the decoding device 112 can perform a decoding procedure that is generally the opposite of the encoding procedure described with respect to the encoding device 104 of Figure 5.

[0225] In the decoding process, decoding device 112 receives video blocks representing encoded video slices and encoded video bitstreams of associated syntax elements sent by encoding device 104. In some embodiments, decoding device 112 may receive encoded video bitstreams from encoding device 104. In some embodiments, decoding device 112 may receive encoded video bitstreams from network entity 79 (e.g., a server, a media-aware network component (MANE), a video editor / assembler, or other such devices configured to implement one or more of the techniques described above). Network entity 79 may or may not include encoding device 104. Some of the techniques described herein may be implemented by network entity 79 before it sends the encoded video bitstream to decoding device 112. In some video decoding systems, network entity 79 and decoding device 112 may be part of a separate device, while in other cases, the same device containing decoding device 112 may perform the functions described regarding network entity 79.

[0226] The entropy decoding unit 80 of the decoding device 112 performs entropy decoding on the bitstream to generate quantization coefficients, motion vectors, and other syntax elements. The entropy decoding unit 80 forwards the motion vectors and other syntax elements to the prediction processing unit 81. The decoding device 112 can receive syntax elements at the video slice level and / or the video block level. The entropy decoding unit 80 can process and analyze both fixed-length and variable-length syntax elements from one or more parameter sets such as VPS, SPS, and PPS.

[0227] When a video slice is decoded into an intra-frame decoded (I) slice, the intra-frame prediction processing unit 84 of the prediction processing unit 81 can generate prediction data for the video block of the current video slice based on the intra-frame prediction mode notified by the signal transmission and data from the previous decoded block of the current frame or picture. When a video frame is decoded into an inter-frame decoded (i.e., B, P, or GPB) slice, the motion compensation unit 82 of the prediction processing unit 81 generates prediction blocks for the video block of the current video slice based on the motion vector and other syntax elements received from the entropy decoding unit 80. The prediction block can be generated from one of the reference pictures in the reference picture list. The decoding device 112 can construct the reference frame list (list 0 and list 1) based on the reference pictures stored in the picture memory 92 using a preset construction technique.

[0228] The motion compensation unit 82 determines the prediction information of the video blocks in the current video slice by parsing motion vectors and other syntax elements, and uses the prediction information to generate prediction blocks for the decoded current video block. For example, the motion compensation unit 82 may use one or more syntax elements in the parameter set to determine the prediction mode (e.g., intra-frame or inter-frame prediction), the inter-frame prediction slice type (e.g., B-slice, P-slice, or GPB-slice), the construction information of one or more reference picture lists for the slice, the motion vector of each inter-frame encoded video block of the slice, the inter-frame prediction state of each inter-frame decoded video block of the slice, and other information for decoding the video blocks in the current video slice.

[0229] The motion compensation unit 82 can also perform interpolation based on an interpolation filter. The motion compensation unit 82 can use the interpolation filter used by the encoding device 104 during the encoding of the video block to calculate the interpolation of the sub-integer primitives of the reference block. In this case, the motion compensation unit 82 can determine the interpolation filter used by the encoding device 104 from the received syntax elements and can use that interpolation filter to generate the prediction block.

[0230] The inverse quantization unit 86 inverse-quantizes or dequantizes the quantized transform coefficients provided in the bitstream and decoded by the entropy decoding unit 80. The inverse quantization process may include determining the quantization level using quantization parameters calculated by the encoding device 104 for each video block of the video slice, and similarly determining the inverse quantization level to be applied. The inverse transform processing unit 88 applies an inverse transform (e.g., inverse DCT or other suitable inverse transform), an inverse integer transform, or a conceptually similar inverse transform to the transform coefficients to produce residual blocks in the primitive domain.

[0231] After the motion compensation unit 82 generates a prediction block for the current video block based on motion vectors and other syntax elements, the decoding device 112 forms a decoded video block by adding the residual block from the inverse transform processing unit 88 to the corresponding prediction block generated by the motion compensation unit 82. Adder 90 represents one or more components performing this addition operation. If needed, loop filters (in or after the decoding loop) can also be used to smooth primitive transformations or otherwise improve video quality. Filter unit 91 is intended to represent one or more loop filters, such as a deblocking filter, an auto-adjusting loop filter (ALF), and a sample auto-adjusting offset (SAO) filter. Although filter unit 91 is shown as an in-loop filter in FIG. 6, in other configurations, filter unit 91 can be implemented as a post-loop filter. Subsequently, the decoded video block in a given frame or picture is stored in picture memory 92, which stores reference pictures for subsequent motion compensation. Image memory 92 also stores decoded video for later presentation on a display device, such as video target device 122 shown in Figure 1.

[0232] In this way, the decoding device 112 of Figure 6 represents an instance of a video decoder configured to perform any of the techniques described herein.

[0233] As used herein, the term "computer-readable media" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or carrying instructions and / or data. Computer-readable media may include non-transitory media capable of storing data, but does not include carrier waves and / or transient electronic signals propagated wirelessly or via wired connections. Examples of non-transitory media may include, but are not limited to, magnetic disks or magnetic tapes, optical storage media such as optical discs (CDs) or digital versatile disks (DVDs), flash memory, memory, or storage devices. Computer-readable media may store code and / or machine-executable instructions thereon, which may represent any combination of programs, functions, subroutines, programs, routines, modules, software packages, software components, or instructions, data structures, or program statements. Code segments may be coupled to another code segment or hardware circuitry by transmitting and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. can be transmitted, forwarded, or sent through any suitable means (including memory sharing, message passing, symbol passing, network sending, etc.).

[0234] In some embodiments, computer-readable storage devices, media, and memory may include cable or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer-readable storage media explicitly excludes media such as energy, carrier signals, electromagnetic waves, and the signals themselves.

[0235] Specific details are provided in the foregoing description to provide a thorough understanding of the embodiments and examples provided herein. However, those skilled in the art to which this invention pertains will understand that embodiments can be practiced without these specific details. For clarity, in some cases, the technology may be presented as comprising individual functional blocks, including functional blocks comprising methods embodied in software or a combination of hardware and software, including devices, device components, steps, or conventional functional blocks. Other components may be used in addition to those shown in the figures and / or described herein. For example, circuits, systems, networks, programs, and other components may be shown as components in block diagram form so as not to obscure the embodiments with unnecessary details. In other cases, known circuits, programs, algorithms, structures, and techniques may be shown without unnecessary details to avoid obscuring the embodiments.

[0236] The various embodiments can be described above as programs or methods, which are illustrated as flowcharts, data flow diagrams, structure diagrams, or block diagrams. Although a flowchart can describe operations as a sequential procedure, many operations can be executed in parallel or concurrently. Furthermore, the order of operations can be rearranged. A program terminates when its operations are completed, but there may be other steps not included in the diagram. A program can correspond to a method, function, procedure, subroutine, subroutine, etc. When a program corresponds to a function, its termination can correspond to the function returning to the calling function or the main function.

[0237] The programs and methods according to the examples described above can be implemented using computer-executable instructions stored in or otherwise obtained from a computer-executable medium. These instructions may include, for example, instructions and data that cause or otherwise configure a general-purpose computer, special-purpose computer, or processing device to perform a function or group of functions. Parts of the computer resources may be accessible via a network. The computer-executable instructions may be, for example, binary files, intermediate format instructions (e.g., assembly language, firmware, source code, etc.). Examples of computer-executable media that can be used to store instructions, information used, and / or information created during the methods according to the examples include magnetic disks or optical disks, flash memory, USB devices with non-volatile memory, network storage devices, etc.

[0238] The devices implementing the procedures and methods disclosed herein may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, the program code or code snippets (e.g., computer program products) that perform the necessary tasks may be stored on computer-readable or machine-readable media. A processor may perform the necessary tasks. Typical examples of form factors include laptops, smartphones, mobile phones, tablet computers or other small personal computers, personal digital assistants, rack-mounted devices, standalone devices, etc. The functionality described herein may also be embodied in peripheral devices or add-in cards. By further example, this functionality may also be implemented on circuit boards between different chips, or on different programs running in a single device.

[0239] Instructions, media for transmitting such instructions, computing resources for executing such instructions, and other structures for supporting such computing resources are instance components for providing the functionality described in this case.

[0240] In the foregoing description, various aspects of this application have been described with reference to specific embodiments thereof; however, those skilled in the art will recognize that this application is not limited thereto. Therefore, although illustrative embodiments of this application have been described in detail herein, it should be understood that the concepts of this application may be embodied and applied differently in other ways, and the appended claims are intended to be construed as including such variations, unless limited by prior art. Various features and aspects of the above-described applications may be used individually or in combination. Furthermore, embodiments may be used in any number of environments and applications beyond those described herein without departing from the broader spirit and scope of this specification. Therefore, the specification and drawings are to be considered illustrative rather than restrictive. For illustrative purposes, the methods are described in a particular order. It should be understood that in alternative embodiments, these methods may be performed in a different order than that described.

[0241] Those skilled in the art will understand that the symbols or terms less than (“<”) and greater than (“>”) used in this document can be replaced with less than or equal to (“<”). ") and greater than or equal to (" The symbol 」) is used without departing from the scope of this specification.

[0242] When a component is described as being "configured" to perform certain operations, the operations can be performed, for example, by designing electronic circuits or other hardware, by programming programmable electronic circuits (such as microprocessors, or other suitable electronic circuits), or any combination thereof.

[0243] The phrase "coupled to" means any component that is physically connected directly or indirectly to another component, and / or communicates with another component (e.g., connected to other components via wired or wireless connections, and / or other suitable communication interfaces).

[0244] The claim language or other language in the disclosure that lists "at least one of" and / or "one or more" of a set indicates that one or more members of the set (in any combination) satisfy the claim. For example, "at least one of A and B" in the claim language means A, B, or A and B. In another instance, "at least one of A, B, and C" in the claim language means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language that lists "at least one of" and / or "one or more" of a set does not limit the set to the items listed in the set. For example, "at least one of A and B" in the claims language can mean A, B, or A and B, and can additionally include items not listed in the A and B set.

[0245] The various illustrative logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, firmware, or a combination thereof. To clearly illustrate this interchangeability between hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above in general terms of their functionality. Whether these functions are implemented as hardware or software depends on the specific application and design constraints imposed on the overall system. Those skilled in the art can implement the described functions in different ways for each specific application; however, such implementation decisions should not be construed as departing from the scope of this document.

[0246] The techniques described herein can also be implemented in electronic hardware, computer software, firmware, or any combination thereof. These techniques can be implemented in any of a variety of devices, such as general-purpose computers, wireless communication handheld devices, or multi-purpose integrated circuit devices, including applications in wireless communication handheld devices and other devices. Any feature described as a module or component can be implemented together in an integrated logic device or separately as a discrete but interoperable logic device. If implemented in software, these techniques can be implemented at least in part by a computer-readable data storage medium including program code that includes instructions for executing one or more of the methods described above when executed. The computer-readable data storage medium can form part of a computer program product, which may include packaging materials. Computer-readable media may include memory or data storage media, such as random access memory (RAM), such as synchronous dynamic random access memory (SDRAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electronically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, etc. Additionally or alternatively, these technologies may be implemented at least in part by computer-readable communication media that carries or transmits program code in the form of instructions or data structures, and that the program code can be accessed, read, and / or executed by a computer (e.g., a propagating signal or wave).

[0247] The code can be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such processors can be configured to perform any of the techniques described herein. A general-purpose processor may be a microprocessor, but alternatively, the processor may be any known processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a DSP and a microprocessor, a combination of multiple microprocessors, a combination of one or more microprocessors with a DSP core, or any other such configuration. Therefore, as used herein, the term "processor" may refer to any of the foregoing structures, any combination of the foregoing structures, or any other structure or apparatus suitable for implementing the techniques described herein. Additionally, in some cases, the functionality described herein may be provided within a dedicated software module or a hardware module configured for encoding and decoding, or incorporated into a combined video transcoder-decoder (CODEC).

[0248] The explanatory features of this case include:

[0249] 1. An apparatus for processing video data, comprising: at least one memory; and at least one processor coupled to the at least one memory, the at least one processor being configured to: acquire a bitstream; retrieve a subtlety type syntax element associated with the bitstream, the subtlety type syntax element specifying the subtlety type of one or more pictures to which a complexity metric (CM) associated with the bitstream applies; retrieve a time period type syntax element associated with the bitstream, the time period type syntax element indicating an upcoming time period or a set of pictures to which the CM applies; and decode a portion of the bitstream based on the subtlety type syntax element and the time period type syntax element.

[0250] State 2. The apparatus according to request 1, wherein the value of the subtlety type syntax element specifies that CM applies to a bitstream of a picture or a portion of a picture, the portion of which is smaller than the whole picture.

[0251] State 3. The apparatus according to request 1, wherein the value of the subtlety type syntax element specifies that the CM is applicable to at least one of the following rows of slices, tiles, subpictures, extensible layers, or decoder tree unit (CTU) of one or more images in a bitstream.

[0252] Format 4. The apparatus according to request 1, wherein the time period type syntax element indicates a specified time interval of the upcoming time period, the number of pictures of the upcoming time period, the upcoming time period including all pictures up to including the next slice, or the upcoming time period including at least one of a single picture.

[0253] 5. The apparatus according to request 1, wherein at least one processor is configured to retrieve a subtlety-level CM syntax structure associated with a bitstream, the subtlety-level CM syntax structure specifying a subtlety-level complexity measure of one or more subtle segments of the bitstream in an upcoming time period.

[0254] 6. The apparatus according to claim 1, wherein at least one processor is configured to: retrieve an additional time-type syntax element associated with a bitstream, the additional time-type syntax element being associated with a subtlety syntax element, wherein the additional time-type syntax element is different from the time-type syntax element; and decode a portion of the bitstream based on the subtlety syntax element and the additional time-type syntax element.

[0255] Format 7. The apparatus according to claim 1, wherein at least one processor is configured to retrieve at least one of the following: a subpicture syntax element associated with the bitstream, the subpicture syntax element indicating a signaling notification subpicture identifier (ID) when the CM is applied to multiple pictures; a decode tree block (CTB) number syntax element associated with the bitstream, which, when the detail type is equal to a slice or tile and the upcoming time spans multiple pictures, can be a signaling notification in the CM indicating the total number of decode tree detail blocks in the upcoming time span; or an average decode tree block (CTB) number syntax element associated with the bitstream, the average CTB number syntax element indicating the average number of CTBs per detail or 4×4 blocks per picture.

[0256] 8. The apparatus according to claim 1, wherein when an available in-frame decoder is present in at least a portion of the bit stream, a signal transmission associated with at least a portion of the stream notifies the in-frame decoder statistics.

[0257] 9. The apparatus according to claim 1, wherein when an available inter-frame decoding block exists in at least a portion of the bit stream, a signal transmission associated with at least a portion of the bit stream notifies the inter-frame decoding block statistics.

[0258] 10. The apparatus according to claim 1, wherein at least one processor is configured to determine the operating frequency of the apparatus based on a CM associated with a bit stream.

[0259] 11. The apparatus according to claim 1 also includes a display configured to display at least a portion of a bitstream.

[0260] 12. The apparatus according to claim 1, wherein the apparatus is one of a mobile device, a wearable device, an augmented reality device, a camera, a personal computer, a vehicle, a robotic device, a television set, or a computing device.

[0261] 13. A method for processing video data, comprising: obtaining a bitstream; retrieving a subtlety type syntax element associated with the bitstream, the subtlety type syntax element specifying the subtlety type of one or more pictures to which a complexity metric (CM) associated with the bitstream applies; retrieving a time period type syntax element associated with the bitstream, the time period type syntax element indicating an upcoming time period or a set of pictures to which the CM applies; and decoding a portion of the bitstream based on the subtlety type syntax element and the time period type syntax element.

[0262] Format 14. According to the method of request item 13, wherein the value of the subtlety type syntax element specifies that CM applies to a bitstream of an image or a portion of an image, the portion of which is smaller than the whole image.

[0263] Format 15. According to the method of request item 13, wherein the value of the subtlety type syntax element specifies that the CM applies to at least one of the slices, tiles, subpictures, extensible layers, or decoder tree unit (CTU) rows of one or more pictures in the bitstream.

[0264] Format 16. According to the method of request item 13, wherein the time period type syntax element indicates the specified time interval of the upcoming time period, the number of pictures in the upcoming time period, the upcoming time period including all pictures up to including the next slice, or the upcoming time period including at least one of a single picture.

[0265] Example 17. The method according to request 13 also includes retrieving a subtlety-level CM syntax structure associated with the bitstream, the subtlety-level CM syntax structure specifying the subtlety-level complexity measure of one or more subtle segments of the bitstream in the upcoming time period.

[0266] Sample 18. The method according to request 13 also includes: retrieving an additional time-type syntax element associated with the bitstream, the additional time-type syntax element being associated with a subtlety type syntax element, wherein the additional time-type syntax element is different from the time-type syntax element, and decoding a portion of the bitstream based on the subtlety type syntax element and the additional time-type syntax element.

[0267] Format 19. The method according to claim 13 also includes retrieving at least one of the following: a subpicture syntax element associated with the bitstream, the subpicture syntax element indicating a signaling notification subpicture identifier (ID) when the CM is applied to multiple pictures; a decode tree block (CTB) number syntax element associated with the bitstream, which, when the detail type is equal to a slice or tile and the upcoming time spans multiple pictures, can be a signaling notification in the CM indicating the total number of decode tree detail blocks on the upcoming time; or an average decode tree block (CTB) number syntax element associated with the bitstream, the average CTB number syntax element indicating the average number of CTBs per detail or 4×4 blocks per picture.

[0268] 20. The method of claim 13, wherein when an available in-frame decoder exists in at least a portion of the bit stream, a signal transmission associated with at least a portion of the bit stream notifies the in-frame decoder statistics.

[0269] 21. The method of claim 13, wherein when an available inter-frame decoding block exists in at least a portion of the bit stream, signal transmission associated with at least a portion of the bit stream notifies the inter-frame decoding block statistics.

[0270] 22. The method according to claim 13 also includes displaying at least a portion of the bitstream on a display.

[0271] 23. The method according to claim 13 also includes determining the operating frequency of the device based on the CM associated with the bit stream.

[0272] 24. An apparatus for processing video data, comprising: at least one memory; and at least one processor coupled to the at least one memory, the at least one processor being configured to: acquire video data; generate a subtlety type syntax element for a bitstream, the subtlety type syntax element specifying the subtlety type of one or more pictures to which a complexity metric (CM) associated with the bitstream applies; generate a time period type syntax element associated with the bitstream, the time period type syntax element indicating an upcoming time period or a set of pictures to which the CM applies; generate a bitstream associated with the video data, the bitstream including the subtlety type syntax element and the time period type syntax element; and output the generated bitstream.

[0273] 25. The apparatus according to request 24, wherein the value of the subtlety type syntax element specifies that CM applies to a bitstream of a picture or a portion of a picture, the portion of which is smaller than the whole picture.

[0274] 26. The apparatus according to request 24, wherein the value of the subtlety type syntax element specifies that the CM is applicable to at least one of the following rows of slices, tiles, subpictures, extensible layers, or decoder tree unit (CTU) of one or more pictures in a bitstream.

[0275] 27. The apparatus according to claim 24, wherein the time period type syntax element indicates a specified time interval of the upcoming time period, the number of pictures of the upcoming time period, the upcoming time period including all pictures up to including the next slice, or the upcoming time period including at least one of a single picture.

[0276] 28. The apparatus according to request 24, wherein one or more processors are configured to generate a subtlety-level CM syntax structure for a bitstream, the subtlety-level CM syntax structure specifying a subtlety-level complexity metric for one or more entities in an upcoming time period.

[0277] 29. The apparatus according to claim 24, wherein the time period type syntax element indicates a specified time interval of the upcoming time period, the number of pictures of the upcoming time period, the upcoming time period including all pictures up to including the next slice, or the upcoming time period including at least one of a single picture.

[0278] 30. The apparatus according to claim 24, wherein at least one processor is configured to generate associated additional time-type syntax elements for a bitstream, the additional time-type syntax elements being associated with a subtlety type syntax element, wherein the additional time-type syntax elements are different from the time-type syntax elements, and wherein the additional time-type syntax elements are used to decode a portion of the bitstream having subtlety type syntax elements.

[0279] 31. The apparatus according to claim 24, wherein at least one processor is configured to generate for a bitstream at least one of the following: a subpicture syntax element associated with the bitstream, the subpicture syntax element indicating a signaling notification subpicture identifier (ID) when a CM is applied to multiple pictures; a decode tree block (CTB) number syntax element associated with the bitstream, which, when the detail type is equal to a slice or tile and the upcoming time spans multiple pictures, can be a signaling notification in the CM indicating the total number of decode tree detail blocks in the upcoming time span; or an average decode tree block (CTB) number syntax element associated with the bitstream, the average CTB number syntax element indicating the average number of CTBs per detail or 4×4 blocks per picture.

[0280] 32. The apparatus according to claim 24, wherein when an available in-frame decoder block exists in at least a portion of the bit stream, a signal transmission associated with at least a portion of the bit stream notifies the in-frame decoder block statistics.

[0281] 33. The apparatus according to claim 24, wherein when an available inter-frame decoding block exists in at least a portion of the bit stream, signal transmission associated with at least a portion of the bit stream notifies the inter-frame decoding block statistics.

[0282] 34. The apparatus according to claim 24 also includes a camera configured to capture video data.

[0283] 35. The apparatus according to claim 24, wherein the apparatus is one of a mobile device, a wearable device, an augmented reality device, a camera, a personal computer, a vehicle, a robotic device, a television set, or a computing device.

[0284] 36. The apparatus according to claim 1, wherein at least one processor is configured to retrieve one or more quality recovery metrics associated with one or more minor segments of a bitstream.

[0285] 37. The apparatus according to claim 1, wherein at least one processor is configured to: receive supplemental enhancement information (SEI) messages; and retrieve subtle type syntax elements from the SEI messages.

[0286] 38. The apparatus according to claim 1, wherein the apparatus includes a decoder.

[0287] 39. The apparatus according to claim 1, wherein the apparatus includes a camera configured to capture one or more images.

[0288] 40. The method according to request 13, wherein at least one processor is configured to retrieve one or more quality recovery metrics associated with one or more minor segments of a bitstream.

[0289] 41. According to the method of request item 13, at least one processor is configured to: receive supplemental enhancement information (SEI) messages; and retrieve subtle type syntax elements from the SEI messages.

[0290] 42. The method according to claim 13, wherein the device includes a decoder.

[0291] 43. The method according to claim 13, wherein the device includes a camera configured to capture one or more images.

[0292] 44. The apparatus according to claim 24, wherein at least one processor is configured to encode one or more quality recovery metrics associated with one or more minor segments of a bitstream.

[0293] 45. The apparatus according to claim 24, wherein at least one processor is configured to: encode a Supplemental Enhancement Information (SEI) message having subtle type syntax elements.

[0294] 46. ​​The apparatus according to claim 24, wherein the apparatus includes a decoder.

[0295] 47. The apparatus according to claim 24, wherein the apparatus includes a camera configured to capture one or more images of video data.

[0296] Mode 48: A method for processing video data, comprising one or more operations according to any one of modes 24 to 47.

[0297] Format 49: A non-transitory computer-readable medium having instructions stored thereon, which, when executed by one or more processors, cause one or more processors to perform one or more operations according to any one of formats 13 to 23 and 48.

[0298] Mode 50: An apparatus for processing video data, comprising components for performing one or more of the operations according to any one of modes 13 to 23 and mode 48.

[0299] Format 1A: An apparatus for processing video data, comprising: at least one memory; and one or more processors coupled to the at least one memory, the one or more processors being configured to: acquire a bitstream; and retrieve a subtlety type syntax element associated with the bitstream, the subtlety type syntax element specifying the subtlety type to which a complexity metric (CM) associated with the bitstream applies.

[0300] State 2A: A device based on State 1A, wherein the value of the subtlety type syntax element specifies that CM is applicable to the image of the bitstream.

[0301] State 3A: A device based on State 2A, wherein the value of the subtlety type syntax element specifies that CM applies to slices of bitstreams.

[0302] State 4A: A device based on State 2A, wherein the value of the subtlety type syntax element specifies the CM applicable to the bitstream of the plot.

[0303] State 5A: A device based on State 2A, wherein the value of the subtlety type syntax element specifies the sub-image for which the CM applies to the bitstream.

[0304] State 6A: A device based on State 2A, wherein the value of the subtlety type syntax element specifies the CM applicable to the scalable layer of the bitstream.

[0305] State 7A: A device based on State 2A, wherein the value of the subtlety type syntax element specifies the CM applicable to the decode tree unit (CTU) line of the bit stream.

[0306] State 8A: A device based on any of the states 1A to 7A, wherein one or more processors are configured to retrieve a time period type syntax element associated with the bit stream, the time period type syntax element specifying the type of the upcoming time period to which the CM applies.

[0307] State 9A: A device based on any of the states 1A to 8A, wherein one or more processors are configured to retrieve a picture-level CM syntax structure associated with a bitstream, the picture-level CM syntax structure specifying a complexity measure of one or more pictures within a time period.

[0308] State 10A: A device based on any of the states 1A to 9A, wherein one or more processors are configured to retrieve a subtlety-level CM syntax structure associated with a bitstream, the subtlety-level CM syntax structure specifying a subtlety-level complexity measure of one or more entities over a period of time.

[0309] State 11A: An apparatus according to state 10A, wherein one or more entities include at least one of slice, tile, sub-picture and layer.

[0310] State 12A: A device according to any of states 1A to 11A, wherein one or more processors are configured to retrieve sub-picture syntax elements associated with a bitstream, and when the time spans multiple pictures, to signal in the CM to notify the sub-picture syntax element indicating the sub-picture identifier (ID).

[0311] State 13A: A device according to any of states 1A to 12A, wherein one or more processors are configured to retrieve the CTB number syntax elements associated with the bit stream, and when the nuance type is equal to a slice or tile and the time period spans multiple pictures, the CTB number syntax elements can be signaled in the CM to indicate the total number of CTB numbers in a time period.

[0312] State 14A: A device according to any of the states 1A to 13A, wherein one or more processors are configured to retrieve the average decode tree block (CTB) number syntax element associated with the bit stream, the average CTB number syntax element indicating the average number of CTBs or 4×4 blocks per picture per minute.

[0313] State 15A: A device according to any one of states 1A to 14A, wherein when an available in-frame decoder block exists in at least a portion of the bit stream, a signal transmission associated with at least a portion of the bit stream notifies the in-frame decoder block statistics.

[0314] State 16A: A device according to any one of states 1A to 15A, wherein when an available inter-frame decoding block exists in at least a portion of the bit stream, a signal transmission associated with at least a portion of the bit stream notifies the inter-frame decoding block statistics.

[0315] State 17A: A device according to any one of states 1A to 16A, wherein one or more processors are configured to retrieve one or more quality recovery metrics associated with one or more minor segments of a bitstream.

[0316] State 18A: The apparatus according to State 17A, wherein one or more micro-segments of the bit stream include at least one of slices, tiles and sub-pictures.

[0317] State 19A: A device according to any one of states 1A to 18A, wherein one or more processors are configured to: receive supplemental enhancement information (SEI) messages; and retrieve subtle type syntax elements from the SEI messages.

[0318] State 20A: A device according to any of the states 1A to 19A, wherein one or more processors are configured to determine the operating frequency of the device based on the CM associated with the bit stream.

[0319] State 21A: A device according to any one of states 1A to 20A, wherein the device includes a decoder.

[0320] Format 22A: The apparatus according to any one of formats 1A to 21A also includes a display configured to display one or more output images.

[0321] Format 23A: A device according to any one of formats 1A to 22A, including a camera configured to capture one or more images.

[0322] State 24A: A device according to any one of states 1A to 23A, wherein the device is a mobile device.

[0323] Format 25A: A method for processing video data, comprising one or more operations based on any one of formats 1A to 24A.

[0324] Format 26A: A non-transitory computer-readable medium having instructions stored thereon, which, when executed by one or more processors, cause one or more processors to perform one or more operations according to any one of formats 1A to 24A.

[0325] Format 27A: An apparatus for processing video data, comprising one or more components for performing operations according to any one of formats 1A to 24A.

[0326] 28A: An apparatus for processing video data, comprising: a memory; and one or more processors coupled to the memory, the one or more processors being configured to: acquire video data; generate a bitstream associated with the video data; and generate a subtle type syntax element for the bitstream, the subtle type syntax element specifying the subtle type to which a complexity metric (CM) associated with the bitstream applies.

[0327] State 29A: A device according to State 28A, wherein the value of the subtlety type syntax element specifies that the CM is applicable to the image of the bitstream.

[0328] State 30A: A device based on State 29A, wherein the value of the subtlety type syntax element specifies that the CM is applicable to slices of bitstreams.

[0329] State 31A: A device according to State 29A, wherein the value of the subtlety type syntax element specifies the CM applicable to the plot of the bit stream.

[0330] State 32A: A device according to State 29A, wherein the value of the subtlety type syntax element specifies the sub-image for which the CM applies to the bitstream.

[0331] State 33A: A device according to State 29A, wherein the value of the subtlety type syntax element specifies the CM applicable to the scalable layer of the bitstream.

[0332] State 34A: A device according to State 29A, wherein the value of the subtlety type syntax element specifies the CM applicable to the decode tree unit (CTU) line of the bit stream.

[0333] Format 35A: A device based on any of formats 28A to 34A, wherein one or more processors are configured to generate a time-type syntax element for a bitstream that specifies the type of the upcoming time-stream to which the CM applies.

[0334] State 36A: A device according to any of the states 28A to 35A, wherein one or more processors are configured to generate a picture-level CM syntax structure for a bitstream that specifies a complexity metric for one or more pictures over a period of time.

[0335] State 37A: A device according to any of the states 28A to 36A, wherein one or more processors are configured to generate a fine-grained CM syntax structure for a bitstream, the fine-grained CM syntax structure specifying a fine-grained complexity metric for one or more entities over a period of time.

[0336] State 38A: A device according to state 37A, wherein one or more entities include at least one of slice, tile, sub-picture and layer.

[0337] State 39A: A device according to any of states 28A to 38A, wherein one or more processors are configured to generate sub-picture syntax elements for a bitstream, and when the time spans multiple pictures, signal transmission in the CM notifies the sub-picture syntax elements of the sub-picture identifier (ID).

[0338] State 40A: A device according to any of the states 28A to 39A, wherein one or more processors are configured to generate a decode tree block (CTB) number syntax element for a bit stream, wherein when the detail type is equal to a slice or tile and the time period spans multiple pictures, a decode tree block (CTB) number syntax element can be signaled in the CM to indicate the total number of decode tree luminance blocks in a time period.

[0339] Sample 41A: A device according to any of the samples 28A to 40A, wherein one or more processors are configured to generate an average code tree block (CTB) number syntax element for a bitstream, the average code tree block (CTB) number syntax element indicating the average number of CTBs or 4×4 blocks per image per minute.

[0340] State 42A: A device according to any one of states 28A to 41A, wherein when an available in-frame decoder block exists in at least a portion of the bit stream, a signal transmission associated with at least a portion of the bit stream notifies the in-frame decoder block statistics.

[0341] State 43A: A device according to any one of states 28A to 42A, wherein when an available inter-frame decoding block exists in at least a portion of the bit stream, a signal transmission associated with at least a portion of the bit stream notifies the inter-frame decoding block statistics.

[0342] State 44A: A device according to any one of states 28A to 43A, wherein one or more processors are configured to generate one or more quality recovery metrics associated with one or more fine segments of the bit stream.

[0343] State 45A: A device according to state 44A, wherein one or more micro-segments of the bit stream include at least one of slices, tiles and sub-pictures.

[0344] State 46A: A device according to any one of states 28A to 45A, wherein one or more processors are configured to: generate supplemental enhancement information (SEI) messages; and include subtle type syntax elements in the SEI messages.

[0345] State 47A: A device according to any one of states 28A to 46A, wherein one or more processors are configured to store bit streams.

[0346] State 48A: A device according to any one of states 28A to 47A, wherein one or more processors are configured to transmit a bit stream.

[0347] State 49A: A device according to any one of states 28A to 48A, wherein the device includes an encoder.

[0348] Format 50A: The apparatus according to any one of formats 28A to 49A also includes a display configured to display one or more output images.

[0349] Format 51A: A device according to any one of formats 28A to 50A, including a camera configured to capture one or more images.

[0350] State 52A: A device according to any one of states 28A to 51A, wherein the device is a mobile device.

[0351] Format 53A: A method for processing video data, comprising one or more operations based on any one of formats 28A to 52A.

[0352] Format 54A: A non-transitory computer-readable medium having instructions stored thereon, which, when executed by one or more processors, cause one or more processors to perform one or more operations according to any one of formats 28A to 52A.

[0353] Format 55A: An apparatus for processing video data, comprising one or more components for performing operations according to any one of formats 28A to 52A.

[0354] 35: Segmentation Unit 41: Predictive Processing Unit 42: Motion Estimation Unit 44: Motion Compensation Unit 46: In-frame prediction processing unit 50: Adder 52: Transformation Processing Unit 54: Quantization unit 56: Entropy Coding Unit 57: Post-processing equipment 58: Inverse quantization unit 60: Inverse Transformation Processing Unit 62: Adder 63: Filter Unit 64: Image Memory 79: Network Entities 80: Entropy Decoding Unit 81: Predictive Processing Unit 82: Motion Compensation Unit 84: In-frame prediction processing unit 86: Inverse quantization unit 88: Inverse Transformation Processing Unit 90: Adder 91: Filter Unit 92: Image Memory 100: System 102: Video Source 104: Encoding devices 106: Encoder Engine 108: Memory 110: Output 112: Decoding equipment 114: Input 116: Decoder Engine 118: Memory 120: Communication Link 122: Video target device 200: Video Images 202: Cyclists 204: Image Gallery 206: Images 208: Slice 210: Plot 212: Sub-image 300: Decoded Encoded Video 302: Operation 304: Operation 306: Operation 308: Operation 400: Program 402: Operation 404: Operation 406: Operation

[0355] Domestic storage information (please note in order of storage institution, date, and number) none Overseas storage information (please note in the order of storage country, institution, date, and number) none< / if>

Claims

1. An apparatus for processing video data, comprising: At least one memory cell; And at least one processor, coupled to the at least one memory, the at least one processor being configured to: obtain a one-bit stream; Retrieve a subtlety type syntax element associated with the bitstream, the subtlety type syntax element specifying a subtlety type of one or more images to which a complexity metric (CM) associated with the bitstream applies, wherein a value of the subtlety type syntax element specifies that the CM applies to an image or a portion of the image in the bitstream, the portion of the image being smaller than the entire image; retrieve a time period type syntax element associated with the bitstream, the time period type syntax element indicating an upcoming time period or a set of images to which the CM applies; and decode a portion of the bitstream based on the subtlety type syntax element and the time period type syntax element.

2. The apparatus according to claim 1, wherein a value of the subtlety type syntax element specifies that the CM applies to at least one of all slices, a tile, a sub-picture, an extensible layer, or a decoder tree unit (CTU) row of the one or more pictures in the bitstream.

3. The apparatus according to claim 1, wherein the time period type syntax element indicates a specified time interval of the upcoming time period, a number of pictures of the upcoming time period, the upcoming time period including all pictures up to a picture including the next slice, or the upcoming time period including at least one of a single picture.

4. The apparatus according to claim 1, wherein the at least one processor is configured to retrieve a subtlety-level CM syntax structure associated with the bitstream, the subtlety-level CM syntax structure specifying a subtlety-level complexity measure of one or more subtle segments of the bitstream in the upcoming time period.

5. The apparatus according to claim 1, wherein the at least one processor is configured to: retrieve an additional time-type syntax element associated with the bitstream, the additional time-type syntax element being associated with the subtlety type syntax element, wherein the additional time-type syntax element is different from the time-type syntax element; and decode a portion of the bitstream based on the subtlety type syntax element and the additional time-type syntax element.

6. The apparatus according to claim 1, wherein the at least one processor is configured to retrieve at least one of the following: a sub-picture syntax element associated with the bitstream, the sub-picture syntax element indicating a sub-picture identifier (ID) when the CM is applied to multiple pictures; a decode tree block (CTB) number syntax element associated with the bitstream, the CTB number syntax element being able to indicate a total number of decode tree blocks in the upcoming period when the detail type is equal to all slices or a tile and the upcoming period spans multiple pictures; or an average CTB number syntax element associated with the bitstream, the average CTB number syntax element indicating an average number of CTBs per detail or 4×4 blocks per picture.

7. The apparatus according to claim 1, wherein when an available in-frame decoder block exists in at least a portion of the bit stream, a signal transmission associated with the at least portion of the bit stream notifies the in-frame decoder block statistics.

8. The apparatus according to claim 1, wherein when an available inter-frame decoding block exists in at least a portion of the bit stream, a signal transmission associated with the at least portion of the bit stream notifies the inter-frame decoding block statistics.

9. The apparatus according to claim 1, wherein the at least one processor is configured to determine an operating frequency of the apparatus based on the CM associated with the bit stream.

10. The apparatus according to claim 1 also includes a display configured to display at least a portion of the bitstream.

11. The apparatus according to claim 1, wherein the apparatus is one of a mobile device, a wearable device, an augmented reality device, a camera, a human computer, a vehicle, a robotic device, a television set, or a computing device.

12. A method for processing video data, comprising the steps of: obtaining a bitstream; retrieving a subtlety type syntax element associated with the bitstream, the subtlety type syntax element specifying a subtlety type of one or more pictures to which a complexity metric (CM) associated with the bitstream applies, wherein a value of the subtlety type syntax element specifies that the CM applies to a picture or a portion of the picture in the bitstream, the portion of the picture being smaller than the whole picture; retrieving a time period type syntax element associated with the bitstream, the time period type syntax element indicating an upcoming time period or a set of pictures to which the CM applies; and decoding a portion of the bitstream based on the subtlety type syntax element and the time period type syntax element.

13. According to the method of request item 12, wherein a value of the subtlety type syntax element specifies that the CM applies to at least one of all slices, a tile, a sub-picture, an extensible layer, or a decoder tree unit (CTU) line of the one or more pictures in the bitstream.

14. According to the method of request item 12, wherein the time period type syntax element indicates a specified time interval of the upcoming time period, a number of pictures of the upcoming time period, the upcoming time period includes all pictures up to a picture including the next slice, or the upcoming time period includes at least one of a single picture.

15. The method according to request 12 also includes retrieving a subtlety-level CM syntax structure associated with the bitstream, the subtlety-level CM syntax structure specifying a subtlety-level complexity measure of one or more subtle segments of the bitstream in the upcoming time period.

16. The method according to claim 12 also includes the steps of: retrieving an additional time-type syntax element associated with the bitstream, the additional time-type syntax element being associated with the subtlety type syntax element, wherein the additional time-type syntax element is different from the time-type syntax element; and decoding a portion of the bitstream based on the subtlety type syntax element and the additional time-type syntax element.

17. The method according to claim 12 also includes retrieving at least one of the following: a sub-picture syntax element associated with the bitstream, the sub-picture syntax element indicating a sub-picture identifier (ID) when the CM is applied to multiple pictures; a decode tree block (CTB) number syntax element associated with the bitstream, which, when the detail type is equal to all slices or a block and the upcoming period spans multiple pictures, can be used in the CM to indicate the total number of decode tree luminance blocks in the upcoming period; or an average CTB number syntax element associated with the bitstream, the average CTB number syntax element indicating an average of the CTB per detail or 4×4 block per picture.

18. The method of claim 12, wherein when an available in-frame decoder block exists in at least a portion of the bitstream, a signal transmission associated with the at least portion of the bitstream notifies the in-frame decoder block statistics.

19. The method of claim 12, wherein when an available inter-frame decoding block exists in at least a portion of the bitstream, a signal transmission associated with the at least portion of the bitstream notifies the inter-frame decoding block statistics.

20. The method according to claim 12 also includes displaying at least a portion of the bitstream on a display.

21. The method according to claim 13 also includes determining an operating frequency of a device based on the CM associated with the bitstream.

22. An apparatus for processing video data, comprising: At least one memory cell; And at least one processor, coupled to the at least one memory, the at least one processor being configured to: acquire video data; Generate a subtlety type syntax element for a bitstream, the subtlety type syntax element specifying a subtlety type of one or more images to which a complexity metric (CM) associated with the bitstream applies; generate a time period type syntax element associated with the bitstream, the time period type syntax element indicating an upcoming time period or a set of images to which the CM applies, wherein a value of the subtlety type syntax element specifies that the CM applies to an image or a portion of the image to the bitstream, the portion of the image being smaller than the entire image; generate the bitstream associated with the video data, the bitstream including the subtlety type syntax element and the time period type syntax element; and output the resulting bitstream.

23. The apparatus according to claim 22, wherein a value of the subtlety type syntax element specifies that the CM applies to at least one of all slices, a tile, a sub-picture, an extensible layer, or a decoder tree unit (CTU) row of the one or more pictures in the bitstream.

24. The apparatus according to claim 22, wherein the time period type syntax element indicates a specified time interval of the upcoming time period, a number of pictures of the upcoming time period, the upcoming time period including all pictures up to a picture including the next slice, or the upcoming time period including at least one of a single picture.

25. The apparatus according to claim 22, wherein the at least one processor is configured to generate a subtlety-level CM syntax structure for the bitstream, the subtlety-level CM syntax structure specifying a subtlety-level complexity metric for one or more entities in the upcoming time period.

26. The apparatus according to claim 22, wherein the time period type syntax element indicates a specified time interval of the upcoming time period, a number of pictures of the upcoming time period, the upcoming time period including all pictures up to a picture including the next slice, or the upcoming time period including at least one of a single picture.

27. The apparatus according to claim 22, wherein the at least one processor is configured to generate an associated additional time-type syntax element for the bitstream, the additional time-type syntax element being associated with the subtlety type syntax element, wherein the additional time-type syntax element is different from the time-type syntax element, and wherein the additional time-type syntax element is used to decode a portion of the bitstream having the subtlety type syntax element.

28. The apparatus according to claim 22, wherein the at least one processor is configured to generate for the bitstream at least one of the following: a subpicture syntax element associated with the bitstream, the subpicture syntax element indicating a subpicture identifier (ID) when the CM is applied to multiple pictures; a decode tree block (CTB) number syntax element associated with the bitstream, the CTB number syntax element indicating a total number of decode tree blocks in the upcoming period when the detail type is equal to all slices or a tile and the upcoming period spans multiple pictures; or an average CTB number syntax element associated with the bitstream, the average CTB number syntax element indicating an average number of CTBs per detail or 4×4 blocks per picture.

29. The apparatus according to claim 22, wherein when an available in-frame decoder block exists in at least a portion of the bit stream, a signal transmission associated with the at least portion of the bit stream notifies the in-frame decoder block statistics.

30. The apparatus according to claim 22, wherein when an available inter-frame decoding block exists in at least a portion of the bit stream, a signal transmission associated with the at least portion of the bit stream notifies the inter-frame decoding block statistics.

31. The apparatus according to claim 22 also includes a camera configured to capture the video data.

32. The apparatus according to claim 22, wherein the apparatus is one of a mobile device, a wearable device, an augmented reality device, a camera, a human computer, a vehicle, a robotic device, a television set, or a computing device.

Images