Storage and delivery of video data for video coding

Abstract

An apparatus for processing video data may be configured to receive a configuration record for decoding a bitstream of video data, where the configuration record for the bitstream includes a tool set indication syntax element, the tool set indication syntax element including information, the information identifies a tool required to decode a bitstream associated with the configuration record from a set of video decoding tools; determining whether to retrieve a bitstream associated with the configuration record based on the tool set indicating a syntax element; and based on the determination that a bit stream associated with the configuration record is to be retrieved, retrieving the bit stream and outputting the bit stream to a video decoder for decoding.

Description

[0001] This application is a divisional application of the Chinese invention patent application with application date January 8, 2021 and application number 202180008003.8.

[0002] This application claims priority to U.S. application No. 17 / 143,611, filed on January 7, 2021, which claims the benefit of U.S. Provisional Patent Application No. 62 / 958,561, filed on January 8, 2020, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to the storage and delivery of video data. Background Art

[0004] Digital video capabilities can be integrated into a variety of devices, including digital televisions, digital direct broadcast systems, wireless broadcast systems, personal digital assistants (PDAs), laptop or desktop computers, tablet computers, e-book readers, digital cameras, digital recording devices, digital media players, video game devices, video game consoles, cellular or satellite radio telephones, so-called "smart phones", video teleconferencing devices, video streaming devices, etc. Digital video devices implement video coding techniques, such as those described in the standards defined by MPEG-2, MPEG-4, ITU-T H.263, ITU-T H.264 / MPEG-4, Part 10, Advanced Video Coding (AVC), ITU-T H.265 / High Efficiency Video Coding (HEVC), and extensions of such standards. By implementing such video coding techniques, video devices can more efficiently transmit, receive, encode, decode, and / or store digital video information.

[0005] Video coding techniques include spatial (intra-picture) prediction and / or temporal (inter-picture) prediction to reduce or eliminate redundancy inherent in video sequences. For block-based video coding, a video slice (e.g., a video picture or a portion of a video picture) can be partitioned into video blocks, which may also be referred to as coding tree units (CTUs), coding units (CUs), and / or coding nodes. Video blocks in an intra-coded (I) slice of a picture are encoded using spatial prediction for reference samples in neighboring blocks in the same picture. Video blocks in an inter-coded (P or B) slice of a picture may use spatial prediction for reference samples in neighboring blocks in the same picture, or temporal prediction for reference samples in other reference pictures. Pictures may be referred to as frames, and reference pictures may be referred to as reference frames. Summary of the Invention

[0006] In general, the present disclosure describes techniques for the storage and delivery of bitstreams that conform to video coding standards based on file formats. The present disclosure also describes examples of media format profiles for video coding standards, and delivery using streaming techniques. As an example, the video coding standard is the Essential Video Coding (EVC) standard, and the file format is the International Organization for Standardization's base media file format. The International Organization for Standardization's base media file format is known as ISOBMFF. An example of a media format profile is the Common Media Application Format (CMAF) profile, and an example of a streaming technique is Dynamic Adaptive Streaming over HTTP (DASH). The present disclosure may also describe encryption requirements and codec parameters for use with Multipurpose Internet Mail Extensions (MIME) types. Although described with respect to EVC, ISOBMFF, CMAF, and DASH, the example techniques should not be considered limiting.

[0007] According to an example of the present disclosure, a method for processing a video stream or video file includes receiving a configuration record for decoding a bitstream of video data, wherein the configuration record for the bitstream includes a toolset indication syntax element, the toolset indication syntax element including information that identifies a tool required to decode the bitstream associated with the configuration record from a set of video decoding tools; based on the toolset indication syntax element, determining whether to retrieve the bitstream associated with the configuration record; and based on the determination to retrieve the bitstream associated with the configuration record, retrieving the bitstream and outputting the bitstream to a video decoder for decoding.

[0008] According to another example of the present disclosure, a device for processing video data includes a memory configured to store a video stream, and one or more processors implemented in a circuit, coupled to the memory, and can be configured to receive a configuration record for decoding a bitstream of video data, wherein the configuration record for the bitstream includes a toolset indication syntax element, the toolset indication syntax element including information that identifies a tool required to decode the bitstream associated with the configuration record from a set of video decoding tools; based on the toolset indication syntax element, determine whether to retrieve the bitstream associated with the configuration record; and based on the determination to retrieve the bitstream associated with the configuration record, retrieve the bitstream and output the bitstream to a video decoder for decoding.

[0009] According to another example of the present disclosure, a computer-readable storage medium storing instructions, which, when executed by one or more processors, causes the one or more processors to receive a configuration record for decoding a bitstream of video data, wherein the configuration record for the bitstream includes a toolset indication syntax element, the toolset indication syntax element including information that identifies a tool required to decode the bitstream associated with the configuration record from a set of video decoding tools; based on the toolset indication syntax element, determining whether to retrieve a bitstream associated with the configuration record; and based on the determination to retrieve the bitstream associated with the configuration record, retrieving the bitstream and outputting the bitstream to a video decoder for decoding.

[0010] According to another example of the present disclosure, an apparatus for processing a video stream or video file includes: a component for receiving a configuration record for decoding a bitstream of video data, wherein the configuration record for the bitstream includes a toolset indication syntax element, the toolset indication syntax element including information that identifies a tool required to decode the bitstream associated with the configuration record from a set of video decoding tools; a component for determining whether to retrieve the bitstream associated with the configuration record based on the toolset indication syntax element; a component for retrieving the bitstream associated with the configuration record based on the determination to retrieve the bitstream; and a component for outputting the bitstream to a video decoder for decoding.

[0011] The details of one or more examples are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a block diagram illustrating an example video encoding and decoding system that may perform the techniques of this disclosure.

[0013] Figure 2A and 2B is a conceptual diagram illustrating an example quadtree plus binary tree (QTBT) structure and a corresponding coding tree unit (CTU).

[0014] Figure 3 is a block diagram illustrating an example video encoder that may perform the techniques of this disclosure.

[0015] Figure 4 is a block diagram illustrating an example video decoder that may perform the techniques of this disclosure.

[0016] Figure 5 is a conceptual diagram illustrating an example structure of a file in accordance with one or more techniques of this disclosure.

[0017] Figure 6is a conceptual diagram illustrating elements of example multimedia content.

[0018] Figure 7 is a block diagram illustrating elements of an example video file.

[0019] Figure 8 is a flowchart illustrating an example method for processing video data according to the techniques of this disclosure. DETAILED DESCRIPTION

[0020] The Essential Video Coding (EVC) standard, developed by ISO / IEC JTC 1 / SC 29 / WG 11 (MPEG), provides an efficient and low-complexity video coding solution. EVC elementary streams are structured as Network Abstraction Layer (NAL) units. The storage of NAL units in the ISO Base Media File Format (ISOBMFF) follows similar principles to other NAL-structured video formats, such as the High Efficiency Video Coding (HEVC) and / or Versatile Video Coding (VVC) standards.

[0021] The storage of EVC elementary streams can be divided into two parts: static information used globally in the elementary stream and dynamic information that may vary for each sample. The sequence parameter set (SPS) and picture parameter set (PPS) may be part of the information that rarely changes and can be considered static. A set of flags can be used to indicate whether the parameter set is expected to change in the stream. In this case, sample grouping will be defined to indicate the samples for which the parameter set does change.

[0022] The Adaptive Parameter Set (APS) can be dynamic information that may change on a per-sample basis. The APS is used to carry Adaptive Loop Filter (ALF) information. The presence of the ALF is signaled via a flag, and samples carrying the APS information may belong to the same sample group.

[0023] A box may refer to a basic syntactic structure in ISOBMFF, including a four-character encoded box type, a box byte count, and a payload. An ISOBMFF file consists of a series of boxes, and boxes may contain other boxes. A movie box ("moov") contains metadata for the continuous media streams present in the file, each represented as a track in the file.

[0024] The metadata for a track can be enclosed in a track box ("trak"), while the media content of the track can be enclosed either in a media data box ("mdat") or directly in a separate file. The media content of a track comprises a series of samples, such as audio or video access units. An access unit is typically a data unit that includes coded media (e.g., picture) data for a common time instance. A sample is an access unit defined by a specific specification, such as the Video Coding Specification described herein. A sample entry can provide a description of the corresponding sample.

[0025] ISOBMFF specifies the following types of tracks: a media track, which contains elementary media streams; a hint track, which either includes media transmission instructions or represents a received packet stream; and a timed metadata track, which includes metadata for time synchronization.

[0026] The present disclosure describes a configuration record for a bitstream of video data, the configuration record including a toolset indication syntax element, the toolset indication syntax element including information that identifies tools needed to decode the bitstream associated with the configuration record. The present disclosure also describes techniques for including a Multipurpose Internet Mail Extensions (MIME) type parameter in, for example, an HTTP transmission, the MIME type parameter including a key-value pair. The key may indicate a MIME type identifying a video decoding tool, and the value may identify the tool needed to decode the bitstream. Typically, a tool may be considered needed if the tool is enabled in at least one parameter set for the media stream. By using such a toolset indication syntax element and a MIME type, the techniques of the present disclosure may advantageously enable a video processing device to determine whether a video decoder of the video processing device is capable of decoding a file without having to parse a large portion of the file.

[0027] Figure 1 1 is a block diagram illustrating an example video encoding and decoding system 100 that can perform the techniques of the present disclosure. The techniques of the present disclosure are generally directed to encoding and decoding (encoding and / or decoding) video data. Generally, video data includes any data used to process video. Thus, video data can include original, unencoded video, encoded video, decoded (e.g., reconstructed) video, and video metadata (such as signaling data).

[0028] like Figure 1As shown, system 100 includes a source device 102, which, in this example, provides encoded video data to be decoded and displayed by a destination device 116. Specifically, source device 102 provides the video data to destination device 116 via a computer-readable medium 110. Source device 102 and destination device 116 can include any of a wide range of devices, including desktop computers, notebook (i.e., laptop) computers, tablet computers, set-top boxes, mobile phones (such as smartphones), televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, etc. In some cases, source device 102 and destination device 116 can be equipped for wireless communication and, therefore, can be referred to as wireless communication devices.

[0029] exist Figure 1 In an example of , source device 102 includes a video source 104, a memory 106, a video encoder 200, and an output interface 108. Destination device 116 includes an input interface 122, a video decoder 300, the memory 120, and a display device 118. According to the present disclosure, source device 102 and destination device 116 can be configured to apply techniques for storage and delivery of bitstreams that comply with a video coding standard based on a file format (e.g., an Essential Video Coding (EVC) elementary bitstream based on the ISO Base Media File Format (ISOBMFF)). The present disclosure also describes media profiles (e.g., a Common Media Application Format (CMAF media profile for EVC)), describes delivery using Dynamic Adaptive Streaming over HTTP (DASH), describes encryption requirements, and describes codec parameters to be used with Multipurpose Internet Mail Extensions (MIME) types that can be utilized by the video encoder 200 and the video decoder 300. Thus, source device 102 represents an example of a client device configured to perform video encoding and transmission, while destination device 116 represents an example of a destination device configured to receive and decode encoded video. In other examples, the source and destination devices may include other components or arrangements. For example, source device 102 may receive video data from an external video source (such as an external camera). Similarly, destination device 116 may interface with an external display device rather than including an integrated display device.

[0030] like Figure 1The system 100 shown in is merely an example. In general, any digital video encoding and / or decoding device can perform the techniques for storage and delivery of bitstreams compliant with a video coding standard based on a file format (e.g., an EVC elementary bitstream based on ISOBMFF). The present disclosure also describes media profiles (e.g., a CMAF media profile for EVC), describes delivery using DASH, describes encryption requirements, and describes codec parameters to be used with MIME types that can be utilized by the video encoder 200 and the video decoder 300. The source device 102 and the destination device 116 are merely examples of such coding devices, where the source device 102 generates encoded video data for transmission to the destination device 116. The present disclosure refers to a "coding" device as a device that performs coding (encoding and / or decoding) of data. Therefore, the video encoder 200 and the video decoder 300 represent examples of coding devices (particularly, a video encoder and a video decoder, respectively). In some examples, source device 102 and destination device 116 can operate in a substantially symmetrical manner, such that each of source device 102 and destination device 116 includes video encoding and decoding components. Thus, system 100 can support one-way or two-way video transmission between source device 102 and destination device 116, e.g., for video streaming, video playback, video broadcasting, or video telephony.

[0031] Generally, video source 104 represents a source of video data (i.e., raw, unencoded video data) and provides a series of consecutive pictures (also referred to as "frames") of video data to video encoder 200, which encodes the picture data. Source device 102 of video source 104 may include a video capture device (such as a video camera), a video archive containing previously captured raw video, and / or a video feed interface for receiving video from a video content provider. Alternatively, video source 104 may generate computer graphics-based data as the source video, or a combination of live video, archived video, and computer-generated video. In each case, video encoder 200 encodes captured, pre-captured, or computer-generated video data. Video encoder 200 may rearrange the pictures from the order in which they were received (sometimes referred to as "display order") into a coding order for encoding. Video encoder 200 may generate a bitstream comprising the encoded video data. Source device 102 may then output the encoded video data onto computer-readable medium 110 via output interface 108 for receipt and / or retrieval by, for example, input interface 122 of destination device 116 .

[0032] Memory 106 of source device 102 and memory 120 of destination device 116 represent general purpose memory. In some examples, memories 106 and 120 can store raw video data, e.g., raw video from video source 104 and raw decoded video data from video decoder 300. Additionally or alternatively, memories 106 and 120 can store, for example, software instructions executable by video encoder 200 and video decoder 300, respectively. Although memory 106 and memory 120 are shown separately from video encoder 200 and video decoder 300 in this example, it should be understood that video encoder 200 and video decoder 300 can also include internal memory to achieve functionally similar or equivalent purposes. Furthermore, memories 106 and 120 can store, for example, encoded video data output from video encoder 200 and input to video decoder 300. In some examples, portions of memories 106 and 120 can be allocated as one or more video buffers, e.g., to store raw, decoded, and / or encoded video data.

[0033] The computer-readable medium 110 can represent any type of medium or device capable of transmitting encoded video data from the source device 102 to the destination device 116. In one example, the computer-readable medium 110 represents a communication medium for enabling the source device 102 to send the encoded video data directly to the destination device 116 in real time, for example, via a radio frequency network or a computer-based network. According to a communication standard such as a wireless communication protocol, the output interface 108 can modulate a transmission signal including the encoded video data, and the input interface 122 can demodulate the received transmission signal. The communication medium can include any wireless or wired communication medium, such as a radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium can form part of a packet-based network such as a local area network, a wide area network, or a global network such as the Internet. The communication medium can include routers, switches, base stations, or any other equipment that facilitates communication from the source device 102 to the destination device 116.

[0034] In some examples, source device 102 may output the encoded data from output interface 108 to storage device 112. Similarly, destination device 116 may access the encoded data from storage device 112 via input interface 122. Storage device 112 may include any of a variety of distributed or locally accessed data storage media, such as a hard drive, Blu-ray disc, DVD, CD-ROM, flash memory, volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded video data.

[0035] In some examples, source device 102 may output the encoded video data to a file server 114 or another intermediate storage device that may store the encoded video generated by source device 102. Destination device 116 may access the stored video data from file server 114 via streaming or downloading. File server 114 may be any type of server device capable of storing encoded video data and transmitting the encoded video data to destination device 116. File server 114 may represent a network server (e.g., for a website), a file transfer protocol (FTP) server, a content delivery network device, or a network attached storage (NAS) device. Destination device 116 may access the encoded video data from file server 114 via any standard data connection, including an internet connection. This may include a wireless channel (e.g., a Wi-Fi connection), a wired connection (e.g., a digital subscriber line (DSL), a cable modem, etc.), or a combination of the two suitable for accessing the encoded video data stored on file server 114. File server 114 and input interface 122 may be configured to operate according to a streaming protocol, a download transfer protocol, or a combination thereof.

[0036] The output interface 108 and the input interface 122 may represent wireless transmitters / receivers, modems, wired network components (e.g., Ethernet cards), wireless communication components operating according to any of the various IEEE 802.11 standards, or other physical components. In examples where the output interface 108 and the input interface 122 include wireless components, the output interface 108 and the input interface 122 may be configured to communicate data, such as encoded video data, according to a cellular communication standard such as 4G, 4G-LTE (Long Term Evolution), LTA-Advanced, 5G, etc. In some examples where the output interface 108 includes a wireless transmitter, the output interface 108 and the input interface 122 may be configured to communicate data, such as encoded video data, according to other wireless standards such as the IEEE 802.11 specification, the IEEE 802.15 specification (e.g., ZigBee TM ), Bluetooth TM Standards, etc.) for transferring data such as encoded video data. In some examples, source device 102 and / or destination device 116 may include respective system-on-chip (SoC) devices. For example, source device 102 may include an SoC device to perform the functions attributed to video encoder 200 and / or output interface 108, and destination device 116 may include an SoC device to perform the functions attributed to video decoder 300 and / or input interface 122.

[0037] The techniques of the present disclosure may be applied to video encoding to support any of a variety of multimedia applications, such as over-the-air television broadcasting, cable television transmission, satellite television transmission, Internet streaming video transmission (such as Dynamic Adaptive Streaming over HTTP (DASH)), digital video encoded onto a data storage medium, decoding of digital video stored on a data storage medium, or other applications.

[0038] The input interface 122 of the destination device 116 receives an encoded video bitstream from the computer-readable medium 110 (e.g., a communication medium, a storage device 112, a file server 114, etc.). The encoded video bitstream may include signaling information defined by the video encoder 200 and used by the video decoder 300, such as syntax elements that describe the characteristics and / or processing of video blocks or other coded units (e.g., slices, pictures, picture groups, sequences, etc.). The display device 118 displays decoded pictures of the decoded video data to a user. The display device 118 may represent any of a variety of display devices, such as a cathode ray tube (CRT), a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, or another type of display device.

[0039] Although not in Figure 1 , but in some examples, the video encoder 200 and the video decoder 300 may be integrated with an audio encoder and / or an audio decoder, respectively, and may include appropriate MUX-DEMUX units or other hardware and / or software to handle a multiplexed stream including both audio and video in a common data stream. If applicable, the MUX-DEMUX units may conform to the ITU H.223 multiplexer protocol or other protocols such as the User Datagram Protocol (UDP).

[0040] Each of the video encoder 200 and the video decoder 300 can be implemented as any of a variety of applicable encoder and / or decoder circuits, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combination thereof. When the technology is implemented partially in software, the device can store the instructions of the software in an applicable non-transitory computer-readable medium and use one or more processors to execute the instructions in hardware to perform the technology of the present disclosure. Each of the video encoder 200 and the video decoder 300 can be included in one or more encoders or decoders, either of which can be integrated as part of a combined encoder / decoder (CODEC) in the corresponding device. A device including the video encoder 200 and / or the video decoder 300 can include an integrated circuit, a microprocessor, and / or a wireless communication device, such as a cellular phone.

[0041] The video encoder 200 and the video decoder 300 may operate according to other proprietary or industry standards, such as the Joint Exploration Test Model (JEM) or ITU-T H.266, also known as Versatile Video Coding (VVC). A recent draft of the VVC standard is described in "Versatile Video Coding (Draft 7)" by Bross et al., ITU-T SG 16 WP3 Joint Video Experts Group (JVET) and ISO / IEC JTC 1 / SC 29 / WG 11, 16th Meeting: Geneva, CH, October 1-11, 2019, JVET-P2001-v14, Bross et al. (hereinafter referred to as "VVC Draft 7"). The video encoder 200 and the video decoder 300 may operate according to the EVC standard developed by ISO / IEC JTC 1 / SC 29 / WG 11 (MPEG). However, the technology of the present disclosure is not limited to any particular coding standard.

[0042] Typically, the video encoder 200 and the video decoder 300 can perform block-based encoding of a picture. The term "block" generally refers to a structure that includes data to be processed (e.g., encoded, decoded, or otherwise used in an encoding and / or decoding process). For example, a block can include a two-dimensional matrix of samples of luminance and / or chrominance data. Typically, the video encoder 200 and the video decoder 300 can encode and decode video data represented in a YUV (e.g., Y, Cb, Cr) format. That is, rather than encoding and decoding the red, green, and blue (RGB) data of the samples of a picture, the video encoder 200 and the video decoder 300 can encode and decode the luminance and chrominance components, where the chrominance components can include both red and blue chrominance components. In some examples, the video encoder 200 converts the received RGB formatted data into a YUV representation before encoding, and the video decoder 300 converts the YUV representation into an RGB format. Alternatively, pre-processing and post-processing units (not shown) can perform these conversions.

[0043] This disclosure may generally refer to the coding (e.g., encoding and decoding) of a picture as a process that includes encoding or decoding the data of the picture. Similarly, this disclosure may refer to the coding of a block of a picture as a process that includes encoding or decoding the data of the block, such as prediction and / or residual coding. The encoded video bitstream generally includes a series of values ​​of syntax elements that represent coding decisions (e.g., coding mode) and the partitioning of the picture into blocks. Therefore, references to coding a picture or block should generally be understood to refer to coding the values ​​of the syntax elements that form the picture or block.

[0044] Various video coding standards define various blocks, sometimes referred to as coding units (CUs), prediction units (PUs), and transform units (TUs). A video encoder (e.g., video encoder 200) partitions a coding tree unit (CTU) into CUs according to a quadtree structure. That is, the video encoder partitions the CTU and CU into four equal non-overlapping squares, and each node of the quadtree has zero or four child nodes. A node without child nodes may be referred to as a "leaf node," and the CU of such a leaf node may include one or more PUs and / or one or more TUs. The video encoder may further partition the PUs and TUs. For example, a residual quadtree (RQT) represents the partitioning of a TU, and a PU represents inter-frame prediction data, while a TU represents residual data. An intra-predicted CU includes intra-frame prediction information, such as an intra-frame mode indication.

[0045] As another example, the video encoder 200 and the video decoder 300 can be configured to operate according to JEM, VVC, EVC, or any other such standard. A video encoder (such as the video encoder 200) partitions a picture into multiple coding tree units (CTUs). The video encoder 200 can partition the CTUs according to a tree structure (such as a quadtree plus binary tree (QTBT) structure or a multi-type tree (MTT) structure). The QTBT structure eliminates the concept of multiple partition types, such as the separation between CU, PU, ​​and TU. The QTBT structure includes two levels: a first level partitioned according to quadtree partitioning, and a second level partitioned according to binary tree partitioning. The root node of the QTBT structure corresponds to the CTU. The leaf nodes of the binary tree correspond to the coding units (CUs).

[0046] In the MTT partitioning structure, blocks can be partitioned using quadtree (QT) partitioning, binary tree (BT) partitioning, and one or more types of ternary tree (TT) (also known as ternary tree (TT)) partitioning. Triple tree or ternary tree partitioning is a partitioning method that divides a block into three sub-blocks. In some examples, ternary tree or ternary tree partitioning divides a block into three sub-blocks without dividing the original block through the center. The partitioning types in MTT (e.g., QT, BT, and TT) can be symmetric or asymmetric.

[0047] In some examples, the video encoder 200 and the video decoder 300 may use a single QTBT or MTT structure to represent each of the luma and chroma components, while in other examples, the video encoder 200 and the video decoder 300 may use two or more QTBT or MTT structures, such as one QTBT / MTT structure for the luma component and another QTBT / MTT structure for the two chroma components (or two QTBT / MTT structures for the respective chroma components).

[0048] The video encoder 200 and the video decoder 300 can be configured to use quadtree segmentation, QTBT segmentation, MTT segmentation, or other segmentation structures. For illustrative purposes, the description of the technology of the present disclosure is presented with respect to QTBT segmentation. However, it should be understood that the technology of the present disclosure is also applicable to video encoders configured to use quadtree segmentation or other types of segmentation.

[0049] Blocks (e.g., CTUs or CUs) can be grouped in a picture in various ways. As an example, a brick can refer to a rectangular area of ​​a CTU row in a particular tile in a picture. A tile can be a rectangular area of ​​a CTU in a particular tile column and a particular tile row in a picture. A tile column refers to a rectangular area of ​​a CTU with a height equal to the picture height and a width specified by a syntax element (e.g., such as in a picture parameter set). A tile row refers to a rectangular area of ​​a CTU with a height specified by a syntax element (e.g., such as in a picture parameter set) and a width equal to the picture width.

[0050] In some examples, a tile can be partitioned into multiple bricks, each of which can include one or more CTU rows in the tile. A tile that is not partitioned into multiple bricks can also be referred to as a brick. However, a brick that is a proper subset of a tile cannot be referred to as a tile.

[0051] Tiles in a picture can also be arranged in slices. A slice can be an integer number of tiles of a picture that can be uniquely contained in a single Network Abstraction Layer (NAL) unit. In some examples, a slice includes several complete tiles or only a contiguous sequence of complete tiles of a tile.

[0052] This disclosure may use "N x N" and "N by N" interchangeably to refer to the sample dimensions of a block (such as a CU or other video block) in terms of vertical and horizontal dimensions, for example, 16x16 samples or 16 by 16 samples. Typically, a 16x16 CU will have 16 samples in the vertical direction (y=16) and 16 samples in the horizontal direction (x=16). Likewise, an N x N CU typically has N samples in the vertical direction and N samples in the horizontal direction, where N represents a non-negative integer value. The samples in a CU can be arranged in rows and columns. Furthermore, a CU does not necessarily have the same number of samples in the horizontal direction as in the vertical direction. For example, a CU may contain N x M samples, where M is not necessarily equal to N.

[0053] The video encoder 200 encodes the video data of the CU representing prediction and / or residual information and other information. The prediction information indicates how the CU will be predicted to form a prediction block for the CU. The residual information typically represents the sample-by-sample difference between the samples of the CU and the prediction block before encoding.

[0054] To predict a CU, the video encoder 200 may typically form a prediction block for the CU through inter-frame prediction or intra-frame prediction. Inter-frame prediction typically refers to predicting a CU from data of a previously encoded picture, while intra-frame prediction typically refers to predicting a CU from previously encoded data of the same picture. To perform inter-frame prediction, the video encoder 200 may use one or more motion vectors to generate a prediction block. The video encoder 200 may typically perform a motion search to identify a reference block that closely matches the CU, for example, in terms of the difference between the CU and a reference block. The video encoder 200 may calculate a difference metric using the sum of absolute differences (SAD), the sum of squared differences (SSD), the mean absolute difference (MAD), the mean squared difference (MSD), or other such difference calculations to determine whether the reference block closely matches the current CU. In some examples, the video encoder 200 may use unidirectional prediction or bidirectional prediction to predict the current CU.

[0055] Some video coding standards also provide an affine motion compensation mode, which can be considered an inter-frame prediction mode. In the affine motion compensation mode, the video encoder 200 can determine two or more motion vectors representing non-translational motion (such as zooming in or out, rotation, perspective motion, or other irregular motion types).

[0056] To perform intra prediction, the video encoder 200 can select an intra prediction mode to generate a prediction block. For example, the video encoder 200 can utilize 67 or some other number of intra prediction modes, including various directional modes as well as planar and DC modes. Typically, the video encoder 200 selects an intra prediction mode that describes neighboring samples of a current block (e.g., a block of a CU) from which samples of the current block are predicted. Assuming that the video encoder 200 encodes CTUs and CUs in raster scan order (left to right, top to bottom), such samples can typically be located above, to the upper left, or to the left of the current block in the same picture as the current block.

[0057] The video encoder 200 encodes data indicating a prediction mode for the current block. For example, for inter-frame prediction mode, the video encoder 200 may encode data indicating which of various available inter-frame prediction modes to use, as well as motion information for the corresponding mode. For unidirectional or bidirectional inter-frame prediction, for example, the video encoder 200 may encode motion vectors using Advanced Motion Vector Prediction (AMVP) or Merge Mode. The video encoder 200 may use a similar mode to encode motion vectors for affine motion compensation mode.

[0058] After a prediction, such as intra-frame prediction or inter-frame prediction, of a block, the video encoder 200 may calculate residual data for the block. The residual data, such as a residual block, represents the sample-by-sample difference between the block and a prediction block for the block formed using the corresponding prediction mode. The video encoder 200 may apply one or more transforms to the residual block to produce transformed data in a transform domain rather than a sample domain. For example, the video encoder 200 may apply a discrete cosine transform (DCT), an integer transform, a wavelet transform, or a conceptually similar transform to the residual video data. In addition, the video encoder 200 may apply a secondary transform, such as a mode-dependent non-separable secondary transform (MDNSST), a signal-dependent transform, a Karhunen-Loeve transform (KLT), etc., after the first transform. The video encoder 200 generates transform coefficients after applying one or more transforms.

[0059] As described above, after performing any transforms to produce transform coefficients, the video encoder 200 can perform quantization of the transform coefficients. Quantization generally refers to the process of quantizing transform coefficients to potentially reduce the amount of data used to represent the transform coefficients, which provides further compression. By performing the quantization process, the video encoder 200 can reduce the bit depth associated with some or all transform coefficients. For example, the video encoder 200 can round down an n-bit value to an m-bit value during quantization, where n is greater than m. In some examples, to perform quantization, the video encoder 200 can perform a bitwise right shift of the value to be quantized.

[0060] After quantization, the video encoder 200 may scan the transform coefficients to produce a one-dimensional vector from the two-dimensional matrix including the quantized transform coefficients. The scan may be designed to place transform coefficients with higher energy (and therefore lower frequency) at the front of the vector and transform coefficients with lower energy (and therefore higher frequency) at the back of the vector. In some examples, the video encoder 200 may scan the quantized transform coefficients using a predefined scan order to produce a serialized vector and then entropy encode the quantized transform coefficients of the vector. In other examples, the video encoder 200 may perform adaptive scanning. After scanning the quantized transform coefficients to form the one-dimensional vector, the video encoder 200 may entropy encode the one-dimensional vector, for example, according to context-adaptive binary arithmetic coding (CABAC). The video encoder 200 may also entropy encode the values ​​of syntax elements that describe metadata associated with the encoded video data for use by the video decoder 300 when decoding the video data.

[0061] To perform CABAC, the video encoder 200 may assign context to symbols to be transmitted in a context model. The context may relate to, for example, whether the adjacent values ​​of the symbol are zero. The probability determination may be based on the context assigned to the symbol.

[0062] The video encoder 200 may further generate syntax data such as block-based syntax data, picture-based syntax data, and sequence-based syntax data, or other syntax data such as a sequence parameter set (SPS), a picture parameter set (PPS), or a video parameter set (VPS), for example, in a picture header, a block header, or a slice header, to the video decoder 300. The video decoder 300 may similarly decode such syntax data to determine how to decode the corresponding video data.

[0063] In this way, the video encoder 200 can generate a bitstream including the encoded video data, for example, describing the partitioning of the picture into blocks (e.g., CUs) and syntax elements for prediction and / or residual information for the blocks. Ultimately, the video decoder 300 can receive the bitstream and decode the encoded video data.

[0064] In general, the video decoder 300 performs processing that is inverse to the processing performed by the video encoder 200 to decode the encoded video data of the bitstream. For example, the video decoder 300 may decode the values ​​of the syntax elements of the bitstream using CABAC in a manner substantially similar to, but inverse to, the CABAC encoding process of the video encoder 200. The syntax elements may define information about the partitioning of a picture into CTUs, and the partitioning of each CTU according to a corresponding partitioning structure (such as a QTBT structure) to define the CUs of the CTU. The syntax elements may further define prediction and residual information for a block (e.g., a CU) of video data.

[0065] The residual information may be represented by, for example, quantized transform coefficients. The video decoder 300 may inverse quantize and inverse transform the quantized transform coefficients of the block to reproduce a residual block for the block. The video decoder 300 uses the signaled prediction mode (intra-frame or inter-frame prediction) and associated prediction information (e.g., motion information for inter-frame prediction) to form a prediction block for the block. The video decoder 300 may then combine the prediction block and the residual block (on a sample-by-sample basis) to reproduce the original block. The video decoder 300 may perform additional processing, such as deblocking to reduce visual artifacts along block boundaries.

[0066] The present disclosure may generally refer to "signaling" certain information, such as syntax elements. The term "signaling" generally refers to the communication of syntax element values ​​and / or other data used to decode encoded video data. That is, video encoder 200 may signal the values ​​of syntax elements in a bitstream. Generally, signaling refers to generating values ​​in the bitstream. As described above, source device 102 may transmit the bitstream to destination device 116 in substantially real time or in non-real time, such as may occur when storing syntax elements to storage device 112 for later retrieval by destination device 116.

[0067] Example techniques for EVC are described below. In some examples, EVC techniques may be similar to those of VVC and / or HEVC (eg, such as block-based coding techniques).

[0068] EVC samples contain access units as defined in ISO / IEC 23094-1, clause 3.1: Information technology – Generic video codec – Part 1: Essential video coding. In EVC, there can be a canonical order for the bitstream to conform to EVC, and certain restrictions may apply. The canonical stream format is an EVC elementary stream that, in addition to the general conditions in ISO / IEC 14496-15: Information technology – Codecs for audiovisual objects – Part 15: Advanced Video Codec (AVC) file format, clause 4.3.2, satisfies the following conditions:

[0069] a. Access unit delimiter NAL unit: The constraints followed by the access unit delimiter NAL unit are defined in ISO / IEC 23094-1.

[0070] b. SPS and PPS: The SPS or PPS to be used in a picture can (and in some examples must) be sent before or in the sample containing the picture. At least the SPS and PPS with id equal to 0 can be stored in the sample entry of the track containing the EVC elementary stream.

[0071] c.APS: The APS to be used in a slice may (and in some examples must) be sent before the VCL NAL unit containing the slice. The APS may be stored in the sample entry and / or

[0072] or in the sample.

[0073] d. SEI messages: Declarative SEI messages may be stored in sample entries; there is no provision for removing such SEI messages from samples.

[0074] e. Padding data. Video data is naturally represented as a variable bit rate in the file format and should be padded for transmission when needed.

[0075] When operating the HRD in constant bit rate (CBR) mode as specified in ISO / IEC 23094-1 Annex C, the removal or addition of filler data NAL units, start codes, SEI messages, or filler data SEI messages may change the bitstream characteristics with respect to compliance with the Hypothetical Reference Decoder (HRD).

[0076] The following is an EVC decoder configuration record and description. In some examples, the video decoder 300 can be configured according to the EVC decoder configuration. As part of describing the EVC decoder configuration description, the following specifies decoder configuration information for ISO / IEC 23094-1 video content.

[0077] This record contains a version field. If the version number is not recognized, a file format parser (e.g., a reader) may (e.g., should) not attempt to decode this record or the stream to which it applies.

[0078] Compatible extensions of this record may extend it and may not change the configuration version code. A reader may ignore unrecognized data that falls outside the definition of the data that the reader is configured to parse.

[0079] The values ​​of profile_idc, level_idc, toolset_idc, chroma_format_idc, pic_width_in_luma_samples, pic_height_in_luma_samples, bit_depth_luma_minus8, and bit_depth_chroma_minus8 may be (and in some examples should be) valid for all parameter sets that are activated when decoding the stream (referred to as "all parameter sets"). Specifically, the following restrictions may apply:

[0080] a. Profile Indication profile_idc can (e.g., should) indicate the profile to which the stream associated with this configuration record conforms. If the SPS is marked with different profiles, it may be necessary to inspect the stream to determine which profile (if any) the entire stream conforms to. If the entire stream is not inspected, or if the inspection finds that the entire stream does not conform to any profile, the entire stream can (e.g., should) be split into two or more sub-streams with separate configuration records in which these rules can be met.

[0081] b. Level Indication level_idc may (eg, should) indicate a capability level that is equal to or greater than the highest level indicated in all parameter sets of this configuration record.

[0082] c.pic_width_in_luma_samples and pic_height_in_luma_samples may (eg, should) contain the highest values ​​of all parameter sets of this configuration record.

[0083] d. Toolset indication toolset_idc can (e.g., should) signal all tools required to decode the stream associated with this configuration record. This tool tag can (e.g., should) comply with the conformance requirements as provided in Table A.6 of ISO / IEC 23094-1 (reproduced below) and can (e.g., should) be the same as the toolset_idc field signaled in the SPS.

[0084] e. The value of chroma_format_idc in all parameter sets may (eg, should) be the same.

[0085] f. The value of bit_depth_luma_minus8 may (eg, should) be the same in all parameter sets.

[0086] g. The value of bit_depth_chroma_minus8 in all parameter sets can (for example, should)

[0087] are the same.

[0088] The EVC decoder configuration record provides explicit indication of the chroma format and bit depth used by the EVC video elementary stream. Each type of such information can (e.g., should) be the same in all parameter sets (if present) in a single EVC configuration record. If two sequences differ in any type of such information, then two different EVC sample entries can (e.g., should) be used.

[0089] There is a set of arrays that carry initialization NAL units. NAL unit types can be restricted to indicate only SPS, PPS, APS, and SEINAL units. NAL unit types that are reserved in ISO / IEC 23094-1 and this specification can be defined, and NAL units with unrecognized NAL unit types can be ignored. This "tolerant" behavior (e.g., ignoring unrecognized NAL unit types) can be designed so that no errors are raised, which allows the possibility of backward-compatible extensions to these arrays in future specifications.

[0090] If stored in the sample entry, the length field can be used in each sample to indicate the length of the NAL unit and parameter set it contains. In some examples, the array is in the order SPS, PPS, APS, SEI.

[0091] Table A.6 of ISO / IEC 23094-1 is as follows:

[0092]

[0093]

[0094]

[0095] The syntax used by the video encoder 200 and the video decoder 300 is described below.

[0096]

[0097]

[0098] The following description illustrates the semantics used to define the terms in the above syntax. The syntax elements profile_idc, level_idc, toolset_idc, chroma_format_idc, toolset_idc, bit_depth_luma_minus8, and bit_depth_chroma_minus8 contain the matching values ​​of the fields in the PPS for all parameter sets of the configuration record. The comment "(32)" indicates that the syntax element toolset_idc is 32 bits. These 32 bits may include one-bit flags, each corresponding to a specific tool, or in some cases, multiple of the 32 bits may correspond to, for example, a combination of tools or a selection of tools from a toolset.

[0099] When the value of the sps_in_stream field is "0", the syntax elements pic_width_in_luma_samples and pic_height_in_luma_samples contain the maximum value of the fields in all SPSs of this configuration record. When the value of the sps_in_stream field is "1", these syntax elements may contain the maximum value of the fields in all SPSs of this configuration record and all SPSs in the stream. If the maximum value of these fields in the SPSs of all parameter sets in this record is not indicated by this field when the value of the sps_in_stream field is "0", or if the value of these fields in the SPSs in the stream has a greater value than the maximum value of the fields in this record when the value of the sps_in_stream field is "1", the value "0" may (e.g., should) be used.

[0100] The syntax element sps_in_stream indicates that the stream may contain additional SPSs that are not included in the NAL unit array of this configuration record. The syntax element pps_in_stream indicates that the stream may contain additional PPSs that are not included in the NAL unit array of this configuration record. The syntax element aps_in_stream indicates that the stream may contain additional APSs that are not included in the NAL unit array of this configuration record.

[0101] The syntax element numArrays indicates the number of NAL unit arrays of the indicated type. The syntax element NAL_unit_type indicates the type of NAL units in the subsequent array (which may (e.g., should) be all of that type). NAL_unit_type takes the values ​​defined in ISO / IEC 23094-1 and may be restricted to taking one of the values ​​indicating an SPS, PPS, APS, or SEI NAL unit.

[0102] The syntax element numNalus indicates the number of NAL units of the indicated type included in the configuration record for the stream to which this configuration record applies. The syntax element nalUnitLength indicates the byte length of the NAL unit. The syntax element nalUnit contains an SPS, PPS, APS, or SEI NAL unit as specified in ISO / IEC 23094-1.

[0103] Video files according to the ISO base media file format and its extensions store data in a series of objects, which are called "boxes." The following describes the ISO base media file format, including the EVC video stream definition and example entry names and formats.

[0104] a. Sample entry and box type: 'evc1', 'evcC'

[0105] b. Container: Sample table frame ('stb1')

[0106] c. Mandatory: The 'evc1' sample entry is mandatory

[0107] d. Quantity: There may be one or more sample entries

[0108] An EVC Visual Sample entry may (eg, should) contain an EVC Configuration box as defined below. This includes an EVCDecoderConfigurationRecord.

[0109] An optional BitRateBox may be present in an EVC visual sample entry to signal the bitrate information of the EVC video stream.

[0110] Where permitted by the ISO Base Media File Format specification, multiple sample entries may be used to indicate portions of the video that use different configurations or parameter sets.

[0111] When the sample entry name is 'evc1', the stream to which this sample entry applies can (eg, should) be a compatible EVC stream as seen by an EVC decoder (eg, video decoder 300) running under the configuration (including profile, level, and toolset) given in EVCConfigurationBox.

[0112] The 'evc1' sample entry allows parameter sets to be stored both in the sample entry and in the stream. sps_in_stream, pps_in_stream, and aps_in_stream, when set to 0, indicate that the array of NAL units of the corresponding type is complete.

[0113] The following is a set of example syntax for configuration boxes specific to the file format.

[0114]

[0115] The following is an example semantic set for the grammar described above.

[0116] Compressorname in the base class VisualSampleEntry indicates the name of the compressor recommended for use with the value "\012EVC Codec" (\012 is 10, the string length in bytes). EVCDecoderConfigurationRecord is defined in section 5.3.3 (e.g., such as, but not limited to, ISO / IEC 23904-1).

[0117] The parameter sets are described below. As an overview, at least the initial SPS and PPS with id equal to 0 can (e.g., should) be carried in the sample entry. If sps_in_stream and / or pps_in_stream are set to "1", additional SPS and / or PPS may be present in-band in the stream.

[0118] Samples carrying a parameter set may (e.g., should) belong to a sample group corresponding to the parameter set type. Three sample group_type 'pss1' are defined in this specification. Another group type parameter is used to distinguish between SPS, PPS, and APS, where 'sps1' identifies a sample group for samples carrying SPS, 'pps1' identifies a sample group for samples carrying PPS, and 'aps1' identifies a sample group for samples carrying APS.

[0119] A sample grouping entry including defined parameter sets is described below.

[0120] a. Group type: 'pss1'

[0121] b. Container: Sample Group Description box ('sgpd')

[0122] c. Mandatory: None

[0123] d. Quantity: zero or more

[0124] The parameter set sample grouping identifies samples containing parameter sets of type SPS, PPS, or APS. The grouping_type_parameter further identifies the type of parameter set and can take the values ​​'sps1', 'pps1', or 'aps1'.

[0125] The following describes some syntax for parameter set sample grouping entries.

[0126] class PSSSampleEntry()extends VisualSampleGroupEntry('pss1')

[0127] {

[0128] }

[0129] Synchronization examples are described below.The synchronization samples in the 'evc1' track may (eg, should) contain a VCL NAL unit indicating that the coded picture with nuh_temporal_id equal to 0 in the sample is an Instantaneous Decoding Refresh (IDR) picture.

[0130] Table 1 indicates the mapping between EVC VCL NAL unit types, ISOBMFF synchronization sample status and SAP types as documented in ISOBMFF.

[0131] Table 1 - Mapping of synchronization sample status and SAP type to NAL unit type

[0132] NAL unit type ISOBMFF synchronization sample status DASH SAP type IDR_N_LP real 1

[0133] The following is the definition of subsamples for EVC. For use with the SubSampleInformationBox (8.7.7 of ISO / IEC 14496-12) in an EVC stream, subsamples are defined based on the value of the Flags field of the Subsample Information Box as specified below. The presence of this box is optional; however, if present in a track containing EVC data, the 'codec_specific_parameters' field in the box may (e.g., should) have the semantics defined here.

[0134] The flags specify the type of subsample information given in this box as follows:

[0135] a.0: NAL unit-based subsample: A subsample contains one or more consecutive NAL units.

[0136] b.1: Tile-based subsample: A subsample contains the VCL NAL units of all CTUs of a tile, together with any associated non-VCL NAL units (if any).

[0137] c.2: Slice-based subsample: A subsample contains one slice (ie, one VCL NAL unit) and associated non-VCL NAL units (if any).

[0138] d. Other values ​​of flags are reserved.

[0139] The subsample_priority field may (eg, should) be set to a value according to the specification of this field in ISO / IEC 14496-12.

[0140] The discardable field may (eg, should) be set to 1 only if the sample is still decodable if the subsample is discarded (eg, a subsample consisting of a SEI NAL unit).

[0141] When the first byte of a NAL unit is included in a subsample, the preceding length field may also (eg, should) be included in the same subsample.

[0142] The codec_specific_parameters field of SubSampleInformationBox is defined as follows for EVC:

[0143]

[0144] tile_col_idx for tile-based subsamples, this parameter indicates the 0-based index of the tile column of the tile containing this subsample.

[0145] tile_row_idx for tile-based subsamples, this parameter indicates the 0-based index of the tile row of the tile containing this subsample.

[0146] The following describes the CMAF media profile. ISO / IEC 23000-19 Common Media Application Format (CMAF) defines structural constraints on ISOBMFF files beyond ISO / IEC 14496-12, for purposes such as adaptive streaming or delivery of encrypted files. Conformance with these structural constraints is signaled by the presence of CMAF-defined structural branding in the FileTypeBox.

[0147] If an ISO BMFF track uses the flag 'cevc', it is called a CMAF EVC track and the following constraints apply (defined for CMAF media profiles for EVC):

[0148] a. It may (eg, should) use 'evc1' as defined in ISO / IEC 23094-1 clause 6.3

[0149] Example entry.

[0150] b. Tracks may (e.g., should) comply with ISO / IEC 23000-19 Clause 7 General

[0151] CMAF track constraints

[0152] c. Tracks MAY (e.g., SHOULD) conform to the general CMAF video track constraints in ISO / IEC 23000-19 clause 9

[0153] If EVC media is provided in a CMAF exchange set, then

[0154] a. Each CMAF track in a CMAF exchange set can (e.g., should) conform to a CMAF EVC track

[0155] b. CMAF exchange sets may (e.g., should) conform to the general CMAF exchange set constraints in ISO / IEC 23000-19 clause 7, and

[0156] c. The general CMAF video track exchange set constraints defined in ISO / IEC 23000-19 clause 9.

[0157] A CMAF exchange set that adheres to these requirements is defined as the CMAF EVC media profile 'cevc'. Encryption of CMAF EVC tracks and CMAF EVC exchange sets may (e.g., SHOULD) conform to ISO / IEC 23000-19 clause 8, using the 'cenc' AES-CTR scheme or the 'cbcs' AES-CBC subsample mode encryption scheme as specified in ISO / IEC 23001-7 clauses 10.1 and 10.4, respectively.

[0158] Furthermore, if the 'cbcs' mode of general encryption using mode encryption as defined in clause 9.6 of ISO / IEC 23001-7 is used, a mode block length of 10 and an encryption of 1:9:skip mode (e.g., as described in clause 10.4 of ISO / IEC 23001-7) may (e.g., should) be applied.

[0159] The mapping to DASH delivery is described below. If EVC-encoded media is provided in a DASH media representation in an adaptation set, the adaptation set may (e.g., should) conform to the DASH profile for CMAF defined in ISO / IEC 23009-1. The following parameters may (e.g., should) appear at the "adaptation set" level and on sets:

[0160] a.@codecs is set according to Appendix A

[0161] b. @mimeType is set for compatibility with "video / mp4 profile="cevc"". Below is a description of the sub-parameters for the MIME type 'codes' parameter. DASH and other applications require the defined values ​​for the codec parameters specified in IETF RFC 6381 for ISO BMFF media tracks. The codec parameter string for the EVC codec is as follows: <Example Entry 4CC>. <key1> <value1> . <key2> <value2> ..... <keyn> <valuen>

[0162] Keys are defined as 4CCs. An initial set of key and associated value pairs is defined in Table 2. Additional keys may be specified as 4CCs. In some examples, the keys are aligned with ISO / IEC 23091-2.

[0163] If a particular key is not provided, the specified default value applies, or if not stated (n / a), the value for that key is unknown.

[0164] Table 2 Definition of the initial set of keys and values ​​defined for EVC

[0165]

[0166]

[0167] For example, codecs = "evc1.vprf3.vlev51.vtoo03FF.vbit20.vcss420.vcpr09.vtrc16.vmac09.vsar01" indicates the EVC Main profile, level 5.1, with 4:2:0 chroma subsampling concatenated with a (0,0) luma sample, a restricted toolset, ITU-R BT.2100 color primaries, ITU-R BT.2100 PQ transfer characteristics, ITU-R BT.2100 YCbCr color matrix, and a sample aspect ratio of 1:1. If the evc1 sample entry is recognized, all keys in Table 2 may (e.g., must) be recognized. If a key is not recognized, the key-value pair is ignored. In some examples, other keys may be defined, such as only 2CC.

[0168] The key-value pair of 'vtoo' and toolset_idc conveys the same information as the toolset_idc syntax element in the decoder configuration record, but indicates a different way to expose this information to the video processing device.

[0169] According to the above techniques, a destination device 116 may be configured to receive a configuration record for decoding a bitstream of video data, wherein the configuration record for the bitstream includes a toolset-indicating syntax element that includes identifying tools from a set of video decoding tools required to decode the bitstream associated with the configuration record; determine whether to retrieve the bitstream associated with the configuration record based on the toolset-indicating syntax element; and based on the determination to retrieve the bitstream associated with the configuration record, retrieve the bitstream and output the bitstream to a video decoder for decoding. The destination device 116 may additionally or alternatively receive a MIME type parameter including a key-value pair, wherein the key indicates that the MIME type identifies a video decoding tool and the value identifies a tool from the video decoding tool required to decode the bitstream. The bitstream of video data may include one or more parameter sets, and each tool identified in the configuration record as required for decoding the bitstream may be enabled in at least one of the one or more parameter sets.

[0170] A toolset indication syntax element is signaled in a configuration box of the file format information. The toolset indication syntax element, which includes information identifying the tools required to decode the bitstream associated with the configuration record, may identify all tools required to decode the bitstream associated with the configuration record. The toolset indication syntax element may be an unsigned 32-bit integer value, where each bit of the unsigned 32-bit integer value corresponds to a unique tool for decoding the bitstream.

[0171] The configuration record may be formatted according to the necessary EVC standard. The configuration record for a bitstream may include a profile syntax element and / or a level syntax element before the toolset indication syntax element. The configuration record for a bitstream may include a chroma-format syntax element after the toolset indication syntax element.

[0172] Figure 2A and 2B is a conceptual diagram illustrating an example quadtree plus binary tree (QTBT) structure 130 and a corresponding coding tree unit (CTU) 132. Solid lines represent quadtree splits, while dashed lines represent binary tree splits. In each split (i.e., non-leaf) node of the binary tree, a flag is signaled to indicate which split type (i.e., horizontal or vertical) is used, in this example, 0 indicates horizontal split and 1 indicates vertical split. For quadtree splits, since the quadtree node splits a block horizontally and vertically into 4 sub-blocks of equal size, there is no need to indicate the split type. Therefore, the video encoder 200 can encode syntax elements (such as split information) at the region tree level (i.e., solid lines) of the QTBT structure 130 and syntax elements (such as split information) at the prediction tree level (i.e., dashed lines) of the QTBT structure 130, and the video decoder 300 can decode them. The video encoder 200 may encode video data (such as prediction and transform data) of the CU represented by the terminal leaf node of the QTBT structure 130 , and the video decoder 300 may decode it.

[0173] generally, Figure 2B The CTU 132 may be associated with parameters that define the size of blocks corresponding to nodes of the first and second levels of the QTBT structure 130. These parameters may include a CTU size (indicating the size of the CTU 132 in samples), a minimum quadtree size (MinQTSize, indicating the minimum allowed quadtree leaf node size), a maximum binary tree size (MaxBTSize, indicating the maximum allowed binary tree root node size), a maximum binary tree depth (MaxBTDepth, indicating the maximum allowed binary tree depth), and a minimum binary tree size (MinBTSize, indicating the minimum allowed binary tree leaf node size).

[0174] The root node of the QTBT structure corresponding to a CTU can have four child nodes at the first level of the QTBT structure, each of which can be split according to quadtree partitioning. That is, the nodes at the first level are leaf nodes (no child nodes) or have four child nodes. The example of the QTBT structure 130 represents such a node as including a parent node and child nodes with branching solid lines. If the node at the first level is not larger than the maximum allowed binary tree root node size (MaxBTSize), the node can be further partitioned by the corresponding binary tree. The binary tree splitting of a node can be iterative until the node generated by the split reaches the minimum allowed binary tree leaf node size (MinBTSize) or the maximum allowed binary tree depth (MaxBTDepth). The example of the QTBT structure 130 represents such a node as having a dotted line for branching. The binary tree leaf node is called a coding unit (CU), which is used for prediction (e.g., intra-screen or inter-screen prediction) and transformation without any further partitioning. As discussed above, a CU can also be referred to as a "video block" or "block."

[0175] In one example of a QTBT partitioning structure, the CTU size is set to 128x128 (luminance sample and two corresponding 64x64 chroma samples), MinQTSize is set to 16x16, MaxBTSize is set to 64x64, MinBTSize (for both width and height) is set to 4, and MaxBTDepth is set to 4. Quadtree partitioning is first applied to the CTU to generate quadtree leaf nodes. The size of the quadtree leaf node can range from 16x16 (i.e., MinQTSize) to 128x128 (i.e., the CTU size). If the quadtree leaf node is 128x128, since the size exceeds MaxBTSize (i.e., 64x64 in this example), the quadtree leaf node will not be further split by the binary tree. Otherwise, the quadtree leaf node will be further split by the binary tree. Therefore, the quadtree leaf node is also the root node of the binary tree and brings the binary tree depth to 0. When the binary tree depth reaches MaxBTDepth (4 in this example), further splitting is not allowed. When the width of a binary tree node is equal to MinBTSize (4 in this example), it means that no further horizontal splitting is allowed. Similarly, a binary tree node with a height equal to MinBTSize means that the binary tree node is not allowed to be further vertically split. As described above, the leaf nodes of the binary tree are called CUs and are further processed according to prediction and transformation without further segmentation.

[0176] Figure 3 is a block diagram illustrating an example video encoder 200 that may perform the techniques of this disclosure. Figure 3 This is provided for illustrative purposes and should not be considered limiting of the techniques broadly illustrated and described in this disclosure. For illustrative purposes, this disclosure describes the video encoder 200 based on the techniques of JEM, EVC, VVC (ITU-T H.266, under development), and HEVC (ITU-T H.265). However, the techniques of this disclosure can be performed by video encoding devices configured for other video coding standards.

[0177] exist Figure 3 In the example of FIG, the video encoder 200 includes a video data memory 230, a mode selection unit 202, a residual generation unit 204, a transform processing unit 206, a quantization unit 208, an inverse quantization unit 210, an inverse transform processing unit 212, a reconstruction unit 214, a filter unit 216, a decoded picture buffer (DPB) 218, and an entropy coding unit 220. Any or all of the video data memory 230, the mode selection unit 202, the residual generation unit 204, the transform processing unit 206, the quantization unit 208, the inverse quantization unit 210, the inverse transform processing unit 212, the reconstruction unit 214, the filter unit 216, the DPB 218, and the entropy coding unit 220 can be implemented in one or more processors or in processing circuitry. For example, the units of the video encoder 200 can be implemented as one or more circuits or logic elements as part of a hardware circuit, or as part of a processor, ASIC, or FPGA. Furthermore, the video encoder 200 can include additional or alternative processors or processing circuitry to perform these and other functions.

[0178] The video data memory 230 may store video data to be encoded by the components of the video encoder 200. The video encoder 200 may receive video data from, for example, the video source 104 ( Figure 1 ) receives video data stored in the video data memory 230. DPB218 can act as a reference picture memory that stores reference video data for use by the video encoder 200 in the prediction of subsequent video data. The video data memory 230 and the DPB 218 can be formed by any of a variety of different storage devices (such as dynamic random access memory (DRAM), including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM) or other types of storage devices). The video data memory 230 and the DPB 218 can be provided by the same storage device or a separate storage device. In various examples, the video data memory 230 can be on the chip with other components of the video encoder 200 (as shown in the figure), or off the chip relative to those components.

[0179] In the present disclosure, references to the video data memory 230 should not be interpreted as limited to memory internal to the video encoder 200 (unless specifically described as such), or to memory external to the video encoder 200 (unless specifically described as such). Rather, references to the video data memory 230 should be understood as a reference memory that stores video data received by the video encoder 200 for encoding (e.g., video data of a current block to be encoded). Figure 1 The memory 106 may also provide temporary storage of outputs from the various units of the video encoder 200 .

[0180] Show Figure 3 The various units of the video encoder 200 are described to help understand the operations performed by the video encoder 200. These units can be implemented as fixed-function circuits, programmable circuits, or a combination thereof. Fixed-function circuits refer to circuits that provide specific functions and are pre-set in the operations that can be performed. Programmable circuits refer to circuits that can be programmed to perform various tasks and provide flexible functions in the operations that can be performed. For example, a programmable circuit can run software or firmware that causes the programmable circuit to operate in a manner defined by the instructions of the software or firmware. Fixed-function circuits can run software instructions (for example, to receive parameters or output parameters), but the type of operations performed by the fixed-function circuits are generally immutable. In some examples, one or more units can be different circuit blocks (fixed-function or programmable), and in some examples, one or more units can be integrated circuits.

[0181] The video encoder 200 may include an arithmetic logic unit (ALU), an elementary function unit (EFU), a digital circuit, an analog circuit, and / or a programmable core formed by a programmable circuit. In an example where the operation of the video encoder 200 is performed using software executed by a programmable circuit, the memory 106 ( Figure 1 ) may store instructions (eg, object code) for software received and executed by the video encoder 200, or another memory (not shown) in the video encoder 200 may store such instructions.

[0182] The video data memory 230 is configured to store received video data. The video encoder 200 can retrieve frames of video data from the video data memory 230 and provide the video data to the residual generation unit 204 and the mode selection unit 202. The video data in the video data memory 230 can be original video data to be encoded.

[0183] The mode selection unit 202 includes a motion estimation unit 222, a motion compensation unit 224, and an intra prediction unit 226. The mode selection unit 202 may include additional functional units to perform video prediction according to other prediction modes. For example, the mode selection unit 202 may include a palette unit, an intra-block copy unit (which may be part of the motion estimation unit 222 and / or the motion compensation unit 224), an affine unit, a linear model (LM) unit, etc.

[0184] The mode selection unit 202 typically coordinates multiple encoding passes to test combinations of encoding parameters and the resulting rate-distortion values ​​for such combinations. The encoding parameters may include the partitioning of CTUs into CUs, the prediction mode for a CU, the transform type for the residual data of a CU, the quantization parameter for the residual data of a CU, etc. The mode selection unit 202 may ultimately select a combination of encoding parameters that has a better rate-distortion value than the other tested combinations.

[0185] The video encoder 200 may partition a picture retrieved from the video data memory 23 into a series of CTUs and encapsulate one or more CTUs into a slice. The mode selection unit 202 may partition the CTUs of the picture according to a tree structure (such as the QTBT structure or quadtree structure described above). As described above, the video encoder 200 may form one or more CUs by partitioning the CTUs according to the tree structure. Such CUs may also be generally referred to as "video blocks" or "blocks."

[0186] Typically, the mode select unit 202 also controls its components (e.g., the motion estimation unit 222, the motion compensation unit 224, and the intra prediction unit 226) to generate a prediction block for the current block (e.g., the current CU, or the overlapping portion of a PU and TU). For inter prediction of the current block, the motion estimation unit 222 may perform a motion search to identify one or more closely matching reference blocks in one or more reference pictures (e.g., one or more previously encoded pictures stored in the DPB 218). In particular, the motion estimation unit 222 may calculate values ​​indicating how similar the potential reference blocks are to the current block, for example, based on the sum of absolute differences (SAD), the sum of squared differences (SSD), the mean absolute difference (MAD), the mean squared difference (MSD), etc. The motion estimation unit 222 may typically perform these calculations using sample-by-sample difference values ​​between the current block and the reference block under consideration. The motion estimation unit 222 may identify the reference block with the lowest value obtained from these calculations, indicating the reference block that most closely matches the current block.

[0187] Motion estimation unit 222 may form one or more motion vectors (MVs) that define the position of a reference block in a reference picture relative to the position of a current block in a current picture. Motion estimation unit 222 may then provide the motion vectors to motion compensation unit 224. For example, for unidirectional inter prediction, motion estimation unit 222 may provide a single motion vector, while for bidirectional inter prediction, motion estimation unit 222 may provide two motion vectors. Motion compensation unit 224 may then use the motion vectors to generate a prediction block. For example, motion compensation unit 224 may use the motion vectors to retrieve data for the reference block. As another example, if the motion vectors have fractional sample precision, motion compensation unit 224 may interpolate the values ​​of the prediction block based on one or more interpolation filters. Furthermore, for bidirectional inter prediction, motion compensation unit 224 may retrieve data for two reference blocks identified by the corresponding motion vectors and combine the retrieved data, for example, by sample-by-sample averaging or weighted averaging.

[0188] As another example, for intra prediction or intra prediction coding, the intra prediction unit 226 can generate a prediction block from samples adjacent to the current block. For example, for directional mode, the intra prediction unit 226 can generally mathematically combine the values ​​of adjacent samples and pad these calculated values ​​along a defined direction on the current block to produce a prediction block. As another example, for DC mode, the intra prediction unit 226 can calculate the average of the adjacent samples of the current block and generate a prediction block to include the obtained average value for each sample of the prediction block.

[0189] Mode selection unit 202 provides the prediction block to residual generation unit 204. Residual generation unit 204 receives the original, unencoded version of the current block from video data memory 230 and the prediction block from mode selection unit 202. Residual generation unit 204 calculates sample-by-sample differences between the current block and the prediction block. The resulting sample-by-sample differences define a residual block for the current block. In some examples, residual generation unit 204 may also determine differences between sample values ​​in the residual block to generate the residual block using residual differential pulse code modulation (RDPCM). In some examples, residual generation unit 204 may be formed using one or more subtractor circuits that perform binary subtraction.

[0190] In the example where the mode select unit 202 partitions the CU into PUs, each PU may be associated with a luma prediction unit and a corresponding chroma prediction unit. The video encoder 200 and the video decoder 300 may support PUs of various sizes. As indicated above, the size of a CU may refer to the size of the luma coding block of the CU, and the size of a PU may refer to the size of the luma prediction unit of the PU. Assuming that the size of a particular CU is 2Nx2N, the video encoder 200 may support PU sizes of 2Nx2N or NxN for intra prediction, and symmetric PU sizes of 2Nx2N, 2NxN, Nx2N, NxN, or similar for inter prediction. The video encoder 200 and the video decoder 300 may also support asymmetric partitioning of PU sizes of 2NxnU, 2NxnD, nLx2N, and nRx2N for inter prediction.

[0191] In an example where the mode selection unit 202 does not further split the CU into PUs, each CU may be associated with a luma coding block and a corresponding chroma coding block. As described above, the size of a CU may refer to the size of the luma coding block of the CU. The video encoder 200 and the video decoder 300 may support CU sizes of 2N×2N, 2N×N, or N×2N.

[0192] For other video coding techniques such as intra block copy mode coding, affine mode coding, and linear model (LM) mode coding, as a few examples, the mode selection unit 202 generates a prediction block for the current block being encoded via the corresponding unit associated with the coding technique. In some examples, such as palette mode coding, the mode selection unit 202 may not generate a prediction block, but instead generate syntax elements indicating how to reconstruct the block based on the selected palette. In such a mode, the mode selection unit 202 may provide these syntax elements to the entropy coding unit 220 for encoding.

[0193] As described above, the residual generation unit 204 receives video data of a current block and a corresponding prediction block. The residual generation unit 204 then generates a residual block for the current block. To generate the residual block, the residual generation unit 204 calculates the sample-by-sample difference between the prediction block and the current block.

[0194] Transform processing unit 206 applies one or more transforms to the residual block to generate a block of transform coefficients (referred to herein as a "transform coefficient block"). Transform processing unit 206 may apply various transforms to the residual block to form the transform coefficient block. For example, transform processing unit 206 may apply a discrete cosine transform (DCT), a directional transform, a Karhunen-Loeve transform (KLT), or a conceptually similar transform to the residual block. In some examples, transform processing unit 206 may perform multiple transforms on the residual block, such as a primary transform and a secondary transform (such as a rotation transform). In some examples, transform processing unit 206 does not apply a transform to the residual block.

[0195] Quantization unit 208 may quantize the transform coefficients in the transform coefficient block to produce a quantized transform coefficient block. Quantization unit 208 may quantize the transform coefficients of the transform coefficient block according to a quantization parameter (QP) value associated with the current block. Video encoder 200 (e.g., via mode select unit 202) may adjust the degree of quantization applied to the transform coefficient block associated with the current block by adjusting the QP value associated with the CU. Quantization may result in a loss of information, and therefore, the quantized transform coefficients may have lower precision than the original transform coefficients produced by transform processing unit 206.

[0196] The inverse quantization unit 210 and the inverse transform processing unit 212 may apply inverse quantization and inverse transform, respectively, to the quantized transform coefficient block to reconstruct a residual block from the transform coefficient block. The reconstruction unit 214 may generate a reconstructed block corresponding to the current block (although possibly with some degree of distortion) based on the reconstructed residual block and the prediction block generated by the mode selection unit 202. For example, the reconstruction unit 214 may add samples of the reconstructed residual block to corresponding samples from the prediction block generated by the mode selection unit 202 to generate the reconstructed block.

[0197] Filter unit 216 may perform one or more filter operations on the reconstructed block. For example, filter unit 216 may perform a deblocking operation to reduce blocking artifacts along the edges of the CU. In some examples, the operations of filter unit 216 may be skipped.

[0198] The video encoder 200 stores the reconstructed block in the DPB 218. For example, in examples where the operation of the filter unit 216 is not required, the reconstruction unit 214 can store the reconstructed block in the DPB 218. In examples where the operation of the filter unit 216 is required, the filter unit 216 can store the filtered reconstructed block in the DPB 218. The motion estimation unit 222 and the motion compensation unit 224 can retrieve a reference picture from the DPB 218, which is formed by the reconstructed (and potentially filtered) block, to perform inter-frame prediction on blocks of subsequently encoded pictures. In addition, the intra-frame prediction unit 226 can use the reconstructed block of the current picture in the DPB 218 to perform intra-frame prediction on other blocks in the current picture.

[0199] In general, entropy coding unit 220 may entropy encode syntax elements received from other functional components of video encoder 200. For example, entropy coding unit 220 may entropy encode quantized transform coefficient blocks from quantization unit 208. As another example, entropy coding unit 220 may entropy encode prediction syntax elements (e.g., motion information for inter-frame prediction or intra-frame mode information for intra-frame prediction) from mode selection unit 202. Entropy coding unit 220 may perform one or more entropy coding operations on the syntax elements (which are another example of video data) to generate entropy-encoded data. For example, entropy coding unit 220 may perform a context-adaptive variable length coding (CAVLC) operation, a CABAC operation, a variable-to-variable (V2V) length coding operation, a syntax-based context-adaptive binary arithmetic coding (SBAC) operation, a probability interval partitioning entropy (PIPE) coding operation, an exponential Golomb coding operation, or another type of entropy coding operation on the data. In some examples, entropy coding unit 220 may operate in a bypass mode in which syntax elements are not entropy-encoded.

[0200] The video encoder 200 may output a bitstream including entropy-encoded syntax elements required to reconstruct blocks of a slice or picture. In particular, the entropy encoding unit 220 may output the bitstream.

[0201] The above operations are described with respect to blocks. Such descriptions should be understood as operations for luma coding blocks and / or chroma coding blocks. As described above, in some examples, the luma coding blocks and chroma coding blocks are the luma and chroma components of a CU. In some examples, the luma coding blocks and chroma coding blocks are the luma and chroma components of a PU.

[0202] In some examples, the operations performed for luma coding blocks do not need to be repeated for chroma coding blocks. As an example, the operations for identifying the motion vector (MV) and reference picture for the luma coding block do not need to be repeated to identify the MV and reference picture for the chroma blocks. Instead, the MV for the luma coding block can be scaled to determine the MV for the chroma blocks, and the reference picture can be the same. As another example, the intra prediction process can be the same for luma coding blocks and chroma coding blocks.

[0203] Video encoder 200 represents an example of a device configured to encode video data, including a memory configured to store the video data, and one or more processing units implemented in circuitry configured to perform one or more example techniques described in this disclosure.

[0204] In some examples, a source device may be configured to receive output from a video encoder 200 and determine (e.g., generate) a configuration record for a bitstream that encapsulates video data encoded by the video encoder 200 according to the Essential Video Coding (EVC) standard, including information for decoding the bitstream. The configuration record may include a toolset indication syntax element (e.g., toolset_idc). The configuration record may be provided at the file format level (e.g., as file format level data) or at the level of media data encoded at the video coding layer (VCL) level. For example, as discussed herein, the configuration record may be included in a configuration box of a media file, where the configuration box is separate from a movie fragment box or other box that includes the actual encoded media data. In some examples, the toolset indication syntax element may include information about all tools required to decode the bitstream associated with the configuration record. The toolset indication syntax element may indicate tools that match another toolset_idc field signaled in a sequence parameter set (SPS).

[0205] In some examples, the toolset indication syntax element may be considered to include the tools necessary to play the file in the sample entry that is part of the encoded parameters. To enable the toolset indication syntax element, an encoding scheme for encoded parameters in a MIME type may be utilized.

[0206] Figure 4 is a block diagram illustrating an example video decoder 300 that may perform the techniques of this disclosure. Figure 4 The present disclosure provides for illustrative purposes and does not limit the techniques broadly illustrated and described in this disclosure. For illustrative purposes, this disclosure describes a video decoder 300 based on techniques of JEM, EVC, VVC (ITU-T H.266, under development), and HEVC (ITU-T H.265). However, the techniques of this disclosure may be performed by video encoding devices configured for other video coding standards.

[0207] exist Figure 4 In the example of FIG, video decoder 300 includes a coded picture buffer (CPB) memory 320, an entropy decoding unit 302, a prediction processing unit 304, an inverse quantization unit 306, an inverse transform processing unit 308, a reconstruction unit 310, a filter unit 312, and a decoded picture buffer (DPB) 314. Any or all of CPB memory 320, entropy decoding unit 302, prediction processing unit 304, inverse quantization unit 306, inverse transform processing unit 308, reconstruction unit 310, filter unit 312, and DPB 314 may be implemented in one or more processors or processing circuits. For example, the units of video decoder 300 may be implemented as one or more circuits or logic elements as part of a hardware circuit or as part of a processor, ASIC, or FPGA. Furthermore, video decoder 300 may include additional or alternative processors or processing circuits to perform these and other functions.

[0208] The prediction processing unit 304 includes a motion compensation unit 316 and an intra prediction unit 318. The prediction processing unit 304 may include additional units to perform prediction according to other prediction modes. As an example, the prediction processing unit 304 may include a palette unit, an intra-block copy unit (which may form part of the motion compensation unit 316), an affine unit, a linear model (LM) unit, etc. In other examples, the video decoder 300 may include more, fewer, or different functional components.

[0209] CPB memory 320 may store video data, such as an encoded video bitstream, to be decoded by components of video decoder 300. For example, the video bitstream may be encoded from computer readable medium 110 ( Figure 1 ) obtains video data stored in CPB memory 320. CPB memory 320 may include a CPB that stores coded video data (e.g., syntax elements) from the coded video bitstream. Furthermore, CPB memory 320 may store video data other than syntax elements for coded pictures, such as temporary data representing outputs from various units of video decoder 300. DPB 314 typically stores decoded pictures, which video decoder 300 may output and / or use as reference video data when decoding subsequent data or pictures in the coded video bitstream. CPB memory 320 and DPB 314 may be formed from any of a variety of memory devices, such as DRAM (including SDRAM, MRAM, RRAM), or other types of memory devices. CPB memory 320 and DPB 314 may be provided by the same memory device or separate memory devices. In various examples, CPB memory 320 may be on-chip with other components of video decoder 300, or off-chip relative to those components.

[0210] Additionally or alternatively, in some examples, video decoder 300 may retrieve the video from memory 120 ( Figure 1 ) to retrieve the encoded video data. That is, memory 120 may store data using CPB memory 320 as discussed above. Similarly, when some or all of the functionality of video decoder 300 is implemented in software to be executed by the processing circuitry of video decoder 300, memory 120 may store instructions to be executed by video decoder 300.

[0211] Figure 4 The various units shown in FIG are illustrated to aid in understanding the operations performed by the video decoder 300. These units may be implemented as fixed function circuits, programmable circuits, or a combination thereof. Figure 3 , fixed-function circuits refer to circuits that provide specific functions and are pre-set in the operations that can be performed. Programmable circuits refer to circuits that can be programmed to perform various tasks and provide flexible functions in the operations that can be performed. For example, a programmable circuit can run software or firmware that causes the programmable circuit to operate in a manner defined by the instructions of the software or firmware. Fixed-function circuits can run software instructions (for example, to receive parameters or output parameters), but the type of operations performed by the fixed-function circuits is generally immutable. In some examples, one or more units may be different circuit blocks (fixed-function or programmable), and in some examples, one or more of the units may be integrated circuits.

[0212] The video decoder 300 may include an ALU, an EFU, a digital circuit, an analog circuit, and / or a programmable core formed by a programmable circuit. In an example where the operation of the video decoder 300 is performed by software running on the programmable circuit, an on-chip or off-chip memory may store instructions (e.g., object code) of the software received and executed by the video decoder 300.

[0213] Entropy decoding unit 302 may receive encoded video data from the CPB and entropy decode the video data to reproduce syntax elements. Prediction processing unit 304, inverse quantization unit 306, inverse transform processing unit 308, reconstruction unit 310, and filter unit 312 may generate decoded video data based on syntax elements extracted from the bitstream.

[0214] Typically, the video decoder 300 reconstructs a picture on a block-by-block basis. The video decoder 300 may perform a reconstruction operation on each block separately (wherein the block currently being reconstructed (ie, decoded) may be referred to as the "current block").

[0215] The entropy decoding unit 302 may entropy decode syntax elements defining the quantized transform coefficients of the quantized transform coefficient block, as well as transform information such as a quantization parameter (QP) and / or a transform mode indication. The inverse quantization unit 306 may use the QP associated with the quantized transform coefficient block to determine a quantization level and, similarly, determine an inverse quantization level to be applied by the inverse quantization unit 306. The inverse quantization unit 306 may, for example, perform a bitwise left shift operation to inverse quantize the quantized transform coefficients. Thus, the inverse quantization unit 306 may form a transform coefficient block including the transform coefficients.

[0216] After inverse quantization unit 306 forms a transform coefficient block, inverse transform processing unit 308 may apply one or more inverse transforms to the transform coefficient block to generate a residual block associated with the current block. For example, inverse transform processing unit 308 may apply an inverse DCT, an inverse integer transform, an inverse Karhunen-Loeve transform (KLT), an inverse rotational transform, an inverse directional transform, or another inverse transform to the transform coefficient block.

[0217] In addition, prediction processing unit 304 generates a prediction block based on the prediction information syntax elements entropy decoded by entropy decoding unit 302. For example, if the prediction information syntax elements indicate that the current block is inter-predicted, motion compensation unit 316 may generate the prediction block. In this case, the prediction information syntax elements may indicate a reference picture in DPB 314, from which to retrieve the reference block, and a motion vector identifying the position of the reference block in the reference picture relative to the position of the current block in the current picture. Motion compensation unit 316 may generally generate a prediction block in the same manner as described for motion compensation unit 224 ( Figure 3 ) is performed in a manner substantially similar to that described in the foregoing.

[0218] As another example, if the prediction information syntax element indicates that the current block is intra-predicted, the intra-prediction unit 318 may generate a prediction block according to the intra-prediction mode indicated by the prediction information syntax element. Again, the intra-prediction unit 318 may generally generate a prediction block in the same manner as described with respect to the intra-prediction unit 226 ( Figure 3 The intra prediction process is performed in a manner substantially similar to that described in the preceding claims. The intra prediction unit 318 may retrieve data of neighboring samples of the current block from the DPB 314.

[0219] The reconstruction unit 310 may reconstruct the current block using the prediction block and the residual block. For example, the reconstruction unit 310 may add samples of the residual block to corresponding samples of the prediction block to reconstruct the current block.

[0220] The filter unit 312 may perform one or more filter operations on the reconstructed block. For example, the filter unit 312 may perform a deblocking operation to reduce blocking artifacts along the edges of the reconstructed block. The operations of the filter unit 312 may not necessarily be performed in all examples.

[0221] The video decoder 300 may store the reconstructed block in the DPB 314. For example, in examples where the operation of the filter unit 312 is not performed, the reconstruction unit 310 may store the reconstructed block to the DPB 314. In examples where the operation of the filter unit 312 is performed, the filter unit 312 may store the filtered reconstructed block to the DPB 314. As discussed above, the DPB 314 may provide reference information such as samples of the current picture for intra prediction and previously decoded pictures for subsequent motion compensation to the prediction processing unit 304. In addition, the video decoder 300 may output decoded pictures (e.g., decoded video) from the DPB 314 for subsequent use in, for example, Figure 1 The display device 118 is displayed on a display device.

[0222] In this manner, video decoder 300 represents an example of a video decoding device that includes memory configured to store video data, and one or more processing units implemented in circuitry and configured to perform the example techniques described in this disclosure.

[0223] In some examples, a client device may be configured to receive a configuration record for decoding a bitstream of video data according to the Essential Video Coding (EVC) standard. The configuration record may include a toolset indication syntax element (e.g., toolset_idc). In some examples, the toolset indication syntax element may include information about all tools required to decode the bitstream associated with the configuration record. The toolset indication syntax element of the configuration record may match the toolset indication syntax element signaled in a sequence parameter set (SPS). Based on the toolset indication syntax element, the client device may determine whether to retrieve the bitstream associated with the configuration record. Based on the determination to retrieve the bitstream associated with the configuration record, the client device may retrieve the bitstream and output the bitstream to the video decoder 300 for decoding.

[0224] In some examples, the toolset indication syntax element may be considered to include the tools needed to play the file in the sample entry that is part of the encoding parameters. To enable the toolset indication syntax element, an encoding scheme for encoded parameters in a MIME type may be utilized.

[0225] Figure 5 is a conceptual diagram illustrating an example structure of a file 500 according to one or more techniques of this disclosure. Figure 5 In the example of , the file 500 includes a movie box 502 and a plurality of media data boxes 504. Figure 5 In the example of FIG. 5 , the movie box 502 and the media data box 504 are shown in the same file, but in other examples, the movie box 502 and the media data box 504 can be in separate files. As indicated above, a box can be an object-oriented building block defined by a unique type identifier and length. For example, a box can be a basic syntactic structure in ISOBMFF, including a four-character encoded box type, a byte count of the box, and a payload.

[0226] Movie box 502 may contain metadata for tracks of file 500. Each track of file 500 may include a continuous stream of media data. Each media data box 504 may include one or more samples 505. Each sample 505 may include an audio or video access unit. Each access unit may include multiple coded pictures for multi-view coding or scalable video coding. For example, an access unit may include one or more coded pictures for each layer.

[0227] In addition, Figure 5 In the example of , movie box 502 includes track box 506. Track box 506 can enclose metadata of a track of file 500. In other examples, movie box 502 can include multiple track boxes for different tracks of file 500. Track box 506 includes track reference box 508 and media box 510. Track reference box 508 can include track reference type box 509. Track reference type box 509 can be associated with a type (e.g., 'tbas') and a track identifier that identifies another track. According to the sixth technique of the present disclosure, the track reference box of a tile track may or may not include a track reference type box associated with the type identifier 'tbas'.

[0228] The media box 510 may contain all objects that declare information about the media data within the track. The media box 510 includes a media information box 512. The media information box 512 may contain all objects that declare characteristic information about the media of the track. The media information box 512 includes a sample table box 514. The sample table box 514 may specify sample-specific metadata.

[0229] exist Figure 5 In the example of , the sample table box 514 includes a sample description box 516, and the sample description box 516 includes a sample entry 518. The sample entry 518 includes an EVC decoder configuration 520, which, as described above, may include a toolset indication syntax element. In other examples, the sample table box 514 may include other boxes in addition to the sample description box 516 and the sample entry 518. Although in Figure 5 Not shown, but sample entry 518 may include a decoder configuration record. As described in more detail above, this disclosure describes techniques for including new types of important information in sample entries, such as sample entry 518.

[0230] Figure 6 is a conceptual diagram illustrating elements of example multimedia content 620. Figure 6 In the example of , multimedia content 620 includes an MPD 622 and multiple representations 624A-624N (representations 624). Representation 624A includes optional header data 626 and segments 628A-628N (segment 628), while representation 624N includes optional header data 630 and segments 632A-632N (segment 632). For convenience, the letter N is used to designate the last movie fragment in each representation 624. In some examples, there may be different numbers of movie fragments between representations 624.

[0231] The MPD 622 may include a data structure separate from the representation 624. In general, the MPD 422 may include data generally describing characteristics of the representation 424, such as encoding and rendering characteristics, adaptation sets, a profile to which the MPD 422 corresponds, text type information, camera angle information, ratings information, trick mode information (e.g., information indicating that a representation includes a temporal subsequence), and / or information for retrieving remote periods (e.g., information for inserting targeted advertisements into media content during playback).

[0232] Header data 626, when present, may describe characteristics of segment 628, such as the temporal location of a random access point (RAP, also known as a stream access point (SAP)), which of the segments 628 include a random access point, the byte offset to a random access point within the segment 628, a uniform resource locator (URL) for the segment 628, or other aspects of the segment 628. Header data 630, when present, may describe similar characteristics of segment 632. Additionally or alternatively, such characteristics may be included entirely within the MPD 622.

[0233] Segments 628, 632 include one or more coded video samples, each of which may include a frame or slice of video data. Each coded video sample of segment 628 may have similar characteristics, such as height, width, and bandwidth requirements. Such characteristics may be described by data in MPD 622, although in Figure 6 Such data is not illustrated in the example of . The MPD 622 may include characteristics as described in the 3GPP specifications, and add any or all of the signaling information described in this disclosure.

[0234] Each segment 628, 632 can be associated with a unique uniform resource locator (URL). Thus, each segment 628, 632 can be independently retrievable using a streaming network protocol such as DASH. In this manner, a destination device can use an HTTP GET request to retrieve a segment 628 or 632. In some examples, a destination device can use an HTTP partial GET request to retrieve a specific byte range of a segment 628 or 632.

[0235] Figure 7 is a block diagram illustrating the elements of an example video file 750. Video file 750 can be considered to encapsulate segments. As described above, video files according to the ISO base media file format and its extensions store data in a series of objects called "boxes." Figure 7 In the example of FIG, a video file 750 includes a file type (FTYP) box 752, a movie (MOOV) box 754, a segment index (sidx) box 762, a movie fragment (MOOF) box 764, and a movie fragment random access (MFRA) box 766. Figure 7 An example of a video file is shown, but it should be understood that other media files may include other types of media data (e.g., audio data, timed text data, etc.) structured similarly to the data of video file 750 according to the ISO base media file format and its extensions.

[0236] The FTYP box 752 generally describes the file type of the video file 750. The file type box 752 may include data identifying specifications describing the optimal use of the video file 750. The file type box 752 may alternatively be placed before the MOOV box 754, the movie fragment box 764, and / or the MFRA box 766.

[0237] In some examples, a segment such as video file 750 may include an MPD update box (not shown) before the FTYP box 752. The MPD update box may include information indicating that the MPD corresponding to the representation including video file 750 is to be updated along with information for updating the MPD. For example, the MPD update box may provide a URI or URL of a resource for updating the MPD. As another example, the MPD update box may include data for updating the MPD. In some examples, the MPD update box may be followed by a segment type (STYP) box (not shown) for video file 750, where the STYP box may define the segment type of video file 750.

[0238] exist Figure 7 , the MOOV box 754 includes a movie header (MVHD) box 756, a track (TRAK) box 758, and one or more movie extension (MVEX) boxes 760. Generally, the MVHD box 756 may describe general characteristics of the video file 750. For example, the MVHD box 756 may include data describing the time when the video file 750 was originally created, the time when the video file 750 was last modified, the time scale of the video file 750, the playback duration of the video file 750, or other data generally describing the video file 750.

[0239] The TRAK box 758 may include data for a track of the video file 750. The TRAK box 758 may include a track header (TKHD) box that describes characteristics of the track corresponding to the TRAK box 758. In some examples, the TRAK box 758 may include coded video pictures, while in other examples, the coded video pictures of the track may be included in a movie fragment 764, which may be referenced by data in the TRAK box 758 and / or the sidx box 762.

[0240] In some examples, video file 750 may include more than one track. Thus, MOOV box 754 may include a number of TRAK boxes equal to the number of tracks in video file 750. TRAK box 758 may describe characteristics of a corresponding track of video file 750. For example, TRAK box 758 may describe temporal and / or spatial information for the corresponding track. When a parameter set track is included in a video file such as video file 750, a TRAK box similar to TRAK box 758 of MOOV box 754 may describe characteristics of the parameter set track. The presence of a sequence-level SEI message may be signaled in the parameter set track within the TRAK box describing the parameter set track.

[0241] The MVEX box 760 may describe characteristics of the corresponding movie fragments 764, e.g., to signal that, in addition to the video data (if any) included within the MOOV box 754, the video file 750 includes the movie fragments 764. In the context of streaming video data, coded video pictures may be included in the movie fragments 764 rather than the MOOV box 754. Thus, all coded video samples may be included in the movie fragments 764 rather than the MOOV box 754.

[0242] The MOOV box 754 may include a number of MVEX boxes 760 equal to the number of movie fragments 764 in the video file 750. Each MVEX box 760 may describe characteristics of a corresponding one of the movie fragments 764. For example, each MVEX box may include a Movie Extended Header box (MEHD) that describes the temporal duration of the corresponding one of the movie fragments 764.

[0243] A sequence data set may be stored in video samples that do not include actual coded video data. A video sample may generally correspond to an access unit, which is a representation of a coded picture at a particular instance in time. In the context of AVC, a coded picture comprises one or more VCL NAL units that contain information for all pixels used to construct an access unit and other associated non-VCL NAL units, such as SEI messages. Thus, a sequence data set that may include a sequence level SEI message may be included in one of the movie fragments 764. The presence of a sequence data set and / or sequence level SEI message may be signaled in one of the movie fragments 764 in one of the MVEX boxes 760 corresponding to one of the movie fragments 764.

[0244] The SIDX box 762 is an optional element of the video file 750. That is, a video file conforming to the 3GPP file format or other such file formats does not necessarily include the SIDX box 762. According to the example of the 3GPP file format, the SIDX box can be used to identify subsegments of a segment (e.g., a segment contained in the video file 750). The 3GPP file format defines a subsegment as "a self-contained set of one or more consecutive movie fragment boxes, which have corresponding (multiple) media data boxes, the media data boxes containing data referenced by the movie fragment box must follow that movie fragment box and precede the next movie fragment box containing information about the same track." The 3GPP file format also indicates that the SIDX box "contains a sequence of references to subsegments of the (sub)segment recorded by that box. The referenced subsegments are consecutive in presentation time. Similarly, the bytes referenced by the segment index box are always consecutive within the segment. The referenced size gives a count of the number of bytes in the referenced material."

[0245] The SIDX block 762 generally provides information representing one or more subsegments of a segment included in the video file 750. For example, such information may include the playback time at which the subsegment begins and / or ends, the byte offset for the subsegment, whether the subsegment includes (e.g., starts at) a stream access point (SAP), the type of SAP (e.g., whether the SAP is an Instantaneous Decoder Refresh (IDR) picture, a Clean Random Access (CRA) picture, a Breakpoint Linked Access (BLA) picture, etc.), the location of the SAP within the subsegment (in terms of playback time and / or byte offset), etc.

[0246] The movie fragment 764 may include one or more coded video pictures. In some examples, the movie fragment 764 may include one or more groups of pictures (GOPs), each of which may include multiple coded video pictures, such as frames or pictures. Additionally, as described above, in some examples, the movie fragment 764 may include a sequence data set. Each movie fragment 764 may include a movie fragment header box (MFHD, Figure 7 (not shown in FIG. 1 ). The MFHD box may describe characteristics of the corresponding movie fragment, such as a sequence number of the movie fragment. The movie fragments 764 may be included in the video file 750 in the order of the sequence numbers.

[0247] MFRA box 766 can describe random access points in the movie fragments 764 of the video file 750. This can help perform trick modes, such as performing seeks to specific time positions (i.e., playback times) in the segments encapsulated by the video file 750. In some examples, the MFRA box 766 is generally optional and does not need to be included in the video file. Likewise, a client device does not have to reference the MFRA box 766 to correctly decode and display the video data of the video file 750. The MFRA box 766 can include a number of track fragment random access (TFRA) boxes (not shown) equal to the number of tracks in the video file 750, or, in some examples, equal to the number of media tracks (e.g., non-hint tracks) in the video file 750.

[0248] In some examples, the movie fragment 764 may include one or more stream access points (SAPs), such as IDR pictures. Similarly, the MFRA box 766 may provide an indication of the location of the SAPs in the video file 750. Thus, a temporal subsequence of the video file 750 may be formed from the SAPs of the video file 750. The temporal subsequence may also include other pictures, such as P frames and / or B frames that depend on the SAPs. The frames and / or slices of the temporal subsequence may be arranged within a segment so that the frames / slices of the temporal subsequence that depend on other frames / slices of the subsequence may be properly decoded. For example, in a hierarchical arrangement of data, data used for prediction of other data may also be included in the temporal subsequence.

[0249] In this example, the video file 750 also includes a sample description box 768. In particular, in this example, the sample description box 768 is included within the TRAK box 758. Figure 7 In the example shown, sample description block 768 does not include video data encoded according to the codec.

[0250] An example sample description box 768 may be defined as follows:

[0251] Sample entries and box types: 'hvc2', 'hev2', 'lhv1', 'lhe1', 'lhvC'

[0252] Container: Sample description box ('stsd')

[0253] · Mandatory: 'hvc1', 'hev1', 'hvc2', 'hev2', 'lhv1' or 'lhe1' sample entries are mandatory · Quantity: One or more sample entries may be present

[0254] In this example definition of sample description box 768, the default and mandatory value of array_completeness is 4 for arrays of all types of parameter sets when the sample entry name is 'lhv1' and 0 for all other types. When the sample entry name is 'lhe1', the default value of array_completeness is 0 for all arrays.

[0255] Figure 8 is a flowchart illustrating an example method for processing video data according to the techniques of this disclosure. Figure 1 The destination device 116 is described, but it should be understood that other devices may be configured to perform similar Figure 8 method.

[0256] The destination device 116 may be configured to receive a configuration record for decoding a bitstream of video data (800). The configuration record for the bitstream may include a toolset indication syntax element that includes information identifying a tool from a set of video decoding tools required to decode the bitstream associated with the configuration record. Based on the toolset indication syntax element in the configuration record, the destination device 116 may determine whether to retrieve the bitstream associated with the configuration record (802). The destination device 116 may then retrieve the bitstream (804). The destination device 116 may output the bitstream to a video decoder for decoding (806). The destination device 116 may additionally or alternatively receive a MIME type parameter including a key-value pair, wherein the key indicates a MIME type identifying a video decoding tool and the value identifies a tool from the video decoding tool required to decode the bitstream. The bitstream of video data may include one or more parameter sets, and each tool identified in the configuration record as required to decode the bitstream may be enabled in at least one of the one or more parameter sets.

[0257] A toolset indication syntax element is signaled in a configuration box of the file format information. The toolset indication syntax element, which includes information identifying tools required to decode a bitstream associated with the configuration record, may identify all tools required to decode the bitstream associated with the configuration record. The toolset indication syntax element may be an unsigned 32-bit integer value, where each bit of the unsigned 32-bit integer value corresponds to a unique tool for decoding the bitstream.

[0258] The configuration record may be formatted according to the necessary EVC standard. The configuration record for a bitstream may include a profile syntax element and / or a level syntax element before the toolset indication syntax element. The configuration record for a bitstream may include a chroma-format syntax element after the toolset indication syntax element.

[0259] It should be appreciated that, depending on the example, certain actions or events of any of the techniques described herein may be performed in a different sequence, may be added, combined, or omitted altogether (e.g., not all described actions or events are required to practice these techniques). Furthermore, in some examples, actions or events may be performed concurrently rather than sequentially, such as through multithreading, interrupt handling, or multiple processors.

[0260] The following clauses represent example implementations of the above-described systems and techniques.

[0261] Clause 1. A method of processing a video stream or video file, the method comprising: receiving a configuration record for decoding a bitstream of video data, wherein the configuration record for the bitstream includes a toolset indication syntax element, the toolset indication syntax element including information identifying a tool from a set of video decoding tools required to decode the bitstream associated with the configuration record; determining, based on the toolset indication syntax element, whether to retrieve a bitstream associated with the configuration record; and based on the determination to retrieve the bitstream associated with the configuration record, retrieving the bitstream and outputting the bitstream to a video decoder for decoding.

[0262] Clause 2. The method of clause 1, further comprising: receiving a Multipurpose Internet Mail Extensions (MIME) type parameter comprising a key-value pair, wherein the key indicates a MIME type identifying a video decoding tool, and the value identifies a tool from the video decoding tool required to decode the bitstream.

[0263] Clause 3. The method of clause 1 or 2, wherein the bitstream of video data comprises one or more parameter sets, and wherein each tool identified in the configuration record as required for decoding the bitstream is enabled in at least one of the one or more parameter sets.

[0264] Clause 4. The method of any of clauses 1-3, wherein the toolset-indicating syntax element comprising information identifying tools required to decode the bitstream associated with the configuration record identifies all tools required to decode the bitstream associated with the configuration record.

[0265] Clause 5. The method of any of clauses 1-4, wherein the configuration record is formatted according to the Essential Video Coding (EVC) standard.

[0266] Clause 6. The method of any of clauses 1-5, wherein the toolset indication syntax element is signaled in a configuration box of the file format information.

[0267] Clause 7. The method of any of clauses 1-6, wherein the toolset indicates that the syntax element comprises an unsigned 32-bit integer value.

[0268] Clause 8. The method of clause 7, wherein each bit of the unsigned 32-bit integer value corresponds to a unique tool for decoding the bitstream.

[0269] Clause 9. The method of any of clauses 1-8, wherein the configuration record for the bitstream includes a profile syntax element prior to the toolset indication syntax element.

[0270] Clause 10. The method of any of clauses 1-9, wherein the configuration record for the bitstream includes a level syntax element prior to the toolset indication syntax element.

[0271] Clause 11. The method of any of clauses 1-10, wherein the configuration record for the bitstream includes a chroma format syntax element following the toolset indication syntax element.

[0272] Clause 12. The method of any of clauses 1-11, wherein the configuration record includes file format level data that encapsulates video coding layer (VCL) level encoded media data in a bitstream of video data, wherein the VCL level encoded media data includes one or more parameter sets, and wherein each tool identified in the configuration record as required for decoding the bitstream is enabled in at least one of the one or more parameter sets.

[0273] Clause 13. An apparatus for processing video data, the apparatus comprising: a memory; and one or more processors implemented in circuitry, coupled to the memory, and configured to: receive a configuration record for decoding a bitstream of video data, wherein the configuration record for the bitstream includes a toolset indication syntax element, the toolset indication syntax element including information identifying a tool from a set of video decoding tools required to decode the bitstream associated with the configuration record; determine, based on the toolset indication syntax element, whether to retrieve a bitstream associated with the configuration record; and based on a determination to retrieve the bitstream associated with the configuration record, retrieve the bitstream and output the bitstream to a video decoder for decoding.

[0274] Clause 14. The device of clause 13, wherein the one or more processors are further configured to: receive a Multipurpose Internet Mail Extensions (MIME) type parameter comprising a key-value pair, wherein the key indicates a MIME type identifying a video decoding tool, and the value identifies a tool from the video decoding tool required to decode the bitstream.

[0275] Clause 15. The apparatus of clause 13 or 14, wherein the bitstream of video data comprises one or more parameter sets, and wherein each tool identified in the configuration record as required for decoding the bitstream is enabled in at least one of the one or more parameter sets.

[0276] Clause 16. The apparatus of any of clauses 13-15, wherein the toolset indicating syntax element comprising information identifying tools required to decode the bitstream associated with the configuration record identifies all tools required to decode the bitstream associated with the configuration record.

[0277] Clause 17. The apparatus of any of clauses 13-16, wherein the configuration record is formatted according to the Essential Video Coding (EVC) standard.

[0278] Clause 18. The apparatus of any of clauses 13-17, wherein the toolset indication syntax element is signaled in a configuration box of the file format information.

[0279] Clause 19. The apparatus of any of clauses 13-15, wherein the toolset indicates that the syntax element comprises an unsigned 32-bit integer value.

[0280] Clause 20. The apparatus of any of Clause 19, wherein each bit of the unsigned 32-bit integer value corresponds to a unique means for decoding the bitstream.

[0281] Clause 21. The apparatus of any of clauses 13-20, wherein the configuration record for the bitstream includes a profile syntax element preceding the toolset indication syntax element.

[0282] Clause 22. The apparatus of any of clauses 13-21, wherein the configuration record for the bitstream includes a level syntax element prior to the toolset indication syntax element.

[0283] Clause 23. The apparatus of any of clauses 13-22, wherein the configuration record for the bitstream includes a chroma format syntax element following the toolset indication syntax element.

[0284] Clause 24. The apparatus of any of clauses 13-23, wherein the configuration record includes file format level data that encapsulates video coding layer (VCL) level encoded media data in a bitstream of video data, wherein the VCL level encoded media data includes one or more parameter sets, and wherein each tool identified in the configuration record as required for decoding the bitstream is enabled in at least one of the one or more parameter sets.

[0285] Clause 25. A computer-readable storage medium having instructions stored thereon that, when executed by one or more processors, cause the one or more processors to: receive a configuration record for decoding a bitstream of video data, wherein the configuration record for the bitstream includes a toolset indication syntax element, the toolset indication syntax element including information identifying a tool from a set of video decoding tools required to decode the bitstream associated with the configuration record; determine, based on the toolset indication syntax element, whether to retrieve a bitstream associated with the configuration record; and based on the determination to retrieve the bitstream associated with the configuration record, retrieve the bitstream and output the bitstream to a video decoder for decoding.

[0286] Clause 26. The computer-readable storage medium of clause 25, wherein the instructions cause one or more processors to: receive a Multipurpose Internet Mail Extensions (MIME) type parameter comprising a key-value pair, wherein the key indicates a MIME type identifying a video decoding tool, and the value identifies a tool from the video decoding tool required to decode the bitstream.

[0287] Clause 27. The computer-readable storage medium of clause 25 or 26, wherein the bitstream of video data comprises one or more parameter sets, and wherein each tool identified in the configuration record as required for decoding the bitstream is enabled in at least one of the one or more parameter sets.

[0288] Clause 28. The computer-readable storage medium of any of clauses 25-27, wherein the toolset indicating syntax element comprising information identifying tools required to decode the bitstream associated with the configuration record identifies all tools required to decode the bitstream associated with the configuration record.

[0289] Clause 29. The computer-readable storage medium of any of clauses 25-28, wherein the configuration record includes file format level data that encapsulates video coding layer (VCL) level encoded media data in a bitstream of video data, wherein the VCL level encoded media data includes one or more parameter sets, and wherein each tool identified in the configuration record as required for decoding the bitstream is enabled in at least one of the one or more parameter sets.

[0290] Clause 30. An apparatus for processing a video stream or video file, the apparatus comprising: means for receiving a configuration record for decoding a bitstream of video data, wherein the configuration record for the bitstream includes a toolset indication syntax element, the toolset indication syntax element including information identifying a tool from a set of video decoding tools required to decode the bitstream associated with the configuration record; means for determining whether to retrieve a bitstream associated with the configuration record based on the toolset indication syntax element; means for retrieving the bitstream associated with the configuration record based on a determination that the bitstream is to be retrieved; and means for outputting the bitstream to a video decoder for decoding.

[0291] Clause 31. A method of processing video data, the method comprising: receiving a configuration record for decoding a bitstream of video data according to the Essential Video Coding (EVC) standard, wherein the configuration record for the bitstream includes a toolset indication syntax element, the toolset indication syntax element including information about all tools required to decode the bitstream associated with the configuration record; determining, based on the toolset indication syntax element, whether to retrieve a bitstream associated with the configuration record; and based on the determination to retrieve the bitstream associated with the configuration record, retrieving the bitstream and outputting the bitstream to a video decoder for decoding.

[0292] Clause 32. A method of processing video data, the method comprising: determining a configuration record for a bitstream of video data generated by a video encoder according to the Essential Video Coding (EVC) standard, wherein the configuration record for the bitstream includes information for decoding the bitstream, and wherein the configuration record includes a toolset indication syntax element that includes information of all tools required to decode the bitstream associated with the configuration record; and signaling the configuration record.

[0293] Clause 33. The method of clause 31 or 32, wherein the toolset indication syntax element is signaled in a configuration box of the file format information.

[0294] Clause 34. An apparatus for processing video data, the apparatus comprising: processing circuitry configured to perform the method of any one or a combination of clauses 1-3.

[0295] Clause 35. The apparatus of clause 34, further comprising a memory for storing the video data.

[0296] Clause 36. The apparatus of any of clauses 34 and 35, further comprising a display configured to display the decoded video data.

[0297] Clause 37. The device of any of clauses 34-36, wherein the device comprises one or more of a camera, a computer, a mobile device, a broadcast receiver device, or a set-top box.

[0298] Clause 38. A computer-readable storage medium having stored thereon instructions that, when executed, cause one or more processors to perform the method of any of clauses 31-33.

[0299] Clause 39. An apparatus for processing video data, the apparatus comprising means for performing the method of any one of clauses 31-33.

[0300] In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted via a computer-readable medium as one or more instructions or codes and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media, which corresponds to tangible media such as data storage media, or communication media, including, for example, any media that facilitates the transfer of a computer program from one place to another according to a communication protocol. In this manner, computer-readable media may generally correspond to (1) a non-transitory tangible computer-readable storage medium, or (2) a communication medium such as a signal or carrier wave. Data storage media may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, codes, and / or data structures for implementing the techniques described in this disclosure. A computer program product may include computer-readable media.

[0301] As an example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage, or other magnetic storage devices, flash memory, or any other medium that can be used to store the required program code in the form of instructions or data structures and that can be accessed by a computer. Moreover, any connection is appropriately referred to as a computer-readable medium. For example, if instructions are sent from a website, server or other remote source using a coaxial cable, optical cable, twisted pair, digital subscriber line (DSL) or wireless technology (such as infrared, radio and microwave), the definition of the medium includes coaxial cable, optical cable, twisted pair, DSL or wireless technology such as infrared, radio and microwave. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals or other temporary media, but are directed to non-temporary tangible storage media. Disks and optical disks as used in this application include compact disks (CDs), laser optical disks, optical optical disks, digital versatile disks (DVDs), floppy disks and Blu-ray disks, wherein disks typically reproduce data magnetically, while optical disks reproduce data optically with lasers. The above combinations should also be included within the scope of computer-readable media.

[0302] Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Thus, the terms "processor" and "processing circuitry" as used in this application may refer to any of the aforementioned structures or any other structures suitable for implementing the techniques described in this application. Additionally, in some aspects, the functionality described in this application may be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated into a combined codec. Similarly, the technology may be fully implemented in one or more circuits or logic elements.

[0303] The techniques of this disclosure can be implemented in a variety of devices or apparatuses, including wireless handsets, integrated circuits (ICs), or a set of ICs (e.g., a chipset). Various components, modules, or units are described in this disclosure to emphasize functional aspects of devices configured to perform the disclosed techniques, but they do not necessarily need to be implemented by different hardware units. Rather, as described above, the various units can be combined in a codec hardware unit, or provided by a collection of interoperable hardware units, including one or more processors as described above in combination with appropriate software and / or firmware.

[0304] Various examples have been described. These and other examples are within the scope of the following claims.< / valuen> < / keyn> < / value2> < / key2> < / value1> < / key1>

Claims

1. A method for processing a video stream or a video file, the method comprising: A configuration record is received for decoding a bitstream of video data, wherein the configuration record for the bitstream applies to a plurality of sequence parameter sets and includes a toolset indication syntax element comprising information identifying a tool from a set of video decoding tools required to decode the bitstream associated with the configuration record; determining, based on the toolset indication syntax element, tools required to decode the bitstream associated with the configuration record; as well as The bitstream associated with the configuration record is decoded using one or more of the tools.

2. The method according to claim 1, further comprising: A Multipurpose Internet Mail Extensions (MIME) type parameter is received, comprising a key-value pair, wherein the key indicates that the MIME type identifies the video decoding tool, and the value identifies the tool from the video decoding tool required to decode the bitstream.

3. The method according to claim 1, wherein The bitstream of video data includes one or more parameter sets, and wherein each of the tools identified in the configuration record as required for decoding the bitstream is enabled in at least one of the one or more parameter sets.

4. The method according to claim 1, wherein The toolset indication syntax element, including information identifying the tools required to decode the bitstream associated with the configuration record, identifies all tools required to decode the bitstream associated with the configuration record.

5. The method according to claim 1, wherein The configuration record is formatted according to the Essential Video Coding (EVC) standard.

6. The method according to claim 1, wherein The toolset is signaled in a configuration box of the file format information indicating syntax elements.

7. The method according to claim 1, wherein The toolset indicates that the syntax element comprises an unsigned 32-bit integer value.

8. The method according to claim 7, wherein: Each bit of the unsigned 32-bit integer value corresponds to a unique tool for decoding the bitstream.

9. The method according to claim 1, wherein The configuration record for the bitstream includes a profile syntax element preceding the toolset indication syntax element.

10. The method according to claim 1, wherein The configuration record for the bitstream includes a level syntax element prior to the toolset indication syntax element.

11. The method according to claim 1, wherein The configuration record for the bitstream includes a chroma format syntax element following the toolset indication syntax element.

12. The method according to claim 1, wherein The configuration record includes file format level data that encapsulates video coding layer (VCL) level encoded media data in the bitstream of the video data, wherein the VCL level encoded media data includes one or more parameter sets, and wherein each of the tools identified in the configuration record as required for decoding the bitstream is enabled in at least one of the one or more parameter sets.

13. A device for processing video data, the device comprising: Memory; as well as one or more processors implemented in circuitry, coupled to the memory, and configured to: receiving a configuration record for decoding a bitstream of video data, wherein the configuration record for the bitstream applies to a plurality of sequence parameter sets and includes a toolset indication syntax element including information identifying a tool from a set of video decoding tools required to decode the bitstream associated with the configuration record; determining, based on the toolset indication syntax element, tools required to decode the bitstream associated with the configuration record; and The bitstream associated with the configuration record is decoded using one or more of the tools.

14. The apparatus according to claim 13, wherein The one or more processors are further configured to: A Multipurpose Internet Mail Extensions (MIME) type parameter is received, comprising a key-value pair, wherein the key indicates that the MIME type identifies the video decoding tool, and the value identifies the tool from the video decoding tool required to decode the bitstream.

15. The apparatus according to claim 13, wherein The bitstream of video data includes one or more parameter sets, and wherein each of the tools identified in the configuration record as required for decoding the bitstream is enabled in at least one of the one or more parameter sets.

16. The apparatus according to claim 13, wherein The toolset indication syntax element, including information identifying the tools required to decode the bitstream associated with the configuration record, identifies all tools required to decode the bitstream associated with the configuration record.

17. The apparatus according to claim 13, wherein The configuration record is formatted according to the Essential Video Coding (EVC) standard.

18. The apparatus according to claim 13, wherein The toolset is signaled in a configuration box of the file format information indicating syntax elements.

19. The apparatus according to claim 13, wherein The toolset indicates that the syntax element comprises an unsigned 32-bit integer value.

20. The apparatus according to claim 19, wherein Each bit of the unsigned 32-bit integer value corresponds to a unique tool for decoding the bitstream.

21. The apparatus according to claim 13, wherein The configuration record for the bitstream includes a profile syntax element preceding the toolset indication syntax element.

22. The apparatus of claim 13, wherein: The configuration record for the bitstream includes a level syntax element prior to the toolset indication syntax element.

23. The apparatus of claim 13, wherein: The configuration record for the bitstream includes a chroma format syntax element following the toolset indication syntax element.

24. The apparatus of claim 13, wherein: The configuration record includes file format level data that encapsulates video coding layer (VCL) level encoded media data in the bitstream of the video data, wherein the VCL level encoded media data includes one or more parameter sets, and wherein each of the tools identified in the configuration record as required for decoding the bitstream is enabled in at least one of the one or more parameter sets.

25. A non-transitory computer-readable storage medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processors to: A configuration record is received for decoding a bitstream of video data, wherein the configuration record for the bitstream applies to a plurality of sequence parameter sets and includes a toolset indication syntax element comprising information identifying a tool from a set of video decoding tools required to decode the bitstream associated with the configuration record; determining, based on the toolset indication syntax element, tools required to decode the bitstream associated with the configuration record; as well as The bitstream associated with the configuration record is decoded using one or more of the tools.

26. The non-transitory computer-readable storage medium of claim 25, wherein: The instructions cause the one or more processors to: A Multipurpose Internet Mail Extensions (MIME) type parameter is received, comprising a key-value pair, wherein the key indicates that the MIME type identifies the video decoding tool, and the value identifies the tool from the video decoding tool required to decode the bitstream.

27. The non-transitory computer-readable storage medium of claim 25, wherein: The bitstream of video data includes one or more parameter sets, and wherein each of the tools identified in the configuration record as required for decoding the bitstream is enabled in at least one of the one or more parameter sets.

28. The non-transitory computer-readable storage medium of claim 25, wherein: The toolset indication syntax element, including information identifying the tools required to decode the bitstream associated with the configuration record, identifies all tools required to decode the bitstream associated with the configuration record.

29. The non-transitory computer-readable storage medium of claim 25, wherein: The configuration record includes file format level data that encapsulates video coding layer (VCL) level encoded media data in the bitstream of the video data, wherein the VCL level encoded media data includes one or more parameter sets, and wherein each of the tools identified in the configuration record as required for decoding the bitstream is enabled in at least one of the one or more parameter sets.

30. An apparatus for processing a video stream or a video file, the apparatus comprising: means for receiving a configuration record for decoding a bitstream of video data, wherein the configuration record for the bitstream applies to a plurality of sequence parameter sets and comprises a toolset indication syntax element comprising information identifying a tool from a set of video decoding tools required to decode the bitstream associated with the configuration record; means for determining, based on the toolset indication syntax element, tools required to decode the bitstream associated with the configuration record; and Means for decoding the bitstream associated with the configuration record using one or more of the tools.

Images