Identification of tile from network abstraction unit header

Abstract

To provide a method and apparatus for video decoding.SOLUTION: A method comprises: decoding a binary coded syntax element carrying an identifier of a picture segment in a high level syntax structure comprising fixed length codewords; including information identifying a tile in the high level syntax structure; and reconstructing the picture segment. A parameter set is active for a picture to which the high level syntax structure belongs.SELECTED DRAWING: None

Description

[Technical Field]

[0001] The disclosed technical subject matter relates to video encoding and decoding, and more particularly to including information that identifies tiles in high-level syntax structures such as fixed-length codepoints and network abstraction layer unit headers.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority under 35 U.S.C. §119 to U.S. Provisional Patent Application No. 62 / 783,152, filed in the U.S. Patent and Trademark Office on December 20, 2018, and U.S. Provisional Patent Application No. 16 / 403,799, filed in the U.S. Patent and Trademark Office on May 6, 2019, the disclosures of which are incorporated herein by reference in their entireties. [Background technology]

[0003] Video encoding and decoding using inter-picture prediction with motion compensation has been known for decades. Uncompressed digital video can consist of a series of images, each having spatial dimensions of, for example, 1920 x 1080 luminance samples and associated chrominance samples. The series of images can have a fixed or variable picture rate (also informally known as the frame rate), for example, 60 pictures per second or 60 Hz. Uncompressed video has significant bitrate requirements. For example, 1080p60 4:2:0 video (1920 x 1080 luminance sample resolution at a 60 Hz frame rate) at 8 bits per sample requires a bandwidth approaching 1.5 Gigabits per second (Gbit / s). One hour of such video requires more than 600 Gbytes of storage.

[0004] One goal of video encoding and decoding can be the reduction of redundancy in an input video signal through compression. Compression helps reduce the aforementioned bandwidth or storage requirements, sometimes by more than two orders of magnitude. Both lossless and lossy compression, as well as combinations thereof, can be utilized. Lossless compression refers to a technique that allows an exact copy of the original signal to be reconstructed from a compressed version of the original signal. When using lossy compression, the reconstructed signal may not be identical to the original signal, but the distortion between the original and reconstructed signals is small enough to make the reconstructed signal useful for the intended application. In the case of video, lossy compression is widely used. The amount of acceptable distortion is application-dependent; for example, a user of a given consumer streaming application may tolerate more distortion than a user of a television contribution application. The achievable compression ratio can reflect that a larger allowable distortion can result in a higher compression ratio.

[0005] Video encoders and decoders can utilize techniques from several broad categories, including, for example, motion compensation, transforms, quantization, and entropy coding, some of which are introduced below.

[0006] The concept of dividing a coded video bitstream into packets for transport over packet networks has been in use for decades. Early on, video coding standards and technologies were mostly optimized for bit-oriented transport, and bitstreams were defined. Packetization occurred at the system layer interface specified, for example, in the Real-Time Transport Protocol (RTP) payload format. With the advent of Internet connectivity suitable for the mass use of video over the Internet, video coding standards reflected their prominent use cases through the conceptual differentiation of the video coding layer (VCL) and the network abstraction layer (NAL). NAL units were introduced in H.264 in 2003 and, with only minor modifications, have been maintained in given video coding standards and technologies ever since.

[0007] NAL units can often be considered as the smallest entity on which a decoder can operate without necessarily decoding all preceding NAL units of a coded video sequence. NAL units enable certain error resilience techniques, as well as certain bitstream manipulation techniques, including bitstream pruning by Media Aware Network elements (MANEs), such as Selective Forwarding Units (SFUs) or Multipoint Control Units (MCUs).

[0008] Figure 1 shows relevant parts of the syntax diagram of the NAL unit header according to H.264 (101) and H.265 (102), in both cases without their respective extensions. In both cases, the "forbidden_zero_bit" is a zero bit used for start code emulation prevention in a given system layer environment. The "nal_unit_type" syntax element indicates the type of data the NAL unit carries, which may be, for example, one of a given slice type, parameter set type, Supplementary Enhancement Information (SEI) message, etc. The H.265 NAL unit header further includes "nuh_layer_id" and "nuh_temporal_id_plus1," which indicate the spatial / SNR and temporal layer of the coded image to which the NAL unit belongs.

[0009] It can be seen that NAL unit headers contain only easily parseable fixed-length codewords that do not have any parsing dependency on other data in the bitstream, e.g., other NAL unit headers, parameter sets, etc. Because NAL unit headers are the first octets in a NAL unit, a MANE can easily extract, parse, and act on them. Other high-level syntax elements, e.g., slice or tile headers, in contrast, are less accessible to a MANE because they may require maintaining parameter set context and / or processing variable-length or arithmetically encoded codepoints. However, even structures such as tile group headers can be designed with properties that make them easily accessible to MANEs, although existing video compression technologies and standards may not do so.

[0010] It can be further observed that the NAL unit header does not contain information that can associate the NAL unit with a segment of a coded image, such as a slice, tile, or similar portion of a bitstream representing a spatial area of ​​the coded image, as shown in FIG. 1. In related art, such information exists in slice headers, in certain cases in the form of a macroblock or CU address. This address is an integer n that indicates the segment, slice, or tile, as the case may be, starting with the nth macroblock / CU in scan order when counting down from the top-left of the image. Thus, n can depend on both the image and macroblock / CU size and can be small (e.g., fits into 8 bits in binary code) or large (e.g., 32400, requiring 16 bits in binary code) for small image sizes. In both cases, it is assumed that the macroblock / CU size is 16x16 samples.

[0011] Historically, picture segments such as tiles or slices were used mostly to facilitate bitstream partitioning and parallelization consistent with maximum transmission unit size constraints. In both cases, identification of tiles or slices in MANEs, SFUs, or similar devices was typically not required. Decoders can obtain relevant information from relatively complex slice headers and / or similar information, along with state obtained from decoding parameter sets.

[0012] More recently, however, picture segments, and in particular tiles (and tile groups, which are collections of tiles in scan order, rectangular order, or other suitable order), have been used for purposes such as a collection of CUs representing a given view in a composed 360 projection, among other applications. In some of these applications, the MANE and SFU can advantageously remove certain tiles or other segments from the coded picture if they are not needed for the application. For example, if a cube projection is used, rendering a scene from an external viewpoint requires a maximum of three of the six cubic faces. Sending CUs and segments representing the remaining minimum of three faces to an endpoint can be a waste of resources. However, in a scenario where a sender sends a complete representation (including all six faces of the cubic projection) to the MANE, and the MANE forwards only a required subset to multiple receivers, and the required subsets differ between receivers, the MANE will accommodate potentially different bitstreams, including potentially different cubic faces, for each receiver. To do so, the MANE currently processes complex variable length coded slice headers and requires maintaining state in the form of parameter sets, etc., as needed to decode the slice headers. Summary of the Invention

[0013] To enable improved identification of tiles based on syntax structures, techniques are disclosed that include tile identifiers in syntax structures that include fixed-length codewords in video codecs.

[0014] A method for video decoding includes decoding binary-encoded syntax elements that carry identifiers of picture segments in a high-level syntax structure that includes fixed-length codewords, and reconstructing the picture segments.

[0015] An apparatus for decoding a video sequence includes at least one memory configured to store program code and at least one processor configured to read the program code and operate as directed by the program code, the program code including: decoding code configured to cause the at least one processor to decode binary-encoded syntax elements carrying identifiers of picture segments in a high-level syntax structure including fixed-length codewords; and reconstructing code configured to cause the at least one processor to reconstruct the picture segments.

[0016] A non-transitory computer-readable medium storing one or more instructions, when executed by one or more processors of the device, causes the one or more processors to decode binary-encoded syntax elements carrying identifiers of picture segments in a high-level syntax structure including fixed-length codewords and to reconstruct the picture segments. [Brief explanation of the drawings]

[0017] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings. [Figure 1] FIG. 1 is a schematic diagram of a NAL unit header according to H.264 and H.265. [Figure 2] FIG. 2 is a simplified block diagram of a communication system according to one embodiment. [Figure 3]FIG. 3 is a simplified block diagram of a communication system according to one embodiment. [Figure 4] FIG. 4 is a simplified block diagram of a decoder according to one embodiment. [Figure 5] FIG. 5 is a simplified block diagram of an encoder according to one embodiment. [Figure 6] FIG. 6 is a schematic diagram of a NAL unit header including a CU address or tile ID syntax element according to one embodiment. [Figure 7] FIG. 7 is a schematic diagram of a tile layout according to one embodiment. [Figure 8] FIG. 8 is a schematic diagram of NAL unit decoding / transport according to one embodiment. [Figure 9] FIG. 9 is a schematic diagram of a computer system according to one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0018] Problems to be solved Video coding syntax lacks an easily identifiable / parseable syntax element that identifies a tile or other image in a high-level syntax structure such as a NAL unit header.

[0019] Figure 2 shows a simplified block diagram of a communication system 200 according to one embodiment. The system 200 may include at least two terminals 210-220 interconnected via a network 250. For unidirectional data transmission, a first terminal 210 may encode video data locally for transmission to the other terminal 220 via the network 250. The second terminal 220 may receive the other terminal's encoded video data from the network 250, decode the encoded data, and display the recovered video data. Unidirectional data transmission may be common in media provisioning applications, etc.

[0020] 2 illustrates a second pair of terminals (230, 240) provided to support bidirectional transmission of encoded video, such as may occur during a video conference. For the bidirectional transmission of data, each terminal (230, 240) may encode video data captured at a local location for transmission to the other terminal over the network (250). Each terminal (230, 240) may also receive encoded video data transmitted by the other terminal, decode the encoded data, and display the recovered video data on a local display device.

[0021] While the terminals (210-240) may be depicted in FIG. 2 as servers, personal computers, and smartphones, the principles of the present disclosure are not so limited. Embodiments of the present disclosure find application with laptop computers, tablet computers, media players, and / or dedicated videoconferencing devices. Network (250) represents any number of networks that convey coded video data between the terminals (210-240), including, for example, wired and / or wireless communication networks. The communication network (250) may exchange data over circuit-switched and / or packet-switched channels. Exemplary networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For purposes of this description, the architecture and topology of network (250) are not important to the operation of the present disclosure, unless otherwise described below.

[0022] 3 shows the arrangement of a video encoder and decoder in a streaming environment as one example of an application for the disclosed subject matter. The disclosed subject matter is equally applicable to other video-enabled applications, including, for example, video conferencing, digital TV, storage of compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0023] The streaming system may include a capture subsystem (313), such as a video source (301), e.g., a digital camera, that generates an uncompressed video sample stream (302). This sample stream (302), shown as a thick line to emphasize its large data volume when compared to an encoded video bitstream, may be processed by an encoder (303) coupled to the camera (301). The encoder (303) may include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter, as described in more detail below. The encoded video bitstream (304), shown as a thin line to emphasize its smaller data volume when compared to the sample stream, may be stored on a streaming server (305) for future use. One or more streaming clients (306, 308) may access the streaming server (305) to retrieve copies (307, 309) of the encoded video bitstream (304). The client (306) may include a video decoder (310) that decodes an input copy of the encoded video bitstream (307) and generates an output video sample stream (311) that can be rendered on a display (312) or other rendering device (not shown). In some streaming systems, the video bitstreams (304, 307, 309) may be encoded according to a predetermined video encoding / compression standard. Examples of these standards include ITU-T Recommendation H.265. A developing video encoding standard is informally known as Versatile Video Coding, or VVC. The disclosed subject matter may be used in the context of VVC.

[0024] FIG. 4 may be a functional block diagram of a video decoder (310) according to one embodiment of the present invention.

[0025] The receiver (410) can receive one or more codec video sequences to be decoded by the decoder (310). In the same or another embodiment, one coded video sequence at a time, with the decoding of each coded video sequence being independent of the other coded video sequences. The coded video sequences can be received from a channel (412), which can be a hardware / software link to a storage device that stores the coded video data. The receiver (410) can receive the coded video data along with other data, such as coded audio data and / or auxiliary data streams, which can be transferred using respective entities (not shown). The receiver (410) can separate the coded video sequences from the other data. To attempt to eliminate network jitter, a buffer memory (415) can be coupled between the receiver (410) and the entropy decoder / parser (420) (hereinafter, the “parser”). If the receiver 410 is receiving data from a store-and-forward device with sufficient bandwidth and controllability, or from an isosynchronous network, the buffer 415 may not be needed or may be small. For use with best-effort packet networks such as the Internet, the buffer 415 is needed, but may be relatively large and advantageously of adaptive size.

[0026] The video decoder (310) may include a parser (420) for reconstructing symbols (421) from the entropy-coded video sequence. These symbol categories include information used to manage the operation of the decoder (310) and, as shown in FIG. 3, potentially information for controlling a rendering device, such as a display (312), which is not an integral part of the decoder but may be connected to the decoder. The control information for the rendering device may be in the form of Supplemental Enhancement Information (SEI) messages or Video Usability Information (VUI) parameter set fragments (not shown). The parser (420) may parse / entropy-decode the received coded video sequence. The coding of the coded video sequence may follow a video coding technique or standard and may follow principles well known to those skilled in the art, including variable-length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser (420) may extract from the coded video sequence a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder based on at least one parameter corresponding to the group. The subgroups may include Groups of Pictures (GOPs), images, tiles, slices, macroblocks, coding units (CUs), blocks, transform units (TUs), prediction units (PUs), etc. The entropy decoder / parser may also extract from the coded video sequence information such as transform coefficients, quantization parameter values, motion vectors, etc.

[0027] The parser (420) can perform entropy decoding / parsing operations on the video sequence received from the buffer (415) to generate symbols (421).

[0028] The reconstruction of the symbols (421) can involve several different units, depending on the type of coded video image or portion thereof (inter- and intra-picture, inter- and intra-block, etc.) and other factors. Which units participate and how can be controlled by subgroup control information parsed from the coded video sequence by the parser (420). The flow of such subgroup control information between the parser (420) and the following units is not shown for clarity.

[0029] In addition to the functional blocks described above, the decoder 310 may be conceptually subdivided into several functional units, as described below. In a practical implementation operating under commercial constraints, many of these units may closely interact with each other and may be at least partially integrated with each other. However, for purposes of describing the disclosed technical subject matter, a conceptual subdivision into the following functional units is appropriate:

[0030] The first unit is a scalar / inverse transform unit (451), which receives quantized transform coefficients as symbols (421) from the parser (420) as well as control information, including the transform used, block size, quantization coefficients, quantization scaling matrices, etc. It can output blocks containing sample values ​​that can be input to an aggregator (455).

[0031] In some cases, the output samples of the scaler / inverse transform (451) may belong to intra-coded blocks, i.e., blocks that do not use prediction information from a previously reconstructed image but can use prediction information from a previously reconstructed portion of the current image. Such prediction information may be provided by an intra-image prediction unit (452). In some cases, the intra-image prediction unit (452) uses surrounding already reconstructed information fetched from the current (partially reconstructed) image (456) to generate blocks of the same size and shape as the block being reconstructed. The aggregator (455) optionally adds, on a sample-by-sample basis, the prediction information generated by the intra-prediction unit (452) to the output sample information provided by the scaler / inverse transform unit (451).

[0032] In other cases, the output samples of the scalar / inverse transform unit (451) may belong to an inter-coded and potentially motion-compensated block. In such cases, the motion-compensated prediction unit (453) may access a reference picture memory (457) to fetch samples used for prediction. After motion-compensating the fetched samples according to the symbols (421) belonging to the block, these samples may be added by an aggregator (455) to the output of the scalar / inverse transform unit (in this case, referred to as residual samples or residual signals) to generate output sample information. The addresses in the reference picture memory from which the motion compensation unit fetches prediction samples may be controlled by motion vectors available to the motion compensation unit, for example, in the form of symbols (421) that may have X, Y, and reference picture components. Motion compensation may also include interpolation of sample values ​​fetched from the reference picture memory when sub-sample accurate motion vectors are used, motion vector prediction mechanisms, etc.

[0033] The output samples of the aggregator (455) may be subjected to various loop filtering techniques in a loop filter unit (456). Video compression techniques may include in-loop filter techniques that are controlled by parameters contained in the coded video bitstream and made available to the loop filter unit (456) as symbols (421) from the parser (420), but may also respond to meta-information obtained during decoding of previous portions of the coded image or coded video sequence (in decoding order), as well as to previously reconstructed and loop-filtered sample values.

[0034] The output of the loop filter unit (456) may be a sample stream that may be output to the rendering device (312) as well as stored in a reference image memory (456) for use in future inter-image prediction.

[0035] Once a given coded image is fully reconstructed, it can be used as a reference image for future prediction. Once a coded image is fully reconstructed and identified as a reference image (e.g., by the parser (420)), the current reference image (456) becomes part of the reference image buffer (457), and fresh current image memory can be reallocated before starting the reconstruction of the next coded image.

[0036] The video decoder 420 may perform decoding operations in accordance with a predetermined video compression technique, which may be documented in a standard, such as ITU-T Rec. H.265. An encoded video sequence may conform to the syntax specified by the video compression technique or standard being used, in the sense of adhering to the syntax of the video compression technique or standard as specified in the video compression technique document or standard, and particularly in the profile documents therein. Compliance also requires that the complexity of the encoded video sequence be within a range defined by the level of the video compression technique or standard. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. The limits set by the level may be further limited, in some cases, through a Hypothetical Reference Decoder (HRD) specification and HRD buffer metadata signaled with the encoded video sequence.

[0037] In one embodiment, the receiver (410) may receive additional (redundant) data along with the coded video. The additional data may be included as part of the coded video sequence. The additional data may be used by the video decoder (420) to properly decode the data and / or more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial, or SNR enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0038] FIG. 5 may be a functional block diagram of a video encoder (303) according to one embodiment of the present disclosure.

[0039] The encoder (303) can receive video samples from a video source (301) (not part of the encoder) that can capture video images to be encoded by the encoder (303).

[0040] The video source (301) can provide a source video sequence to be encoded by the encoder (303) in the form of a digital video sample stream, which can be in any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 Y CrCB, RGB, etc.), and any suitable sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media serving system, the video source (301) can be a storage device storing previously prepared video. In a video conferencing system, the video source (303) can be a camera capturing local image information as a video sequence. The video data can be provided as multiple individual images that convey motion when viewed in sequence. The images themselves can be organized as a spatial array of pixels, where each pixel can contain one or more samples, depending on the sampling structure, color space, etc., in use. Those skilled in the art can easily understand the relationship between pixels and samples. The following discussion focuses on samples.

[0041] According to one embodiment, the encoder (303) can encode and compress images of a source video sequence into a coded video sequence (543) in real time or under any other time constraint required by the application. Enforcing the appropriate coding rate is one function of the controller (550). The controller controls other functional units, as described below, and is operatively coupled to these units. Coupling is not shown for clarity. Parameters set by the controller may include rate control-related parameters (picture skip, quantizer, lambda value for rate-distortion optimization techniques, etc.), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. Those skilled in the art can readily identify other functions of the controller (550) as they may be relevant to a video encoder (303) optimized for a given system design.

[0042] Some video encoders operate in what those skilled in the art will readily recognize as a "coding loop." As an overly simplistic explanation, the coding loop consists of the encoding portion of the encoder (530) (hereinafter the "source coder") (responsible for generating symbols based on the input picture to be encoded and a reference picture), and a (local) decoder (533) embedded within the encoder (303) that reconstructs the symbols to generate sample data, which the (remote) decoder will also generate (so that in the video compression techniques contemplated in the disclosed subject matter, any compression between the symbols and the encoded video bitstream is lossless). The reconstructed sample stream is input to a reference picture memory (534). Because decoding of the symbol stream yields bit-exact results independent of the decoder location (local or remote), the contents of the reference picture buffer are also bit-perfect between the local and remote encoders. In other words, the prediction portion of the encoder "sees" the exact same sample values ​​as the reference picture samples that the decoder will "see" when using the prediction during decoding. This basic principle of reference picture synchrony (and the resulting drift when synchrony cannot be maintained, e.g., due to channel errors) is well known to those skilled in the art.

[0043] The operation of the "local" decoder (533) may be the same as the "remote" decoder (310), as already described in detail above in connection with Figure 4. However, with brief reference again to Figure 4, because symbols are available and the encoding / decoding of symbols for the coded video sequence by the entropy coder (545) and parser (420) may be lossless, the entropy decoding portion of the decoder (310), including the channel (412), receiver (410), buffer (415), and parser (420), may not be implemented entirely within the local decoder (533).

[0044] An observation that can be made at this point is that any decoder technique, with the exception of analysis / entropy decoding, that is present in the decoder must also be present in a corresponding encoder, in substantially identical functional form. For this reason, the technical matters disclosed will focus on the operation of the decoder. A description of the encoder techniques can be omitted, as they are the reverse of the decoder techniques that have been comprehensively described. Only in certain areas is a more detailed explanation necessary and is provided below.

[0045] As part of its operation, the source coder (530) can perform motion compensated predictive coding, which predictively codes an input frame with respect to one or more previously coded frames from the video sequence designated as “reference frames.” In this way, the coding engine (532) codes differences between pixel blocks of the input frame and pixel blocks of reference frames that can be selected as prediction references for the input frame.

[0046] The local video decoder (533) can decode the encoded video data of frames that may be designated as reference frames based on symbols generated by the source coder (530). The operation of the encoding engine (532) can advantageously be a lossy process. If the encoded video data can be decoded by a video decoder (not shown in FIG. 5), the reconstructed video sequence will typically be a replica of the source video sequence, with some errors. The local video decoder (533) replicates the decoding process performed on the reference frames by the video decoder and can cause the reconstructed reference frames to be stored in a reference image cache (534). In this way, the encoder (303) can locally store (without transmission errors) copies of reconstructed reference frames that have common content as reconstructed reference frames obtained by a far-end video decoder.

[0047] The predictor (535) can perform prediction searches for the coding engine (532). That is, for a new frame to be encoded, the predictor (535) can search the reference picture memory (534) for sample data (as candidate reference pixel blocks) or predefined metadata, such as reference picture video vectors, block shapes, etc., that can serve as appropriate prediction references for the new picture. The predictor (535) can operate on a sample block-pixel block basis to find appropriate prediction references. In some cases, as determined by the search results obtained by the predictor (535), the input picture can have prediction references drawn from multiple reference pictures stored in the reference picture memory (534).

[0048] The controller (550) can manage the encoding operations of the video coder (530), including, for example, setting the parameters and subgroup parameters used to encode the video data.

[0049] The output of all the aforementioned functional units may undergo entropy coding in an entropy coder (545), which converts the symbols produced by the various functional units into a coded video sequence by losslessly compressing the symbols according to techniques known to those skilled in the art, such as Huffman coding, variable length coding, arithmetic coding, etc.

[0050] The transmitter (540) can buffer the coded video sequence as it is generated by the entropy encoder (545) and prepare it for transmission over a communication channel (560), which may be a hardware / software link to a storage device that stores the coded video data. The transmitter (540) can merge the coded video data from the video coder (530) with other data to be transmitted, such as coded audio data and / or auxiliary data streams (sources not shown).

[0051] The controller (550) can manage the operation of the encoder (303). During encoding, the controller (550) can assign a predetermined encoding image type to each encoded image, which can affect the encoding technique that can be applied to the respective image. For example, a picture can often be assigned as one of the following frame types:

[0052] An intra picture (I-picture) may be one that can be coded and decoded without using any other frame in the sequence as a source of prediction. Some video codecs allow different types of intra pictures, including, for example, Independent Decoder Refresh Pictures. Those skilled in the art are aware of these variations of I-pictures and their respective applications and functions.

[0053] A predictive picture (P picture) may be one that can be coded and decoded using intra- or inter-prediction, using at most one motion vector and reference index to predict the sample values ​​of each block.

[0054] A Bi-directionally Predictive Picture (B-picture) may be one that can be coded and decoded using intra- or inter-prediction, using up to two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, a multiple predicted picture may use two or more reference pictures and associated metadata for the reconstruction of a single block.

[0055] A source image is typically spatially subdivided into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples each) and coded block by block. Blocks may be predictively coded with reference to other (already coded) blocks, as determined by the coding assignment applied to the image of each block. For example, blocks of an I-picture may be non-predictively coded, or they may be predictively coded with reference to already coded blocks of the same image (spatial prediction or intra-prediction). Pixel blocks of a P-picture may be coded non-predictively, via spatial prediction, or via temporal prediction, with reference to one previously coded reference image. Blocks of a B-picture may be coded non-predictively, via spatial prediction, or via temporal prediction, with reference to one or two previously coded reference images.

[0056] The video coder (303) may perform encoding operations in accordance with a pre-defined video encoding technique or standard, such as ITU-T Rec. H.265. In its operation, the video coder (303) may perform various compression operations, including predictive encoding operations that exploit temporal and spatial redundancy in the input video sequence. The encoded video data may therefore conform to a syntax specified by the video encoding technique or standard being used.

[0057] In one embodiment, the transmitter (540) can transmit additional data along with the coded video. The video coder (530) may include such data as part of the coded video sequence. The additional data may include temporal, spatial, or SNR enhancement layers, other types of redundant data such as redundant pictures and slices, Supplemental Enhancement Information (SEI) messages, Video Usability Information (VUI) parameter set fragments, etc.

[0058] According to one embodiment, information identifying picture segments such as tiles, groups of tiles, slices, groups of blocks (GOBs), etc. (hereafter referred to as tiles) is placed in an easily accessible high-level syntax structure such as a NAL unit header (NUH) or a similar structure that contains fixed-length codewords and is designed for easy processing by a MANE (hereafter referred to as NUH).

[0059] The information that identifies the tiles can take different forms. In designing this information, several design considerations should be kept in mind. Some of these design considerations are listed below:

[0060] The number of possible tiles in a given picture can be small, for example, compared to the number of possible slices in conventional video coding techniques or standards. For example, in H.264, it is possible to have a slice that covers a single macroblock (for a given picture size), and there are as many slices possible as there are macroblocks. In contrast, when representing a tiled cube map, six tiles may be sufficient, regardless of the image resolution. In many practical cases, a maximum number of tiles of 64, 128, or 256 can be safely assumed.

[0061] The tile layout can be fixed, and while the video coding technology itself may allow flexibility in the tile layout between pictures, a system standard or technology may limit that flexibility to the point where the tile layout remains the same throughout the session. This allows the tile layout to be made available to the MANE through non-video bitstream-specific means, such as during session setup. This may prevent undesirable context dependencies between parameter sets in video coding and MANE operation.

[0062] Under at least the above assumptions, the mechanism for identifying tiles carried by NAL units so that they can be removed by the MANE can be significantly simplified compared to related technologies such as H.264 and H.265. For example, in H.264 and H.265, the MANE would need to identify the correct sequence parameter set to know the length of the slice / tile address codeword in the slice header. Such length information is encoded as a variable-length codeword in the sequence parameter set. Thus, the MANE would, at a minimum, need to follow the parameter set activation sequence to identify the currently active sequence parameter set, and then decode the variable-length codeword to identify the length of the binary-encoded slice / tile address carried in the slice header (possibly not in this order, since parameter sets are parsing-independent). The MANE would then need to decode the variable-length codeword in the slice header to obtain the starting macroblock / CU address. This information can be matched against the tile layout decoded from the parameter set to identify the tile.

[0063] In the same or another embodiment, the tile's identification information could be the address of the first macroblock / CU of the tile. In effect, such a mechanism moves the starting address from the slice header to the NUH. Doing so may be a minimal-change approach to codec design, but has the drawback of significantly increasing the NUH. However, the increase in the size of the NUH may even be acceptable from a coding efficiency standpoint, since the same amount of bits would have been removed from the slice / tile header.

[0064] As noted above, macroblock / CU addresses can be moderately small for small picture sizes and large macroblock / CU sizes, or quite large for small CU sizes and large picture sizes. For this reason, the H.265 SPS includes an indication of the length of macroblock / CU addresses carried in slice headers. In the same or another embodiment, this mechanism could be retained for NAL unit headers. However, doing so could have two drawbacks. First, context dependency established by determining the size of syntax elements in NAL unit headers through parameter set values ​​could require the MANE to track the activation of parameter settings, which could be redundant. Second, NAL unit headers, at least until now, have been octet-aligned to simplify processing in the MANE. Maintaining that octet alignment could require padding—wasting bits. In such cases, the size of the macroblock / CU address indicated by the parameter set, along with the remaining NAL unit header syntax elements, is not added up to a number of bits divisible by 8.

[0065] In the same or another embodiment, the macroblock / CU address—or the size of any other syntax element in the NAL unit header—can be determined by other fields in the NAL unit header. This mechanism avoids content dependency between the parameter set and the NAL unit header and can be desirable in many cases. A drawback can be the use of bits or codepoints in other fields of the NAL unit header. More details are provided below.

[0066] However, if we do not consider slices in the traditional sense, only tiles or tiles, or similar allocation mechanisms of CUs to bitstream entities, more advanced options are available. To explain these options, the terms slice and tile will be briefly reviewed. A slice is a collection of CUs or macroblocks, usually in scan order, and is identified by two factors: a starting macroblock / CU address, usually coded in a slice header, and the end of the slice, which is often identified by the start of a new slice (and similarly indicated by the presence of the next slice header). While certain video compression technologies and standards impose certain relatively minor restrictions on the number and layout of slices, in most cases the layout of slices can change from coded picture to coded picture and is often determined by mechanisms such as rate control and MTU size matching.

[0067] A tile, on the other hand, can typically refer to a rectangular arrangement of CUs, and the size and shape of the rectangle (that rectangular tile and other rectangular tiles that combine to make up the picture) are coded in a parameter set. In other words, the tile layout is somewhat static, in that changing from one tile layout to another requires the activation of a different parameter set. Furthermore, the number of tiles can be advantageously limited to enable efficient hardware implementation. As a result, in many video compression techniques and standards, relatively short fixed-length binary codewords, e.g., 8 bits, allow for handling the maximum number of tiles for all practically used image sizes. Thus, a fixed-length codeword for tile ID is used to identify tiles in the NAL unit header, thereby avoiding parsing and context-dependence between tile identification NAL unit header codewords and parameter sets. Similarly, a fixed-length codeword for tile group ID can be used to identify tile groups. Of course, mechanisms supporting variable-length codewords for macroblock / CU addresses in the NAL unit header can be applied to tile ID codewords as well, if so desired, at the expense of similar architectural disadvantages.

[0068] Referring to Figure 6, several implementation options for NAL unit header design are presented.

[0069] A NAL unit (601) may be part of a coded video bitstream. In some cases, the NAL unit is octet-aligned and is smaller than or equal to the common Maximum Transfer Unit (MTU) size of the data network. One such common MTU size is approximately 1500 octets, which stems from the limitations imposed by early Ethernet technology. The NAL unit may initially include a NAL unit header (602). Framing of the NAL unit within the coded video bitstream may be via start codes, alignment with the packet structure of the underlying packet-oriented transport network, etc.

[0070] Referring again to FIG. 6, an exemplary NAL unit header syntax diagram (603) is also shown, similar to that used in H.265. The disclosed technical subject matter may also be used with NAL unit headers of similar structure, such as the NAL unit header of H.264, or any other high-level syntax structure including VVC or fixed-length codewords. The NAL unit header (603) may include a syntax element, CU address or tile ID (604). The length of that syntax element may be fixed and may be selected so that the NAL unit header remains octet-aligned. The syntax element (604) may be in a format that is easily processable by video encoders and decoders as well as MANEs. By way of example, and not limitation, the CU address or tile ID (604) may be represented by a 6-bit unsigned integer, as represented by Descriptor u(6). In the example presented, the CU address or tile ID occupies the same bits as used for layer_id in H.265. A different presentation of the same technical matter is shown using the NAL unit header (605) and the CU address or tile ID (606).

[0071] Also shown is a NAL unit (607) that preserves fields of the H.265 NAL unit header. A syntax element (608) is added, for example, to the end of the NAL unit header. The location of the syntax element is merely exemplary; it may also be inserted somewhere in the middle of other syntax elements in the NAL unit header. The syntax element may be fixed or variable-sized. If variable-sized, its size may be determined by one of the mechanisms described above (e.g., through a parameter set syntax element, through a NAL unit type, etc.), or by any other suitable mechanism.

[0072] The syntax element (608) can carry any form of picture segment identification information, such as a macroblock / CU address or tile identifier, such as a tile number, or a number indicating a group of tiles. The numbering range of the syntax element can be predetermined. In the case of a macroblock / CU address, the numbering range can be from 0 to the maximum number of macroblocks / CUs in the picture. For a tile ID, the numbering range depends on the maximum number of tiles and can be defined by mechanisms known to those skilled in the art, such as the profile, level, tiers, the maximum or actual image size coded in the parameter set, etc. If the syntax element is present for non-tile / slice NAL units (such as parameter set NAL units, SEI NAL units, etc.), the value of the syntax element can be limited to a predetermined number, such as 0. Alternatively, the presence of the syntax element can be gated by NAL unit type, whereby the syntax element need not be present for a given NAL unit type. Alternatively, overriding semantics other than those described above may be assigned to syntax elements for certain non-tile / slice NAL unit types.

[0073] In the same or another embodiment, tile IDs may be identified, for example, as follows: Figure 7 shows a picture (701) in the spatial domain subdivided by example tile layouts (702) shown through bold lines. The presented tile layout may be expressible, for example, through the syntax available in H.265, or through a more advanced syntax that would be required to express the tile layout (702). Each tile in the tile layout may have an assigned tile ID through any suitable numbering mechanism, but preferably through scan order numbering of the tiles. In Figure 7, scan order tile numbering is shown through tile IDs 1 through 8, e.g., the second tile in scan order has assigned tile ID 2 (703).

[0074] An encoder can write NAL unit headers containing syntax elements covering tile IDs or macroblock / CU addresses located as described above in a similar manner to writing existing NAL unit header syntax known to those skilled in the art.

[0075] A decoder or MANE can parse the NAL unit header—or more precisely, the syntax elements that make up the NAL unit header—from the coded video bitstream, regardless of the presence or absence of syntax elements carrying macroblock / CU addresses, or tile IDs, or other forms of tile identification information, in a manner known to those skilled in the art. However, it should be noted that the syntax elements may in some cases be coded without requiring state information, as described above, and may be in an accessible entropy-coded format, e.g., fixed-length, binary code. Thus far, parsing the NAL unit header in accordance with the disclosed technical subject matter may not involve any additional burden on the decoder or MANE beyond the actual presence of the syntax element tile_id itself.

[0076] According to the disclosed technical subject matter, however, a decoder or MANE can identify tiles in an encoded image with little effort compared to the operations required without the disclosed technical subject matter. As an example, assume that the decoder or MANE has been informed by external, non-video encoding means that a certain tile does not need to be reconstructed for a given application. For example, consider a scene such as that shown in FIG. 7, i.e., a village road. Assume that the road is being captured by a surveillance camera. Consider the tile with tile ID 2 (703). Assume that this tile mostly covers a wall, and the surveillance system configurator considers that area irrelevant for surveillance. Thus, while the camera can encode all tiles, the tile with ID 2 may not be needed for the application. Thus, if a bitstream generated by a camera is routed through one or more MANEs to its final destination, and one MANE observes a bandwidth shortage and must remove something from the bitstream, it can advantageously remove that tile because it is not needed for the application. In the absence of the disclosed technical matter, this would require, at a minimum, that the payload of the NAL unit (slice or tile) be parsed to the extent necessary to extract the macroblock / CU address of the first macroblock in the tile, and then map that macroblock / CU address to the tile layout (if tiles are used). Depending on the video coding technology or standard being used, and as mentioned above, both processing of variable length codewords and maintaining parameter set context in the MANE may be required, although both are undesirable from an implementation and computational complexity standpoint. Instead, in the same or another embodiment, the MANE may obtain all the information necessary to identify which tile is carried by the NAL unit through NAL unit header processing of binary encoded codewords.

[0077] Referring to FIG. 8, a decoder or MANE can use the disclosed technical subject matter, for example, as follows.

[0078] A decoder may parse a NAL unit header from a video bitstream, including syntax elements covering macroblock / CU addresses or tile IDs (801). Using that information, the decoder or MANE can identify the tile IDs (802). The tile IDs may be directly coded, or a priori information about the tile layout, established, for example, by decoding a parameter set and subsequent activation sequence, may be matched with the coded macroblock / CU addresses in the NAL unit header. The decoder can match the tile IDs against a list of tiles that require reconstruction or forwarding by the decoder or MANE, respectively (803). If a match exists (803), the decoder can reconstruct, or the MANE can forward, the NAL unit carrying the tile. However, if a match does not exist (804), the decoder or MANE can discard the NAL unit, perhaps silently.

[0079] The techniques for image referencing in network abstraction unit headers described above may be implemented as computer software using computer-readable instructions and physically stored on one or more computer-readable media. For example, Figure 9 illustrates a computer system 900 suitable for implementing certain embodiments of the disclosed subject matter.

[0080] Computer software may be encoded using any suitable machine code or computer language that may be subject to assembly, compilation, linking, or similar mechanisms to create code including instructions that may be executed by a computer central processing unit (CPU), graphics processing unit (GPU), etc. directly, or through interpretation, microcode execution, etc.

[0081] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet of Things, etc.

[0082] 9 for computer system 900 are exemplary in nature and are not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing embodiments of the present disclosure, nor should the arrangement of components be interpreted as having any dependency or requirement related to any one or combination of components illustrated in the exemplary embodiment of computer system 900.

[0083] The computer system 900 may include certain human interface input devices that may respond to input by one or more human users, for example, through tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, claps), visual input (e.g., gestures), and olfactory input (not shown). The human interface devices may also be used to capture certain media that do not necessarily involve direct conscious human input, such as audio (e.g., voice, music, ambient sounds), images (e.g., scanned images, photographic images acquired from a still image camera), and video (e.g., two-dimensional video, three-dimensional video, including stereoscopic images).

[0084] The input human interface devices may include one or more of a keyboard 901, a mouse 902, a trackpad 903, a touchscreen 910, a data glove 904, a joystick 905, a microphone 906, a scanner 907, and a camera 908 (only one of each is depicted).

[0085] The computer system 900 may also include certain human interface output devices. Such human interface output devices may stimulate one or more of the human user's senses, for example, through tactile output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touchscreen 910, data gloves 904, or joystick 905; however, haptic feedback devices that do not function as input devices may also be present), audio output devices (e.g., speakers 909, headphones (not shown)), visual output devices (screens 910, including CRT screens, LCD screens, plasma screens, and OLED screens, each with or without touchscreen input capability and each with or without haptic feedback capability—some of which may output two-dimensional visual output or three-dimensional or higher-dimensional output via means such as stereoscopic output, such as virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).

[0086] Computer system 900 may also include human-accessible storage devices and their associated media, such as optical media including CD / DVD ROM / RW 920 using media such as CD / DVD 921, thumb drives 922, removable hard drives or solid state drives 923, legacy magnetic media such as tape and floppy disks, specialized ROM / ASIC / PLD-based devices such as security dongles (not shown), etc.

[0087] Those skilled in the art should also understand that the term "computer readable media" as used in connection with the subject matter disclosed herein does not encompass transmission media, carrier waves, or other transitory signals.

[0088] The computer system 900 may also include interfaces to one or more communications networks. Networks may be, for example, wireless, wired, or optical. Networks may further be local, wide-area, metropolitan, vehicular, and industrial, real-time, delay-tolerant, and the like. Examples of networks include cellular networks, including Ethernet, GSM, 3G, 4G, 5G, LTE, and the like; wired or wireless wide-area digital networks, including cable, satellite, and terrestrial broadcast television; vehicular and industrial networks, including CANBus; and the like. Certain networks generally require an external network interface adapter that attaches to a particular general-purpose data port or peripheral bus (949), such as a USB port on the computer system 900; others are generally integrated into the core of the computer system 900 by attaching to a system bus, described below (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system 900 can communicate with other entities. Such communications may be uni-directional, receive-only (e.g., broadcast television), uni-directional transmit-only (e.g., a CAN bus to a given CAN bus device), or bi-directional, e.g., to other computer systems using local or wide area digital networks. Certain protocols and protocol stacks may be used in each of these networks and network interfaces, as described above.

[0089] The human interface devices, human-accessible storage devices, and network interfaces described above may be attached to the core 940 of the computer system 900 .

[0090] The core 940 may include one or more central processing units (CPUs) 941, graphics processing units (GPUs) 942, specialized programmable processing units in the form of field programmable gate arrays (FPGAs) 943, hardware accelerators 1044 for certain tasks, etc. These devices may be connected through a system bus 948, along with read-only memory (ROM) 945, random access memory 946, and internal mass storage devices 947 such as internal non-user-accessible hard drives, SSDs, etc. In some computer systems, the system bus 948 may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may be attached directly to the core's system bus 948 or through a peripheral bus 949. Peripheral bus architectures include PCI, USB, etc.

[0091] The CPU 941, GPU 942, FPGA 943, and accelerator 944 may execute predetermined instructions that, in combination, may constitute the above-described computer code. That computer code may be stored in ROM 945 or RAM 946. Transient data may also be stored in RAM 946, while permanent data may be stored, for example, in internal mass storage device 947. Rapid storage and retrieval from any of the memory devices may be enabled through the use of cache memory, which may be closely associated with one or more of the CPU 941, GPU 942, mass storage device 947, ROM 945, RAM 946, etc.

[0092] The computer-readable medium may have computer code thereon for performing various computer-implemented operations. The media and computer code may be specially designed and constructed for the purposes of the present disclosure, or they may be of the kind well known and available to those skilled in the computer software arts.

[0093] As an example, and not by way of limitation, a computer system having architecture 900, and specifically core 940, can provide functionality as a result of a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media can be user-accessible mass storage devices, as described above, as well as storage devices associated with core 940 that are non-transitory in nature, such as core internal mass storage device 947 or ROM 945. Software implementing various embodiments of the present disclosure can be stored in such devices and executed by core 940. Computer-readable media can include one or more memory devices or chips, depending on particular needs. Software can cause core 940, and specifically the processors therein (including a CPU, GPU, FPGA, etc.), to perform particular processes or portions of particular processes described herein, including defining data structures stored in RAM 946 and modifying those data structures according to software-defined processes. Additionally or alternatively, a computer system may provide functionality as a result of logic hardwired or otherwise embodied in circuitry (e.g., accelerator 944). Circuitry may operate in place of, or in conjunction with, software to perform particular processes or portions of particular processes described herein. References to software include logic, and vice versa, where appropriate. References to computer-readable media may include circuitry (such as an integrated circuit (IC)) that stores software for execution, circuitry that embodies logic for execution, or both, where appropriate. The present disclosure encompasses any appropriate combination of hardware and software.

[0094] While this disclosure has described several exemplary embodiments, alterations, substitutions, and various substitute equivalents exist, which are within the scope of this disclosure. Thus, it will be appreciated that those skilled in the art will be able to devise many systems and methods that, although not explicitly shown or described herein, embody the principles of the present disclosure and, therefore, are within the spirit and scope of the present disclosure. [Explanation of symbols]

[0095] 200 Communication Systems 210 Terminal 1 220 Terminal 2 250 Network 701 Pictures 702 tile layout 900 Computer Systems 940 cores

Claims

1. A method for video encoding, A step of encoding a picture segment in a video sequence, In a high-level syntax structure including a fixed-length codeword, the steps include encoding the identifier of the picture segment as a binary-encoded syntax element, Includes, The size of the syntax element is determined through at least one syntax element in the parameter set. method.

2. The aforementioned picture segment is one of a tile, a group of tiles, or a subpicture. The method according to claim 1.

3. The syntax element is one of the following in the picture segment: a macroblock address, a coding unit address of the first macroblock or coding unit, or a tile address of the first coding unit. The method according to claim 2.

4. The aforementioned syntax element is a tile identifier. The method according to claim 2.

5. Given a tile layout, each tile in the tile layout has a unique tile identifier, and the unique tile identifier is assigned according to the scan order of the picture segments in the tile layout. The method according to claim 4.

6. The scan order is associated with the tile in the upper left corner of the tile layout and is incremented one by one in order from top to bottom and right to left. The method according to claim 5.

7. The syntax element is in a fixed position in the high-level syntax structure. The method according to claim 1.

8. The parameter set is active for the picture to which the high-level syntax structure belongs. The method according to claim 1.

9. The identifier of the picture segment encoded as a binary-encoded syntax element indicates the identifier of the tile to be decoded. The method according to claim 1, including the method described in claim 1.

10. A device for video encoding, At least one memory configured to store program code, The system includes at least one processor configured to read the program code and operate as instructed by the program code, The aforementioned program code is: The encoding code is configured to cause at least one processor to encode picture segments in a video sequence, The encoding code is configured such that at least one processor encodes the identifier of the picture segment as a binary-encoded syntax element in a high-level syntax structure including a fixed-length codeword. Includes, The size of the syntax element is determined through at least one syntax element in the parameter set. Device.

11. The aforementioned picture segment is one of a tile, a group of tiles, or a subpicture. The apparatus according to claim 10.

12. The syntax element is one of the following in the picture segment: a macroblock address, a coding unit address of the first macroblock or coding unit, or a tile address of the first coding unit. The apparatus according to claim 11.

13. The aforementioned syntax element is a tile identifier. The apparatus according to claim 11.

14. Given a tile layout, each tile in the tile layout has a unique tile identifier, and the unique tile identifier is assigned according to the scan order of the picture segments in the tile layout. The apparatus according to claim 13.

15. The scan order is associated with the tile in the upper left corner of the tile layout and is incremented one by one in order from top to bottom and right to left. The apparatus according to claim 14.

16. The syntax element is in a fixed position in the high-level syntax structure. The apparatus according to claim 10.

17. The parameter set is active for the picture to which the high-level syntax structure belongs. The apparatus according to claim 10.

18. A method of video encoding performed by an encoder, the method being: The steps include: generating an encoded bitstream; The step includes storing the encoded bitstream, To generate the aforementioned encoded bitstream: A step of encoding a picture segment in a video sequence, In a high-level syntax structure including a fixed-length codeword, the steps include encoding the identifier of the picture segment as a binary-encoded syntax element, Includes, The size of the syntax element is determined through at least one syntax element in the parameter set. method.

Images