Identification of Tiles from a Network Abstraction Unit Header

Abstract

To provide a method and apparatus for video decoding allowing easy identification of a tile in a high-level syntax structure.SOLUTION: A method for video decoding comprises: decoding a binary coded syntax element carrying a layer identifier in a NAL unit header comprising fixed length codewords; determining whether the layer identifier is included in a list of tiles to be decoded; and reconstructing the tile based on determining whether the layer identifier is included in the list of tiles to be decoded.SELECTED DRAWING: None

Description

Technical Field

[0001] The disclosed technical matter relates to video encoding and decoding. And more specifically, it relates to including information for identifying tiles in high-level syntax structures such as fixed-length code points and network abstraction layer unit headers.

[0002] Cross-reference to Related Applications This application claims the benefit of U.S. Provisional Patent Application No. 62 / 783,152, filed on December 20, 2018, and U.S. Provisional Patent Application No. 16 / 403,799, filed on May 6, 2019, in the United States Patent and Trademark Office, and the disclosures of these are hereby incorporated by reference in their entirety.

Background Art

[0003] Video encoding and decoding using inter-picture prediction with motion compensation has been known for decades. Uncompressed digital video can be composed of a series of images, and each image has, for example, a spatial dimension of 1920×1080 luminance samples and related chrominance samples. A series of images can have, for example, a fixed or variable picture rate (informally also known as the frame rate), such as 60 pictures per second or 60 Hz. Uncompressed video has significant bitrate requirements. For example, 8-bit / sample 1080p60 4:2:0 video (1920×1080 luminance sample resolution at a frame rate of 60 Hz) requires a bandwidth close to 1.5 gigabits per second (Gbit / s). One hour of such video requires a storage area of more than 600 gigabytes.

[0004] One purpose of video encoding and decoding can be the reduction of redundancy in the input video signal through compression. Compression helps to reduce the aforementioned bandwidth or storage requirements, in some cases by more than two orders of magnitude. Both lossless and lossy compression, as well as combinations thereof, can be utilized. Lossless compression refers to a technique in which an exact copy of the original signal can be reconstructed from the compressed original signal. When using lossy compression, the reconstructed signal may not be identical to the original signal, but the distortion between the original signal and the reconstructed signal is small enough to make the reconstructed signal useful for the intended application. In the case of video, lossy compression is widely used. The amount of allowable distortion depends on the application. For example, a user of a given consumer streaming application may tolerate more distortion than a user of a television contribution application. The achievable compression ratio can reflect the fact that a larger allowable / tolerable distortion can result in a higher compression ratio.

[0005] Video encoders and decoders can utilize techniques from several broad categories, including, for example, motion compensation, transformation, quantization, and entropy encoding. Some of these are introduced below.

[0006] For decades, the concept of splitting an encoded video bitstream into packets for transfer over a packet network has been used. Initially, most video coding standards and techniques were optimized for bit-oriented transport, and a bitstream was defined. Packetization occurred, for example, at the system layer interface specified by the Real-Time Transport Protocol (RTP) payload format. The emergence of Internet connectivity suitable for the mass use of video on the Internet has led video coding standards to reflect their prominent use cases through the conceptual differentiation of the video coding layer (VCL) and the network abstraction layer (NAL). The NAL unit was introduced in H.264 in 2003 and has since been maintained in certain video coding standards and techniques with only minor modifications.

[0007] The NAL unit can often be regarded as the smallest entity for which a decoder can operate without necessarily decoding all preceding NAL units of an encoded video sequence. The NAL unit enables certain error recovery techniques and certain bitstream manipulation techniques, including bitstream pruning, by media aware network elements (MANEs) such as selective forwarding units (SFUs) or multipoint control units (MCUs).

[0008] Figure 1 shows the relevant parts of the syntax diagrams of NAL unit headers according to H.264 (101) and H.265 (102), both without their respective extensions. In both cases, the "forbidden_zero_bit" is a zero bit used for start code emulation prevention in a given system layer environment. The syntax element of "nal_unit_type" refers to the type of data carried by the NAL unit, which can be, for example, one of a given slice type, parameter set type, Supplementary Enhancement Information (SEI) message, etc. The H.265 NAL unit header further includes "nuh_layer_id" and "nuh_temporal_id_plus1", which indicate the spatial / SNR and temporal layer of the encoded picture to which the NAL unit belongs.

[0009] It can be seen that the NAL unit header only contains easily parseable fixed-length codewords that have no parsing dependency on other data in the bitstream, such as other NAL unit headers, parameter sets, etc. Since the NAL unit header is the first octet in the NAL unit, the MANE can easily extract, parse, and act based on them. In contrast, other high-level syntax elements, such as slice or tile headers, are difficult for the MANE to access because they may need to maintain a parameter set context and / or process variable-length or arithmetically encoded code points. However, even a structure such as a tile group header can be designed to have characteristics that allow easy access by the MANE. However, existing video compression technologies and standards may not do so.

[0010] As shown in FIG. 1, it can be observed that the NAL unit header does not include information that can associate the NAL unit with a segment of the encoded image, such as a slice, tile, or similar part of the bitstream representing the spatial area of the encoded image. In related art, such information exists in the slice header, in the form of a macroblock or CU address in certain cases. This address is, in some cases, an integer n indicating a segment, slice, or tile starting with the nth macroblock / CU in scan order when counting down from the upper left of the image. Therefore, n can depend on both the image and the macroblock / CU size, and can be small (e.g., fitting into 8 bits in binary code) for a small image size, or large (e.g., 32400, requiring 16 bits in binary code). In both cases, it is assumed that the macroblock / CU size is 16×16 samples.

[0011] Historically, picture segments such as tiles or slices have mostly been used to facilitate the splitting and parallelization of the bitstream that conforms to the maximum transfer unit size constraint. In both cases, the identification of tiles or slices in a MANE, SFU, or similar device has not usually been required. The decoder can obtain relevant information from relatively complex slice headers and / or similar information, along with the state obtained from the decoding of the parameter set.

[0012] However, more recently, picture segments, and in particular tiles (and tile groups, which are collections of tiles in a scan order, rectangular order, or other suitable order) have been used for purposes such as representing a collection of CUs that represent a given view in a configured 360 projection in several applications. In some of these applications, the MANE and SFU can advantageously remove a given tile or other segment from the encoded picture when not needed for the application. For example, when a cube projection is used, rendering a scene from an external viewpoint requires at most 3 out of 6 cube faces. Sending CUs and segments representing the remaining at least 3 faces to endpoints can be a waste of resources. However, in a scenario where the sender sends a complete representation (including all 6 surfaces of the cube projection) to the MANE, and the MANE transfers only the required subset to multiple receivers, and the required subset is different among receivers, the MANE will adjust potentially different bitstreams that include potentially different cube surfaces for each receiver. To do so, currently, the MANE is required to process complex variable length coded slice headers and maintain state in the form of a parameter set, etc., as required to decode the slice headers.

Summary of the Invention

[0013] Techniques are disclosed for including a tile identifier in a syntax structure that includes fixed length codewords in a video codec to enable improved identification of tiles based on the syntax structure.

[0014] A method for video decoding includes decoding a binary-encoded syntax element that carries an identifier of a picture segment in a high-level syntax structure including fixed-length codewords, and reconstructing the picture segment.

[0015] An apparatus for decoding a video sequence includes at least one memory configured to store program code, and at least one processor configured to read the program code and operate as instructed by the program code. The program code includes steps of decoding, by the at least one processor, code configured to decode a binary-encoded syntax element that carries an identifier of a picture segment in a high-level syntax structure including fixed-length codewords, and reconstructing, by the at least one processor, code configured to reconstruct the picture segment.

[0016] A non-transitory computer-readable medium storing one or more instructions, when executed by one or more processors of a device, causes the one or more processors to decode a binary-encoded syntax element that carries an identifier of a picture segment in a high-level syntax structure including fixed-length codewords and reconstruct the picture segment.

Brief Description of the Drawings

[0017] Further features, properties, and various advantages of the disclosed technical subject matter will become more apparent from the following detailed description and the accompanying drawings.

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

[0018] Problems to be Solved Video coding syntax lacks easily distinguishable / parsable syntax elements that identify tiles or other pictures in high-level syntax structures such as NAL unit headers.

[0019] FIG. 2 shows a simplified block diagram of a communication system (200) according to one embodiment. The system (200) may include at least two terminals (210-220) interconnected via a network (250). For unidirectional data transmission, the first terminal (210) may encode video data at a local location for transmission to the other terminal (220) via the network (250). The second terminal (220) may receive the encoded video data of the other terminal from the network (250), decode the encoded data, and display the recovered video data. Unidirectional data transmission may be common in media delivery applications, etc.

[0020] FIG. 2 shows a second pair of terminals (230, 240) provided to support bidirectional transmission of encoded video that may occur during a video conference, for example. For bidirectional data transmission, each terminal (230, 240) may encode video data captured at a local location for transmission to the other terminal via the network (250). Each terminal (230, 240) may also receive the encoded video data transmitted by the other terminal, decode the encoded data, and display the recovered video data on a local display device.

[0021] In FIG. 2, the terminals (210-240) can be shown as a server, a personal computer, and a smartphone, but the principles of the present disclosure are not so limited. Embodiments of the present disclosure find applications using a laptop computer, a tablet computer, a media player, and / or a dedicated video conferencing device. The network (250) represents any number of networks that transmit encoded video data between terminals (210-240), including, for example, wired and / or wireless communication networks. The communication network (250) can exchange data in circuit-switched and / or packet-switched channels. Representative networks include a communication network, a local area network, a wide area network, and / or the Internet. For the purposes of this description, the architecture and topology of the network (250) are not important for the operation of the present disclosure unless otherwise described below.

[0022] FIG. 3 shows the arrangement of video encoders and decoders in a streaming environment as one example of an application for the disclosed technical matters. The disclosed technical matters are similarly applicable to other video-enabled applications, including, for example, storage of compressed video in digital media such as video conferencing, digital TV, CD, DVD, memory stick, and the like.

[0023] The streaming system may include a capture subsystem (313) and may include, for example, a video source (301), such as a digital camera, that generates an uncompressed video sample stream (302). This sample stream (302) is shown as a thick line to emphasize the large data volume when compared to the encoded video bitstream and may be processed by an encoder (303) coupled to the camera (301). The encoder (303) can include hardware, software, or a combination thereof to enable or implement aspects of the disclosed technical matter, as described in more detail below. The encoded video bitstream (304) is shown as a thin line to emphasize the smaller data volume when compared to the sample stream and may be stored at a streaming server (305) for future use. One or more streaming clients (306, 308) can access the streaming server (305) to retrieve copies (307, 309) of the encoded video bitstream (304). The client (306) may include a video decoder (310) that decodes an input copy of the encoded video bitstream (307) and generates an output video sample stream (311) that can be rendered on a display (312) or other rendering device (not shown). In some streaming systems, the video bitstreams (304, 307, 309) can be encoded according to a given video encoding / compression standard. Examples of these standards include ITU-T Recommendation H.265. A video encoding standard under development is known informally as Versatile Video Coding, or VVC. The disclosed technical matter can be used in the context of VVC.

[0024] FIG. 4 may be a functional block diagram of a video decoder (310) according to one embodiment of the present invention.

[0025] The receiver (410) can receive one or more codec video sequences to be decoded by the decoder (310). In the same or another embodiment, it is one encoded video sequence at a time, and the decoding of each encoded video sequence is independent of other encoded video sequences. The encoded video sequence can be received from a channel (412) that can be a hardware / software link to a storage device storing the encoded video data. The receiver (410) can receive the encoded video data together with other data, such as encoded audio data and / or auxiliary data streams, and the data can be transferred using respective entities (not shown). The receiver (410) can separate the encoded video sequence from other data. A buffer memory (415) can be coupled between the receiver (410) and the entropy decoder / parser (420) (hereinafter, “parser”). The buffer (415) may not be needed or may be small if the receiver (410) is receiving data from a storage / transfer device with sufficient bandwidth and controllability or from an isosynchronous network. For use in a best-effort packet network such as the Internet, the buffer (415) is needed, relatively large, and can advantageously be of an adaptable size.

[0026] The video decoder (310) may include a parser (420) for reconstructing symbols (421) from an entropy-coded video sequence. The categories of these symbols include information used to manage the operation of the decoder (310), and potentially information for controlling rendering devices such as a display (312) that, while not an essential part of the decoder, can be connected to the decoder, as shown in FIG. 3. The control information for the rendering device can be in the form of supplementary enhancement information (SEI messages) or a video usability information (VUI) parameter set fragment (not shown). The parser (420) may parse / entropy-decode the received encoded video sequence. The encoding of the encoded video sequence can follow a video encoding technique or standard and can follow principles well known to those skilled in the art, including variable length encoding, Huffman encoding, arithmetic encoding with or without context sensitivity, etc. The parser (420) may extract a set of subgroup parameters for at least one of the subgroups of pixels within the video decoder based on at least one parameter corresponding to the group from the encoded video sequence. The subgroups can include groups of pictures (GOPs), pictures, tiles, slices, macroblocks, coding units (CUs), blocks, transform units (TUs), prediction units (PUs), etc. The entropy decoder / parser can also extract from the encoded video sequence information such as transform coefficients, quantization parameter values, motion vectors, etc.

[0027] The parser (420) can perform an entropy decoding / parsing operation on the video sequence received from the buffer (415) to generate symbols (421).

[0028] The reconstruction of symbol (421) can include multiple different units depending on the type of the encoded video image or a portion thereof (such as between pictures and within a picture, between blocks and within a block, etc.) and other factors. Which unit is involved and how it is involved can be controlled by subgroup control information syntactically analyzed from the encoded video sequence by a parser (420). The flow of such subgroup control information between the parser (420) and the following multiple units is not shown for clarity.

[0029] In addition to the functional blocks described above, decoder 310 can conceptually be subdivided into several functional units as described below. In a practical implementation operating under commercial constraints, many of these units can interact closely with each other and can be at least partially integrated with each other. However, for the purpose of explaining the disclosed technical matters, it is appropriate to conceptually subdivide into the following functional units.

[0030] The first unit is a scaler / inverse transform unit (451). The scaler / inverse transform unit (451) receives from the parser (420), as symbols (421), quantized transform coefficients, as well as control information including the transform to be used, block size, quantization coefficients, quantization scaling matrices, etc. That unit can output a block containing sample values that can be input to an aggregator (455).

[0031] In some cases, the output samples of the scaler / inverse transform (451) can belong to an intra-coded block. That is, it is a block that does not use prediction information from a previously reconstructed image but can use prediction information from a previously reconstructed part of the current image. Such prediction information can be provided by the intra-image prediction unit (452). In some cases, the intra-image prediction unit (452) uses the surrounding already reconstructed information fetched from the current (partially reconstructed) image (456) to generate a block of the same size and shape as the block being reconstructed. The aggregator (455) sometimes adds, for each sample, the prediction information generated by the intra-prediction unit (452) to the output sample information provided by the scaler / inverse transform unit (451).

[0032] In other cases, the output samples of the scaler / inverse transform unit (451) can belong to an inter-coded and potentially motion-compensated block. In such cases, the motion-compensation prediction unit (453) can access the reference picture memory (457) to fetch the samples used for prediction. After motion-compensating the samples fetched according to the symbol (421) belonging to the block, these samples can be added by the aggregator (455) to the output of the scaler / inverse transform unit (in this case called the residual samples or residual signal) to generate the output sample information. The address in the reference picture memory form where the motion-compensation unit fetches the prediction samples can be in the form of, for example, a symbol (421) having X, Y, and reference picture components and can be controlled by the motion vector available to the motion-compensation unit. Motion compensation can also include interpolation of the sample values fetched from the reference picture memory when an exact motion vector of sub-samples is used, a motion vector prediction mechanism, etc.

[0033] The output samples of the aggregator (455) can be subject to various loop filtering techniques within the loop filter unit (456). Video compression techniques can include in-loop filter techniques. In-loop filter techniques are controlled by parameters included in the encoded video bitstream and are made available to the loop filter unit (456) as symbols (421) from the parser (420), but can also respond to meta information obtained during the decoding of previous portions (in decoding order) of the encoded image or encoded video sequence, and can also respond to previously reconstructed and loop-filtered sample values.

[0034] The output of the loop filter unit (456) can be a sample stream that is output to the rendering device (312) and stored in the reference image memory (456) for use in future inter-picture prediction.

[0035] A given encoded image, once fully reconstructed, can be used as a reference image for future prediction. Once the encoded image is fully reconstructed and identified as a reference image (e.g., by the parser (420)), the current reference image (456) becomes part of the reference image buffer (457), and a fresh current image memory can be reallocated before starting the reconstruction of the next encoded image.

[0036] Video decoder 420 can perform decoding operations according to established video compression techniques that can be documented in standards such as ITU-T Rec. H.265. The encoded video sequence can conform to the syntax defined by the video compression technique or standard being used, in the sense that it follows the syntax of the video compression technique or standard as defined in the video compression technique document or standard, and in particular, in the profile document therein. Also, for compliance, it is necessary that the complexity of the encoded video sequence be within the range defined by the level of the video compression technique or standard. In some cases, the level restricts the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. The limitations set by the level can, in some cases, be further restricted through the Hypothetical Reference Decoder (HRD) specification and the metadata of the HRD buffer signaled in the encoded video sequence.

[0037] In one embodiment, receiver (410) can receive additional (redundant) data along with the encoded video. The additional data can be included as part of the encoded video sequence. The additional data can be used by video decoder (420) to properly decode the data and / or to more accurately reconstruct the original video data. The additional data can be in the form of, for example, temporal, spatial, or SNR enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0038] FIG. 5 can be a functional block diagram of video encoder (303) according to one embodiment of the present disclosure.

[0039] Encoder (303) can receive video samples from a video source (301) (which is not part of the encoder) that can capture the video images to be encoded by encoder (303).

[0040] The video source (301) can provide a source video sequence to be encoded by an encoder (303) in the form of a digital video sample stream, and the form can be any suitable bit depth (e.g., 8 bits, 10 bits, 12 bits, ...), any color space (e.g., BT.601 Y CrCB, RGB, etc.), and any suitable sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media serving system, the video source (301) can be a storage device storing previously prepared videos. In a video conferencing system, the video source (303) can be a camera that captures local image information as a video sequence. The video data may be provided as a plurality of individual images that convey motion when viewed in sequence. The image itself may be configured as a spatial array of pixels, where each pixel can include one or more samples depending on the sampling structure, color space, etc. in use. One skilled in the art can easily understand the relationship between a pixel and a sample. The following description focuses on samples.

[0041] According to one embodiment, the encoder (303) can encode and compress the images of the source video sequence into an encoded video sequence (543) in real time or under any other time constraints required by the application. Implementing an appropriate encoding speed is one function of the controller (550). The controller controls other functional units and is functionally coupled to these units, as will be described below. The couplings are not shown for clarity. The parameters set by the controller can include rate control related parameters (picture skip, quantizer, lambda value of rate distortion optimization techniques, etc.), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. A person skilled in the art can easily identify other functions of the controller (550). This is because they may be related to a video encoder (303) optimized for a given system design.

[0042] Some video encoders operate in what those skilled in the art would readily recognize as a “coding loop.” As an overly simplified explanation, the coding loop consists of an encoding part of the encoder (530) (hereinafter, “source coder”) (which is responsible for generating symbols based on the input picture to be encoded and reference pictures), and a (local) decoder (533) embedded within the encoder (303) that reconstructs the symbols to generate sample data. The (remote) decoder will also generate (such that any compression between the symbols and the encoded video bitstream is lossless in the video compression technology considered in the disclosed technical matter). The reconstructed sample stream is an input to the reference picture memory (534). Since the decoding of the symbol stream results in a bit-exact result independent of the decoder location (local or remote), the content of the reference picture buffer is also bit-exact between the local encoder and the remote encoder. In other words, the prediction part of the encoder “sees” the same sample values as the reference picture samples that the decoder “sees” when using the prediction during decoding. This basic principle regarding reference picture synchronization (and the resulting drift in the case where synchronization cannot be maintained, for example, due to channel errors) is well known to those skilled in the art.

[0043] The operation of the "local" decoder (533) can be the same as that of the "remote" decoder (310). This has already been detailed above in connection with FIG. 4. However, referring briefly to FIG. 4 again, symbols are available and the encoding / decoding of symbols for the encoded video sequence by the entropy encoder (545) and the parser (420) can be reversible. So, the entropy decoding part of the decoder (310), including the channel (412), the receiver (410), the buffer (415), and the parser (420), may not be fully implemented within the local decoder (533).

[0044] An observation that can be made at this point is that any decoder technology that exists within the decoder, excluding parsing / entropy decoding, must also exist in a substantially identical functional form within the corresponding encoder. For this reason, the disclosed technical matters focus on the operation of the decoder. The description of the encoder technology can be omitted because it is the reverse of the decoder technology described comprehensively. Only in a given field is a more detailed description necessary and is provided below.

[0045] As part of its operation, the source coder (530) can perform motion compensated predictive coding, which predictively encodes the input frame with respect to one or more previously coded frames from the video sequence designated as the "reference frame". In this way, the encoding engine (532) encodes the difference between the pixel block of the input frame and the pixel block of the reference frame that can be selected as the prediction reference for the input frame.

[0046] The local video decoder (533) can decode the encoded video data of a frame that can be specified as a reference frame based on the symbols generated by the source coder (530). The operation of the encoding engine (532) can advantageously be an irreversible process. If the encoded video data can be decoded by a video decoder (not shown in FIG. 5), the reconstructed video sequence can typically be a replica of the source video sequence with some errors. The local video decoder (533) replicates the decoding process that is executed on the reference frame by the video decoder and stores the reconstructed reference frame in the reference picture cache (534). In this way, the encoder (303) can locally store (without transmission errors) a copy of the reconstructed reference frame having common content as the reconstructed reference frame obtained by the far-end video decoder.

[0047] The predictor (535) can perform a prediction search for the encoding engine (532). That is, for a new frame to be encoded, the predictor (535) can search the reference picture memory (534) for sample data (as candidate reference pixel blocks), or for predetermined metadata such as reference picture motion vectors, block shapes, etc., that can serve as an appropriate prediction reference for the new picture. The predictor (535) can operate on a sample block-pixel block basis to find an appropriate prediction reference. In some cases, the input picture can have a prediction reference drawn from a plurality of reference pictures stored in the reference picture memory (534) as determined by the search results obtained by the predictor (535).

[0048] The controller (550) can manage the encoding operation of the video coder (530), including, for example, setting the parameters and subgroup parameters used to encode the video data.

[0049] The outputs of all the aforementioned functional units can undergo entropy encoding in an entropy encoder (545). The entropy encoder converts the symbols generated by the various functional units into an encoded video sequence by reversibly compressing the symbols according to techniques known to those skilled in the art, such as, for example, Huffman coding, variable length coding, arithmetic coding, etc.

[0050] When generated by the entropy encoder (545), the transmitter (540) can buffer the encoded video sequence and prepare it for transmission via a communication channel (560). The communication channel may be a hardware / software link to a storage device that stores the encoded video data. The transmitter (540) can merge the encoded video data from the video coder (530) with other data to be transmitted, such as, for example, encoded audio data and / or an auxiliary data stream (source not shown).

[0051] The controller (550) can manage the operation of the encoder (303). During encoding, the controller (550) can assign a predetermined encoded picture type to each encoded picture, which can affect the encoding technique applicable to each picture. For example, a picture can often be assigned as one of the following frame types.

[0052] An intra picture (I picture) can be encoded and decoded without using any other frame within the sequence as a prediction source. Some video codecs allow different types of intra pictures, including, for example, Independent Decoder Refresh Pictures. Those skilled in the art are aware of these variations of I pictures, as well as their respective applications and functions.

[0053] A predicted picture (P picture) can be encoded and decoded using intra prediction or inter prediction using at most one motion vector and a reference index to predict the sample values of each block.

[0054] A bi-directionally predictive picture (B picture) can be encoded and decoded using intra prediction or inter prediction using at most two motion vectors and reference indices to predict the sample values of each block. Similarly, multiple predicted pictures can use two or more reference pictures and associated metadata for the reconstruction of a single block.

[0055] A source image is generally spatially subdivided into a plurality of sample blocks (e.g., blocks of 4×4, 8×8, 4×8, or 16×16 samples each), and each block is encoded. The blocks can be predictively encoded with reference to other (already encoded) blocks as determined by the encoding assignment applied to each block of the image. For example, blocks of an I picture may be encoded non-predictively, or they may be predictively encoded with reference to already encoded blocks of the same image (spatial prediction or intra prediction). Pixel blocks of a P picture can be encoded non-predictively, via spatial prediction, or via temporal prediction with reference to one previously encoded reference image. Blocks of a B picture can be encoded non-predictively, via spatial prediction, or via temporal prediction with reference to one or two previously encoded reference images.

[0056] The video coder (303) can perform encoding operations according to a given video encoding technology or standard, such as ITU-T Rec. H.265. In such operations, the video coder (303) can perform various compression operations, including predictive encoding operations that utilize the temporal and spatial redundancies in the input video sequence. The encoded video data can thus conform to the syntax specified by the video encoding technology or standard being used.

[0057] In one embodiment, the transmitter (540) can transmit additional data along with the encoded video. The video coder (530) may include such data as part of the encoded video sequence. The additional data can include other forms of redundant data such as temporal / spatial / SNR enhancement layers, redundant picture pictures and slices, supplementary enhancement information (SEI) messages, video user utility information (VUI) parameter set fragments, etc.

[0058] According to one embodiment, information identifying picture segments such as tiles, tile groups, slices, groups of blocks (GOBs), etc. (hereinafter referred to as tiles) is arranged in an easily accessible high-level syntax structure such as a NAL unit header (NUH), or in a similar structure that includes a fixed-length codeword and is designed for easy processing by MANE (hereinafter referred to as NUH).

[0059] The information identifying the tiles can take different forms. Some design considerations should be kept in mind when designing this information. Some of these design considerations are listed below.

[0060] The number of possible tiles within a given picture can be made small, for example, when compared to the number of possible slices in conventional video coding techniques or standards. For example, in H.264, it is possible to have a slice that covers a single macroblock (for a given picture size), and as many slices as there are macroblocks are possible. In contrast, when representing a tiled cube map, six tiles may be sufficient regardless of the resolution of the image. In many practical cases, a maximum number of 64, 128, or 256 tiles can be safely assumed.

[0061] The tile layout can be fixed, and while the video coding technology itself may allow flexibility in the tile layout between pictures, the system standard or technology can limit that flexibility to the point where the tile layout remains the same throughout the session. This allows the tile layout to be made available to the MANE via non-video-bitstream-specific means such as during session setup. Unwanted context dependencies between parameter sets in video encoding and MANE operations can thereby be prevented.

[0062] At least under the above assumptions, the mechanism for identifying the tiles carried by the NAL unit to enable the NAL unit to be removed by the MANE can be significantly simplified compared to related technologies such as H.264 and H.265. For example, in H.264 and H.265, the MANE would need to identify the correct sequence parameter set in order to know the length of the slice / tile address codeword in the slice header. Such length information is encoded as a variable length codeword within the sequence parameter set. Thus, the MANE would need to, at a minimum, follow the activation sequence of the parameter set to identify the current active sequence parameter set and, (since the parameter set is parsing-independent, perhaps not in this order) would need to decode the variable length codeword to identify the length of the binary-encoded slice / tile address carried in the slice header. The MANE would then need to decode the variable length codeword in the slice header to obtain the starting macroblock / CU address. This information can be collated against the tile layout decoded from the parameter set to identify the tile.

[0063] In the same or another embodiment, the identification information of the tile can be the address of the first macroblock / CU of the tile. In fact, such a mechanism moves the starting address from the slice header to the NUH. Doing so can be a minimal change approach to codec design but has the drawback of significantly increasing the NUH. However, the increase in the size of the NUH may be acceptable even from the perspective of encoding efficiency. Because the same amount of bits would have been removed from the slice / tile header.

[0064] As pointed out above, the macroblock / CU address can be moderately small for small image sizes and large macroblock / CU sizes, or can be quite large for small CU sizes and large image sizes. For this reason, the H.265 SPS includes an indication of the length of the macroblock / CU address carried in the slice header. In the same or another embodiment, this mechanism can be maintained for the NAL unit header. However, doing so can have two drawbacks. First, the context-dependency established by determining the size of the syntax element in the NAL unit header through the parameter set value may require the MANE to track the activation of the parameter setting, which can be redundant. Second, the NAL unit header has been, at least until now, octet aligned to simplify processing in the MANE. Maintaining that octet alignment may require wasting padding bits. In such a case, the size of the macroblock / CU address indicated by the parameter set is not added up to a number of bits divisible by 8 together with the remaining NAL unit header syntax elements.

[0065] In the same or another embodiment, the size of the macroblock / CU address - or any other syntax element within the NAL unit header - can be determined by other fields within the NAL unit header. This mechanism can avoid the content-dependency between the parameter set and the NAL unit header and, in many cases, can be desirable. The drawback can be the use of bits or code points in other fields of the NAL unit header. Details are provided below.

[0066] However, when not considering slices in the conventional sense, only tiles or tile groups, or a similar assignment mechanism of CUs to bitstream entities, more advanced options, are available. To explain these options, the terms slice and tile are briefly revisited. A slice is usually a collection of CUs or macroblocks in scan order and is identified by two factors. Namely, the starting macroblock / CU address, usually encoded in the slice header, and the end of the slice, often identified by the start of a new slice (similarly, indicated by the presence of the next slice header). Given video compression techniques and standards impose certain relatively small restrictions on the number and layout of slices, but in most cases, the slice layout can be changed from encoded image to encoded image and is often determined by mechanisms such as rate control and MTU size matching.

[0067] Tiles, on the one hand, can typically refer to a rectangular arrangement of CUs, and the size and shape of the rectangles (from which the rectangular tiles and other rectangular tiles are combined to form a picture) are encoded within a parameter set. In other words, the tile layout is somewhat static in that a change from one tile layout to another requires the activation of a different parameter set. Further, the number of tiles can be advantageously limited to enable an efficient hardware implementation. As a result, in many video compression techniques and standards, for example, 8-bit relatively short fixed-length binary codewords are used to handle the maximum number of tiles for all image sizes actually used. Thus, a fixed-length codeword for the tile ID is used to identify the tile within the NAL unit header, thereby avoiding the parsing and context-dependency between the tile identification NAL unit header codeword and the parameter set. Similarly, a fixed-length codeword for the tile group ID can be used to identify a tile group. Of course, a mechanism that supports variable-length codewords for macroblock / CU addresses within the NAL unit header can be similarly applied to tile ID codewords, sacrificing such architectural drawbacks if so desired.

[0068] Referring to FIG. 6, several implementation options for the NAL unit header design are presented.

[0069] A NAL unit (601) can be part of an encoded video bitstream. Optionally, the NAL unit is octet-aligned and smaller than or equal to the common maximum transfer unit (MTU) size of the data network. One such common MTU size is approximately 1500 octets, which is derived from a given limit related to the initial Ethernet technology. The NAL unit may first include a NAL unit header (602). The framing of NAL units in the encoded video bitstream is through a start code, through alignment with the packet structure of the underlying packet-oriented transport network, etc.

[0070] Referring back to FIG. 6, an exemplary syntax diagram of a NAL unit header (603) is also shown, which is similar to that used in H.265. The disclosed technical matters can likewise be used with NAL unit headers of similar structure. For example, the NAL unit header of H.264, or VVC, or any other high-level syntax structure including fixed-length codewords. The NAL unit header (603) may include a syntax element CU address or tile ID (604). The length of the syntax element may be fixed and selected such that the NAL unit header remains octet-aligned. The syntax element (604) can be in a format that is easily processable not only by video encoders and decoders but also by MANE. By way of example and not limitation, the CU address or tile ID (604) can be represented by a 6-bit unsigned integer, such as represented by a descriptor u(6). In the example presented, the CU address or tile ID occupies the same bits as used for layer_id in H.265. Different presentations of similar technical matters are shown using the NAL unit header (605) and the CU address or tile ID (606).

[0071] Also shown is a NAL unit (607) that stores fields of the H.265 NAL unit header. A syntax element (608) is added, for example, at the end of the NAL unit header. The position of the syntax element is merely exemplary and it may also be inserted anywhere in the middle of other syntax elements of the NAL unit header. The syntax element may be of fixed or variable size. And in the case of variable size, the size may be determined by any of the mechanisms described above (e.g., through parameter set syntax elements, through NAL unit types, etc.), or by any other suitable mechanism.

[0072] The syntax element (608) can carry any form of picture segment identification information. For example, it is a macroblock / CU address or tile identifier such as a tile number, or a number indicating a tile group. The numbering range of the syntax element may be determined in advance. In the case of a macroblock / CU address, the numbering range may be from 0 to the maximum number of macroblocks / CUs in the picture. For tile IDs, the numbering range depends on the maximum number of tiles and may be defined by mechanisms known to those skilled in the art such as, for example, profile, level, tiers, the maximum or actual image size encoded in the parameter set, etc. When the syntax element exists for non-tile / slice NAL units (such as parameter set NAL units, SEI NAL units, etc.), the value of the syntax element may be restricted to a predetermined number, for example 0. Alternatively, the existence of the syntax element can be gated in the NAL unit type, and thereby the syntax element may not exist for a given NAL unit type. Alternatively, overriding semantics other than those described above may be assigned to the syntax element for a given non-tile / slice NAL unit type.

[0073] In the same or another embodiment, the tile ID can be identified, for example, as follows. FIG. 7 shows a picture (701) in a spatial domain subdivided by an exemplary tile layout (702) indicated by the thick lines. The presented tile layout can be represented, for example, by the syntax available in H.265 or by a more advanced syntax that would be required to represent the tile layout (702). Each tile within the tile layout can have an assigned tile ID through any suitable numbering mechanism, but preferably through scan-order numbering of the tiles. In FIG. 7, the scan-order tile numbering is shown through tile IDs 1 to 8. For example, the second tile in the scan order has the assigned tile ID 2 (703).

[0074] The encoder can write the syntax element covering the tile ID, or the header of the NAL unit containing the macroblock / CU address arranged as described above, in a manner similar to that describing the header syntax of existing NAL units known to those skilled in the art.

[0075] The decoder or MANE can analyze from the encoded video bitstream the syntax elements that make up the NAL unit header - more precisely - regardless of the presence or absence of a syntax element that conveys a macroblock / CU address, or a tile ID, or another form of tile identification information, in a manner known to those skilled in the art. However, it should be noted that the syntax elements may in some cases be encoded without requiring state information as described above and can be an accessible entropy coding format, for example, fixed-length, binary code. So far, parsing the NAL unit header according to the disclosed technical matters may not involve an additional burden on the decoder or MANE beyond the actual presence of the syntax element tile_id itself.

[0076] In accordance with the disclosed technical matters, however, a decoder or MANE can identify tiles in an encoded image with little effort compared to the operations required in the absence of the disclosed technical matters. As one example, assume that a decoder or MANE is informed by external, non-video encoding means that a given tile need not be reconstructed for a given application. Consider, for example, a scene as shown in FIG. 7. That is, a village road. Assume that the road is being captured by a surveillance camera. Consider the tile with tile ID 2 (703). That tile mostly covers a wall surface, and assume that the configurator of the surveillance system considers that area to be irrelevant for surveillance. Thus, the camera can encode all tiles, but the tile with ID 2 may not be required for the application. Thus, if the bitstream generated by the camera is routed through one or more MANEs to a final destination and one MANE observes a bandwidth shortage and has to remove something from the bitstream, that tile can be advantageously removed as it is not required for the application. In the absence of the disclosed technical matters, at a minimum, the payload of the NAL unit (slice or tile) would need to be parsed to the extent necessary to extract the macroblock / CU address of the first macroblock in the tile, and then that macroblock / CU address would be mapped against the tile layout (if the tile is being used). Depending on the video encoding technology or standard being used and as described above, both variable length codeword processing and parameter set context retention in the MANE may be required, but neither is desirable from the perspective of implementation and computational complexity. Instead, in the same or another embodiment, the MANE can obtain all the information necessary to identify which tiles are being carried by the NAL unit through NAL unit header processing of binary encoded codewords.

[0077] Referring to FIG. 8, the decoder or MANE can use the disclosed technical matters as follows, for example.

[0078] The decoder can parse a NAL unit header (801) that includes a syntax element covering a macroblock / CU address or tile ID from a video bitstream. Using that information, the decoder or MANE can identify the tile ID (802). The tile ID may be directly encoded or can match an encoded macroblock / CU address in the NAL unit header with priori information regarding the tile layout that is established, for example, by decoding a parameter set and a subsequent activation sequence. The decoder can match the tile ID against a list of tiles that respectively require reconstruction or transfer by the decoder or MANE (803). When a match exists (803), the decoder can reconstruct the NAL unit carrying the tile, or the MANE can transfer it. However, when no match exists (804), the decoder or MANE can probably silently discard the NAL unit.

[0079] The technique for image reference in the network abstraction unit header described above is implemented as computer software using computer-readable instructions and can be physically stored on one or more computer-readable media. For example, FIG. 9 shows a computer system 900 suitable for implementing a predetermined embodiment of the disclosed technical matters.

[0080] Computer software can be encoded using any suitable machine code or computer language that can be the subject of assembly, compilation, linking, or similar mechanisms to create code containing instructions. The instructions can be executed directly by a computer central processing unit (CPU), a graphics processing unit (GPU), etc., or through interpretation, microcode execution, etc.

[0081] The instructions can be executed on various types of computers or their components, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, the Internet of Things, etc.

[0082] The components shown in FIG. 9 for computer system 900 are exemplary in nature and are not intended to suggest any limitation as to the use or functionality scope of the computer software implementing the embodiments of the present disclosure. Also, the component configuration should not be construed as having any dependency or requirement regarding any one or combination of the components shown in the exemplary embodiment of computer system 900.

[0083] Computer system 900 can include a predetermined human interface input device. Such human interface input devices can respond to input by one or more human users through, for example, tactile input (e.g., keystrokes, swipes, movement of a data glove), audio input (e.g., voice, applause), visual input (e.g., gestures), olfactory input (not shown). The human interface device can also be used to capture a predetermined medium that is not necessarily directly related to conscious human input, such as audio (e.g., voice, music, ambient sound), images (e.g., scanned images, photographic images obtained from a still image camera), video (e.g., two-dimensional video, three-dimensional video including stereoscopic images).

[0084] The input human interface device may include one or more of a keyboard 901, a mouse 902, a track pad 903, a touch screen 910, a data glove 904, a joystick 905, a microphone 906, a scanner 907, a camera 908 (only one of each is depicted).

[0085] The computer system 900 may also include a predetermined human interface output device. Such human interface output devices can stimulate the senses of one or more human users, for example, through tactile output, sound, light, and smell / taste. Such human interface output devices include tactile output devices (e.g., tactile feedback by a touch screen 910, a data glove 904, or a joystick 905, but there may also be a tactile feedback device that does not function as an input device), audio output devices (e.g., speakers 909, headphones (not shown)), visual output devices (screens 910 including CRT screens, LCD screens, plasma screens, OLED screens, each of which may or may not have a touch screen input function, each of which may or may not have a tactile feedback function - some of these can output two-dimensional visual output or three-dimensional or higher output through means such as stereoscopic image output, virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).

[0086] The computer system 900 may also include human-accessible storage devices and their associated media. Optical media including CD / DVD ROM / RW 920 using media 921 such as CD / DVD, thumb drives 922, removable hard drives or solid state drives 923, legacy magnetic media such as tapes and floppy (registered trademark) disks, specialized ROM / ASIC / PLD-based devices such as security dongles (not shown), etc.

[0087] One of ordinary skill in the art should also understand that the term "computer readable media" as used in connection with the technical matters disclosed herein does not include a transmission medium, a carrier wave, or other transient signals.

[0088] Computer system 900 may also include an interface to one or more communication networks. The network can be, for example, wireless, wired, or optical. The network can further be local, wide area, metropolitan, vehicle and industrial, real-time, delay-tolerant, etc. Examples of networks include cellular networks such as Ethernet, GSM, 3G, 4G, 5G, LTE, etc., cable television, satellite television, and wired or wireless wide area digital networks including over-the-air broadcast television, vehicle and industrial including CANBus, etc. A given network generally requires an external network interface adapter attached to a given general-purpose data port or peripheral bus (949). For example, a USB port of computer system 900, and others are generally incorporated into the core of computer system 900 by attaching to the system bus described below (e.g., an Ethernet interface to a PC computer system, or a cellular network interface to a smartphone computer system). Using any of these networks, computer system 900 can communicate with other entities. Such communication can be uni-directional, receive-only (e.g., broadcast television), uni-directional transmit-only (e.g., CAN bus to a given CAN bus device), or bi-directional, for example, to other computer systems using a local or wide area digital network. A given protocol and protocol stack can be used for each of those networks and network interfaces as described above.

[0089] The above-described human interface device, human-accessible storage device, and network interface can be attached to the core 940 of computer system 900.

[0090] The core 940 may include programmable processing devices specialized in the form of one or more central processing units (CPUs) 941, graphics processing units (GPUs) 942, field programmable gate arrays (FPGAs) 943, hardware accelerators 1044 for a given task, and the like. These devices may be connected through a system bus 948 together with internal mass storage devices 947 such as read only memories (ROMs) 945, random access memories 946, internal non-user-accessible hard drives, solid state drives (SSDs), and the like. In some computer systems, the system bus 948 may be accessible in the form of one or more physical plugs to enable expansion by additional CPUs, GPUs, and the like. Peripheral devices may be attached directly to the core's system bus 948 or through a peripheral bus 949. The architecture of the peripheral bus includes PCI, USB, and the like.

[0091] The CPU 941, GPU 942, FPGA 943, and accelerator 944 can, in combination, execute certain instructions that can constitute the computer code described above. The computer code can be stored in the ROM 945 or the RAM 946. Transient data can also be stored in the RAM 946, while permanent data can be stored, for example, in the internal mass storage device 947. Fast storage and retrieval to any of the memory devices can be enabled through the use of cache memory, which can be closely associated with one or more CPUs 941, GPUs 942, mass storage device 947, ROM 945, RAM 946, and the like.

[0092] A computer-readable medium can have computer code thereon for performing various computer-implemented operations. The medium and the computer code can be specially designed and constructed for the purposes of this disclosure or they can be of the kind well known and available to those skilled in the computer software arts.

[0093] As an example, and not by way of limitation, a computer system having architecture 900, and specifically core 940, can provide functionality as a result of software embodied on one or more tangible computer-readable media being executed by a processor (including a CPU, GPU, FPGA, accelerator, etc.). Such computer-readable media can be the mass storage devices accessible to the user as described above, as well as certain storage devices associated with core 940 of a non-transitory nature, such as internal core mass storage device 947 or ROM 945. The software implementing various embodiments of the present disclosure can be stored in such devices and executed by core 940. The computer-readable media can include one or more memory devices or chips depending on specific needs. The software can cause core 940, and specifically the processors (including CPU, GPU, FPG, etc.) therein, to execute a specific process or specific portions of a specific process described herein. This includes defining a data structure stored in RAM 946 and modifying such data structures according to the processes defined by the software. Additionally or alternatively, the computer system can provide functionality as a result of logic wired in a circuit (e.g., accelerator 944) or otherwise embodied. The circuit can operate instead of or in conjunction with the software to execute a specific process or specific portions of a specific process described herein. References to software include logic and, where appropriate, vice versa. References to computer-readable media can include circuits (such as integrated circuits (ICs)) that store software for execution, circuits that embody logic for execution, or, where appropriate, both. The present disclosure encompasses any suitable combination of hardware and software.

[0094] Although several exemplary embodiments have been described, changes, substitutions, and various alternative equivalents exist and are within the scope of this disclosure. Thus, it will be correctly understood by those skilled in the art that, even if not explicitly shown or described herein, the principles of this disclosure can be embodied and, accordingly, many systems and methods within the spirit and scope of this disclosure can be devised.

Description of Reference Numerals

[0095] 200 Communication system 210 First terminal 220 Second terminal 250 Network 701 Picture 702 Tile layout 900 Computer system 940 Core

Claims

Claim 1 A method for video encoding, comprising: encoding a video sequence; decoding a binary-coded syntax element that carries an identifier of a picture segment in a high-level syntax structure including fixed-length codewords; reconstructing the picture segment; wherein a parameter set is active for a picture to which the high-level syntax structure belongs; the method. Claim 2 The method according to claim 1, wherein the picture segment is one of a tile, a group of tiles, or a sub-picture. The method according to claim 1. Claim 3 The method according to claim 2, wherein the syntax element is one of a macroblock address, an encoding unit address of a first macroblock or an encoding unit, or a tile address of a first encoding unit, respectively, in the picture segment. The method according to claim 2. Claim 4 The method according to claim 2, wherein the syntax element is a tile identifier. The method according to claim 2. Claim 5 For a given tile layout, each tile in the tile layout has a unique tile identifier, and the unique tile identifier is assigned according to a scan order of the picture segment in the tile layout. The method according to claim 4. Claim 6 The method according to claim 5, wherein the scan order is associated with a tile at the upper left corner of the tile layout and is incremented one by one in the order from top to bottom and from right to left. The method according to claim 5. Claim 7 The method according to claim 1, wherein the syntax element is at a fixed position in the high-level syntax structure. The method according to claim 1. Claim 8 The method according to claim 1, wherein a size of the syntax element is determined through at least one syntax element in the parameter set. The method according to claim 1. Claim 9 The method further comprises: determining, based on decoding of the binary-coded syntax element, that a tile identifier associated with the binary-coded syntax element is included in a list of tiles to be decoded; reconstructing the picture segment based on the determination that the tile identifier is included in the list of tiles to be decoded; The method according to claim 1, including the above steps. Claim 10 An apparatus for encoding a video sequence, comprising: At least one memory configured to store program code, At least one processor configured to read the program code and operate as instructed by the program code, The program code includes Encoding code configured to cause the at least one processor to encode a video sequence, Decoding code configured to cause the at least one processor to decode a binary-coded syntax element that carries an identifier of a picture segment in a high-level syntax structure including fixed-length codewords, Reconstruction code configured to cause the at least one processor to reconstruct the picture segment, and A parameter set is active for a picture to which the high-level syntax structure belongs, Device.

11. The picture segment is one of a tile, a group of tiles, or a sub-picture, The device according to claim 10.

12. The syntax element is one of a macroblock address, an encoding unit address of a first macroblock or an encoding unit, or a tile address of a first encoding unit, respectively, in the picture segment, The device according to claim 11.

13. The syntax element is a tile identifier, The device according to claim 11.

14. For a given tile layout, each tile in the tile layout has a unique tile identifier, and the unique tile identifier is assigned according to the scan order of the picture segments in the tile layout, The device according to claim 13.

15. The scan order is associated with the tile in the upper left corner of the tile layout and increases one by one according to the order from top to bottom and from right to left, The device according to claim 14.

16. The syntax element is in a fixed position in the high-level syntax structure, The device according to claim 10.

17. The size of the syntax element is determined through at least one syntax element in the parameter set, The device according to claim 10. A method of video encoding performed by an encoder, the method comprising: generating an encoded bitstream; and storing the encoded bitstream, wherein generating the encoded bitstream comprises: encoding a video sequence; decoding a binary-coded syntax element that carries an identifier of a picture segment in a high-level syntax structure including fixed-length codewords; reconstructing the picture segment; and a parameter set is active for a picture to which the high-level syntax structure belongs, method.

Images