Low-latency image coding

Abstract

To provide a decoder, an encoder, a decoding method, an encoding method and a program that increase the encoding efficiency and enable parallel decoding in accordance with wavefront parallel processing.SOLUTION: A decoder 5 reconstructs a picture 10 from a data stream 12 into which the picture 10 is coded in units of slices 14 into which the picture 10 is partitioned, where the decoder is configured to decode the slices from the data stream in accordance with a slice order 16. Naturally, the decoder is not restricted to serially decode the slices. Rather, the decoder uses wavefront parallel processing in order to decode the slices, provided that the picture partitioned into slices is appropriate for wavefront parallel processing. In addition, the decoder is responsive to a syntax element portion 18 within a current slice of the slices, in order to decode the current slice in accordance with one of at least two modes 20, 22.SELECTED DRAWING: Figure 24

Description

[Technical Field]

[0001] The present invention relates to low-delay coding of images. [Background technology]

[0002] In the current HEVC design slices, entropy slice (formerly lightweight slice) tiles and WPP (wavefront parallel processing) are included as tools for parallelization.

[0003] For parallelization of video encoders and decoders, picture-level partitioning has several advantages over other approaches. In conventional video codecs, such as H.264 / AVC [1], picture partitioning is only possible with regular slices, which has a high cost in terms of coding efficiency. For scalable parallel H.264 / AVC decoding, it is necessary to combine macroblock-level parallelism for picture reconstruction and frame-level parallelism for entropy decoding. However, this approach results in limited reduction in picture latency and high memory usage. To overcome these limitations, new picture partitioning strategies are included in the HEVC codec. The current reference software version (HM-6) includes four different approaches: regular or normal slices, entropy slices, wavefront parallel processing (WPP) substreams, and tiles. Typically, these picture partitions include a set of largest coding units (LCUs), or, synonymously, coding tree units (CTUs), as defined in HEVC or a subset thereof.

[0004] 1 shows an image 898 in which every row 902 of LCUs or macroblocks in the image is illustratively arranged into a regular slice 900. Regular or normal slices (as defined in H.264 [1]) have the largest coding penalty, as they destroy entropy decoding and prediction dependencies. Entropy slicing, like slicing, breaks the entropy decoding dependency, but allows prediction (and filtering) to cross slice boundaries.

[0005] In WPP, image partitions are row-interleaved, and both entropy decoding and prediction are enabled to use data from blocks in other partitions. In this way, coding loss is minimized while simultaneously taking advantage of wavefront parallelism. However, interleaving disrupts bitstream causality, such that a conventional partition requires the next partition to be decoded.

[0006] 2 exemplarily shows an image 898 divided into two rows 904a, 904b of horizontally partitioning tiles 906. The tiles define horizontal 908 and vertical boundaries 910 that partition the image 898 into tile columns 912a, b, c and rows 904a, b. Like regular slices 900, tiles 906 break entropy decoding and prediction dependencies but do not require a header per tile.

[0007] For each of these techniques, the number of partitions can be freely chosen by the encoder. Generally, having more partitions results in higher compression loss. However, in WPP, loss propagation is not so high, and therefore the number of image partitions can even be fixed at one per row. This also brings several advantages. First, for WPP, bitstream causality is guaranteed. Second, the decoder implementation can assume a certain amount of parallelism is available, which also increases with resolution. Furthermore, and finally, neither context selection nor prediction dependencies need to be broken when decoding in wavefront order, resulting in relatively low coding loss.

[0008] However, to date, all-parallel encoding in transform concepts has not been able to achieve high compression efficiency while maintaining low latency. This is also true for WPP concepts: a slice is the smallest unit of transfer in the coding pipeline, and several WPP substreams still have to be transferred consecutively. Summary of the Invention [Problem to be solved by the invention]

[0009] It is therefore an object of the present invention to provide an image coding concept that allows for increased efficiency, e.g. by further reducing the end-to-end delay or by reducing the coding overhead consumed, thus enabling parallel decoding, e.g. according to wavefront parallelism. [Means for solving the problem]

[0010] This object is achieved by the subject matter of the independent claims.

[0011] The basic insight of the present invention is that if the normal slice concept, in which slices are coded / decoded completely independently of the regions of the image outside each slice, or at least independently of the regions outside each slice as far as entropy coding is concerned, is abandoned in favor of different modes of slices, i.e., some called dependent slices that allow interdependencies across slice boundaries, and others called normal slices that do not, then parallel processing concepts such as wavefront parallel processing can be realized with reduced end-to-end delay.

[0012] A further fundamental finding of the present invention, which can be combined with the first one or used individually, is that the WPP processing concept can be made more efficient if the starting syntax part of the slice is used to locate the WPP entry point.

[0013] Preferred embodiments of the present application are described below with reference to the drawings, advantageous embodiments are the subject matter of the dependent claims. [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 1 shows an image that is exemplary partitioned into regular slices by rows of LCUs or macroblocks in the image. [Figure 2] FIG. 2 shows an image that is exemplarily divided into two rows of horizontally partitioned tiles. [Figure 3] FIG. 3 exemplarily illustrates the allocation of parallel-encoded partitions to slices or network transport segments. [Figure 4] FIG. 4 shows a schematic diagram illustrating a general fragmentation of a frame in a tile coding approach for minimum end-to-end delay. [Figure 5] FIG. 5 shows a schematic diagram illustrating an exemplary fragmentation of a frame in a WPP coding approach for minimum end-to-end delay. [Figure 6]FIG. 6 shows a schematic block diagram illustrating a conversation scenario using video services. [Figure 7] FIG. 7 shows a schematic diagram of a possible time scheduling of encoding, transmission and decoding for tiles with a common subset with minimum end-to-end delay. [Figure 8] FIG. 8 shows a schematic timing schedule for achieving a common end-to-end delay. [Figure 9] FIG. 9 shows an image with an exemplary 11×9 coding treeblock partitioned into two slices. [Figure 10] FIG. 10 shows an image with an exemplary 13×8 coding treeblock partitioned into three tiles. [Figure 11a] FIG. 11a shows an example for the sequence parameter set syntax. [Figure 11b] FIG. 11b shows an example for the sequence parameter set syntax. [Figure 11c] FIG. 11c shows an example for the sequence parameter set syntax. [Figure 12a] FIG. 12a shows an example for an image parameter set syntax. [Figure 12b] FIG. 12b shows an example for the image parameter set syntax. [Figure 13a] Figure 13a shows an example for slice header syntax. [Figure 13b] Figure 13b shows an example for slice header syntax. [Figure 13c] Figure 13c shows an example for slice header syntax. [Figure 14] FIG. 14 illustrates partitioning of an image for WPP processing on regular slices and for low latency processing on dependent slices. [Figure 15] FIG. 15 shows examples for parts in the image parameter set syntax. [Figure 16] Figure 16 shows a possible slice header syntax. [Figure 17] FIG. 17 shows a schematic representation of the coding interdependencies for normal slices (and dependent slices). [Figure 18] FIG. 18 shows a schematic diagram comparing coding for low-latency transmission of tiles (wavefront parallelism with dependent slices). [Figure 19] FIG. 19 shows a timing schedule illustrating exemplary WPP encoding with pipelined low-latency transmission when using wavefront parallel processing with dependent slices as shown on the right side of FIG. [Figure 20] FIG. 20 shows a schematic diagram illustrating the robustness improvement by using regular slices as anchors. [Figure 21] FIG. 21 shows another embodiment for slice header syntax. [Figure 22] FIG. 22 shows another embodiment for the image parameter set syntax. [Figure 23] FIG. 23 shows a schematic diagram illustrating the symbol probability initialization process for dependent slices when starting at the left image boundary. [Figure 24] FIG. 24 shows a schematic diagram of the decoder. [Figure 25] Figure 25 shows a schematic diagram of the partitioning of an image into coding blocks and slices, as well as a block diagram of a decoder. [Figure 26] FIG. 26 shows a schematic block diagram of the encoder. [Figure 27] FIG. 27 shows schematically an image partitioned into normal and dependent slices, referred to herein as slice segments. [Figure 28] Figures 28a and 28b show schematically an image partitioned into normal and subordinate slices, here called slice segments on the one hand and tiles on the other hand. [Figure 29] FIG. 29 shows a flow diagram illustrating the context initialization process using dependent slices. [Figure 30]FIG. 30 shows a flow diagram illustrating the context storage process for using dependent slices. [Figure 31] Figure 31 shows schematically the different possibilities for signalling the WPP entry point. DETAILED DESCRIPTION OF THE INVENTION

[0015] In the following, the description begins with a description of current concepts for enabling parallel image processing and low-delay encoding, respectively. The problems that arise when one desires to have both capabilities are outlined. In particular, as will be seen from the following description, the WPP substream concept as taught so far contradicts the desire for low delay due to the need to transmit WPP substreams by grouping them into a slice. The following embodiments represent a parallel processing concept, such as the WPP concept, that can be applied to applications requiring even lower delay by extending the slice concept, i.e., by introducing another type of slice, later called dependent slice.

[0016] Minimizing the end-to-end video delay from capturing to display is one of the primary objectives in applications such as video conferencing.

[0017] The signal processing chain for digital video transmission consists of a camera, a capturing device, an encoder, an encapsulation stage, a transmission stage, a demultiplexer, a decoder, a renderer, and a display stage. Each of these stages contributes to the end-to-end delay by buffering image data before its serial transmission to the subsequent stage.

[0018] Some applications require minimization of such delays, for example remote manipulation of objects in hazardous areas without direct line of sight to the manipulated object, or minimally invasive surgery. Even short delays can result in serious problems with proper operation or even serious mistakes.

[0019] Often, entire video frames are buffered within a processing stage, for example to allow intra-frame processing. Some stages collect data to form packets that are sent to the next stage. There is generally a lower bound for delay resulting from local processing requirements. This is analyzed in more detail for each individual stage below.

[0020] In-camera processing does not necessarily require intra-frame signal processing, as the minimum delay is given by the sensor's integration time, which is limited by the frame rate, and some design choices made by the hardware manufacturer. The camera output is typically associated with a scan order, where processing usually begins in the upper left corner, moves to the upper right corner, and continues line by line to the lower right corner. As a result, it takes approximately one frame period for all data to be transferred from the sensor to the camera output.

[0021] Although a capturing device can send camera data immediately after receiving it, it typically buffers some data and generates bursts to optimize data access to memory or storage. Furthermore, the connection between the camera / capture and computer memory typically limits the bit rate at which captured image data can be sent to memory for further processing (encoding). Typically, cameras are connected via USB 2.0 or immediately USB 3.0, which always involves transferring image data partially to the encoder. This limits parallelization capabilities on the encoder side in extremely low-latency scenarios; the encoder tries to start encoding as soon as data becomes available from the camera, e.g., in raster scan order from the top to the bottom of the image.

[0022] In the encoder, there is a degree of freedom that allows one to trade off coding efficiency for reduced processing delay with respect to the data rate required for a particular video fidelity.

[0023] The encoder uses data already sent to predict a later-encoded image. Generally, the difference between the actual image and the prediction can be coded with fewer bits than would be required without the prediction. This prediction must be made available to the decoder; thus, the prediction is based on previously decoded portions of the same image (intra-frame prediction) or other previously processed images (inter-frame prediction). Pre-HEVC video coding standards only use portions of the image above or within the same line but to the left, which have been previously coded, for intra-frame prediction, motion vector prediction, and entropy coding (CABAC).

[0024] In addition to optimizing the prediction structure, the impact of parallel processing can be considered. Parallel processing requires the identification of image regions that can be processed independently. For practical reasons, contiguous regions, such as horizontal or vertical rectangles, are selected, often referred to as "tiles." Given low latency constraints, these regions should enable parallel encoding of data entering memory from capture as soon as possible. Given raster-scan memory transfer, vertical partitioning of raw data makes sense to begin encoding immediately. By dividing the image into vertical partitions (see the diagram below), intra-prediction, motion vector prediction, and entropy coding (CABAC) within such tiles can yield significant coding efficiency. To minimize latency, starting from the top, only portions of the image are transferred to the encoder's frame memory. Furthermore, parallel processing should begin in vertical tiles.

[0025] Another way to enable parallel processing is to use WPP within regular slices compared to tiles, where a "row" of tiles is contained in a single slice. Data within a slice can be coded in parallel using WPP substreams within the slice. Image separation into slices 900 and tiles / WPP substreams 914 is shown in example Figure 3 / 1.

[0026] Thus, FIG. 3 illustrates the allocation of parallel coded partitions, such as 906 or 914, to slices or network transport segments (single network packets or multiple network 900 packets).

[0027] The encapsulation of coded data into Network Abstraction Layer (NAL) units, as defined in H.264 or HEVC, if applicable, adds several headers to the data blocks before transmission or during the coding process, which allow identification of each block and reordering of the blocks. In the case of the standard, no additional signaling is required since the order of the coding elements is always the decoding order, which is given by an implicit assignment of the position of tiles or general coded fragments.

[0028] If parallelism is considered with an additional transport layer for low-latency parallel transmission, i.e., the transport layer can reorder the image partitions for tiles to enable low-latency transmission, which means sending fragments as they are encoded, as shown in Figure 4. Those fragments may be slices that are not fully encoded, they may be subsets of slices, or they may be included in dependent slices.

[0029] When creating additional fragments, there is a trade-off between efficiency, which is highest for large data blocks due to header information adding a certain number of bytes, and delay due to the parallel encoder's need to buffer large data blocks before transmission. The overall delay can be reduced if the coded representation of a vertical tile 906 is separated into multiple fragments 916 that are transmitted as soon as the fragment is fully coded. The size of each fragment can be determined in terms of a fixed image area, such as a macroblock, LCU, etc., or in terms of a maximum data size as shown in FIG. 4.

[0030] Thus, FIG. 4 shows a general fragmentation of a frame in a tile coding approach for minimum end-to-end delay.

[0031] Similarly, FIG. 5 shows the fragmentation of a frame in the WPP coding approach for minimum end-to-end delay.

[0032] The transmission may add further delays, for example, when additional block-oriented processing is applied, such as forward error correction codes, to increase the robustness of the transmission. Also, the network infrastructure (such as routers) or physical links may add delays, which are typically known as the latency for the connection. In addition to the latency, the transmission bit rate determines the time (latency) required to transfer data from participant a to participant b in a conversation using video services, such as that shown in Figure 6.

[0033] If encoded data blocks are transmitted out of order, reordering delays must be taken into account: decoding can begin as soon as a data unit arrives, assuming that other data units that must be decoded before this one are available.

[0034] In the case of tiles, there are no dependencies between tiles, so tiles can be decoded immediately. If fragments are made up of tiles, such as a separate slice per fragment as shown in Figure 4, the fragments can be forwarded directly as soon as they are each coded and their contained LCUs or CUs are coded.

[0035] The renderer assembles the outputs of the parallel decoding engines and sends the combined image line by line to the display.

[0036] The display does not necessarily add any delay, but may in fact do some intra-frame processing before the image data is actually displayed - this is a design choice made by the hardware manufacturer.

[0037] In summary, we can influence the stages encoding, encapsulation, transmission, and decoding to achieve the minimum end-to-end delay. If we use parallel processing, tiles, and fragmentation within tiles, the overall delay can be significantly reduced as shown in Figure 7, compared to commonly used processing chains that add approximately one frame delay at each of these stages as shown in Figure 8.

[0038] In particular, FIG. 7 shows the encoding, transmission and decoding for tiles with a generic subset with minimum end-to-end delay, while FIG. 8 shows the commonly achieved end-to-end delay.

[0039] HEVC further enables the use of slice partitioning, tile partitioning in the following ways:

[0040] Tile: An integer number of treeblocks occurring together in one column and one row, and ordered consecutively in a treeblock raster scan of the tile. The division of each image into tiles is a partitioning. In an image, tiles are ordered consecutively in a tile raster scan of the image. Although a slice contains treeblocks that are consecutive in a treeblock raster scan of the tile, these treeblocks are not necessarily consecutive in a treeblock raster scan of the image.

[0041] Slice: An integer number of treeblocks ordered consecutively in a raster scan. The division of each image into slices is a partitioning. Treeblock addresses are derived from the address of the first treeblock in the slice (as represented in the slice header).

[0042] Raster Scan: The mapping of a rectangular two-dimensional pattern onto a one-dimensional pattern such that the first entry in the one-dimensional pattern is the first row at the top of the two-dimensional pattern, scanned from left to right, followed similarly by the second, third, etc. rows of the pattern, each scanned (down) from left to right.

[0043] Treeblock: An NxN block of luma samples and two corresponding blocks of chroma samples for an image with three sample arrays, or an NxN block of samples for a monochrome image or an image coded with three separate color planes. The division of a slice into treeblocks is a partitioning.

[0044] Partitioning: The division of a set into subsets such that each element of the set is in exactly one of the subsets.

[0045] Quadtree: A tree in which a parent node can be split into four child nodes, and a child node can become a parent node for another split into four child nodes.

[0046] In the following, spatial subdivision of images, slices, and tiles is described. In particular, the following description specifies how an image is partitioned into slices, tiles, and coding tree blocks. An image is divided into slices and tiles. A slice is a sequence of coding tree blocks. Similarly, a tile is a sequence of coding tree blocks.

[0047] Samples are processed in units of coding treeblocks. The luma array size per treeblock in samples, both width and height, is CtbSize. The width and height of the chroma array per coding treeblock are CtbWidthC and CtbHeightC, respectively. For example, an image may be divided into two slices as shown in the following figure. As another example, an image may be divided into three tiles as shown in the second below figure.

[0048] Unlike slices, tiles are always rectangular and always contain an integer number of coding tree blocks in a coding tree block raster scan. A tile can consist of coding tree blocks contained in more than one slice. Similarly, a slice can consist of coding tree blocks contained in more than one tile.

[0049] FIG. 9 shows an image 898 having an 11x9 coding treeblock 918 partitioned into two slices 900a,b.

[0050] FIG. 10 shows an image with a 13x8 coding tree block 918 that is partitioned into three tiles.

[0051] Each coding 898 tree block 918 is assigned a partition signaling to identify the block size for intra or inter prediction and for transform coding. The partitioning is a recursive quad-tree partitioning. The root of the quad-tree is associated with the coding tree block. The quad-tree is split until it reaches a leaf, which is called a coding block. The coding block is the root node of two trees, the prediction tree and the transform tree.

[0052] The prediction tree specifies the location and size of the prediction block. A prediction block and associated prediction data is called a prediction unit.

[0053] FIG. 11 shows an exemplary sequence parameter set RBSP syntax.

[0054] The transform tree specifies the location and size of the transform blocks. A transform block and its associated transform data is called a transform unit.

[0055] The split information for luma and chroma is the same as the prediction tree, and may or may not be the same as the transform tree.

[0056] A coding block, associated coded data, and associated prediction and transform units together form a coding unit.

[0057] The process for the conversion of coding tree block addresses in coding tree block raster order to tile scan order may be as follows: The output of this process is - an array CtbAddrTS[ctbAddrRS] with ctbAddrRS in the range 0 to PicHeightInCtbs*PicWidthInCtbs-1; - An array TileId[ctbAddrTS] with ctbAddrTS in the range 0 to PicHeightInCtbs*PicWidthInCtbs-1. The array CtbAddrTS[] is derived as follows: TIFF0007769162000001.tif62155

[0058] The array TileId[] is derived as follows: TIFF0007769162000002.tif29134

[0059] Corresponding example syntaxes are shown in Figures 11, 12 and 13, where Figure 12 has an example picture parameter set RBSP syntax and Figure 13 shows an example slice header syntax.

[0060] In the syntax examples, the following semantics may apply:

[0061] entropy_slice_flag equal to 1 specifies that the value of the absent slice header syntax element is inferred to be equal to the value of the slice header syntax element in the proceeding slice, where the proceeding slice is defined as the slice containing the coding tree block with position (SliceCtbAddrRS-1). entropy_slice_flag is equal to 0 when SliceCtbAddrRS is equal to 0.

[0062] tiles_or_entropy_coding_sync_idc equal to 0 specifies that there is only one tile in each image in the coded video sequence, and furthermore, no specific synchronization process for the context variables is invoked before decoding the first coding tree block in a row of coding tree blocks.

[0063] tiles_or_entropy_coding_sync_idc equal to 1 specifies that there may be more than one tile in each image in the coded video sequence, and furthermore, no specific synchronization process for the context variables is invoked before decoding the first coding tree block in a row of coding tree blocks.

[0064] tiles_or_entropy_coding_sync_idc equal to 2 specifies that there is only one tile in each image in the coded video sequence, and a specific synchronization process for the context variables is invoked before decoding the first coding tree block in a row of coding tree blocks, and a specific storage process for the context variables is invoked after decoding two coding tree blocks in a row of coding tree blocks.

[0065] The value of tiles_or_entropy_coding_sync_idc is in the range 0 to 2.

[0066] num_tile_columns_minus1+1 specifies the number of tile columns into which the image is partitioned.

[0067] num_tile_rows_minus1+1 specifies the number of tile rows to partition the image into. When num_tile_columns_minus1 is equal to 0, num_tile_rows_minus1 is not equal to 0.

[0068] One or both of the following conditions are met for each slice and tile: - All coded blocks in a slice belong to the same tile. - All coded blocks in a tile belong to the same slice.

[0069] NOTE - Within the same image, there may be both slices containing multiple tiles and tiles containing multiple slices.

[0070] uniform_spacing_flag equal to 1 specifies that column borders, and similarly row borders, are uniformly distributed across the image. uniform_spacing_flag equal to 0 specifies that column borders, and similarly row borders, are not uniformly distributed across the image, but are explicitly indicated using the syntax elements column_width[i] and row_height[i].

[0071] column_width[i] specifies the width of the i-th tile column in units of coding tree blocks.

[0072] row #height[i] specifies the height of the ith tile row in units of coding tree blocks. The value of ColumnWidth[i], which specifies the width of the ith tile column in units of coding tree blocks, and the value of ColumnWidthInLumaSamples[i], which specifies the width of the ith tile column in units of luma samples, are derived as follows: TIFF0007769162000003.tif40155

[0073] The value of RowHeight[i], which specifies the height of the ith tile row in units of coding treeblocks, is derived as follows: TIFF0007769162000004.tif34161

[0074] The value of ColBd[i], which specifies the left column boundary position of the ith tile column in terms of the coding tree block, is derived as follows: TIFF0007769162000005.tif13140

[0075] The value of RowBd[i], which specifies the location of the top row boundary of the ith tile row in terms of the coding treeblock, is derived as follows: TIFF0007769162000006.tif12140

[0076] num_substreams_minus1+1 specifies the maximum number of subsets included in a slice when tiles_or_entropy_coding_sync_idc is equal to 2. When not present, the value of num_substreams_minus1 is inferred to be equal to 0.

[0077] num_entry_point_offsets specifies the number of entry_point_offset[i] syntax elements in the slice header. When tiles_or_entropy_coding_sync_idc is equal to 1, the value of num_entry_point_offsets ranges from 0 to (num_tile_columns_minus1+1)*(num_tile_rows_minus1+1)-1. When tiles_or_entropy_coding_sync_idc is equal to 2, the value of num_entry_point_offsets ranges from 0 to num_substreams_minus1. When not present, the value of num_entry_point_offsets is inferred to be equal to 0.

[0078] offset_len_minus1+1 specifies the length, in bits, of the entry_point_offset[i] syntax element.

[0079] entry_point_offset[i] specifies the ith entry point offset in bytes and is represented by offset_len_minus1+1 bits. A coded slice NAL unit consists of num_entry_point_offsets+1 subsets, with subset index values ​​ranging from 0 to num_entry_point_offsets. Subset 0 consists of bytes 0 to entry_point_offset[0]-1 of the coded slice NAL unit, subset k, with k in the range 1 to num_entry_point_offsets-1, consists of bytes entry_point_offset[k-1] to entry_point_offset[k]+entry_point_offset[k-1]-1 of the coded slice NAL unit, and the last subset (with subset index equal to num_entry_point_offsets) consists of the remaining bytes of the coded slice NAL unit.

[0080] NOTE - NAL unit headers and slice headers of coded slice NAL units are always included in subset 0.

[0081] When tiles_or_entropy_coding_sync_idc is equal to 1 and num_entry_point_offsets is greater than 0, each subset contains all coded bits of one or more complete tiles, and the number of subsets is less than or equal to the number of tiles in the slice. When tiles_or_entropy_coding_sync_idc is equal to 2 and num_entry_point_offsets is greater than 0, the subset k includes all bits used during the initialization process for the current bitstream pointer k, for each of all possible k values.

[0082] Regarding slice data semantics, the following may apply:

[0083] An end_of_slice_flag equal to 0 indicates that another macroblock follows in the slice. An end_of_slice_flag equal to 1 indicates the end of the slice, and no further macroblocks follow.

[0084] entry_point_marker_two_3bytes is a 3-byte fixed-value sequence equal to 0x000002. This syntax element is called the entry marker prefix.

[0085] tile_idx_minus_1 specifies the TileID in raster scan order. The first tile in an image has a TileID of 0. The value of tile_idx_minus_1 ranges from 0 to (num_tile_columns_minus1+1)*(num_tile_rows_minus1+1)-1.

[0086] The CABAC analysis process for slice data may be as follows:

[0087] This process is invoked when parsing a syntax element with descriptor ae(v).

[0088] The inputs to this process are the value of the syntax element and a request for the value of the conventional parsed syntax element.

[0089] The output of this process is the value of the syntax element.

[0090] The initialization process of the CABAC parsing process is called when starting to parse the slice data of a slice. When tiles_or_entropy_coding_sync_idc is equal to 2 and num_substreams_minus1 is greater than 0, a mapping table BitStreamTable with num_substreams_minus1+1 entries that specify the bitstream pointer table to use for subsequent current bitstream pointer derivation is derived as follows: - BitStreamTable[0] contains the initialized bitstream pointer. - For every index i greater than 0 and less than num_substreams_minus1+1, BitStreamTable[i] contains a bitstream pointer to entry_point_offset[i] bytes after BitStreamTable[i-1]. The current bitstream pointer is set to BitStreamTable[0].

[0091] The minimum coding block address of the coding tree block containing the spatial neighboring block T, ctbMinCbAddrT, is derived using the position (x0, y0) of the top-left luma sample of the current coding tree block, for example, as follows: x=x0+2< <Log2CtbSize-1 y=y0-1 ctbMinCbAddrT=MinCbAddrZS[x>>Log2MinCbSize][y>>Log2MinCbSize]

[0092] The variable availableFlagT is obtained by invoking the appropriate coding block availability derivation process with ctbMinCbAddrT as input. Start parsing the coding tree and when tiles_or_entropy_coding_sync_idc is equal to 2 and num_substreams_minus1 is greater than 0, the following applies: - If CtbAddrRS%PicWidthInCtbs is equal to 0, the following applies: When availableFlagT is equal to 1, the synchronization process of the CABAC parsing process is invoked as specified in the subclause "Synchronization process for context variables". - The decoding process for binary decisions before termination is called, followed by an initialization process for the arithmetic decoding engine. - The current bitstream pointer is set to point to BitStreamTable[i] with index i, which is derived as follows: i=(CtbAddrRS / PicWidthInCtbs)%(num_substreams_minus1+1) Otherwise, if CtbAddrRS%PicWidthInCtbs is equal to 2, the storage process of the CABAC parsing process is invoked as specified in the subclause "Storage process for context variables".

[0093] The initialization process can be as follows:

[0094] The output of this process is the initialized CABAC internal variables.

[0095] That particular process is invoked when starting to parse slice data for a slice, or when starting to parse data for a coding tree and the coding tree is the first coding tree in a tile.

[0096] The storage process for the context variables can be as follows:

[0097] The input to this process is the CABAC context variable indexed by ctxIdx.

[0098] The output of this process is the variables TableStateSync and TableMPSSync, which contain the values ​​of the variables m and n used in the initialization process of the context variables assigned to the syntax elements except for the end-of-slice flag. For each context variable, the corresponding entries n and m in the tables TableStateSync and TableMPSSync are initialized to the corresponding pStateIdx and valMPS.

[0099] The synchronization process for context variables can be as follows:

[0100] The inputs to this process are the variables TableStateSync and TableMPSSync, which contain the values ​​of the variables n and m used in the storage process of the context variables assigned to the syntax elements except for the end flag of the slice.

[0101] The output of this process is a CABAC context variable indexed by ctxIdx.

[0102] For each context variable, the corresponding context variables pStateIdx and valMPS are initialized to the corresponding entries n and m of the tables TableStateSync and TableMPSSync.

[0103] In the following, low-delay encoding and transmission using WPP is described. In particular, the following description clarifies how low-delay transmission as described in FIG. 7 can be applied to WPP.

[0104] First of all, it is important that a subset of the image can be sent before the entire image is completed. Typically, this can be achieved using slices, as already shown in FIG.

[0105] To reduce delay compared to tiles, it is necessary to apply a single WPP sub-bitstream per row of an LCU and further enable separate transmission of each of those rows, as shown in the figure below. To keep coding efficiency high, slices per row / substream cannot be used. Therefore, in the following, so-called dependent slices are introduced, as defined in the next section. These slices, for example, have fields used for entropy slicing, but not all fields of the full HEVC slice header. Furthermore, there may be a switch to turn off CABAC breaking between rows. In the case of WPP, the use of CABAC context (arrows in Figure 14) and row prediction are enabled to preserve the coding efficiency gain of WPP over tiles.

[0106] In particular, FIG. 14 illustrates an image 10 for WPP to a regular slice 900 (regular SL) and for low latency processing to a dependent slice (DS) 920.

[0107] Currently, the upcoming HEVC standard provides two types of partitioning for slices: regular (normal) slices and entropy slices. Regular slices are completely independent image partitions except for some dependency that can be exploited to deblock the filter process at slice boundaries. Entropy slices are independent only with respect to entropy coding. The idea of ​​Figure 14 is to generalize the slicing concept. Thus, the upcoming HEVC standard should provide two general types of slices: independent (regular) or dependent. Therefore, a new type of slice is introduced: dependent slices.

[0108] A dependent slice is a slice that has a dependency on a previous slice, where the dependency is specific data available between slices in the entropy decoding and / or pixel reconstruction process.

[0109] The concept of dependent slices is exemplarily shown in FIG. 14. An image, for example, always starts with a regular slice. It should be noted that in this concept, the behavior of regular slices is slightly modified. Typically, in standards such as H264 / AVC or HEVC, regular slices are completely independent partitions and, furthermore, do not need to retain any data after decoding, except for some data for the deblocking filter process. However, processing of this next dependent slice 920 is only possible by referencing the data of the regular slice 900 mentioned above, here in the first row. To establish this, the regular slice 900 should retain the data of the last CU row. This data is - CABAC encoding engine data (context model state of one CU in which the entropy decoding process of a dependent slice can be initialized); - all decoded syntax elements of a CU for the regular CABAC decoding process of a subordinate CU, - Intra and motion vector prediction data Includes.

[0110] Consequently, each subordinate slice 920 should follow the same procedure, preserving the data for this next subordinate slice in the same image.

[0111] In practice, these additional steps should not be a problem, since the decoding process is generally always forced to store some data such as syntax elements.

[0112] In the following sections, possible changes to the HEVC standard syntax that would be required to allow for the concept of dependent slices are presented.

[0113] FIG. 5, for example, shows possible changes in the picture parameter set RBSP syntax.

[0114] The picture parameter set semantics for the dependent slices may be as follows:

[0115] A dependent_slices_present_flag equal to 1 specifies that the image contains dependent slices, and the decoding process of each (regular or dependent) slice should store the state of entropy decoding and the data of intra and motion vector prediction for the next slice, which may be a dependent slice that may follow a regular slice. The following dependent slices can refer to the stored data.

[0116] Figure 16 shows a possible slice_header syntax with changes relevant to the current state of HEVC.

[0117] dependent_slice_flag equal to 1 specifies that the value of the absent slice header syntax element is inferred to be equal to the value of the slice header syntax element in a progressive (regular) slice, where a progressive slice is defined as the slice containing a coding tree block with position (SliceCtbAddrRS-1). dependent_slice_flag is equal to 0 when SliceCtbAddrRS is equal to 0.

[0118] no_cabac_reset_flag equal to 1 specifies CABAC initialization from the saved state of the previously decoded slice (which has no initialization value). Otherwise, i.e., equal to 0, specifies CABAC initialization independent of any state of the previously decoded slice, i.e., with initialization value.

[0119] last_ctb_cabac_init_flag equal to 1 specifies CABAC initialization from the saved state of the last coded tree block of the previously decoded slice (e.g., for tiles, it is always equal to 1). Otherwise (if equal to 0), the initialization data is referenced from the saved state of the second coded tree block of the last (adjacent) ctb-row of the previously decoded slice if the first coded tree block of the current slice is the first coded tree block in the row (i.e., WPP mode); otherwise, the CABAC initialization is pre-formed from the saved state of the last coded tree block of the previously decoded slice.

[0120] A comparison of dependent slices and other partitioning schemes (information) is provided below.

[0121] In FIG. 17 the difference between normal and dependent slices is shown.

[0122] As shown in Figure 18, the possible encoding and transmission of WPP substreams in a dependent slice (DS) is comparable to the encoding for tile (left) and WPP / DS (right) low-latency transmission. The thick, continuous crosses in Figure 18 indicate the same point in time for the two methods, assuming that encoding a WPP row takes the same time as encoding a single tile. Due to the encoding dependency, only the first row of a WPP is prepared after all tiles have been encoded. However, using the dependent slice approach, the WPP approach allows the first row to be sent as soon as it is encoded. This differs from the earlier WPP substream allocation, where a "substream" is defined for WPP as the concatenation of CU rows of a slice that are WPP decoded by the same decoder thread, i.e., the same core / processor. However, row-by-row and entropy slice-by-entropy slice substreams are also possible before entropy slicing breaks the entropy encoding dependency and therefore has lower coding efficiency, i.e., WPP efficiency gains are lost.

[0123] In addition, the delay difference between both approaches can be made really low, assuming a transmission as shown in Figure 19. In particular, Figure 19 shows WPP encoding with pipelined low-delay transmission.

[0124] Assuming that the encoding of the two CUs after DS#1.1 in the WPP approach in Figure 18 is not longer than the transmission of the first row SL#1, there is no difference between tile and WPP in the low-delay case. However, the encoding efficiency of WP / DS outperforms the tile concept.

[0125] To increase robustness for the WPP low latency mode, FIG. 20 shows that robustness improvement is achieved by using a regular slice (RS) as an anchor. In the image shown in FIG. 20, a (regular) slice (RS) is followed by a dependent slice (DS). Here, the (regular) slice acts as an anchor that breaks the dependency on the previous slice, so that more robustness is provided at such an insertion point of the (regular) slice. In principle, this is no different from inserting a (regular) slice anyway.

[0126] The concept of dependent slices can also be implemented as follows.

[0127] Here, Figure 21 shows a possible slice header syntax.

[0128] Slice header semantics are as follows:

[0129] dependent_slice_flag equal to 1 specifies that the value of each absent slice header syntax element is inferred to be equal to the value of the corresponding slice header syntax element in the previous slice containing the coding tree block whose coding tree block address is SliceCtbAddrRS-1. When absent, the value of dependent_slice_flag is inferred to be equal to 0. The value of dependent_slice_flag is equal to 0 when SliceCtbAddrRS is equal to 0.

[0130] slice_address specifies the address in slice granularity resolution where the slice starts. The slice_address syntax element is (Ciel(Log2(PicWidthInCtbs*PicHeightInCtbs))+SliceGranularity) bits in length.

[0131] The variable SliceCtbAddrRS, which indicates the coding tree block where the slice starts in coding tree block raster scan order, is derived as follows: SliceCtbAddrRS=(slice_address>>SliceGranularity)

[0132] The variable SliceCbAddrZS, which denotes the address of the first coding block in the slice at the smallest coding block granularity in z-scan order, is derived as follows: SliceCbAddrZS=slice_address <<((log2_diff_max_min_coding_block_size-SliceGranularity)<<1)

[0133] Slice decoding starts at the largest possible coding unit, or in other terms, at the slice start coordinate, CTU.

[0134] first_slice_in_pic_flag indicates whether the slice is the first slice of the picture. If first_slice_in_pic_flag is equal to 1, the variables SliceCbAddrZS and SliceCtbAddrRS are both set to 0, and decoding starts at the first coding tree block in the picture.

[0135] The pic_parameter_set_id specifies the picture parameter set in use. The value of pic_parameter_set_id is in the range 0 to 255.

[0136] num_entry_point_offsets specifies the number of entry_point_offset[i] syntax elements in the slice header. When tiles_or_entropy_coding_sync_idc is equal to 1, the value of num_entry_point_offsets ranges from 0 to (num_tile_columns_minus1+1)*(num_tile_rows_minus1+1)-1. When tiles_or_entropy_coding_sync_idc is equal to 2, the value of num_entry_point_offsets ranges from 0 to PicHeightInCtbs-1. When not present, the value of num_entry_point_offsets is inferred to be equal to 0.

[0137] offset_len_minus1+1 specifies the length, in bits, of the entry_point_offset[i] syntax element.

[0138] entry_point_offset[i] specifies the i-th entry point offset in bytes and is represented by offset_len_minus1+1 bits. The coded slice data after the slice header consists of num_entry_point_offsets+1 subsets, with subset index values ​​ranging from 0 to num_entry_point_offsets. Subset 0 consists of bytes 0 to entry_point_offset[0]-1 of the coded slice data, subset k, with k in the range 1 to num_entry_point_offsets-1, consists of bytes entry_point_offset[k-1] to entry_point_offset[k]+entry_point_offset[k-1]-1 of the coded slice data, and the last subset (with subset index equal to num_entry_point_offsets) consists of the remaining bytes of the coded slice data.

[0139] When tiles_or_entropy_coding_sync_idc is equal to 1 and num_entry_point_offsets is greater than 0, each subset contains all coded bits of exactly one tile, and the number of subsets (i.e., the value of num_entry_point_offsets+1) is less than or equal to the number of tiles in the slice.

[0140] Note - When tiles_or_entropy_coding_sync_idc is equal to 1, each slice must contain either a subset of one tile (if entry point signaling is unnecessary) or an integer number of complete tiles.

[0141] When tiles_or_entropy_coding_sync_idc is equal to 2 and num_entry_point_offsets is greater than 0, each subset k, with k in the range from 0 to num_entry_point_offsets-1, contains all coded bits of exactly one row of the coding tree block, and the last subset (with subset index equal to num_entry_point_offsets) contains all coded bits of the remaining coding blocks contained in the slice, which consist of exactly one row of the coding tree block or subsets of one row of the coding tree block, and the number of subsets (i.e., the value of num_entry_point_offsets+1) is equal to the number of rows of the coding tree block in the slice, and subsets of one row of the coding tree block in the slice are also counted.

[0142] NOTE A slice may contain multiple rows of coding tree blocks and subsets of rows of coding tree blocks when tiles_or_entropy_coding_sync_idc is equal to 2. For example, if a slice contains 2.5 rows of coding tree blocks, the number of subsets (i.e., the value of num_entry_point_offsets+1) is equal to 3.

[0143] The corresponding image parameter set RBSP syntax can be selected as shown in FIG.

[0144] Image parameter set RBSP semantics can be as follows:

[0145] dependent_slice_enabled_flag equal to 1 specifies the presence of the syntax element dependent_slice_flag in the slice header for a coded picture that references a picture parameter set. dependent_slice_enabled_flag equal to 0 specifies the absence of the syntax element dependent_slice_flag in the slice header for a coded picture that references a picture parameter set. The value of dependent_slice_enabled_flag is equal to 1 when tiles_or_entropy_coding_sync_idc is equal to 3.

[0146] tiles_or_entropy_coding_sync_idc equal to 0 specifies that there is only one tile in each image referencing the picture parameter set, there is no specific synchronization process for context variables that is invoked before decoding the first coding tree block of a row of coding tree blocks in each image referencing the picture parameter set, and further the values ​​of cabac_independent_flag and dependent_slice_flag for the coded images referencing the picture parameter set are not both equal to 1.

[0147] Note that when cabac_independent_flag and dependent_slice_flag are both equal to 1 for a slice, the slice is an entropy slice.

[0148] tiles_or_entropy_coding_sync_idc equal to 1 specifies that there may be more than one tile in each image that references the picture parameter set, and there is no specific synchronization process for context variables that is invoked before decoding the first coding tree block of a row of coding tree blocks in each image that references the picture parameter set, and further, the values ​​of cabac_independent_flag and dependent_slice_flag for coded images that reference the picture parameter set are not both equal to 1.

[0149] tiles_or_entropy_coding_sync_idc equal to 2 specifies that there is only one tile in each image that references the picture parameter set, a specific synchronization process for the context variables is invoked before decoding the first coding tree block of a row of coding tree blocks in each image that references the picture parameter set, and a specific storage process for the context variables is invoked after decoding two coding tree blocks of a row of coding tree blocks in each image that references the picture parameter set, and the values ​​of cabac_independent_flag and dependent_slice_flag for coded images that reference the picture parameter set are not both equal to 1.

[0150] tiles_or_entropy_coding_sync_idc equal to 3 specifies that there is only one tile in each image referencing the picture parameter set, and there is no specific synchronization process for context variables that is invoked before decoding the first coding tree block of a row of coding tree blocks in each image referencing the picture parameter set; furthermore, the values ​​of cabac_independent_flag and dependent_slice_flag for coded images referencing the picture parameter set may both be equal to 1.

[0151] When dependent_slice_enabled_flag is equal to 0, tiles_or_entropy_coding_sync_idc is not equal to 3.

[0152] It is a bitstream conformance requirement that the value of tiles_or_entropy_coding_sync_idc be the same for all picture parameter sets activated within a coded video sequence.

[0153] For each slice that references the picture parameter set, when tiles_or_entropy_coding_sync_idc is equal to 2 and the first coding block in the slice is not the first coding block in the first coding tree block of a row of coding tree blocks, the last coding block in the slice belongs to the same row of coding tree blocks as the first coding block in the slice.

[0154] num_tile_columns_minus1+1 specifies the number of tile columns into which the image is partitioned. num_tile_rows_minus1+1 specifies the number of tile rows into which the image is partitioned.

[0155] When num_tile_columns_minus1 is equal to 0, num_tile_rows_minus1 is not equal to 0. uniform_spacing_flag equal to 1 specifies that the column borders, and similarly the row borders, are uniformly distributed across the image. uniform_spacing_flag equal to 0 specifies that the column borders, and similarly the row borders, are not uniformly distributed across the image, but are explicitly indicated with the syntax elements column_width[i] and row_height[i].

[0156] column_width[i] specifies the width of the i-th tile column in units of coding tree blocks.

[0157] row_height[i] specifies the height of the ith tile row in units of coding tree blocks.

[0158] The vector colWidth[i] specifies the width of the ith tile column in units of CTBs with column i ranging from 0 to num_tile_columns_minus1.

[0159] The vector CtbAddrRStoTS[ctbAddrRS] specifies the correspondence from CTB addresses in raster scan order to CTB addresses in tile scan order, with index ctbAddrRS ranging from 0 to (picHeightInCtbs*picWidthInCtbs)-1.

[0160] The vector CtbAddrTStoRS[ctbAddrTS] specifies the correspondence from CTB addresses in tile scan order to CTB addresses in raster scan order, with index ctbAddrTS ranging from 0 to (picHeightInCtbs*picWidthInCtbs)-1.

[0161] The vector TileId[ctbAddrTS] specifies the correspondence from CTB address to tile id in tile scan order, with ctbAddrTS ranging from 0 to (picHeightInCtbs*picWidthInCtbs)-1.

[0162] The values ​​for colWidth, CtbAddrRStoTS, CtbAddrTStoRS and TileId are derived by calling a tile scanning conversation process with the CTB raster and PicHeightInCtbs and PicWidthInCtbs as inputs, and the output is assigned to colWidth, CtbAddrRStoTS and TileId.

[0163] The value of ColumnWidthInLumaSamples[i], which specifies the width of the ith tile column in units of luma samples, is set equal to colWidth[i], and colWidth[i]< <Log2CtbSizeである。

[0164] The array MinCbAddrZS[x][y], which specifies the correspondence from location (x, y) to the minimum CB address in z scan order in units of minimum CB, with x ranging from 0 to picWidthInMinCbs-1 and y ranging from 0 to picHeightInMinCbs-1, is derived by invoking the Z scanning order array initialization process with Log2MinCbSize, Log2CtbSize, PicHeightInCtbs, PicWidthInCtbs and the vector CtbAddrRStoTS as inputs, and the output is assigned to MinCbAddrZS.

[0165] loop_filter_across_tiles_enabled_flag equal to 1 specifies that in-loop filtering operations are performed across tile boundaries. loop_filter_across_tiles_enabled_flag equal to 0 specifies that in-loop filtering operations are not performed across tile boundaries. In-loop filtering operations include deblocking filter, sample adaptive offset, and adaptive loop filter operations. When not present, the value of loop_filter_across_tiles_enabled_flag is inferred to be equal to 1.

[0166] cabac_independent_flag equal to 1 specifies that CABAC decoding of coded blocks in a slice is independent of any state of previously decoded slices. cabac_independent_flag equal to 0 specifies that CABAC decoding of coded blocks in a slice is dependent on the state of previously decoded slices. When not present, the value of cabac_independent_flag is inferred to be equal to 0.

[0167] The derivation process for the availability of the coding block with the smallest coding block address may be as follows:

[0168] The inputs to this process are: - the minimum coded block address in z-scan order, minCbAddrZS - current minimum coded block address in z-scan order, currMinCBAddrZS is.

[0169] The output of this process is the availability of the coding block with the smallest coding block address cbAddrZS in the z-scan order cbAvailable.

[0170] NOTE 1 - The meaning of availability is determined when this process is invoked. NOTE 2 - Any coding block, regardless of its size, is associated with a minimum coding block address, which is the address of the coding block with the minimum coding block size in the z-scan order. - cbAvailable is set to false if one or more of the following conditions are true: - minCbAddrZS is less than 0 - minCbAddrZS is greater than currMinCBAddrZS - the coding block with the minimum coding block address minCbAddrZS belongs to a different slice than the coding block with the current minimum coding block address currMinCBAddrZS, and furthermore, the dependent_slice_flag of the slice containing the coding block with the current minimum coding block address currMinCBAddrZS is equal to 0. - The coding block with the minimum coding block address minCbAddrZS is included in a different tile than the coding block with the current minimum coding block address currMinCBAddrZS. - Otherwise, cbAvailable is set to true.

[0171] The CABAC analysis process for slice data may be as follows:

[0172] This process is invoked when parsing a particular syntax element with descriptor ae(v).

[0173] The inputs to this process are the value of a syntax element and a request for the value of a conventional parsed syntax element.

[0174] The output of this process is the value of the syntax element.

[0175] When starting to analyze the slice data for a slice, the initialization process of the CABAC analysis process is called.

[0176] FIG. 23 shows how the spatial neighborhood T is used to invoke the coding tree block availability derivation process associated with the current coding tree block (information).

[0177] The minimum coding block address of the coding tree block containing the spatial neighboring block T (Figure 23), ctbMinCbAddrT, is derived using the position (x0, y0) of the top-left luma sample of the current coding tree block as follows: x=x0+2< <Log2CtbSize-1 y=y0-1 ctbMinCbAddrT=MinCbAddrZS[x>>Log2MinCbSize][y>>Log2MinCbSize]

[0178] The variable availableFlagT is obtained by invoking the coding block availability derivation process with ctbMinCbAddrT as input.

[0179] When starting to parse the coding tree as specified, the following ordered steps are applied:

[0180] The arithmetic decoding engine is initialized as follows:

[0181] If CtbAddrRS is equal to slice_address, dependent_slice_flag is equal to 1, and entropy_coding_reset_flag is equal to 0, then the following applies: The CABAC parsing process synchronization process is called with TableStateIdxDS and TableMPSValDS as inputs. The decoding process for binary decisions before termination is invoked, followed by an initialization process for the arithmetic decoding engine.

[0182] Otherwise, if tiles_or_entropy_coding_sync_idc is equal to 2 and CtbAddrRS % PicWidthInCtbs is equal to 0, then the following applies: When availableFlagT is equal to 1, the CABAC parsing process synchronization process is called with TableStateIdxWPP and TableMPSValWPP as inputs. The decoding process for the binary decisions before termination is called, followed by the process for the arithmetic decoding engine.

[0183] When cabac_independent_flag is equal to 0 and dependent_slice_flag is equal to 1, or tiles_or_entropy_coding_sync_idc is equal to 2, the storage process is applied as follows: When tiles_or_entropy_coding_sync_idc is equal to 2 and CtbAddrRS%PicWidthInCtbs is equal to 2, the storage process of the CABAC analysis process is called with TableStateIdxWPP and TableMPSValWPP as outputs. When cabac_independent_flag is equal to 0, dependent_slice_flag is equal to 1, and end_of_slice_flag is equal to 1, the storage process of the CABAC parsing process is called with TableStateIdxDS and TableMPSValDS as outputs.

[0184] Parsing of syntax elements proceeds as follows:

[0185] For each requested value of the syntax element, a binarization is derived.

[0186] The binarization for the syntax elements and the sequence of parsed bins determines the decoding process flow.

[0187] For each bin of the binarization of a syntax element, indexed by the variable binIdx, a context index ctxIdx is derived.

[0188] For each ctxIdx, the arithmetic decoding process is invoked.

[0189] The resulting sequence of parsed bins (b0..bbinIdx) is compared with the set of bin strings given by the binarization process after decoding of each bin. When the sequence matches a bin string in the given set, the corresponding value is assigned to the syntax element.

[0190] If a request for a syntax element value is processed for the syntax element pcm-flag and the decoded value of pcm_flag is equal to 1, the decoding engine is initialized after decoding any pcm_alignment_zero_bit, num_subsequent_pcm, and all pcm_sample_luma and pcm_sample_chroma data.

[0191] Thus, the above description reveals a decoder as shown in Figure 24. This decoder, generally designated by reference numeral 5, reconstructs image 10 from data stream 12 in which image 10 is encoded by slices 14 into which image 10 is partitioned, and decoder 5 is configured to decode slices 14 from data stream 12 according to slice order 16. Of course, decoder 5 is not limited to decoding slices 14 consecutively. Rather, decoder 5 can use wave-front parallel processing to decode slices 14, provided that image 10's partitioning into slices 14 is suitable for wave-front parallel processing. Thus, decoder 5 may be a decoder that can decode slices 14 in a staggered manner, for example, starting the decoding of slices 14 by taking into account slice order 16 to enable wave-front processing, as described above and further below.

[0192] Decoder 5 responds to syntax element portion 18 within slice 14 to decode the current slice according to one of at least two modes 20 and 22. According to the first of the at least two modes, i.e., mode 20, the current slice is decoded from data stream 12 using context-adaptive entropy decoding, including derivation of context across slice boundaries, i.e., across the dotted line in FIG. 24, by using information resulting from the encoding / decoding of other "previous slices in slice order 16." Furthermore, decoding of the current slice from data stream 12 using first mode 20 includes continuous updating of codec symbol probabilities and initialization of symbol probabilities at the start of decoding of the current slice that depend on the saved state of symbol probabilities of previously decoded slices. Such dependency is described above, for example, in connection with the "Synchronization Process for Codec Variables." Finally, first mode 20 also includes predictive decoding across slice boundaries. Such predictive decoding across slice boundaries may include, for example, intra-prediction across slice boundaries, i.e., predicted sample values ​​within a current slice based on already reconstructed sample values ​​of a previous slice "in slice order 16," or prediction of coding parameters, prediction modes, coding modes, etc. across slice boundaries, such as prediction of motion vectors.

[0193] According to the second mode 22, the decoder 5 decodes a current slice, i.e., a currently decoded slice, from the data stream 12 using context-adaptive entropy decoding, but restricting the derivation of the context so as not to cross slice boundaries. For example, if a template of adjacent positions used to derive a context for a particular syntax element associated with a block in the current slice extends into an adjacent slice, thereby crossing the slice boundary of the current slice, the corresponding attributes of the respective portions of the adjacent slice, such as the values ​​of the corresponding syntax elements of this adjacent portion of the adjacent slice, are set to default values ​​to suppress interdependencies between the current slice and the adjacent slice. While continuous updating of the symbol probabilities of the context can occur as in the first mode 20, the initialization of the symbol probabilities in the second mode 22 is independent of any previously decoded slice. Furthermore, predictive decoding is performed, restricting the predictive decoding so as not to cross slice boundaries.

[0194] To facilitate understanding of the description of Figure 24 and the following description, reference is made to Figure 25, which shows a possible implementation of decoder 5 in a more structural sense compared to Figure 24. As is the case in Figure 24, decoder 5 is a predictive decoder that uses, for example, context-adaptive entropy decoding to decode the data stream to obtain prediction residuals and prediction parameters.

[0195] As shown in Figure 25, the decoder 5 may include an entropy decoder 24, an inverse quantization and inverse transform module 26, a combiner 28 implemented, for example, as an adder and a predictor 28, as shown in Figure 25. The entropy decoder 24, module 26, and adder 27 are serially connected between the input and output of the decoder 5 in the order they are mentioned, and the predictor 28 is further connected between the output of the adder 28 and its further input to form a prediction loop together with the combiner 27. Thus, the decoder 24 has its output further connected to the coding parameter input of the predictor 28.

[0196] Although Figure 25 gives the impression that the decoder decodes the current image serially, decoder 5 may be implemented to decode, for example, image 10 in parallel. Decoder 5 may, for example, include multiple cores, each operating according to elements 24-28 in Figure 25. However, parallel processing is optional, and furthermore, decoder 5 operating serially may also decode the data stream coming into the input of entropy decoder 24.

[0197] To efficiently achieve the just-described ability to decode the current image 10 serially or in parallel, the decoder 5 operates in units of coding blocks 30 to decode the image 10. The coding blocks 30 are, for example, coding tree blocks or leaf blocks, into which the largest coding blocks 32 are partitioned by recursive multi-tree partitioning, such as quad-tree partitioning. The code tree blocks 32 may then be regularly arranged in columns and rows to form a regular partitioning of the image 10 into these code tree blocks 32. In FIG. 25, the code tree blocks 32 are shown with solid lines, while the coding blocks 30 are shown with dotted lines. For illustrative purposes, only one code tree block 32 is shown to be further partitioned into coding blocks 30, while the other code tree block 32 is shown not to be further partitioned to directly form coding blocks. The data stream 12 may include a syntax portion signaling how the image 10 is partitioned into code blocks 30.

[0198] Data stream 12 conveys, for each coding block 30, syntax elements that clarify how modules 24-28 recover the image content within that coding block 30. For example, these syntax elements include: 1) optionally, partitioning data for further partitioning the coding block 30 into prediction blocks; 2) optionally, partitioning data for further partitioning the coding block 30 into residual and / or transform blocks; 3) A prediction mode that signals whether a prediction mode is used to derive a prediction signal for a coding block 30, where the granularity at which this prediction mode is signaled can depend on the coding block 30 and / or the prediction block. 4) Prediction parameters may be signaled for each coding block, or for each prediction block with certain prediction parameters, if present, being sent depending on the prediction mode, for example. Possible prediction modes may include, for example, intra-prediction and / or inter-prediction. 5) Other syntax elements may be present, such as, for example, filtering information for filtering the image 10 in the coding block 30 to obtain a predicted signal and / or a reconstructed signal to be reproduced. 6) Finally, residual information, particularly in the form of transform coefficients, may be included in the data stream for coding block 30, and the residual data may be signaled on a residual block basis, and for each residual block, if present, spectral decomposition may be performed, for example on a transform block basis as described above.

[0199] The entropy decoder 24 is responsible for obtaining the aforementioned syntax elements from the data stream. For this purpose, the entropy decoder 24 uses context-adaptive entropy decoding, i.e., it provides several contexts. To derive a particular syntax element from the data stream 12, the entropy decoder 24 selects a particular context from among the possible contexts. The selection among the possible contexts is performed depending on the neighborhood attributes of the portion of the image 10 to which the current syntax element belongs. For each possible context, the entropy decoder 24 manages symbol probabilities, i.e., probability estimates for each possible symbol of the symbol alphabet on which the entropy decoder 24 operates. "Managing" includes the aforementioned continuous updating of the symbol probabilities of the contexts in order to apply the symbol probabilities associated with each context to the actual image context. With this measure, the symbol probabilities are applied to the actual probability statistics of the symbols.

[0200] Another environment in which neighboring attributes influence the reconstruction of a current portion of image 10, such as current coding block 30, is predictive decoding within predictor 28. Prediction is not limited to predicting content within current coding block 30, but can also include predicting parameters, partitioning data, or transform coefficients contained in data stream 12 for current coding block 30, such as prediction parameters. That is, predictor 28 can predict image content or such parameters from the aforementioned neighbors to obtain a written signal that is then combined with a prediction residual, as obtained by module 26 from data stream 12. When predicting parameters, predictor 28 can use syntax elements contained in the data stream as prediction residuals to obtain the actual values ​​of the prediction parameters. Predictor 28 uses the later prediction parameter values ​​to obtain the aforementioned predicted signal, which is then combined with the prediction residual in combiner 27.

[0201] The "neighborhood" mentioned above mainly covers the upper left part of the environment of the current part to which the currently entropy decoded or currently predicted syntax element belongs. In Figure 25, such a neighborhood is exemplarily indicated by 34 for one coding block 30.

[0202] The encoding / decoding order is defined within the coding blocks 30: at the coarsest level, the code tree blocks 32 of the image 10 are scanned in a scan order 36, shown here as a raster scan leading row by row from top to bottom. Within each code tree block, the coding blocks 30 are scanned in a depth-first traversal order, such that at each hierarchical level, the code tree blocks 32 are also scanned in a raster scan leading row by row from top to bottom.

[0203] The coding order defined in the coding block 30 is consistent with the definition of the neighborhood 34 used to derive attributes in the neighborhood for selecting a context and / or performing spatial prediction, in that the neighborhood 34 primarily covers portions of the image 10 that have already undergone decoding according to the coding order. Whenever a portion of the neighborhood 34 covers an unavailable portion of the image 10, default data, for example, is used instead. For example, the neighborhood template 34 could extend outside the image 10. However, another possibility is that the neighborhood 34 extends to adjacent slices.

[0204] The image 10, i.e., each slice, along the encoding / decoding order defined by the slice division, e.g., the coding blocks 30, is a continuous, uninterrupted sequence of coding blocks 30 along the coding block order described above. In FIG. 25, slices are indicated by dash-dotted lines 14. The order defined within a slice 14 results from the organization of the execution of consecutive coding blocks 30 as described above. If the syntax element portion 18 of a particular slice 14 indicates that it is decoded in the first mode, the entropy decoder 24 enables context-adaptive entropy decoding to derive a context across slice boundaries. That is, spatial neighborhood 34 is used to select a context in the entropy-decoded data for the current slice 14. In the case of FIG. 25, for example, slice number 3 may be the currently decoded slice, and further, in the entropy decoding syntax element for coding block 30 or some portion thereof, the entropy decoder 24 can use attributes resulting from decoded portions in adjacent slices, such as slice number 1. Predictor 28 operates in the same way: for slices in the first mode 20, predictor 28 uses spatial prediction across slice boundaries surrounding the current slice.

[0205] However, for slices that have the second mode 22 associated with them, i.e., whose syntax element portion 18 indicates the second mode 22, the entropy decoder 24 and predictor 28 are restricted in their entropy context derivation and predictive decoding due to dependencies related to portions present only within the current slice. Clearly, this limits coding efficiency. On the other hand, slices in the second mode 22 allow for the destruction of interdependencies between sequences of slices. Therefore, slices in the second mode 22 can be interspersed within a picture 10 or within the video to which the picture 10 belongs to allow for resynchronization points. However, it is not necessary for each picture 10 to have at least one slice in the second mode 22.

[0206] As already mentioned above, the first and second modes 20 and 22 also differ in their initialization of the symbol probabilities. A slice coded in the second mode 22 causes the entropy decoder 24 to reinitialize the probabilities independent of any previously decoded slices, i.e., slices that are previously decoded in the sense of the order defined in the slice. The symbol probabilities are, for example, set to default values ​​known to both the encoder and the decoder, or initialization values ​​are included in the slice coded in the second mode 22.

[0207] That is, for slices coded / decoded in the second mode 22, the adaptation of the symbol probabilities always starts immediately from the beginning of these slices, and therefore the adaptation accuracy is not good for these slices at the beginning of these slices.

[0208] The situation is different for slices coded / decoded in the first mode 20. The initialization of symbol probabilities performed by the entropy decoder 24 for subsequent slices depends on the saved state of the symbol probabilities of the previously decoded slice. Whenever a slice coded / decoded in the first mode 20 has its beginning located, for example, on a side other than the left side of the image 10, i.e., on a side other than the side where the raster scan 36 begins row by row before proceeding down the next row, the symbol probabilities as they occurred at the end of the entropy decoding of the previous slice are used. This is indicated, for example, in FIG. 2 by the arrow 38 for slice number 4. Slice number 4 begins somewhere between the right and left sides of the image 10, and therefore, when initializing the symbol probabilities, the entropy decoder 24 uses the symbol probabilities obtained in the entropy decoding of the previous slice, i.e., slice number 3, until its end, including the continuous updating of the symbol probabilities during the entropy decoding of slice 3.

[0209] A slice having a second mode 22 associated with it but starting on the left side of the image 10, such as slice number 5, does not have the symbol probabilities applied after completing entropy decoding of the immediately preceding slice number 4, because this would prevent the decoder 5 from decoding the image 10 in parallel using wavefront processing. Rather, as described above, the entropy decoder 24 applies the symbol probabilities after completing entropy decoding of the second code tree block 32 in the encoding / decoding order 36 to the immediately preceding code tree block row in the encoding / decoding order 36, as indicated by arrow 40.

[0210] 25, for example, image 10 is illustratively partitioned into three rows of code tree blocks and four columns of coding tree root blocks 32, and each code tree block row is further subdivided into two slices 14, such that the start of every second slice coincides with the first coding unit in the coding unit order of each code tree root block row. Thus, entropy decoder 24 can use wavefront processing on decoded image 10 by decoding each code tree root block row in parallel, starting with the first or top code tree root block row and then the second, then the third, alternating these code tree root block rows.

[0211] Of course, the partitioning of the blocks 32 in a recursive manner into further coding blocks 30 is arbitrary, and therefore, in a more general sense, the blocks 32 may also be referred to as "coding blocks." That is, more generally, the image 10 may be partitioned into coding blocks 32 arranged in rows and columns and having a raster scan order 36 defined relative to one another, and the decoder 5 may be considered to associate each slice 14 with a consecutive subset of the coding blocks 32 in the raster scan order 36, such that the subsets follow one another along the raster scan order 36 according to the slice order.

[0212] As also becomes clear from the above description, the decoder 5, or more specifically the entropy decoder 24, may be configured to store symbol probabilities as obtained in the context-adaptive entropy decoding of any slice up to the second coding block in the coding block row according to the raster scan order 36. When initializing the symbol probabilities for the context-adaptive entropy decoding of the current slice having the first mode 20 associated therewith, the decoder 5, or more specifically the entropy decoder 24, checks whether the first coding block 32 of the contiguous subset of coding blocks 32 associated with the current slice is the first coding block 32 in the coding block row according to the raster scan order 36. If so, the symbol probabilities for the context-adaptive entropy decoding of the current slice are initialized in response to the stored symbol probabilities as obtained in the context-adaptive entropy decoding of the previously decoded slice up to the second coding block in the coding block row according to the raster scan order 36, as will be explained with reference to arrow 40. Otherwise, initialization of the symbol probabilities for the context-adaptive entropy decoding of the current slice is performed according to the symbol probabilities as obtained in the context-adaptive entropy decoding of the previously decoded slice up to the end of the previously decoded slice, i.e. according to arrow 38. Also, in the case of initialization according to 38, it is meant the state saved at the end of the entropy decoding of the previous slice in slice order 36, while in the case of initialization 40, it is meant that the previously decoded slice includes the end of the second block of the row immediately preceding block 32 in block order 36.

[0213] As shown by the dotted lines in Figure 24, the decoder may be configured to respond to syntax element portion 18 within a current slice of slice 14 to decode the current slice according to one of at least three modes. That is, a third mode 42 may be present alongside the others 20 and 22. The third mode 42 may differ from the second mode 22 in that prediction across slice boundaries is enabled, but entropy encoding / decoding is still restricted from crossing slice boundaries.

[0214] Above, two embodiments are shown for the syntax element portion 18. The table below summarizes these two embodiments.

[0215] TIFF0007769162000007.tif60130

[0216] In one embodiment, the syntax element part 18 is formed by the dependent_slice_flag individually, while in another embodiment, the combination of the dependent_slice_flag and the no_cabac_reset_flag forms the syntax element part. As far as the initialization of the symbol probabilities according to the saved states of the symbol probabilities of the previously decoded slices is concerned, it refers to the synchronization process for the context variables. In particular, the decoder may be configured, when last_ctb_cabac_init_flag=0 and tiles_or_entropy_coding_sync_idc=2, to store symbol probabilities as obtained in context-adaptive entropy decoding of slices previously decoded in succession according to raster scan order up to the second coding block, and further, when initializing the symbol probabilities for context-adaptive entropy decoding of the current slice in accordance with the first mode, to check whether the first coding block of the consecutive subset of coding blocks associated with the current slice is the first coding block in succession according to raster scan order, and if so, to initialize the symbol probabilities for context-adaptive entropy decoding of the current slice according to the stored symbol probabilities as obtained in context-adaptive entropy decoding of slices previously decoded in succession according to raster scan order up to the second coding block, and further, if not, to initialize the symbol probabilities for context-adaptive entropy decoding of the current slice according to the symbol probabilities as obtained in context-adaptive entropy decoding of slices previously decoded up to the end of the previously decoded slice.

[0217] Thus, in other words, according to the second embodiment for the syntax, the decoder reconstructs the image 10 from a data stream 12 in which the image is coded in slices 14 into which the image (10) is partitioned, the decoder is configured to decode the slices 14 from the data stream 12 according to a slice order 16, and further the decoder is responsive to a dependent_slice_flag in the syntax element portion 18, i.e., the current slice of the slice, in order to decode the current slice according to one of at least two modes 20, 22. According to a first mode 20 of the at least two modes, i.e., when dependent_slice_flag=1, the decoder decodes the current slice from the data stream 12 using context adaptive entropy decoding 24, including derivation of context across slice boundaries, continuous updating of symbol probabilities of the context, and initialization of symbol probabilities 38, 40 according to the saved states of symbol probabilities of previously decoded slices, and predictive decoding across slice boundaries; and according to a second mode 22 of the at least two modes, i.e., when dependent_slice_flag=0, the decoder decodes the current slice from the data stream 12 using context adaptive entropy decoding, which restricts derivation of context to not cross slice boundaries, continuous updating of symbol probabilities of the context, and initialization of symbol probabilities independent of any previously decoded slices, and predictive decoding, which restricts predictive decoding to not cross slice boundaries. The image 10 may be partitioned in coding blocks 32 arranged in rows and columns and having a raster scan order 36 defined relative to one another, and the decoder is further configured to associate each slice 14 with a consecutive subset of the coding blocks 32 in the raster scan order 36 such that the subsets follow one another along the raster scan order 36 according to the slice order.The decoder may be configured, i.e., in response to tiles_or_entropy_coding_sync_idc=2, to store symbol probabilities as obtained in context-adaptive entropy decoding of slices previously decoded in succession according to raster scan order 36 up to the second coding block 32, and further configured, when initializing the symbol probabilities for context-adaptive entropy decoding of the current slice in accordance with the first mode, to check whether the first coding block of the consecutive subset of coding blocks 32 associated with the current slice is the first coding block 32 in succession according to raster scan order, and if so, to initialize 40 the symbol probabilities for context-adaptive entropy decoding of the current slice in accordance with the stored symbol probabilities as obtained in context-adaptive entropy decoding of slices previously decoded in succession according to raster scan order 36 up to the second coding block, and further configured, if not, to initialize 38 the symbol probabilities for context-adaptive entropy decoding of the current slice in accordance with the symbol probabilities as obtained in context-adaptive entropy decoding of slices previously decoded up to the end of the previously decoded slice. The decoder may be configured to respond to a syntax element portion (18) within a current slice of slice 14 to decode the current slice according to one of at least three modes, namely, one of a first mode 20 and a third mode 42 or a second mode 22, and the decoder is configured to decode the current slice from the data stream according to the third mode 42, i.e., when dependent_slice_flag=1 and tiles_or_entropy_coding_sync_idc=3, using context-adaptive entropy decoding with continuous updating of symbol probabilities of the context and initialization of symbol probabilities independent of any previously decoded slices, and predictive decoding across slice boundaries, restricting context derivation to not cross slice boundaries, and one of the first and third modes being selected in response to a syntax element, namely, cabac_independent_flag.The decoder may further be configured to store the symbol probabilities as obtained in the context-adaptive entropy decoding of the previously decoded slice up to the end of the previously decoded slice, i.e., when tiles_or_entropy_coding_sync_idc=0, 1, and 3 ("3" when cabac_independentflag=0), and further configured to initialize the symbol probabilities for the context-adaptive entropy decoding of the current slice according to the stored symbol probabilities when initializing the symbol probabilities for the context-adaptive entropy decoding of the current slice according to the first mode. The decoder may be configured in the first and second modes to restrict predictive decoding within tiles into which the image is subdivided, i.e., when tiles_or_entropy_coding_sync_idc=1.

[0218] Naturally, the encoder can therefore set the above syntax to enable the decoder to obtain the above advantages. The encoder may, but need not, be a parallel process, such as a multi-core encoder, etc. To encode the image 10 into the data stream 12 in units of slices 14, the encoder is configured to encode the slices 14 into the data stream 12 according to a slice order 16. The encoder determines a syntax element portion 18 for a current slice of the slice to signal that the syntax element portion is coded according to one of at least two modes 20, 22, and encodes the syntax element portion 18 therein; if the current slice is coded according to a first mode 20 of the at least two modes, encodes the current slice into the data stream 12 using context-adaptive entropy coding 24 including derivation of context across slice boundaries, continuous updating of symbol probabilities of the context, and initialization of symbol probabilities 38, 40 according to saved states of symbol probabilities of previously coded slices, and predictive coding across slice boundaries; if the current slice is coded according to a second mode 22 of the at least two modes, encodes the current slice into the data stream 12 using context-adaptive entropy coding with derivation of context so as not to cross slice boundaries, continuous updating of symbol probabilities of the context, and initialization of symbol probabilities independent of any previously coded slices, and predictive coding that restricts predictive coding so as not to cross slice boundaries. While the image 10 may be partitioned into coding blocks 32 arranged in rows and columns and having a raster scan order 36 defined relative to one another, the encoder may be configured to associate each slice 14 with a consecutive subset of coding blocks 32 in the raster scan order 36 such that the subsets follow one another along the raster scan order 36 according to the slice order.The encoder may be configured to store symbol probabilities as obtained in context adaptive entropy coding of slices previously coded in succession according to a raster scan order 36 up to the second coding block 32, and further, when initializing the symbol probabilities for context adaptive entropy coding of the current slice in accordance with the first mode, check whether the first coding block of the consecutive subset of coding blocks 32 associated with the current slice is the first coding block 32 in succession according to the raster scan order, and if so, initialize 40 the symbol probabilities for context adaptive entropy coding of the current slice in accordance with the stored symbol probabilities as obtained in context adaptive entropy coding of slices previously coded in succession according to the raster scan order 36 up to the second coding block, and otherwise initialize 38 the symbol probabilities for context adaptive entropy coding of the current slice in accordance with the symbol probabilities as obtained in context adaptive entropy coding of slices previously decoded up to the end of the previously coded slice. The encoder may be configured to encode the syntax element portion (18) into a current slice of the slices (14) such that the current slice is signaled to be encoded therein according to one of at least three modes, namely, one of a first mode (20) and a third mode (42) or a second mode (22), and the encoder is configured to encode the current slice into a data stream according to the third mode (42) using context-adaptive entropy coding that restricts derivation of the context so as not to cross slice boundaries, and has continuous updating of symbol probabilities of the context and initialization of symbol probabilities independent of any previously coded slices, and predictive coding across slice boundaries, and the encoder distinguishes between the first and third modes using, for example, a syntax element, namely, cabac_independent_flag.The encoder may be configured to determine a generic syntax element, such as dependent_slices_present_flag, and to operate in one of at least two generic operating modes according to the generic syntax element, i.e., to perform encoding of syntax element portions slice by slice according to a first generic operating mode, and to write the generic syntax element to a data stream necessarily using a different one of at least two modes other than the first mode according to a second generic operating mode. The encoder may be configured to necessarily and uninterruptedly continue updating symbol probabilities from the beginning to the end of a current slice according to the first and second modes. The encoder may be configured to store probabilities for symbols as obtained in context-adaptive entropy encoding of a previously encoded slice until the end of the previously encoded slice, and to initialize symbol probabilities for context-adaptive entropy encoding of the current slice according to the stored symbol probabilities when initializing the symbol probabilities for context-adaptive entropy encoding of the current slice according to the first mode. Furthermore, the encoder may constrain predictive encoding within tiles into which an image is subdivided in the first and second modes.

[0219] A possible structure of the encoder is represented in Figure 26 for completeness. Predictor 70 operates much like predictor 28, i.e., performs prediction, but also determines coding parameters, including, for example, prediction parameters and mode, by optimization. Modules 26 and 27 also occur in the decoder. Subtractor 72 determines a lossless prediction residual, which is lossy coded in transform and quantization module 74 by use of quantization, and optionally further using a spectral decomposition transform. Entropy coder 76 performs context-adaptive entropy coding.

[0220] In addition to the specific syntax examples given above, different examples are outlined below showing correspondence between the terminology used below and the terminology used above.

[0221] In particular, as outlined above, a dependent slice is not only "dependent" in that it can utilize knowledge known from outside its boundaries, but also has a more rapidly adapted entropy context, as outlined above, or achieves better spatial prediction due to tolerances beyond its boundaries. Rather, to save the rate cost that would otherwise be spent on defining slice headers by dividing an image into slices, a dependent slice adopts part of the slice header syntax from a previous slice; that is, this slice syntax header part is not transmitted again for the dependent slice. This is shown, for example, at 100 in FIG. 16 and at 102 in FIG. 21, according to which the slice type is adopted, for example, from a previous slice. By this measure, the subdivision of an image into slices, such as independent slices and dependent slices, is less expensive in terms of expensive bit consumption.

[0222] It is the dependencies just mentioned that result in slightly different wording in the example outlined below: A slice is defined as a unit portion of an image whose slice header syntax can be set individually. Thus, a slice consists of one independent / regular / normal slice, now called an independent slice segment, using the nomenclature described above, and zero, one, or two or more dependent slices, now called dependent slice segments, using the nomenclature described above.

[0223] FIG. 27, for example, shows an image partitioned into two slices: one formed by slice segments 141-143, and the other formed solely by slice segment 144. Indices 1-4 indicate the slice order in the coding order. FIGS. 28a and 28b show different examples of the subdivision of an image 10 into two tiles: in FIG. 28a, one slice covers both tiles 501 and 502 and is formed by all five slice segments 14; in FIG. 28a, two slices are formed by subdividing tile 501 into slice segments 141-142 and 143-144, respectively; and in FIG. 28a, another slice covers tile 502 and is formed by slice segments 145-146.

[0224] The definition could be:

[0225] Dependent slice segment: A slice segment in which the values ​​of some syntax elements of the slice segment header are inferred from the values ​​for the previous independent slice segment in decoding order, previously referred to as a dependent slice in the above embodiments.

[0226] Independent slice segment: A slice segment in which the values ​​of syntax elements in the slice segment header are not inferred from the values ​​for the previous slice segment, previously referred to as a normal slice in the above-described embodiments.

[0227] Slice: An integer number of coding tree units contained in one independent slice segment and all subsequent dependent slice segments (if any) that precede the next independent slice segment (if any) within the same access unit / picture.

[0228] Slice Header: The slice segment header of an independent slice segment that is the current slice segment or that precedes the current dependent slice segment.

[0229] Slice segment: An integer number of coding tree units that are sequentially ordered in a tile scan and contained in a single NAL unit. The division of each image into slice segments is a partition.

[0230] Slice segment header: The portion of an encoded slice segment that contains data elements associated with the first or all coding tree units represented in the slice segment.

[0231] The signaling of "modes" 20 and 22, i.e., "dependent slice segment" and "independent slice segment", can be as follows:

[0232] In some additional NAL units, such as for example PPS, syntax elements may be used to signal whether the use of dependent slices is configured for a particular picture of a sequence for a particular picture:

[0233] dependent_slice_segments_enabled_flag equal to 1 specifies the presence of the syntax element dependent_slice_segment_flag in the slice segment header. dependent_slice_segments_enabled_flag equal to 0 specifies the absence of the syntax element dependent_slice_segment_flag in the slice segment header.

[0234] The dependent_slice_segments_enabled_flag has the same range as the dependent_slices_present_flag described above.

[0235] Similarly, the dependent_slice_flag may be called dependent_slice_segment_flag to account for a different nomenclature for slices.

[0236] dependent_slice_segment_flag equal to 1 specifies that the value of each slice segment header syntax element that is not present in the header of the current slice segment is inferred to be equal to the value of the corresponding slice segment header syntax element in the slice header, i.e., the slice segment header of the previous independent slice segment.

[0237] At the same level, for example the image level, the following syntax elements may be included:

[0238] An entropy_coding_sync_enabled_flag equal to 1 specifies that a specific synchronization process for context variables is invoked before decoding a coding tree unit that includes the first coding tree block of a row of coding tree blocks in a respective tile in each image that references the PPS, and is further invoked after a specific storage process for context variables is decoded a coding tree unit that includes the second coding tree block of a row of coding tree blocks in a respective tile in each image that references the PPS. An entropy_coding_sync_enabled_flag equal to 0 specifies that a specific synchronization process for context variables is not required to be invoked before decoding a coding tree unit that includes the first coding tree block of a row of coding tree blocks in a respective tile in each image that references the PPS, and is further not required to be invoked after a coding tree unit that includes the second coding tree block of a row of coding tree blocks in a respective tile in each image that references the PPS. It is a bitstream conformance requirement that the value of entropy_coding_sync_enabled_flag be the same for all PPSs that are activated within a CVS. It is a bitstream conformance requirement that when entropy_coding_sync_enabled_flag is equal to 1 and the first coding tree block in a slice is not the first coding tree block in a row of coding tree blocks in a tile, the last coding tree block in a slice belongs to the same row of coding tree blocks as the first coding tree block in the slice. It is a bitstream conformance requirement that when entropy_coding_sync_enabled_flag is equal to 1 and the first coding tree block in a slice segment is not the first coding tree block in a row of coding tree blocks in a tile, the last coding tree block in a slice segment belongs to the same row of coding tree blocks as the first coding tree block in the slice segment.

[0239] As already mentioned, the encoding / decoding order in CTB 30 results in a row-by-row start from top to bottom, scanning the first tile and then visiting the next tile in a raster manner if there is more than one tile in the image.

[0240] The decoder 5 and encoder therefore work as follows in entropy decoding (encoding) the slice segments 14 of the image:

[0241] A1) Whenever the currently decoded / encoded syntax element synEl is the first syntax element of a tile 50, slice segment 14 or row of a CTB, the initialization process of Figure 29 is started. A2) Otherwise, decoding of this syntax element occurs using the current entropy context. A3) If the current syntax element is the last syntax element in the CTB 30, then the entropy context storage process is initiated as shown in FIG. A4) The process advances to the next syntax element in A1).

[0242] In the initialization process, a check 200 is made as to whether synEI is the first syntax element of a slice segment 14 or a tile 50. If yes, the context is initialized independently of any previous slice segment in step 202. If no, a check 204 is made as to whether synEI is the first syntax element of a row of a CTB 30 and entropy_coding_sync_enabled_flag is equal to 1. If yes, a check 206 is made as to whether a second CTB 30 is available in the row before the CTB 30 of the equivalent tile (see FIG. 23). If yes, context adoption according to type 40 is performed in step 210 using the currently stored context probability for adoption of type 40. Otherwise, the context is initialized independently of any previous slice segment in step 202. If check 204 reveals no, it is checked in step 212 whether synEl is the first syntax element in the first CTB of the dependent slice segment 14 and whether dependent_slice_segment_flag is equal to 1, and if yes, a context adoption according to 38 is performed in step 214 using the currently stored context probability for adoption of type 38. After any of steps 214, 212, 210 and 202, the decoding / encoding actually starts.

[0243] Dependent slice segments with dependent_slice_segment_flag equal to 1 help to further reduce the encoding / decoding delay with little coding efficiency penalty.

[0244] In the storage process of Figure 30, it is checked in step 300 whether the encoded / decoded synEl is the last syntax element of the second CTB 30 in a row of CTBs 30 and whether the entropy_coding_sync_enabled_flag is equal to 1. If yes, the current entropy context is stored in step 302, i.e., the entropy coding probability of the context is stored in a storage specified for the adoption of a 40 type. Similarly, in addition to step 300 or 302, it is checked in step 304 whether the encoded / decoded synEl is the last syntax element of a slice segment 14 and whether the dependent_slice_segment_flag is equal to 1. If yes, the current entropy context is stored in step 306, i.e., the entropy coding probability of the context is stored in a storage specified for the adoption of a 38 type.

[0245] Note that any check asking whether a syntax element is the first synEl in a CTB row makes use of, for example, the syntax element slice_address400 in the slice segment header, i.e., the start syntax element that identifies the location of the start of each slice segment along the decoding order.

[0246] When reconstructing image 10 from data stream 12 using WPP processing, the decoder can accurately utilize the later start syntax portion 400 to search for WPP substream entry points. Because each slice segment includes a start syntax portion 400 that indicates the location within image 10 where the decoding of that slice segment begins, the decoder can identify the entry point of the WPP substream into which the slice segments are grouped by identifying slice segments that begin on the left side of the image using the start syntax portion 400 of the slice segment. The decoder then sequentially begins decoding the WPP substreams according to slice order. (sequentially)The encoder can start encoding the WPP substreams in a staggered, parallel manner, starting with the first slice in the image (10) and decoding the WPP substreams in a staggered, parallel manner. Slice segments may be smaller than one image width, i.e., one row of the CTB, so that their transmission can be interleaved within the WPP substreams to further reduce the overall end-to-end transmission delay. The encoder provides each slice (14) with a start syntax portion (400) indicating the location within the image (10) where the encoding of the respective slice begins, and further groups the slices into WPP substreams such that, for each WPP substream, the first slice in the slice order begins on the left side of the image. The encoder can even use the WPP process alone when encoding an image: the encoder encodes the WPP substreams in a staggered, parallel manner, starting encoding the WPP substreams consecutively according to the slice order.

[0247] Incidentally, the latter embodiment, which uses the start syntax part of a slice segment as a means for locating the entry point of a WPP substream, can be used without the dependent slice concept.

[0248] It is possible to achieve this for all parallel processing images 10 by setting the variables mentioned above as follows:

[0249] TIFF0007769162000008.tif99159

[0250] It is feasible to mix WPP with tile partitioning. In that case, one can consider tiles as individual images: each with WPP consists of slices with one or more dependent slice segments, and the checks in steps 300 and 208 refer to a second CTB in the CTB row above in the same tile, just as steps 204 and A1 refer to a first CTB 30 in the CTB 30 row of the current tile. In that case, the above table can be expanded:

[0251] TIFF0007769162000009.tif240161

[0252] As a brief note, the latter extension is enabled in embodiment 2. Embodiment 2 enables the following processing:

[0253] TIFF0007769162000010.tif103166

[0254] However, for the following extensions, the following table occurs:

[0255] Add to the image parameter set semantics: If tiles_or_entropy_coding_sync_idc is equal to 4, then the first row of each CTB is contained in a different slice that has its dependent slice flag set to 1. CTBs from different rows must not be in the same slice. There may be more than one slice per CTB row.

[0256] If tiles_or_entropy_coding_sync_idc is equal to 5, then each CTB, but the first tile, must be contained in a different slice. CTBs of different tiles must not be in the same slice. There may be more than one slice present per tile.

[0257] For further explanation, see FIG.

[0258] That is, the above table can be expanded:

[0259] TIFF0007769162000011.tif180129

[0260] With regard to the above-described embodiments, it should be noted that the decoder should be configured to, for example, in response to tiles_or_entropy_coding_sync_idc=1, 2 in the first and second modes, read information from the current slice revealing the subdivision of the current slice into parallel subsections, where the parallel subsections may be WPP substreams or tiles, and to discontinue context-adaptive entropy decoding at the end of the first parallel subsection and restart it anew at the start of any subsequent parallel subsection, including, in the first mode, initialization of symbol probabilities depending on the storage state of symbol probabilities of the preceding parallel subsection, and in the second mode, initialization of symbol probabilities independent of the previously decoded slice and the previously decoded parallel subsection.

[0261] Thus, the above description reveals a method for low-delay encoding, decoding, encapsulation and transmission of video data configured as provided by the new HEVC encoding standard, for example, as organized in tiles, wavefront parallel processing (WPP) substreams, slices or entropy slices.

[0262] In particular, it defines how to transfer parallel coded data in a conversational scenario to obtain minimum latency in the coding, decoding and transmission processes. Thus, a pipelined parallel coding, transmission and decoding approach is described to enable minimum latency applications such as gaming, remote surgery, etc.

[0263] Furthermore, the above-described embodiment fills a gap in wavefront parallel processing (WPP) so that it can be used in low-latency transmission scenarios. Therefore, the new encapsulation format for WPP substream 0 shows a dependent slice. This dependent slice can contain entropy slice data, a WPP substream, a complete row of LCUs, or just a fragment of a slice, and the conventional transmitted slice header also applies to the included fragment data. The included data is signaled in the subslice header.

[0264] One final note: the naming for the new slice could also be "subset / lightweight slice", but the name "dependent slice" has been found to be better.

[0265] Signaling is shown that describes the level of parallelism in encoding and forwarding.

[0266] Although some aspects are described in the context of an apparatus, it will be apparent that these aspects also represent a description of a corresponding method, where a block or apparatus corresponds to a method step or feature of a method step. Similarly, aspects described in the context of a method step also represent a description of a corresponding block or item or feature of a corresponding apparatus. Some or all of the method steps may be performed by (or using) a hardware apparatus, such as, for example, a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, one or more of any of the most important method steps may be performed by such an apparatus.

[0267] Depending on specific implementation requirements, embodiments of the present invention can be implemented in hardware or in software. Implementation can be performed using a digital storage medium, such as a floppy disk, DVD, Blu-ray, CD, ROM, PROM, EPROM, EEPROM, or FLASH memory, on which electronically readable control signals are stored that cooperate (or can cooperate) with a programmable computer system so that the respective methods are performed. Thus, the digital storage medium may be computer-readable.

[0268] Some embodiments according to the present invention include a data carrier having electronically readable control signals that can cooperate with a programmable computer system to perform one of the methods described herein.

[0269] Generally, embodiments of the present invention can be implemented as a computer program product having program code that operates to perform one of these methods when the computer program product is run on a computer, which program code may for example be stored on a machine-readable carrier.

[0270] Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

[0271] In other words, an embodiment of the inventive method is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

[0272] A further embodiment of the inventive method is therefore a data carrier (or digital storage medium or computer readable medium) comprising, recorded on it, the computer program for performing one of the methods described herein. The data carrier, digital storage medium or recording medium is typically tangible and / or non-transitory.

[0273] A further embodiment of the inventive method is, therefore, a data stream or sequence of signals representing the computer program for performing one of the methods described herein, which data stream or sequence of signals may be adapted to be transmitted via a data communications connection, for example the Internet.

[0274] A further embodiment comprises a processing means, for example a computer, or a programmable logic device, configured to or adapted to perform one of the methods described herein.

[0275] A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0276] Further embodiments according to the present invention include an apparatus or system configured to transfer (e.g., electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a memory device, etc. The apparatus or system may, for example, include a file server for transferring the computer program to the receiver.

[0277] In some embodiments, a programmable logic device (e.g., a field programmable gate array) may be used to perform some or all of the functions of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor to perform one of the methods described herein. In general, the methods are preferably performed by any hardware apparatus.

[0278] The above-described embodiments are merely illustrative for the principles of the present invention. It is understood that modifications and variations of the arrangements and details described herein will be apparent to others skilled in the art. It is therefore intended that the present invention be limited only by the scope of the appended claims and not by the specific details set forth herein by way of description and illustration of the embodiments.

[0279] literature [1] Thomas Wiegand, Gary J. Sullivan, Gisle Bjontegaard, Ajay Luthra, "Overview of the H.264 / AVC Video Coding Standard", IEEE Trans. Circuits Syst. Video Technol., vol. 13, N7, July 2003. [2] JCT-VC, "High-Efficiency Video Coding (HEVC) text specification Working Draft 6", JCTVC-H1003, February 2012. [3] ISO / IEC 13818-1: MPEG-2 Systems specification.

Claims

1. 1. A decoder for reconstructing an image (10) from a data stream (12) in which the image (10) is coded in slices (14) into which the image (10) is partitioned, the decoder being configured to decode the slices (14) from the data stream (12) according to a slice order (16), and the decoder being responsive to a syntax element portion (18) in a current one of the slices to decode the current slice according to one of at least two modes (20, 22), and According to a first mode (20) of the at least two modes, decoding the current slice from the data stream (12) using context-adaptive entropy decoding (24), including deriving a context across slice boundaries, continuously updating symbol probabilities of the context, and initializing the symbol probabilities (38, 40) according to stored states of symbol probabilities of previously decoded slices, and predictive decoding across the slice boundaries; and According to a second mode (22) of the at least two modes, decoding the current slice from the data stream (12) using context adaptive entropy decoding with a restriction that the derivation of the context does not cross the slice boundary, continuous updating of symbol probabilities of the context and initialization of the symbol probabilities independent of any previously decoded slice, and predictive decoding with a restriction that the predictive decoding does not cross the slice boundary; the image (10) is partitioned into coding blocks (32) arranged in rows and columns and having a defined raster scan order (36) relative to one another, and the decoder is configured to associate each slice (14) with a consecutive subset of the coding blocks (32) in the raster scan order (36) such that the subsets follow one another along the raster scan order (36) according to the slice order; the decoder is configured to store symbol probabilities obtained during context-adaptive entropy decoding of the previously decoded slice up to a second coding block (32) that is consecutive according to the raster scan order (36); and further configured, when initializing the symbol probabilities for the context-adaptive entropy decoding of the current slice according to the first mode, to check whether a first coding block of the consecutive subset of coding blocks (32) associated with the current slice is a first coding block (32) that is consecutive according to the raster scan order, and if so, to initialize (40) the symbol probabilities for the context-adaptive entropy decoding of the current slice according to the stored symbol probabilities obtained during context-adaptive entropy decoding of the previously decoded slice up to a second coding block that is consecutive according to the raster scan order (36); and, otherwise, to initialize (38) the symbol probabilities for the context-adaptive entropy decoding of the current slice according to the symbol probabilities obtained during context-adaptive entropy decoding of the previously decoded slice up to the end of the previously decoded slice. the decoder is configured to use intra-frame and inter-frame prediction; decoder.

2. 2. The decoder of claim 1, wherein the decoder is configured to store symbol probabilities obtained when context-adaptive entropy decoding the previously decoded slice up to an end of the previously decoded slice, and further configured to initialize the symbol probabilities for the context-adaptive entropy decoding of the current slice according to the stored symbol probabilities when initializing the symbol probabilities for the context-adaptive entropy decoding of the current slice according to the first mode.

3. 3. A decoder as claimed in claim 1 or claim 2, wherein the decoder is configured, in the first and second modes, to restrict the predictive decoding within tiles into which the image is subdivided.

4. 4. The decoder of claim 1, wherein the decoder is configured to read information from a current slice that identifies a redivision of the current slice into parallel subsections, stop the context-adaptive entropy decoding at the end of a first parallel subsection, and restart the context-adaptive entropy decoding anew at the beginning of any subsequent parallel subsection, the context-adaptive entropy decoding including initialization of the symbol probabilities according to a saved state of the symbol probabilities of a previous parallel subsection in the first mode and initialization of the symbol probabilities independent of any previously decoded slice and any previously decoded parallel subsection in the second mode.

5. 5. The decoder of claim 1, wherein the decoder is configured to copy, according to the first mode (20) of the at least two modes, a portion of a slice header syntax from a previous slice decoded in the second mode for the current slice.

6. The decoder is configured to reconstruct the image (10) from the data stream (12) using a WPP process, each slice (14) including a start syntax portion (400) indicating a location within the image (10) to start decoding the respective slice, and the decoder: identifying an entry point of a WPP substream in which the slice is grouped by identifying a slice that starts on the left side of the image using a start syntax portion of the slice; and configured to decode the WPP sub-streams in a staggered manner in parallel and to start the decoding of the WPP sub-streams sequentially according to the slice order. A decoder according to any one of claims 1 to 5.

7. 1. An encoder for encoding an image (10) into a data stream (12) in units of slices (14) into which the image (10) is partitioned, the encoder configured to encode the slices (14) into the data stream (12) according to a slice order (16), the encoder comprising: determining a syntax element portion (18) for a current one of the slices, and encoding the syntax element portion (18) into the current slice such that the syntax element portion signals the current slice being encoded according to one of at least two modes (20, 22); If the current slice is coded according to a first mode (20) of the at least two modes, coding the current slice into the data stream (12) using context-adaptive entropy coding (24) including deriving a context across slice boundaries, continuously updating symbol probabilities of the context, and initializing the symbol probabilities (38, 40) according to saved states of symbol probabilities of previously coded slices, and predictive coding across the slice boundaries; and If the current slice is coded according to a second mode (22) of the at least two modes, the current slice is coded into the data stream (12) using context-adaptive entropy coding with a constraint that the derivation of the context does not cross the slice boundary, continuous updating of symbol probabilities of the context and initialization of the symbol probabilities independent of any previously coded slice, and predictive coding with a constraint that the predictive coding does not cross the slice boundary, the image (10) is partitioned into coding blocks (32) arranged in rows and columns and having a defined raster scan order (36) relative to one another, and the encoder is configured to associate each slice (14) with a consecutive subset of the coding blocks (32) in the raster scan order (36) such that the subsets follow one another along the raster scan order (36) according to the slice order; the encoder is configured to store symbol probabilities obtained when context-adaptive entropy encoding the previously encoded slice up to a second coding block (32) that is consecutive according to the raster scan order (36); and is further configured, when initializing the symbol probabilities for the context-adaptive entropy encoding of the current slice according to the first mode, to check whether a first coding block of a contiguous subset of the coding blocks (32) associated with the current slice is a first coding block (32) that is consecutive according to the raster scan order, and if so, to initialize (40) the symbol probabilities for the context-adaptive entropy encoding of the current slice according to the stored symbol probabilities obtained when context-adaptive entropy encoding the previously encoded slice up to a second coding block that is consecutive according to the raster scan order (36); and, if not, to initialize (38) the symbol probabilities for the context-adaptive entropy encoding of the current slice according to the symbol probabilities obtained when context-adaptive entropy encoding the previously decoded slice up to the end of the previously encoded slice. the encoder is configured to use intra-frame and inter-frame prediction; Encoder.

8. A method of transmitting video, comprising transmitting data associated with the video, the data comprising: a data stream containing information relating to images of the video, the images being partitioned into slices (14), the encoded information being encoded into the data stream by a predetermined method, the predetermined method comprising: encoding an image (10) into a data stream (12) in units of slices (14) into which the image (10) is partitioned, the method including encoding the slices (14) into the data stream (12) according to a slice order (16); determining a syntax element portion (18) for a current one of the slices and encoding the syntax element portion (18) into the current slice such that the syntax element portion signals the current slice being encoded according to one of at least two modes (20, 22); If the current slice is coded according to a first mode (20) of the at least two modes, coding the current slice into the data stream (12) using context-adaptive entropy coding (24), including deriving a context across slice boundaries, continuously updating symbol probabilities of the context, and initializing the symbol probabilities (38, 40) according to saved states of symbol probabilities of previously coded slices, and predictive coding across the slice boundaries; if the current slice is coded according to a second mode (22) of the at least two modes, coding the current slice into the data stream (12) using context-adaptive entropy coding with a constraint that the derivation of the context does not cross the slice boundary, continuous updating of symbol probabilities of the context and initialization of the symbol probabilities independent of any previously coded slice, and predictive coding with a constraint that the predictive coding does not cross the slice boundary; Including, wherein the image (10) is partitioned into coding blocks (32) arranged in rows and columns and having a defined raster scan order (36) relative to one another, and the method includes associating each slice (14) with a consecutive subset of the coding blocks (32) in the raster scan order (36) such that the subsets follow one another along the raster scan order (36) according to the slice order; and the predetermined method includes the steps of: storing symbol probabilities obtained when context-adaptive entropy coding the previously coded slice up to a second coding block (32) that is consecutive according to the raster scan order (36); and, when initializing the symbol probabilities for the context-adaptive entropy coding of the current slice according to the first mode, checking whether a first coding block of a contiguous subset of the coding blocks (32) associated with the current slice is a first coding block (32) that is consecutive according to the raster scan order, and if so, initializing (40) the symbol probabilities for the context-adaptive entropy coding of the current slice according to the stored symbol probabilities obtained when context-adaptive entropy coding the previously coded slice up to a second coding block that is consecutive according to the raster scan order (36); and, if not, initializing (38) the symbol probabilities for the context-adaptive entropy coding of the current slice according to the symbol probabilities obtained when context-adaptive entropy coding the previously decoded slice up to the end of the previously coded slice. wherein the predetermined method includes using intra-frame prediction and inter-frame prediction. A method for sending video.

Images