Encoder, decoder, and encoding and decoding methods involving complex processing for flexibly sized image partitions

Abstract

To provide a video encoder where a VCL (video coding layer) includes techniques for fragmentation and spatial partitioning.SOLUTION: A video encoder (101) for encoding a plurality of pictures of a video each including original picture data includes a data encoder (110) configured for generating an encoded video signal. The data encoder (110) encodes the plurality of pictures of the video into encoded picture data. Moreover, the video encoder (101) includes an output interface (120) configured for outputting the encoded picture data of each of the pictures.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] This invention relates to video encoding and video decoding, and in particular, to encoding and decoding methods involving complex processing for encoders, decoders, and flexibly sized image data.

Background Art

[0002] H.265 / HEVC (HEVC = High Efficiency Video Coding) is already a video coder that provides tools for enhancing or even enabling parallel processing in encoders and / or decoders. For example, HEVC assists in the sub - division of an image into an array of tiles that are encoded independently of each other. Another concept supported by HEVC is associated with WPP (CTU = Coding Tree Unit) according to which, in the processing of consecutive CTU lines, if it is provided that a certain minimum CTU offset is preserved, the CTU - columns or CTU - lines of an image are processed parallel from left to right, for example in a striped manner. However, it is preferable to have readily available a video coder that more efficiently supports the parallel processing function of video encoders and / or video decoders.

[0003] In the following, an introduction to VCL partitioning according to the state - of - the - art is explained (VCL = Video Coding Layer).

[0004] Generally, in video coding, the coding process of image samples requires smaller partitions (divisions). Samples are divided into several rectangular areas for joint processing such as predictive or transform coding. Thus, an image is partitioned into blocks of a specific size that is constant during the encoding of a video sequence. In the H.264 / AVC standard, fixed - size blocks of 16×16 samples, so - called macroblocks, are used (AVC = Advanced Video Coding).

[0005] In the latest HEVC standard (see Non-Patent Document 1), there is a maximum size of coded tree block (CTB) or coded tree unit (CTU) of 64 × 64 samples. In further explanations of HEVC, more common CTUs are used for such types of blocks.

[0006] The CTUs are processed in the order of the raster scan, starting with the top left CTU, processing the CTUs within the image in a linear fashion, and then moving down to the bottom right CTU. [Prior art documents] [Non-patent literature]

[0007] [Non-Patent Document 1] [1] ISO / IEC, ITU-T. High efficiency video coding. ITU-T Recommendation H.265 | ISO / IEC 23008 10 (HEVC), edition 1, 2013; edition 2, 2014. [Overview of the project] [Problems that the invention aims to solve]

[0008] The coded CTU data is organized into a type of container called a slice. Originally, in previous video coding standards, a slice meant a segment containing one or more consecutive CTUs of an image. Slices are used for segments of coded data. From another perspective, a complete image can also be defined as one large segment, and historically, the term slice is still used. In addition to the coded image sample, a slice also contains additional information related to the coding process of the slice itself, which is placed in a so-called slice header.

[0009] According to the latest technology, the VCL (Video Coding Layer) also includes techniques for fragmentation and spatial partitioning. Such partitioning is used in video coding for various reasons, such as handling load balancing in parallelization, CTU size matching in network transmission, and error mitigation.

[0010] Therefore, the object of this invention is to provide an improved concept for video encoding and video decoding. [Means for solving the problem]

[0011] The object of this invention is resolved by the main points of the independent claims.

[0012] Preferred embodiments are provided in the dependent claims. [Effects of the Invention]

[0013] According to this invention, an improved concept for video encoding and video decoding is obtained. [Brief explanation of the drawing]

[0014] Embodiments of this invention will be described in detail below with reference to the drawings.

[0015] [Figure 1] Figure 1 shows a video encoder according to an embodiment. [Figure 2] Figure 2 shows a video decoder according to an embodiment. [Figure 3] Figure 3 shows a system according to an embodiment. [Figure 4] Figure 4 illustrates the effects of replacing a partial CTU that generates a tile boundary. [Figure 5] Figure 5 shows luma samples for each image compensation from a partial CTU. [Figure 6]FIG. 6 shows CTU grid alignment incompatibility. [Figure 7] FIG. 7 shows the ratio of CTUs affected by grid incompatibility. [Figure 8] FIG. 8 shows two sets of tile boundaries. [Figure 9] FIG. 9 shows the column of corresponding CTUs in the reference image after a partial CTU in the current image. [Figure 10] FIG. 10 shows a video encoder. [Figure 11] FIG. 11 shows a video decoder. [Figure 12] FIG. 12 shows the relationship between, on the one hand, the reconstructed signal, i.e., the reconstructed image, and, on the other hand, the combination of the predicted residual signal and the predicted signal conveyed in the data stream as a signal. [Figure 13] FIG. 13 shows image segmentation by slices in raster scan order. [Figure 14] FIG. 14 shows image partitioning by tiles. DETAILED DESCRIPTION OF THE INVENTION

[0016] The following description of the drawings begins with providing an explanation of an encoder and a decoder of a predictive coder based on blocks for a coded image of a video in order to form an example for a coding frame operation in which embodiments of this invention are incorporated. Each encoder and decoder is described with respect to FIGS. 10 to 12. In the following, although the embodiments described in FIGS. 1 to 3 and subsequent drawings are also used to form encoders and decoders that do not operate according to the coding frame operation under the encoders and decoders of FIGS. 10 and 11, a description of embodiments of the concept of this invention is provided together with an explanation of how such a concept is incorporated into the encoders and decoders of FIGS. 10 and 11, respectively.

[0017] Figure 10 shows a device for predictively encoding an image 12 into a data stream 14 using a video encoder, which exemplifies transform-based residual coding. The device, or encoder, is denoted by the numeral 10. Figure 11 shows a corresponding video decoder 20, a device 20 configured to predictively decode an image 12' from a data stream 14 using transform-based residual decoding. The apostrophe is used to indicate that the image 12' reconstructed by the decoder 20 is derived from the original encoded image 12 by the device 10, with respect to the coding loss introduced by the quantization of the predictive residual signal. While embodiments of this application are not limited to this type of predictive residual coding, Figures 10 and 11 exemplify transform-based predictive residual coding. This also applies to other details described with respect to Figures 10 and 11, which will be outlined below.

[0018] The encoder 10 is configured to make the predicted residual signal dependent on the transformation from space to spectrum, and thus encode the resulting predicted residual signal into the data stream 14. Similarly, the decoder 20 is configured to decode the predicted residual signal from the data stream 14, and thus make the resulting predicted residual signal dependent on the transformation from spectrum to space.

[0019] Internally, the encoder 10 includes a predictive residual signal generator 22 that generates a predictive residual signal 24 to measure the derivation of the predictive signal 26 from the original signal, i.e., from the image 12. The predictive residual signal generator 22 is, for example, a subtractor that extracts the predictive signal from the original signal, i.e., from the image 12. The encoder 10 further includes a converter 28 that makes the predictive residual signal 24 dependent on a spatial-to-spectral transformation in order to obtain a spectral-domain predictive residual signal 24' dependent on quantization by a quantizer 32 also included in the encoder 10. Thus, the quantized predictive residual signal 24'' is encoded into the bitstream 14. For this purpose, the encoder 10 optionally includes an entropy coder 34 that encodes the predictive residual signal so that the entropy is converted into a datastream 14 and quantized. The predictive signal 26 is generated by the prediction stage 36 of the encoder 10 based on the predictive residual signal 24'' encoded into the datastream 14 and decoded from the datastream 14. For this purpose, the prediction stage 36 internally includes an inverse quantizer 38 that inversely quantizes the predicted residual signal 24'' to obtain a spectral domain predicted residual signal 24'''' corresponding to signal 24'', excluding quantization losses, as shown in Figure 10, and an inverse converter 40 that follows the inverse quantizer 38 and makes the latter predicted residual signal 24'''' subject to an inverse transform, i.e., a spectral-to-space transform, to obtain a predicted residual signal 24'''' corresponding to the original predicted residual signal 24, excluding quantization losses. The coupler 42 of the prediction stage 36 recombines the predicted signal 26 and the predicted residual signal 24'''' by addition or the like to obtain a reconstructed signal 46, i.e., a reconstruction of the original signal 12. The reconstructed signal 46 corresponds to signal 12'. The prediction module 44 of the prediction stage 36 generates a predicted signal 26 based on signal 46, for example, using spatial prediction, i.e., intra-image prediction, and / or temporal prediction, i.e., inter-image prediction.

[0020] Similarly, the decoder 20, as shown in Figure 11, internally includes components corresponding to the prediction stage 36 and is interconnected in a manner corresponding to the prediction stage 36. In particular, the entropy decoder 50 of the decoder 20 entropy decodes the quantized spectral domain prediction residual signal 24'' from the data stream. Furthermore, the inverse quantizer 52, inverse converter 54, coupler 56, and prediction module 58 are interconnected in the same manner as described above for the module of the prediction stage 36 and cooperate to recover the signal reconstructed based on the prediction residual signal 24''. As a result, as shown in Figure 11, the output of the coupler 56 produces the reconstructed signal, i.e., image 12'.

[0021] Although not specifically explained above, it is clear that encoder 10 sets several coding parameters, including, for example, prediction modes and operating parameters, according to some optimization plans, for example, in a way that optimizes coding cost, i.e., based on criteria related to a certain rate and distortion. For example, encoder 10 and decoder 20 and corresponding modules 44, 58 support different prediction modes, such as intra-coding mode and inter-coding mode, respectively. The granularity at which encoders and decoders switch between these prediction mode types corresponds to sub-divisions of images 12, 12' into coded segments or coded blocks, respectively. Within these units of coded segments, for example, the image is sub-segmented into intra-coded blocks and inter-coded blocks. The intra-coded blocks are predicted based on space, with each block having already encoded / decoded neighbors, as outlined in more detail below. Several intracoding modes exist, and each segment is selected for each intracoded segment, including directional or angular intracoding modes, such that each segment is filled by extrapolating neighboring sample values ​​along a specific direction, as defined for each directional intracoding mode. For example, the intracoding modes include one or more other modes, such as a DC coding mode, where the prediction for each intracoded block is to assign DC values ​​to all samples in each intracoded segment, and / or a planar intracoding mode, where the prediction for each block is to approximate or determine a spatial distribution of sample values ​​described by a two-dimensional linear function on the sample position of each intracoded block, having a plane drive slope and offset defined by a two-dimensional linear function based on neighboring samples. In comparison, for example, an intercoded block is predicted in time. For an intercoded block, the action vector is transmitted in the data stream by signal.The action vector directs the spatial substitution of a portion of the previously coded image in the video to which image 12 belongs. The previously coded / decoded image is extracted therein to obtain a predictive signal for each intercoded block. This means, in addition to the residual signal coding constituted by the data stream 14, an entropy-coded transformation coefficient level representing the quantized spectral domain predictive residual signal 24'', etc. The data stream 14 has coded coding mode parameters into them, and predictive parameters for some blocks, such as action parameters for the intercoded segments, and any other parameters, such as parameters for controlling and signaling the subsegments of image 12 and image 12' into the segments, respectively, in order to assign coding modes to the various blocks. The decoder 20 uses these parameters to subdivide the image in the same way as the encoder did, in order to assign the same predictive mode to the segments, and to perform the same predictions so that the same predictive signals result.

[0022] Figure 12 illustrates, on the one hand, the relationship between the reconstructed signals, i.e., the reconstructed image 12', and on the other hand, the combination of the predicted residual signal 24'''' and the predicted signal 26 as signals in the data stream 14. As already shown above, the combination is addition. The predicted signal 26 is shown in Figure 12 as a sub-segment of the image area, in intracoded blocks indicated graphically with diagonal lines, and in interconnected blocks indicated graphically without diagonal lines. Sub-segments are several sub-segments, such as a regular sub-sub

[0023] Furthermore, the data stream 14 has an intra-coding mode encoded therein for the intra-coded blocks 80, and assigns one of several supported intra-coding modes to each intra-coded block 80. For the intercoded blocks 82, the data stream 14 has one or more operational parameters encoded therein. Generally speaking, the intercoded blocks 82 are not limited to being coded in time. Instead, the intercoded blocks 82 are blocks predicted from previously coded portions beyond the current image 12 itself, such as images coded before the video to which the image 12 belongs, or images of another display, or even hierarchically lower layers in the case of encoders and decoders, which are respectively variable-scale encoders and decoders.

[0024] The predicted residual signal 24'''' in Figure 12 is also shown as a subsegment of the image area into block 84. These blocks are called transformation blocks to distinguish them from the coded blocks 80,82. In effect, Figure 12 shows that the encoder 10 and decoder 20 use two different subdivisions of image 12 and image 12' into blocks, namely one subdivision into coded blocks 80,82 and another subdivision into transformation block 84. Both subdivisions are the same; that is, each coded block 80,82 simultaneously forms a transformation block 84. However, Figure 12 shows, for example, the case where the subdivision into transformation block 84 forms an extension of the subdivision into coded blocks 80,82. Thus, any boundary between the two blocks 80,82 overlaps with the boundary between the two blocks 84. In other words, each block 80,82 corresponds either to one of the transformation blocks or to a group of transformation blocks 84. However, the subdivisions are also determined or selected independently of each other, such that the transformed block 84 instead crosses the block boundary between blocks 80 and 82. As far as the subdivision to transformed block 84 is concerned, similar statements apply as those brought to the forefront for the subdivision to blocks 80 and 82, namely, that block 84 is the result of a regular subdivision of an image area into blocks (with or without column and row arrangement), or the result of a recursive multitree subdivision of an image area, or a combination thereof, or any other kind of block class. Incidentally, it should be noted that blocks 80, 82, and 84 are not limited to quadratic expressions, rectangles, or any other form.

[0025] Figure 12 further illustrates that the coupling of the predicted signal 26 and the predicted residual signal 24'''' directly results in the reconstructed signal 12'. However, it should be noted that one or more predicted signals 26 may be coupled with the predicted residual signals 24'''' according to an alternative embodiment, resulting in image 12'.

[0026] In Figure 12, the transformation block 84 has the following significance. The converter 28 and inverse converter 54 perform their transformations within these units of transformation block 84. For example, many encoders use some kind of DST or DCT for all transformation blocks 84. Some encoders allow the omission of transformations for some of the transformation blocks 84 so that the predicted residual signal is encoded directly in the spatial domain. However, according to the embodiments described below, the encoder 10 and decoder 20 are configured in such a way that they support several transformations. For example, the transformations supported by the encoder 10 and decoder 20 include the following:

[0027] ○ DCT-II (or DCT-III), where DCT represents the discrete cosine transform. 〇 DST-IV, where DST represents the discrete sine transformation. 〇 DCT-IV 〇 DST-VII 〇 Identity Transformation (IT)

[0028] Of course, the converter 28 supports all of the previous conversion versions of these conversions, while the decoder 20 or inverse converter 54 supports its corresponding regression or inverse version.

[0029] ○ Reverse DCT-II (or Reverse DCT-III) ○ Reverse DST-IV 〇 Inverse DCT-IV 〇 Reverse DST-VII 〇 Identity Transformation (IT)

[0030] The following explanation will provide more details on how the transformations are supported by the encoder 10 and decoder 20. In any case, it should be noted that the set of supported transformations simply includes one transformation, such as a transformation from spectrum to space or from space to spectrum.

[0031] As already outlined above, Figures 10-12 illustrate, as an example, that the concepts of the invention described further below are implemented to form concrete examples for encoders and decoders according to the present application. So far, the encoder and decoder in Figures 10 and 11 represent possible implementations of the encoder and decoder described below, respectively. However, Figures 10 and 11 are merely examples. The encoder according to the embodiment of the present application performs block-based encoding of image 12, unlike the encoder in Figure 10, etc., using the concepts outlined in more detail below. For example, it is a still image encoder, not a video encoder. It does not support interpretation. Or, subdivision into blocks 80 is performed in a different way than illustrated in Figure 12. Similarly, the decoder according to the embodiment of the present application performs block-based decoding of image 12' from data stream 14, using the coding concepts outlined further below. Unlike the encoder in Figure 10, etc., it performs block-based encoding of image 12. For example, unlike the decoder 20 in Figure 11, it is a still image decoder, not a video decoder. It does not support intra-prediction. Alternatively, it subsegments image 12' into blocks in a different way than described for Figure 12. And / or, it extracts prediction residue from the data stream 14 of the spatial region rather than the transformation region, for example.

[0032] The process of partitioning an image using slices will now be briefly explained.

[0033] When the H.263 standard is implemented, sequences of data representing adjacent blocks are organized into groups called slices, in a specific scanning order. Generally, dependencies between different slices of an image, such as between prediction and entropy coding, are prohibited. Thus, each slice in an image is reconstructed independently.

[0034] Figure 13 shows image segmentation by slicing in raster scan order. The slice size is determined by the number of CTUs (Coded Tree Units) and the size of each coded CTU belonging to the slice shown in Figure 13. Figure 13 includes 50 CTUs, for example, CTU221, CTU224, and CTU251.

[0035] The image partitioned by tiles will now be briefly explained in relation to Figure 14. Figure 14 includes 50 CTUs, for example, CTU223, CTU227, and CTU241.

[0036] Although the concept is quite similar to Flexible Macroblock Ordering (FMO) added to H.264 / AVC, tiling is a concept introduced in HEVC. The concept of tiling allows an image to be segmented into several rectangular regions.

[0037] Therefore, as shown in Figure 14, tiles are the result of segmenting the original image into a given number of columns and rows by specified heights and widths. Consequently, tiles in an HEVC bitstream are required to have common boundaries that form a regular grid.

[0038] In the following, a typical video encoder according to the embodiment is described in Figure 13. A typical video decoder according to the embodiment is described in Figure 2. And a typical system according to the embodiment is described in Figure 3.

[0039] Figure 1 shows a typical video encoder 101 according to an embodiment.

[0040] The video encoder 101 is configured to encode multiple images of a video by generating an encoded video signal. Each of the multiple images contains the original image data.

[0041] The video encoder 101 includes a data encoder 110 configured to generate an encoded video signal containing encoded image data. The data encoder is configured to encode multiple images of the video into the encoded image data.

[0042] Furthermore, the video encoder 101 includes an output interface 120 configured to output encoded image data for each of multiple images.

[0043] Figure 2 shows a typical video decoder 151 according to an embodiment.

[0044] The video decoder 151 is configured to decode an encoded video signal containing encoded image data in order to reconstruct multiple images of the video.

[0045] The video decoder 151 includes an input interface 160 configured to receive an encoded video signal.

[0046] Furthermore, the video decoder includes a data decoder 170 configured to reconstruct multiple images of the video by decoding the encoded image data.

[0047] Figure 3 shows a typical system according to an embodiment.

[0048] The system includes a video encoder 101 shown in Figure 1 and a video decoder 151 shown in Figure 2.

[0049] The video encoder 101 is configured to generate an encoded video signal. The video decoder 151 is configured to decode the encoded video signal in order to reconstruct the video image.

[0050] The first aspect of this invention is required in claims 1 to 5, claims 45 to 49, and claims 89 to 93.

[0051] A second aspect of this invention is required in claims 6 to 13, claims 50 to 57, and claims 94 to 98.

[0052] A third aspect of this invention is required in claims 14 to 18, claims 58 to 62, and claims 99 to 103.

[0053] A fourth aspect of this invention is required in claims 19 to 22, claims 63 to 66, and claims 104 to 108.

[0054] A fifth aspect of this invention is required in claims 24 to 27, claims 68 to 71, and claims 109 to 113.

[0055] A sixth aspect of this invention is required in claims 28 to 32, in claims 72 to 76, and in claims 114 to 118.

[0056] A seventh aspect of this invention is required in claims 33 and 34, in claims 77 and 78, and in claims 119 through 123.

[0057] The eighth aspect of this invention is required in claims 35 to 38, claims 79 to 82, and claims 124 to 128.

[0058] The ninth aspect of this invention is required in claims 40 to 44, claims 84 to 88, and claims 129 to 133.

[0059] Claims 23, 39, 67, and 83 require specific examples, for example, that are used in different aspects of this invention.

[0060] The following details the complexity to be handled for flexibly sized image partitions according to the embodiments.

[0061] In HEVC, tiles are defined as subsegment structures for an image. They are defined in the Image Parameter Set (PPS), where their dimensions are given. Tiles, as now defined, have a size that is a multiple of CTU (generally 64 samples). Only the last tile of an image, i.e., only the tiles at the right or bottom boundary, are allowed to have fewer than 64 CTUs at the right and bottom image boundaries.

[0062] However, such restrictions prevent a couple of use cases.

[0063] Merging MCTS bitstreams: To ensure that the original image boundaries are not the image boundaries in the merged (combined) bitstream, the bitstreams being merged together within a single bitstream must be a multiple of the size chosen for the CTU.

[0064] To achieve proper load balancing, the tiles are arranged in multiples of a predetermined size (CTU size), but this is not always the case, and they adapt to the content.

[0065] Furthermore, the current video coding standard developed by JVET-Multipurpose Video Coding allows for larger values ​​of CTU (up to 128). This makes it all the more important to allow tiles that are not multiples of CTU. While it is possible to define partial CTUs based on whether the requested functionality is necessary, this will affect coding efficiency in such cases. Nevertheless, it is preferable to allow fewer CTUs for the rightmost and bottom boundary of a tile, rather than allowing a tile to contain the full number of CTUs of its determined size.

[0066] However, when such functionality is permitted, different aspects affect the decoding process.

[0067] The following describes how to control the complexity of partial CTUs within an image.

[0068] In particular, the first aspect of this invention will now be described in detail.

[0069] Currently, all CTUs are the same size. The processor is optimized for decoding configurations of a predetermined size. Variable sizes impose an increase in complexity for the decoder. In particular, the number of CTUs can vary from image to image.

[0070] The first aspect of the present invention is that, when such variable-sized CTUs are allowed within each tile, the limitation is that the number of variable-sized CTUs must be kept constant from image to image.

[0071] In the first aspect, restrictions always apply.

[0072] Figure 4 shows the affected CTU by replacing the partial CTU that generates the tile boundary.

[0073] If the tile size is not a multiple of the predetermined CTU size, then a variable-sized CTU smaller than the predetermined CTU size is introduced. As shown in Figure 4, having the same number of such variable-sized CTUs in each image is a requirement for bitstream matching.

[0074] Furthermore, this characteristic is signaled to the decoder using a flag in the parameter set that indicates num_partial_ctus_constant_flag.

[0075] The second aspect of this invention will now be described in detail.

[0076] The second aspect distinguishes partial CTUs that generate tile boundaries from regular tile boundaries in terms of the influence of their complexity. When the decoder checks the level limit for the maximum amount of decoded pel (per) per time unit, it adds a complexity factor for partial CTUs. For example, if a variable-sized CTU is not limited in size by the tile boundary, then a partial CTU is counted at 1.5 times its actual size, or at a number of pel of a predetermined CTU size. This means that the image size limit is affected by the presence of partial CTUs. For example, if all CTUs were the same size, the image limit for the decoder would be 4096 × 2160 at 50 frames per second. However, if all defined 5 × 4 tiles have partial CTUs at the right and bottom boundaries, i.e., 20 partial CTUs, then the limit is 4056 × 2128 at the same frame rate.

[0077] Figure 5 shows luma samples for each image compensation from a partial CTU. In particular, the complexity-adapted image area is shown in Figure 5.

[0078] Furthermore, the lower limit for the size of variable-sized CTUs is imposed so as not to arbitrarily allow small variable CTUs with sizes between a single sample and a predetermined CTU size in any direction, by requiring that the motion vector or substitution information of the reference image be considered granular in the coding process, for example, by being stored with the granularity of 16 × 16 samples. Variable-sized CTUs are not allowed to be smaller than a predetermined minimum CTU size, for example, indicated or derived from the motion vector storage granularity, so that no two CTUs are within the same motion vector storage unit.

[0079] A third aspect of this invention will now be described in detail.

[0080] In a third, separate aspect, the presence of partial CTUs is taken into consideration for signaling the complexity of a tile. For example, HEVC includes the parameter min_spatial_segmentation_idc in the VUI, which describes the complexity of the spatial segment containing the tile. Now, when applied to a tile, it implies that "there are no tiles in a CVS (encoded video sequence) containing more than (4*PicSizeInSamplesY) / (min_spatial_segmentation_idc+4) luma samples." Furthermore, as part of this invention, complexity signaling is limited for tiles so that they do not contain more than a given number of partial CTUs, dependent on the maximum tile size and / or the relationship between the default CTU size and the partial CTU size. Instead, an additional parameter is signaled in the VUI, indicating the relative amount of partial CTUs allowed in the image.

[0081] In the following, the embodiments will explain how the CTU boundary can change from image to image.

[0082] In particular, the fourth aspect of this invention will now be described in detail.

[0083] If tiles have right / bottom boundaries ending with a granularity smaller than a predefined CTU size, and subsequent tiles begin with a predefined CTU size, then when the tile setup changes from image to image, the alignment of CTUs across the image will not be achieved, which is detrimental to certain predictive coding, such as using temporal motion vector prediction. Memory management becomes more difficult, and it is desirable to limit the flexibility of misalignment of CTU boundaries generated from image to image.

[0084] The fourth aspect should be that any tile boundary that generates a partial CTU can only move to a position that generates a partial CTU in different images within the same video sequence, with the same split ratio. This allows for aligned CTUs in areas not covered by the range from the initial last CTU position and the modified last CTU position, as shown in Figure 6.

[0085] Figure 6 shows the CTU grid alignment mismatch.

[0086] The fifth aspect of this invention will now be described in detail.

[0087] In the fifth aspect, the constraint is applied for the rate of blocks where grid misalignment occurs between images, resulting in altered subtile boundaries, as shown in Figure 7. This enforcement is either defined in the default method for a given profile / level or indicated by a syntactic element in the bitstream, e.g., num_misaligned_ctus_per_picture. Figure 7 shows the ratio of CTUs affected by grid misalignment.

[0088] Instead, the number of partial (boundary) CTUs remains constant at the same ratio. The coercion described here is implemented by a simple coercion directive, such as misalignment_constant_flag, or by a less flexible method, while the tile boundary directives remain flexible, as the coercion must be fulfilled.

[0089] The sixth aspect of this invention will now be described in detail.

[0090] In the sixth aspect, misalignment of CTUs is completely avoided. Two sets of tile boundaries are defined. The tile boundaries of the first set correspond to conventional tile boundaries. • Entropy coding reset • Intra-predictive limit • Scan order conversion Furthermore, the tiles of the first set allow for the generation of partial CTUs at the right and bottom tile boundaries.

[0091] The tile boundaries of the second set, at the right and / or bottom tile boundaries, i.e., to the left and / or above the aforementioned tile boundaries, merely generate partial CTUs and do not imply an entropy coding reset or intra-prediction or scan order transformation. Thus, the position of the partial CTUs in the image can (and is forced) remain static while the actual tile configuration is allowed to switch over time.

[0092] Figure 8 shows two sets of tile boundaries. Within the CVS, a tile boundary that changes position is not allowed to generate new subtiles. Figure 8 illustrates an invention based on two images at different time moments, in which the tile boundaries of the first set change position from time moment T-1 to time moment T, and a new second set of tile boundaries is introduced to maintain the position of the subtiles at time moment T, similar to time moment T-1, even if changes occur in the scan order and entropy coding and intra-prediction.

[0093] The seventh aspect of this invention will now be described in detail.

[0094] In another independent embodiment, the problem is addressed in line buffer requirements by introducing a partial CTU to reduce the impact of complexity.

[0095] Figure 9 shows a sequence of corresponding CTUs in the reference image following a partial CTU in the current image.

[0096] As shown in Figure 9, a horizontal column of partial CTUs in one image, without a column of partial CTUs, leads to an incorrect alignment of the CTU grid with respect to the reference image. A single column (the current column of CTUs) accesses the CUs (CU = coded units) of two columns of CTUs for tools such as ATMVP (Alternative Time-Action Vector Prediction). Since the arranged columns of CTUs are often cached in cost-intensive high-speed memory in hardware execution (referenced as line buffers), it is undesirable to hold a large number of columns of CTUs in this cache. Thus, another limitation in maintaining the alignment of columns of CTUs across an image in the presence of partial CTUs that generate tile boundaries is that the horizontal tile boundaries that generate the partial CTUs on them do not allow to change during the coded video sequence and are therefore static.

[0097] The following describes predictor derivation from an array of CTUs according to an embodiment.

[0098] In particular, the eighth aspect of this invention will now be described in detail.

[0099] TMVP (Time-Action Vector Prediction) and ATMVP derivatives currently depend on a predefined CTU size, i.e., they use a constant CTU grid on the image.

[0100] For TMVP, If yCb >> CtbLog2SizeY is equal to yColBr >> CtbLog2SizeY, then yColBr is less than pic_height_in_luma_samples and xColBr is less than pic_width_in_luma_samples. (That is, if the bottom right corner CB of the reference image is in the same column of CTU, it will later be cached in line buffer memory.)

[0101] Then, the predictors arranged at the bottom right are taken. Otherwise (assuming the bottom right CB of the reference image is NOT in the same column of CTU, it will later be NOT in the line buffer), the predictors arranged in the center are taken.

[0102] For ATMVP, the positions (xColSb, yColSb) of the subblocks arranged inside ColPic are extracted as follows: xColSb=Clip3(xCtb, Min(CurPicWidthInSamplesY-1, xCtb+(1<<CtbLog2SizeY)+3)、xSb+(tempMv[0]> >4)) yColSb=Clip3(yCtb, Min(CurPicHeightInSamplesY-1, yCtb+(1<<CtbLog2SizeY)-1)、ySb+(tempMv[1]> >4))

[0103] In other words, the coordinates of the arranged subblocks are truncated in each dimension so that they lie inside the same CTU column. The same line buffer considerations apply.

[0104] Both TMVP and ATMVP use a predefined CTU size CtbLog2SizeY to divide all samples of the encoded image into encoded blocks for the derivation of arrayed blocks and subblocks.

[0105] Because of the tile boundaries that are allowed to generate partial CTUs, the CTU grid is not constant and changes from image to image, generating overlaps between columns of varying numbers of CTUs, as shown in Figure 9. In this part of the invention, the CTU grid of the current image is imposed on a reference frame to organize buffer management and predictor access. That is, the dimensions or size of the current local partial CTU are applied to select a reference area in the (arranged) reference image used (so that it is contrasted with a constant maximum CTU size across the image).

[0106] For TMVP, one embodiment of this invention is as follows:

[0107] yTileStart and xTileStart correspond to the top-left sample position of a given tile that operates the current coded block, corresponding to the tile start and the associated sample position of the current coded block corresponding to the arranged block, and yCbInTile and xCbInTile correspond to the top-left sample position of the tile that operates the current coded block.

[0108] (The difference is emphasized in bold type): yCbInTile = yCb - yTileStart yColBrInTile=yColBr-yTileStart CtbLog2HeightY(y) describes the height of each CTU column. If yCbInTile >> CtbLog2HeightY(yCbInTile) is equal to yColBrInTile >> CtbLog2HeightY(yCbInTile), then yColBr is less than pic_height_in_luma_samples, and xColBr is less than pic_width_in_luma_samples.

[0109] Then, the predictors arranged at the bottom right are taken. Otherwise, the predictors arranged in the center are taken.

[0110] For ATMVP, the following are embodiments of the adjusted clip operation: The positions (xColSb, yColSb) of the subblocks arranged inside ColPic are derived as follows: xColSb=Clip3(xCtb, Min(CurPicWidthInSamplesY-1, xCtb+(1<<CtbLog2WidthY(xCtb))+3)、xSb+(tempMv[0]> >4)) yColSb=Clip3(yCtb, Min(CurPicHeightInSamplesY-1, yCtb+(1< <CtbLog2HeightY(yCtb))-1)、 ySb+(tempMv[1]>>4))

[0111] The following describes the restrictions on the sub-image aspect coding tool.

[0112] In particular, the ninth aspect of this invention will now be described in detail.

[0113] In previous video coding standards, the limitations on coding sub-images (e.g., slices or tiles) were distinguished as follows: I_SLICE-Intracoding (usually random access points) does not use a reference for other images. P_SLICE - Predictive coding. Use one image from a single reference image list for motion-compensated prediction. B_SLICE - A bipredictive coding with two reference image lists. Allows combining action-compensated predictions from images in both lists.

[0114] In this context, the terms slice or tile are interchangeable. That is, a continuous group of CTUs in a bitstream belong to a single encoded image and together represent encoded entropy.

[0115] This difference allows for two things: - From analyzing the components of the slice or tile header, to controlling the analysis and decoding process of the components of each sub-image immediately above, and - Profiling based on such types by prohibiting the use of computationally complex types such as B_SLICES.

[0116] The type indication can also be interpreted as a bag of associated restrictions for each type, for example, the I_SLICE type, which is the most restrictive type; the P_SLICE type, which relaxes the restriction to include interpretations using one reference image; and the B_SLICE type, which relaxes another restriction to include criteria from both temporally preceding and succeeding images by allowing a bitstream order of images different from the order in which they are provided.

[0117] The following problems arise. -I_SLICE types also include "current image references," which are P_SLICE types that are "behavior-compensated" or "translation-compensated" blocks based on predictions that use the current coded image as a reference. -P_SLICE is no longer frequently used and is usually replaced by B_SLICE in applications. - Interpretation evolves beyond the use of two reference frames.

[0118] Current image referencing, as described above, imposes additional complexity on the decoding process compared to conventional intra-prediction. Traditionally, intra-prediction relies only on sample values ​​in the current image that are immediately near a small number (e.g., three) of lines of samples in the direct neighborhood, or in so-called multi-reference line prediction, to generate sample values ​​for the predicted block, for example, through interpolation. Current image referencing, on the other hand, relies on a mechanism previously used in inter-prediction between frames of a video sequence, where sample values ​​of a reference block are copied to the position of the current block. The amount of samples accessed to generate predictors in this technique is much higher than in normal intra-prediction. Therefore, predictor samples are not only drawn from a much smaller number of accessed samples compared, but the lookup on the encoder side to extract suitable action vectors or substitution information is also much higher. Furthermore, the reference image needs to be updated along with the current image decoding process to provide a larger area on top of the current image for reference, i.e., the availability of a reconstructed area.

[0119] In one embodiment of this invention, instead of using the type distinction described above, the actual imposed tool restrictions are signaled in the (tile) header.

[0120] 1) The flag is used to indicate whether a reference image other than the current image is being used (intra / interswitch).

[0121] 2) A flag is used to indicate the use of the current image reference (interpretation using the current encoded image). This is communicated by a signal dependent on intra / inter selection.

[0122] 3) The syntactic element is signaled to indicate the number of reference image lists to be used in the prediction process. This syntactic element is assumed to be equal to 1 when only the current image reference is used. In other cases, a value of 1 applies to what was previously a P_SLICE type, while a value of 2 applies to what was previously a B_SLICE type. Additional values ​​are used, for example, to indicate multi-hypothesis prediction. The example syntax is shown in Table 1.

[0123] [Table 1]

[0124] A current_picture_only_reference_flag equal to 1 indicates that only the current image is allowed to be used as a reference. This is similar to the traditional I_SLICE type.

[0125] A current_picture_inter_reference_flag equal to 1 indicates that the current image is available for inter-prediction (the "Current Image Reference" or "Intra-Block Copy" tool).

[0126] num_ref_pic_lists_active indicates the number of active reference image lists. Assuming current_picture_only_reference_flag is equal to 1, If current_picture_inter_reference_flag is equal to 0, num_ref_pic_lists_active is active to 0 (and therefore excluded from the bitstream). If current_picture_only_reference_flag is equal to 0, A value of num_ref_pic_lists_active equal to 1 indicates a tile type similar to P_SLICE. num_ref_pic_lists_active, which is equal to 2, indicates a tile type similar to B_SLICE. A value of num_ref_pic_lists_active greater than 2 is used to indicate the use of more reference image lists, for example, for multi-hypothesis interpretation.

[0127] While several aspects have been described within the context of the apparatus, it is clear that these aspects also represent a description of the corresponding method. A block or device corresponds to a method step or a function of a method step. Similarly, an aspect described within the context of a method step also represents a description of the corresponding block, item, or function of the corresponding apparatus. Some or all method steps are performed by a hardware device (or the use of a hardware device), such as a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, one or more of the most important method steps are performed by such a device.

[0128] Embodiments of this invention can be implemented in hardware, or in software, or at least partially in hardware, or at least partially in software, depending on the specific implementation requirements. Implementations can be carried out in cooperation with (or potentially in cooperation with) a programmable computer system, using a digital storage medium, such as a floppy disk, DVD, Blu-ray, CD, ROM, PROM, EPRROM, EEPROM, or flash memory, which contains electronically readable control signals. Thus, the digital storage medium is computer-readable.

[0129] Some embodiments of this invention include a data carrier having an electronically readable control signal, which may cooperate with a programmable computer system so that one of the methods described herein can be performed.

[0130] Generally, embodiments of this invention can be implemented as a computer program product having program code. The program code is manipulated to perform one of the methods when the computer program product runs on a computer. The program code is stored, for example, on a machine-readable carrier.

[0131] Other embodiments include a computer program stored on a machine-readable carrier for performing one of the methods described herein.

[0132] In other words, an embodiment of the method of this invention is a computer program having program code for performing one of the methods described herein when the computer program is running on a computer.

[0133] Another embodiment of the method of this invention is a data carrier (or digital storage medium, or computer-readable medium) thereon containing a computer program for performing one of the methods described herein, on which such a program is recorded. The data carrier or digital storage medium or recorded medium is generally tangible and / or non-temporary.

[0134] Another embodiment of the method of this invention is a data stream or sequence of signals representing a computer program for performing one of the methods described herein. The data stream or sequence of signals is configured to be transmitted over a data communication connection, for example, over the Internet.

[0135] Another embodiment includes processing means, such as a computer or a programmable logical device, configured or applied to perform one of the methods described herein.

[0136] Another embodiment includes a computer on which a computer program for performing one of the methods described herein is installed.

[0137] Another embodiment according to this invention includes an apparatus or system configured to transfer (e.g., electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver is, for example, a computer, a mobile device, a memory device, or the like. The apparatus or system includes, for example, a file server for transferring the computer program to the receiver.

[0138] In some embodiments, a programmable logic device (e.g., a field-programmable gate array) is used to perform some or all of the functions of the methods described herein. In some embodiments, the field-programmable gate array works in cooperation with a microprocessor to perform one of the methods described herein. In general, the methods are preferably performed by any hardware device.

[0139] The apparatus described herein is implemented using hardware devices, a computer, or a combination of hardware devices and a computer.

[0140] The methods described herein are performed using hardware devices, computers, or a combination of hardware devices and computers.

[0141] The embodiments described above are presented solely for the sake of the principles of the invention. It will be understood that partial modifications and variations of the arrangement and details described herein will be obvious to those skilled in the art. Accordingly, it is intended that the scope of the claims may be limited only to what is likely to happen, and not by the specific details provided in the manner of the description and explanation of the embodiments herein.

Claims

1. A hardware device for video decoding, A first flag indicating whether to predict a block of tiles based on a sample reconstructed with respect only to the current image and not to other images, wherein the first flag included in the tile header of the current image is decoded from the data stream, A second flag indicating whether to predict the current block of the tile by copying a sample from a reference block of the sample reconstructed with respect to the current image, in response to a first flag indicating that the block of the tile is predicted based on a sample reconstructed with respect to the current image only and not to other images, wherein the second flag included in the header of the tile containing the first flag is decoded from the data stream, Depending on the second flag indicating that the current block is predicted by copying a sample from the reference block, the prediction of the current block in the current image is determined by copying a sample from the reference block. The current block is reconstructed based on the prediction of the current block and the prediction residual decoded from the data stream. A hardware device configured in such a way.

2. The hardware device according to claim 1, characterized in that the tile is composed of one or more coding tree units.

3. A method for video decoding, A first flag indicating whether to predict a block of tiles based on a sample reconstructed with respect only the current image and not other images, the step of decoding the first flag included in the tile header of the current image from a data stream, A step of decoding a second flag from the data stream, which is included in the header of the tile containing the first flag, in response to a first flag indicating that the current block of the tile is predicted by copying a sample from a reference block of the sample reconstructed with respect to the current image, in response to a first flag indicating that the block of the tile is predicted based on a sample reconstructed with respect to the current image only and not to other images; The steps include determining the prediction of the current block in the current image by copying a sample from the reference block, in accordance with the second flag indicating that the current block will be predicted by copying a sample from the reference block, The steps include: reconstructing the current block based on the prediction of the current block and the prediction residual decoded from the data stream; A method that includes this.

4. The method according to claim 3, characterized in that the tile is composed of one or more coding tree units.

5. A non-transient digital storage medium, The aforementioned non-transient digital storage medium stores a computer program. When the computer program is executed by the computer, the computer program A first flag indicating whether to predict a block of tiles based on a sample reconstructed with respect only the current image and not other images, the step of decoding the first flag included in the tile header of the current image from a data stream, A step of decoding a second flag from the data stream, which is included in the header of the tile containing the first flag, in response to a first flag indicating that the current block of the tile is predicted by copying a sample from a reference block of the sample reconstructed with respect to the current image, in response to a first flag indicating that the block of the tile is predicted based on a sample reconstructed with respect to the current image only and not to other images; The steps include determining the prediction of the current block by copying a sample from the reference block, in accordance with the second flag indicating that the current block will be predicted by copying a sample from the reference block, The steps include: reconstructing the current block in the current image based on the prediction of the current block and the prediction residual decoded from the data stream; A non-transient digital storage medium characterized by performing the following:

6. The non-transient digital storage medium according to claim 5, characterized in that the tile is composed of one or more coding tree units.

Images