Device for selecting intra-prediction mode for padding

Abstract

To provide a video codec configured to perform a boundary padding to improve the quality of a time-wise inter-prediction.SOLUTION: The present invention relates to a video codec to support a time-wise inter-prediction, which is configured to perform padding of an area across a reference picture boundary, of a reference part 100 of a reference picture 101. The reference part is referred to by selecting one of a plurality of intra prediction modes and padding an area by using the selected intra prediction mode.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] This application relates to video coding / decoding. [Background technology]

[0002] All video coding standards, including the state-of-the-art ITU T H.265|MPEG H HEVC video coding standard (reference 1: ITU-T and ISO / IEC. High Efficiency Video Coding. ITU-T Recommendation H.265|ISO / IEC 23008 10 (HEVC), edition 1, 2013; edition 2, 2014), since H.261, rely on the basic concept of block-based hybrid video coding. Here, the input signal for each color component is segmented into disjoint blocks that are coded using a hybrid video coding scheme based on prediction, transformation, quantization, and entropy coding. Thus, one of the first steps in the encoding and decoding algorithms is the generation of a prediction signal for the current block, which is done either by spatial or temporal prediction schemes. In the case of temporal prediction, the prediction signal is derived from an already processed frame, denoted as reference frame. For this purpose, a displacement vector specifies the offset of the prediction signal in the reference frame relative to the position of the current coded block. This temporal prediction scheme is also known as translational motion compensated prediction. In this sense, the displacement vector is denoted as a motion vector.

[0003] To allow the motion vectors to have sub-pixel accuracy, the prediction signal is calculated using an interpolation filter. Furthermore, the prediction signal can be constructed by superposition of several prediction signals, each specified by a different motion vector and / or a different reference frame. Regardless of the final calculation and construction of the prediction signal, the basic principle is the same. The reference block area in the reference frame is determined by the position and size of the current coded block displaced by an additional motion vector. If an interpolation filter is used, this area must be expanded on each side depending on the number of filter taps used. The pixels in the reference block area are then used to generate the temporal prediction signal of the current coded block. Usually, the reference block area is in the reconstructed reference frame. Nevertheless, the reference block area may not completely overlap with the reconstructed frame area, which means that part of the reference pixel area may be outside the reference frame area.

[0004] In current video coding standards, the basic approach to handle reference parts, i.e., reference pixel areas, outside the reference picture, i.e., reference frame area, is to constrain motion vectors or extend the reference frame area. For example, in H.261, motion vectors are constrained to picture boundaries so that all reference pixels are within the reference frame area, i.e., all samples of the sample part of the reference picture are within the boundary of the reference frame. Alternatively, in H.264 / H.265, a process called border padding is used to extend the already reconstructed reference frame as shown in Fig. 16. That is, border pixels (border samples 110) are copied to the padding area (area outside the reference picture 101), which can be seen as intra prediction of border pixels perpendicular to the frame boundary 106. Now, at the edge of the reference frame, the edge pixels are further copied to the remaining area outside the reference frame 101 area in the vicinity of the edge pixels as shown in Fig. 16.

[0005] Furthermore, known techniques of border padding are advantageous when dealing with certain video content known as omnidirectional or 360° video, which uses an omnidirectional projection to map the surrounding cameras onto a rectangular video picture plane. One widely used format for such content is based on equirectangular projection (ERP). Omnidirectional content allows areas outside the picture boundary to be filled with the correct content, since the correct content is by definition depicted elsewhere in the same picture. The vertical picture border of ERP video content that depicts 360° horizontally can be padded by copying samples from the opposite vertical picture border. However, the horizontal picture border exhibits different characteristics, especially when the vertical viewing angle is not full coverage. Summary of the Invention [Problem to be solved by the invention]

[0006] In Fig. 17, an exemplary ERP video frame is shown with a horizontal coverage angle of 360° and a vertical coverage angle of less than 180°, i.e., only 30° below the equator at 0°. Although optimal padding for vertical picture boundaries consists of using samples from the opposite vertical picture side, the top and bottom horizontal picture boundaries marked with dashed lines exhibit similar characteristics to the picture boundaries of typical rectangular videos with limited field of view and therefore benefit from the multi-mode border padding described herein. Thus, one aspect of the invention is to change the padding mode for each picture boundary, for example, depending on knowledge of the content characteristics or when signaled in the bitstream.

[0007] In many cases, simple vertical extension of frame boundary pixels leads to a splice that does not closely approximate the current predicted block. Better splices may be achieved by using different intra prediction modes to extend the reference frame boundary, such as angular prediction, DC prediction, or planar prediction. Exemplary intra prediction modes that can be used for boundary extension are those from H.265 described in Reference 1 or from the Joint Exploration Test Model (JEM) described in Reference 2 (J. Chen, E. Alshina, GJ Sullivan, J.-R. Ohm, J. Boyce, Algorithm Description of Joint Exploration Test Model 7 (JEM7), JVET, doc. JVET-G1001, July 2017). Furthermore, in future embodiments of the method, future intra prediction modes can also be used to extend the reference boundary. The spatial extension mode of the frame boundary depends on the characteristics of the current predicted block. Therefore, a particular angular intra prediction mode needs to be chosen / selected from a set of multiple angular intra prediction modes to generate a prediction signal that provides the best approximation of the current prediction block. [Means for solving the problem]

[0008] It is therefore an object of the present invention to provide a video codec arranged to perform boundary padding to improve the quality of temporal inter prediction.

[0009] This object is achieved by the subject matter of the claims of the present application.

[0010] The basic idea underlying this application is that an improvement in the quality of temporal inter prediction is achieved by selecting the most appropriate intra prediction mode for padding, which may be a selection of the angular intra prediction mode among multiple such intra prediction modes based on the characteristics of the current prediction block.

[0011] According to one embodiment, the angular intra prediction mode used for padding is selected based on an evaluation of the reference picture area at the part intersecting the reference picture boundary with the reference part. The evaluation is performed for each of at least a subset of the angular intra prediction modes by filling a probe area further from the boundary than the boundary sample along a direction opposite to the filling direction associated with the respective angular intra prediction mode based on boundary samples of the reference picture at the part of the boundary, to obtain a probe filling, and comparing the probe filling and the reconstructed probe area filling to measure the similarity between them. Thus, the selection is made based on the characteristics of the current prediction block, thus resulting in the best approximation of the current prediction block and improving the prediction quality.

[0012] According to some embodiments of the present application, the angular intra prediction mode used for padding is selected based on the prediction result of an area outside the reference picture of the reference portion. The reference portion is referenced by an inter prediction block of the current picture. Therefore, the angular intra prediction mode is selected according to the characteristics of the area, and thus the most appropriate angular intra prediction mode for padding can be selected. In addition, multiple angular intra prediction modes used to predict the area can be signaled as a subset of supported angular intra prediction modes supported by the codec of the intra prediction block.

[0013] Further advantages are achieved by the claimed codec.Preferred embodiments of the present application are explained below with reference to the drawings. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1 is a block diagram of an apparatus for predictive coding of video, as an example of a video decoder capable of implementing angular intra-prediction mode selection for padding concepts according to some embodiments of the present application. [Diagram 2] 2 is a block diagram of an apparatus for predictively decoding video, compatible with the apparatus of FIG. 1, as an example of a video decoder capable of implementing angular intra-prediction mode selection for padding concepts according to some embodiments of the present application. [Diagram 3] 1A and 1B are schematic diagrams showing examples of relationships between a prediction residual signal, a prediction signal and a reconstructed signal, respectively, to illustrate the functionality of setting subdivisions for defining a prediction signal, for handling a prediction residual signal, etc. [Figure 4] FIG. 2 is a schematic diagram illustrating an example of an angular intra-prediction mode used in some embodiments of the present application. [Diagram 5] FIG. 1 is a schematic diagram illustrating an example of calculating a preferential border padding side according to an embodiment of the present application; [Figure 6] FIG. 13 is a schematic diagram illustrating a further example of calculating a preferential border padding side according to an embodiment of the present application; [Figure 7] FIG. 2 is a schematic diagram illustrating an example of a probe area according to an embodiment of the present application. [Figure 8] FIG. 2 is a schematic diagram showing an example used to explain the process of defining a probe area according to an embodiment of the present application. [Figure 9] FIG. 13 illustrates a sample table for deriving probe area coordinates according to an embodiment of the present application. [Figure 10] FIG. 2 is a schematic diagram illustrating an example of motion vector clipping of a boundary extension reference block according to an embodiment of the present application; [Figure 11]FIG. 2 is a schematic diagram illustrating an example of deriving a template area related to motion vector clipping according to an embodiment of the present application; [Figure 12] FIG. 2 is a schematic diagram illustrating an example of block size adaptive motion vector clipping according to an embodiment of the present application; [Figure 13] FIG. 1 is a schematic diagram illustrating an angular intra-prediction mode used for padding, selected based on prediction results for areas outside the reference picture of the reference portion according to the present application. [Figure 14] FIG. 1 is a schematic diagram illustrating an angular intra-prediction mode used for padding, selected based on prediction results for areas outside the reference picture of the reference portion according to the present application. [Figure 15] FIG. 2 is a schematic diagram illustrating an example of a horizontal boundary between two independent spatial segments according to an embodiment of the present application. [Figure 16] FIG. 1 is a schematic diagram showing vertical prediction to explain the state of the art. [Figure 17] FIG. 2 shows a sample image of a video frame with a horizontal coverage angle of 360° and a vertical coverage angle of less than 180°. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] The following description of the figures begins with the presentation of a description of a video encoder and a video decoder of a block-based predictive codec for coding pictures of a video, to give examples of coding frameworks in which embodiments of the configured predictive codec may be incorporated. The video encoder and the video decoder are described in relation to Figures 1 to 3. Below, a description of embodiments of the configured predictive concept of the present application is presented together with an explanation of how such concepts may be incorporated into the video encoder and the decoder of Figures 1 and 2, respectively, although the embodiments described thereafter in Figures 4 onwards may also be used to give video encoders and video decoders that do not operate according to the coding framework underlying the video encoder and the video decoder of Figures 1 and 2.

[0016] Fig. 1 shows an apparatus for predictively coding a video 11 composed of a sequence of pictures 12 into a data stream 14. Block-wise predictive coding is used for this purpose. Furthermore, transform-based residual coding is exemplarily used. The apparatus, i.e. the encoder, is indicated with the reference number 10. Fig. 2 shows a corresponding decoder 20, i.e. an apparatus 20 arranged to predictively decode a video 11' composed of pictures 12' of picture blocks from the data stream 14, again exemplarily using transform-based residual decoding, the apostrophe being used to indicate that the pictures 12' and the video 11' reconstructed by the decoder 20 deviate respectively from the picture 12 originally coded by the apparatus 10 in terms of coding losses introduced by quantization of the predictive residual signal. Although Figs. 1 and 2 exemplarily use transform-based predictive residual coding, the embodiments of the present application are not limited to this type of predictive residual coding. This also applies to other details explained in connection with Figs. 1 and 2 outlined below.

[0017] The encoder 10 is configured to subject the prediction residual signal to a spatial-to-spectral transform and to encode the resulting prediction residual signal into a data stream 14. Similarly, the decoder 20 is configured to decode the prediction residual signal from the data stream 14 and to subject the resulting prediction residual signal to a spectral-to-spatial transform.

[0018] Internally, the encoder 10 may comprise a prediction residual signal generator 22 for generating a prediction residual 24 in order to measure the deviation of a prediction signal 26 from the original signal, i.e. the video 11 or the current picture 12. The prediction residual signal generator 22 may for example be a subtractor for subtracting the prediction signal from the original signal, i.e. the current picture 12. The encoder 10 then further comprises a transformer 28 for subjecting the prediction residual signal 24 to a spatial to spectral transformation to obtain a spectral domain prediction residual signal 24', which is then subjected to quantization by a quantizer 32 also comprised in the encoder 10. The prediction residual signal 24'' thus quantized is coded into the bitstream 14. For this purpose, the encoder 10 may optionally comprise an entropy coder 34 for entropy coding the transformed and quantized prediction residual signal into the data stream 14. Based on the prediction residual signal 24'' decoded to and decodable from the data stream 14, a prediction residual 26 is generated by a prediction stage 36 of the encoder 10. For this purpose, the prediction stage 36 may internally comprise, as shown in FIG. 1, an inverse quantizer 38 for inverse quantizing the prediction residual signal 24'' to obtain a spectral domain prediction residual signal 24'''' corresponding to the signal 24' except for quantization losses, followed by an inverse transformer 40 for subjecting the prediction residual signal 24''' to an inverse transformation, i.e., a spectral to spatial transformation, to obtain a prediction residual signal 24'''' corresponding to the original prediction residual signal 24 except for quantization losses. A combiner 42 of the prediction stage 36 then recombines the prediction signal 26 and the prediction residual signal 24'''', e.g., by addition, to obtain a reconstructed signal 46, i.e., a reconstruction of the original signal 12. The reconstructed signal 46 may correspond to the signal 12'.

[0019] A prediction module 44 of the prediction stage 36 then generates the predicted signal 26 based on the signal 46, for example by using spatial prediction, i.e. intra-prediction, and / or temporal prediction, i.e. inter-prediction, as will be described in more detail below.

[0020] Similarly, the decoder 20 is internally composed of corresponding components that may be interconnected in a manner corresponding to the prediction stage 36. In particular, an entropy decoder 50 of the decoder 20 is able to entropy decode a quantized spectral domain prediction residual signal 24″ from the data stream, followed by a dequantizer 52, an inverse transformer 54, a combiner 56 and a prediction module 58 interconnected and cooperating in the manner described above with respect to the modules of the prediction stage 36 to recover a reconstructed signal based on the prediction residual signal 24″, such that the output of the combiner 56 results in a reconstructed signal, i.e. the video 11′ or its current picture 12′, as shown in FIG.

[0021] Although not specifically mentioned above, it is clear that the encoder 10 can set certain coding parameters, including, for example, prediction modes, motion parameters, etc., according to some optimization scheme, such as, for example, optimizing some rate- and distortion-related criteria, i.e., coding cost, and / or using some rate control. As will be explained in more detail below, the encoder 10 and the decoder 20 and corresponding modules 44, 58 support different prediction modes, such as intra- and inter-coding modes, respectively, which form a kind of set or pool of primitive prediction modes, according to which the prediction of picture blocks is configured in a manner that will be explained in more detail below. The granularity with which the encoder and the decoder switch between these prediction configurations may correspond to a subdivision of the pictures 12 and 12', respectively, into blocks. It is noted that some of these blocks are only intra-coded blocks, some are only inter-coded blocks, and optionally still other blocks may be blocks obtained using both intra- and inter-coding, but with the details set subsequently. According to the intra-coding mode, the prediction signal of a block is obtained based on the spatial already coded / decoded neighbors of the respective block. There may be several intra-coding sub-modes, the choice of which seemingly represents a kind of intra-prediction parameters. There may be directional or angular intra-coding sub-modes, according to which the prediction signal of the respective block is filled by extrapolating to the respective block the neighboring sample values ​​along a certain direction specific to the respective directional intra-coding sub-mode.The intra-coding sub-modes may also include one or more further sub-modes, such as, for example, a DC coding mode according to which the prediction signal of the respective block assigns a DC value to all samples within the respective block, and / or a Planar intra-coding mode according to which the prediction signal of the respective block is approximated or determined to be a spatial distribution of sample values ​​described by a two-dimensional linear function over the sample positions of the respective block, with the slope and offset of the plane defined by the two-dimensional linear function based on the neighboring samples. In comparison, according to an inter-prediction mode, the prediction signal of a block is obtained, for example, by temporally predicting the interior of the block. For the parameterization of the inter-prediction mode, a motion vector is signaled in the data stream, which motion vector indicates the spatial displacement of the part of the previously coded picture of the video 11 from which the previously coded / decoded picture is sampled in order to obtain the prediction signal of the respective block. This means that in addition to the residual signal coding contained in the data stream 14, such as the entropy coded transform coefficient levels representing the quantized spectral domain prediction residual signal 24'', the data stream 14 also encodes therein prediction related parameters for assigning a prediction mode to the blocks, prediction parameters of the assigned prediction mode, such as motion parameters for an inter prediction mode, and optionally further parameters for controlling the construction of the final prediction signal of the blocks using the assigned prediction mode and prediction parameters, as outlined in more detail below. In addition, the data stream may contain parameters for controlling and signaling the subdivision of the pictures 12 and 12' into blocks. The decoder 20 uses these parameters to subdivide the pictures in the same way as the encoder did, to assign the same prediction modes and parameters to the blocks, to perform the same predictions, resulting in the same prediction signals.

[0022] Fig. 3 illustrates the relationship between, on the one hand, the reconstructed signal, i.e. the reconstructed picture 12', and, on the other hand, the combination of the prediction residual signal 24'''' and the prediction signal 26 signaled in the data stream. As already indicated above, the combination can be additive. The prediction signal 26 is illustrated in Fig. 3 as a subdivision of the picture area into blocks 80 of various sizes, but this is only an example. The subdivision can be any subdivision, such as a regular subdivision of the picture area into rows and columns of blocks, or a multi-tree subdivision into leaf blocks of various sizes, such as a quad-tree subdivision of the picture 12, a mixture of which is illustrated in Fig. 3, where the picture area is first subdivided into rows and columns of tree root blocks and then further subdivided according to a recursive multi-tree subdivision, resulting in the blocks 80.

[0023] The prediction residual signal 24'''' in FIG. 3 is also illustrated as a subdivision of the picture area into blocks 84. These blocks may be called transform blocks to distinguish them from the coding blocks 80. In effect, FIG. 3 illustrates that the encoder 10 and the decoder 20 may use two different subdivisions of the picture 12 and the picture 12' into blocks, respectively, one into coding blocks 80 and the other into blocks 84. Although both subdivisions may be the same, i.e. each block 80 may simultaneously constitute a transform block 84 and vice versa, FIG. 3 illustrates the case where the subdivision into transform blocks 84 constitutes an extension of the subdivision into blocks 80, for example, such that the boundary between two blocks 80 overlaps with the boundary between two blocks 84, or in other words, such that each block 80 coincides with one of the transform blocks 84 or coincides with a cluster of transform blocks 84. However, these subdivisions may also be determined or selected independently of each other, such that the transform blocks 84 may alternatively cross the block boundaries between the blocks 80. Thus, as far as the subdivision into transform blocks 84 is concerned, the same expressions as those mentioned above in relation to the subdivision into blocks 80 apply, i.e. the blocks 84 may be the result of a normal subdivision of the picture area into blocks, arranged in rows and columns, or the result of a recursive multi-tree subdivision of the picture area, or a combination of these, or any other type of segmentation. As an aside, it should be noted that the blocks 80 and 84 are not limited to square, rectangular, or other shapes. Furthermore, the subdivision into blocks 80 of the current picture 12, from which the prediction signal is generated, and into blocks 84 of the current picture 12, from which the prediction residual is coded, are not the only subdivisions used for coding / decoding. These subdivisions from the granularity at which the determination of the prediction signal and the residual coding are performed may differ in that, firstly, the residual coding may alternatively be performed without subdivision, and secondly, at other granularities than these subdivisions, the encoder and decoder may set certain coding parameters, which may include some of the above-mentioned parameters, such as prediction parameters, prediction signal configuration control signals, etc.

[0024] 3 illustrates that the combination of the prediction signal 26 and the prediction residual signal 24'''' directly results in the reconstructed signal 12'. However, it should be noted that according to alternative embodiments, more than one prediction signal 26, e.g. prediction signals obtained from other views or from other coding layers that are coded / decoded in separate prediction loops with separate DPBs, can be combined with the prediction residual signal 24'''' to result in the picture 12'.

[0025] In Fig. 3, the transform block 84 shall have the following meaning: the transformer 28 and the inverse transformer 54 perform the transform in units of the transform block 84. For example, many codecs use some kind of DST or DCT for all transform blocks 84. Some codecs can skip the transform, so that for some of the transform blocks 84, the prediction residual signal is directly coded in the spatial domain. However, according to the embodiments described below, the encoder 10 and the decoder 20 are configured such that they support several transforms. For example, the transforms supported by the encoder 10 and the decoder 20 may include: ○DCT-II (or DCT-III), where DCT stands for discrete cosine transform ○DST-IV, where DST stands for discrete sine transform ○DCT-IV ○DST-VII Identity transformation (IT)

[0026] Naturally, the transformer 28 will support all forward transform versions of these transforms, while the decoder 20 or inverse transformer 54 will support the corresponding backward or inverse versions: ○Inverse DCT-II (or inverse DCT-III) ○Reverse DST-IV ○Reverse DCT-IV ○Reverse DST-VII Identity transformation (IT)

[0027] In any case, it should be noted that the set of supported transforms may only include one transform, such as one spectral-to-spatial or spatial-to-spectral transform.

[0028] As already outlined above, Figures 1 to 3 are presented as examples in which the configured prediction concept, which will be described further below, can be implemented, in order to give specific examples of video encoders and decoders according to the present application. So far, the video encoders and decoders of Figures 1 and 2 respectively represent possible implementations of the video encoders and decoders described further below in this specification. As outlined in more detail below, when the video encoders and decoders of Figures 1 and 2 incorporate the below-described embodiments for configured prediction according to the present application, the video encoders of Figures 1 and 2 support, as at least one option, the processing of block 80 or even all blocks constituting the current picture 12 in a manner outlined in more detail below. The embodiments described below therefore refer inter alia to a video encoder equivalent to the encoder 10 of Figure 1 which handles block 80 in a manner outlined in more detail below, the same being true with respect to the decoder of Figure 2, which therefore represents an example of a video decoder according to an embodiment in which block 80 is handled in a manner outlined in more detail below. However, Figures 1 and 2 are merely illustrative examples. A video encoder according to some embodiments of the present application may perform block-based coding using concepts outlined in more detail below, but may differ from the encoder of FIG. 1, for example, in that the subdivision into blocks 80 is performed differently than illustrated in FIG. 3, or in that the encoder does not use transform predictive residual coding, but instead directly codes the predictive residual, for example, in the spatial domain. Similarly, a video decoder according to some embodiments of the present application may perform decoding from a data stream 14 using configured predictive coding concepts outlined further below, but may differ from the decoder 20 of FIG. 2, for example, in that it subdivisions the picture 12′ into blocks differently than described in connection with FIG. 3, and / or derives the predictive residual from the data stream 14, for example, in the spatial domain instead of the transform domain.

[0029] It should be noted that, in particular, with regard to the block subdivision into blocks 80, the same can be done in the manner outlined in relation to Fig. 3 or in a different manner. The subdivision into transform blocks, if present, can also be done as described in relation to Fig. 3 or in a different manner. In particular, the subdivision into other blocks, such as into blocks on the one hand and into transform blocks on the other hand, can be done by subdivision of the picture 12 into these blocks separately, each independently or dependently. For example, one subdivision, such as into transform blocks, can be an extension of the other subdivision as described above, or both subdivisions can be separate extensions of a common primary subdivision, such as, for example, into the tree-root block array of the picture as described in relation to Fig. 3. This also applies to other sub-picture granularities mentioned below, such as with regard to the definition of certain prediction parameters, prediction modes, contribution weights, etc. Different subdivisions can be used for different of these entities, and the same can be defined independently, partially independently or as extensions from each other.

[0030] Nevertheless, the following description focuses on the situation where the motion vector points towards a portion in the reference picture plane that intersects a reference picture boundary beyond which the picture content is unavailable. To pad a sub-portion that extends beyond the boundary line or is completely outside the reference picture, a selection of one of multiple intra prediction modes according to an embodiment of the present invention is made. The aim is to improve the temporal inter prediction quality by selecting the most appropriate angular intra prediction mode for padding based on the characteristics of the prediction block.

[0031] In the following description, the term "reference pixel area" corresponds to "reference portion", "reference frame" corresponds to "reference picture", "frame boundary" corresponds to "reference picture boundary", "boundary pixel" corresponds to "boundary sample", and "template area" corresponds to "probe area". Each corresponding term may be used in the description. In addition, "padding" has the same meaning as "boundary padding" in this specification. Furthermore, in the following embodiment, one of the multiple Angular intra prediction modes is selected as an example of an intra prediction mode for padding. However, the intra prediction mode of the present invention is not limited to the Angular intra prediction mode, and the multiple intra prediction modes may include, for example, a DC mode and / or a Planar mode, and / or a mode for padding other intra prediction modes. For the DC mode and the Planar mode, the vertical direction (perpendicular to the frame boundary) may be used for the boundary sample.

[0032] 4 illustrates 33 different angular intra prediction modes used in some embodiments of the present application, where the prediction angle in a reference portion (reference pixel area) 100 of a reference picture (reference frame) 101 ranges between 45° and 135° with respect to a frame boundary (reference picture boundary) 106 as shown in FIG.

[0033] The implemented angular border padding follows the angular intra prediction of the Joint Exploration Test Model (JEM) described in Reference 2. It should be noted that the angular intra prediction algorithm of JEM assumes that the left and top boundary pixels of an area are always available for prediction. Nevertheless, the basic prediction algorithm assumes that all boundary pixels are straight lines, and therefore translation of either the left or top boundary pixels is performed in JEM. This translation step is omitted in the preferred embodiment of the present application. Furthermore, JEM uses 65 angular intra prediction modes, consisting of 33 vertical modes and 33 horizontal modes. In the preferred embodiment of the present application, only 33 vertical modes are used, as shown in FIG. 4. It should be noted that the vertical angular prediction method in JEM must be rotated 0°, 90°, 180°, or 270° clockwise to perform border padding of the bottom, left, top, or right reference frame boundary 106, respectively.

[0034] If the reference pixel area 100 is partially or completely outside the reference frame 101 in the horizontal and vertical directions, for example at the edge of the reference frame (i.e., at the corner of the frame), a preferential border padding side is calculated. Hor and N Ver indicates the number of horizontal and vertical lines of the reference pixel area 101 that are outside the reference frame area. Hor and N Ver Depending on the , the preferred border padding side can be calculated as follows: If edge == Top-Left-Edge then If N Ver > N Hor then Top-Side else Left-Side Else if edge == Top-Right-Edge then If N Ver > N Horthen Top-Side else Right-Side Else if edge == Bottom-Right-Edge then If N Ver > N Hor then Bottom-Side else Right-Side Else if edge == Bottom-Left-Edge then If N Ver > N Hor then Bottom-Side else Left-Side

[0035] For this calculated preferred border padding side, the claimed selected angular intra prediction mode is applied, while for the remaining sides, the original vertical border padding process is applied as shown in FIG.

[0036] 5, at the frame boundary 106, the overlapping reference pixel area 100 is divided into an area where multi-mode border padding (MMBP) is used and the remaining area where default padding is used, in this case vertical padding.

[0037] Alternatively, the reference pixel area 100 outside the reference frame 101 is entirely padded with MMBP as shown in Fig. 6. In this case, not all angular intra prediction modes can be used, since depending on the prediction angle, the MMBP padding may require boundary pixels 102 that are not available, as shown for example by the dotted arrows in Fig. 6. It should be further noted that the restriction of the angular intra prediction modes can be implicitly determined by the encoder (10) and the decoder (20) without the need to signal the restricted subset. For this purpose, the prediction direction of the corner pixels of the predicted part of the reference pixel area is flipped by 180° to test whether all necessary boundary pixels are available.

[0038] In this embodiment, the most suitable angular inter prediction mode for boundary padding is derived in the video encoder 10 and the video decoder 20 whenever the reference pixel area is partially or entirely outside the reference frame area. As a prerequisite, the reference pixel area 100 is divided into an area 100' outside the reference frame 101 and an area 100'' inside the reference frame 101, as shown in FIG. 7. All pixels in the reference frame 101 are directly used as input pixels for the motion compensation prediction of the current prediction block. All pixels outside the reference frame 101 in the area 100' are predicted with the help of intra prediction from the neighboring boundary pixels 102 of the reference frame 101, as described below. It should be noted that the angular intra prediction with a vertical angle of 90° corresponds to the state-of-the-art reference frame boundary padding used in H.265 and JEM.

[0039] To select one of multiple angular intra prediction modes in the encoder 10 and the decoder 20, the direction of each intra prediction may be rotated 180 degrees to face inward of the reference frame 101. Then, as shown in FIG. 7, a template area, i.e., a probe area 104, is defined based on the rotated intra prediction in the reference frame 101. The size and shape of the template area 104 are not limited. For example, one possible shape would be a rectangular block. For each mode and associated template area 104, a rotated intra prediction is performed from the neighboring boundary pixels 102 to predict all pixel values ​​within the template area 104. By evaluating the results of the above predictions, the most appropriate angular intra prediction mode is selected.

[0040] For the evaluation, the predicted pixels are compared with the already reconstructed pixels of the reference frame 101 in the template area 104 to derive the final Angular intra prediction mode. According to the claim language, the probe area (template area) 104 is filled and the probe filling is compared with the reconstructed probe area 104 filling to measure the similarity between them. For this comparison, any measure of similarity between blocks can be used, for example, sum of absolute differences (SAD), sum of absolute difference transforms (SATD), sum of squared errors (SSE), Hadamard transform, etc. To improve the prediction, content-dependent transforms can be used to transform information into another domain, e.g., location, which is less sensitive. Such content-dependent transforms can also be trained. The Angular intra prediction mode with the highest similarity measure between the predicted pixels and the reconstructed pixels in the template area 104 can be finally selected for use in padding the area 100' of the reference portion 100 of the reference picture 101 that extends beyond the border 106 of the reference picture 101.

[0041] As a modification of the above embodiment, in order to further improve the selection accuracy, multiple (e.g. number N) different template areas (probe areas) can be used for each Angular intra prediction mode, in the following assuming N=2. For example, a first template area is defined depending on an outermost pixel line 112 of the reference pixel area 100 with respect to the frame boundary 106, as shown in Fig. 8. A second template area is defined depending on a first pixel line 110 of the reference pixel area 100, which is within an area 100' of the reference frame 101, as shown in Fig. 8.

[0042] For both these pixel lines 110, 112 in the reference pixel area 100, the angular intra prediction mode is inverted to determine the leftmost and rightmost frame boundary pixels that need to be used for angular intra prediction. Without any kind of limitation, in the following it is assumed that the reference pixel area 100 overlaps with the reference frame 101 on the frame boundary 106 (i.e., the preferred padding side is on top). In this case, the leftmost and rightmost frame boundary pixels are denoted by coordinates {x left-most ;0} and {x right-most ;0}. Then the associated template area has {x left-most -1;1} and {x right-most 9 , the template area is defined by a rectangle with top-left and bottom-right coordinates of {+1;+3}. This means that the template area includes the next two pixel lines in the reference frame below the used frame boundary pixel 102, extended by one additional pixel in the left and right direction. If the reference pixel area 100 overlaps the reference frame 101 at the left, right or bottom frame boundary (i.e. the preferred padding side is left, right or bottom), the template area needs to be rotated accordingly and is defined by the table shown in FIG. 9 . That is, the area 100′ extends to the top side of the reference picture 101, i.e. above the border 106, and the template area is defined as above. If the area 100′ extends to the left side of the reference picture 101, the template area is rotated with respect to the left border 101 of the reference picture. The coordinates of the defined template area are derived based on the table in FIG. 9, where the height and width are the size of the reference picture 101.

[0043] As mentioned before, a first template area based on a first pixel line 110 and a second template area based on an outermost pixel line 120 are defined taking into account the table of FIG. 9. Then, pixel values ​​inside both of these template areas are predicted by rotating the original Angular intra prediction by 180°, i.e., both template areas (probe areas) are filled based on boundary samples (boundary pixels) 102 of the reference picture 101 at the part of the boundary 106 along the opposite direction to the filling direction associated with the respective Angular intra prediction mode as shown in FIG. 7. Thus, the Angular prediction scheme follows the Angular intra prediction of the Joint Exploration Test Model (JEM) described in Reference 2. Then, the predicted pixel values ​​in both template areas are compared with the reconstructed reference pixels by using a Sum of Absolute Differences (SAD) similarity measure. The final Angular intra prediction mode can be selected as the mode that minimizes the sum of the SAD values ​​of both template areas for N=2, i.e., the Angular intra prediction mode with the highest similarity between the probe filling and the reconstructed filling can be selected. It should be noted that other pixel lines in the reference pixel area can also be used to define the template area. Furthermore, for example, intermediate pixel lines between the outermost pixel lines and the first contour line can also be used to define the template area. In addition, the similarity measure can also include weighting of samples in each probe area for features such as luminance intensity or edge detection to differentiate the importance of samples, thereby weighting the similarity contribution of each sample in the template area to the similarity measure unequal based on the psychovisual weighting performed.

[0044] In addition, high-level parameters that control the behavior of the boundary padding with derivation of the Angular intra prediction mode for padding may include the size and shape of the template area. This means that some additional pixel lines perpendicular to the frame boundary and optional extensions of the template area may be parallel to the left and right frame boundaries. In addition, high-level parameters may be introduced to specify the lines in the reference pixel area used to define the template area, such as the outermost line, the first line in the reference frame, or an intermediate line between the outermost line and the first line. Furthermore, a reduction in the number of Angular intra prediction modes used may be signaled. Additional high-level parameters may be introduced to specify minimum and / or maximum thresholds that define how far outside the reference frame the reference pixel area needs to be to select an Angular intra prediction mode for padding. That is, the thresholds may define the amount of extension area of ​​the reference portion outside the reference picture. The high-level parameters may be signaled in a picture parameter set (PPS), a sequence parameter set (SPS), a slice header, or any other high-level syntax structure.

[0045] Furthermore, known codecs featuring vertical boundary extension can be used with respect to the process for selecting the probe area selection. In the implementation of vertical boundary extension, a technique known as motion vector (MV) clipping is often used to lower the memory cost associated with storing the sample values ​​of boundary extension samples. For the repetitive nature of the sample values ​​at the vertical boundary extension, it is wise to clip the motion vectors resulting in a reference block that contains only boundary extension samples. The content of the reference block resulting from the clipped MV is similar to the original reference block. However, the decoder implementation can save resources using such samples by generating boundary extensions of limited size. Generally, a fixed size restriction is used, such as the maximum coding unit size + the size of the interpolation filter kernel as illustrated in FIG. 10, so that all potential reference block sizes can be fitted into a limited amount of generated boundary extension samples.

[0046] As illustrated in Fig. 11, the distance of the reference pixel area from the frame boundary in multi-mode boundary padding (MMBP) affects the selection process of the probe area at the decoder 20 side. Therefore, in order to avoid inconsistencies due to the use of different probe areas, MV clipping needs to be performed in a coordinated manner at the encoder 10 side and the decoder 20 side. Therefore, one aspect of the present invention is a high-level syntax element of the bitstream structure, such as a parameter set, slice header, etc., that allows the encoder 10 to indicate to the decoder 20 the MV clipping procedure to be used or the corresponding bitstream constraints, as described above. Such syntax elements include, for example, the maximum MV length, or the distance from the reference block to the frame boundary at which MV clipping is performed, as well as the size to which the MV is clipped, or the distance between the frame boundary and the reference pixel area due to the clipped motion vector. The information included is not limited to the above information, but may also include further or other information.

[0047] Furthermore, adaptive MV clipping can be used to control the probe area selection in a beneficial manner by adjusting the amount of clipping to the size of the reference pixel area, i.e., the reference block size. As illustrated in Fig. 12, the same MV is used for two blocks of different sizes B>A, and adaptive MV clipping results in two different MVs, e.g., colored in red, where the reference area is as close as possible to the frame boundary and the distance to the selected probe area is also minimized. That is, for a smaller size of reference portion 110', i.e., block size A (smaller block), the amount of motion vectors is larger than for a larger size of reference portion 110', i.e., size B (larger block). Thus, the motion vector coordinates of the inter-predicted blocks of the current picture are clipped by an amount that depends on the size of the reference portion, such that the amount is larger for smaller sizes. Therefore, another aspect of the present invention is a high-level syntax element of a bitstream structure, such as a parameter set, slice header, etc., that allows the encoder 10 to indicate to the decoder 20 that MV clipping is performed adaptively to the reference block size, or to indicate corresponding bitstream constraints to the decoder 20.

[0048] The amount of coding options evaluated at the encoder side is an important factor for the complexity of the codec. Therefore, one aspect of the present invention is a means for controlling the impact of the complexity of the boundary padding described herein. This means consists of performing motion estimation at the encoder, i.e., searching and evaluating the cost of coding various motion compensation predictor samples for inter prediction by extending the known motion estimation by the second stage. The first motion compensation stage encompasses all steps of the known motion compensation (e.g., generating known boundary padding samples and evaluating the coding cost of residual data for a given predictor), while the second stage is a modification to the known motion compensation, where the best predictor is further enhanced through a more sophisticated boundary sample padding scheme such as the scheme described herein.

[0049] If the reference portion is partially or entirely outside the reference picture, the following other embodiments also apply. As already mentioned above, as a prerequisite, the reference pixel area is divided into a region outside the reference frame area and a region inside the reference frame area. All pixels in the reference frame area are directly used as input pixels for the motion compensation prediction of the current prediction block. As explained in Fig. 13 and Fig. 14, all pixels outside the reference frame area are predicted with the help of angular intra prediction from adjacent boundary pixels of the reference frame. That is, the area 100' of the reference portion 100, i.e. the area 100' outside the boundary 106 of the reference picture 101 in the reference portion 100, is filled based on the boundary samples 110. The filling direction is selected from 33 different angular intra modes. In this case, it is not necessary to select from 33 angular intra modes, but it is possible to select from a limited number of angular intra prediction modes that are defined as parameters in advance. Fig. 13 shows one angular intra prediction mode 120, and Fig. 14 shows another angular intra prediction mode 122. In Figures 13 and 14, the boundary pixels 110' used depend directly on the selected angular intra-prediction mode 120 and 122, respectively. It should be further noted that the angular intra-prediction mode with a vertical angle of 90° corresponds to the state-of-the-art reference frame boundary padding used in H.265 and JEM. The chosen or selected angular intra-prediction mode is signaled to the decoder within the encoded bitstream.

[0050] In addition, the selected Angular intra prediction mode can be coded using context-based adaptive arithmetic coding (CABAC). In addition, an enable flag is used to indicate whether default vertical border padding or border padding based on the selected Angular intra prediction mode is applied. Since the default vertical mode is one of the 33 Angular intra prediction modes, the remaining 32 Angular intra prediction modes can be coded with five additional CABAC bins. The enable flag, as well as the remaining Angular modes, are coded as a series of binary decisions with an adaptive binary probability model (a subset of these bins, e.g., the Angular intra prediction modes, can also be coded with a non-adaptive probability model). To model the enable flag, various context models can be used that can be selected based on the amount by which the reference pixel area is outside the reference frame. For example, various context models can be used for cases where the reference pixel area is outside the reference frame by less than 1 / 4, less than 1 / 2, less than 3 / 4, or more than 3 / 4.

[0051] Alternatively, a list of Most Probable Modes (MPMs) is used to signal frequently occurring angular intra prediction modes for border padding. Possible MPM candidates may be angular intra prediction modes for border padding of previously coded neighboring blocks if angular intra prediction modes for border padding are available for those blocks. Other MPM candidates may be angular intra prediction modes of previously coded neighboring blocks if available. Furthermore, angular intra prediction modes of coded blocks in reference frames collocated in the reference picture may be used as MPM candidates. In addition, angular intra prediction modes of coded blocks in associated random access point pictures collocated in the reference picture area may be used as MPM candidates. For the 33 angular intra prediction modes, the MPM candidate list contains one MPM candidate mode. A CABAC binary flag is used to signal whether the MPM candidate should be used. The remaining 32 angular modes may be signaled with five CABAC bins. The MPM flags, as well as the remaining angular intra prediction modes, are coded as a series of binary decisions with an adaptive binary probability model as described above.

[0052] The angular intra prediction mode for the boundary mode is part of the motion information of the coded block. Therefore, when the motion information is used / copied by a different coded block, for example when merge mode or motion vector prediction is used, the angular intra prediction mode for the boundary padding associated with the copied motion vector is also copied and may be applied when the reference pixel area is outside the reference frame area. In the case of motion vector prediction from a block with MMBP, if MMBP is used for the coded block, the selected angular intra prediction mode for padding may be the MPM for coding of the padding direction of the current coded block. Alternatively, the possible motion prediction candidates may provide a list of possible MPMs.

[0053] High-level parameters controlling the behavior of partial boundary padding with signaling of the selected angular intra prediction mode for padding may include parameters for defining a subset of angular intra prediction modes to be used for boundary padding. In this case, the number of supported angular intra prediction modes for padding may be limited to a smaller number so that fewer CABAC bins need to be used to signal the selected angular intra prediction mode for boundary padding. For example, using only a subset of 18 angular modes in combination with an MPM candidate list of two modes leaves only 16 angular modes that can be coded in four CABAC bins (thus resulting in two bins for signaling MPM modes and five bins for signaling the remaining non-MPM modes). As mentioned above, additional high-level parameters may be introduced to specify minimum and / or maximum thresholds that define how far the reference pixel area needs to be outside the reference frame area to select the most appropriate angular intra prediction mode for padding. The high-level parameters may be signaled in a picture parameter set (PPS), a sequence parameter set (SPS), a slice header, or any other high-level syntax structure.

[0054] The above embodiments can be combined, for example the embodiment using templates and the other embodiments for signaling the selected angular intra prediction mode as described above can be combined in further embodiments as described below.

[0055] The selection of the angular intra prediction mode follows the embodiment described in FIG. 7. In this embodiment, an ordered list of angular intra prediction modes for padding is generated based on the similarity measure described above. The generated ordered list of angular intra prediction modes for padding is truncated by a predefined value of the angular intra prediction mode for the maximum signaled padding. Finally, the angular intra prediction mode for the selected padding in this truncated ordered list is signaled using context-based adaptive arithmetic coding (CABAC) as described above.

[0056] As already mentioned above, an enable flag can be used to indicate whether the default vertical border padding or a truncated ordered list of selected padding modes should be used. In addition, the number of padding modes signaled can be up to 2 N, which can be signaled in N CABAC bins. The selected Angular intra prediction mode is coded as a series of binary decisions with an adaptive binary probability model (a subset of these bins can also be coded with a non-adaptive probability model). When a binary probability model is used to code the selected Angular intra prediction mode, various context models can be used, which can be selected based on the amount of area of ​​the reference portion that is outside the reference frame area, as described above. Furthermore, the truncated ordered list of selected Angular intra prediction modes for padding is used as an MPM candidate list. Here, the MPM candidate list and the remaining Angular intra prediction modes for padding can be coded as described above. Furthermore, the same high-level parameters as described above can also be used, e.g., parameters for defining a subset of Angular intra prediction modes, thresholds for applying the selected Angular intra prediction mode, and parameters for defining the size and shape of the template area. Also, as described above, the high-level parameters can be signaled in a picture parameter set (PPS), a sequence parameter set (SPS), a slice header, or any other high-level syntax structure.

[0057] In a preferred embodiment of the present application, the high level syntax includes syntax elements that control multi-mode boundary padding, which are conveyed in one or more high level syntax structures, such as a sequence parameter set (SPS; valid for all pictures in a coded video sequence), a picture parameter set (PPS; valid for one or more pictures), a slice header (valid for a slice), etc.

[0058] A high-level syntax element that controls an application to select an Angular intra prediction mode may include, but is not limited to, one or more of the following parameters: A flag or IDC signaling which angular inter prediction mode for padding (MMBP, vertical, or omnidirectional padding) should be applied Parameters for signaling or constraints indicating frame boundary wise combinations of padding modes. For example, whether MMBP or omnidirectional padding is used can be signaled in the data stream for parts along the frame boundary or outer picture boundary of the reference picture. As an option, both options can be signaled as available. Different available modes can be signaled so that the encoder and decoder check which parts of the frame boundary of the reference picture overlap or intersect with the reference area to decide which intra prediction mode to use for different parts; Parameters for signaling the selection process of the above embodiment to be used Parameter for signaling angular intra prediction mode that can be used for MMBP A parameter that specifies the exact angular intra prediction mode to be used for a particular reference frame border region. Minimum and / or maximum thresholds that define how far outside the reference frame area the reference pixel area must be for the Angular intra prediction mode selection to apply Parameters for signaling the size and shape of the template area Parameter for signaling the set of lines within the reference pixel area used to determine the template area

[0059] Regarding the second option, it should be further noted that although the embodiment outlined above uses the Angular intra prediction mode as an example, in which an intra prediction mode for padding was ultimately chosen, it is clear that padding in a vertical padding mode, in which padding is performed by vertically extrapolating boundary samples to an area beyond the frame boundary, is also an intra prediction mode, just as the omnidirectional padding mode is an intra prediction mode that fills / pads an area outside the frame boundary using samples in the same frame, i.e., the reference frame. Thus, the high level syntax makes it clear which intra prediction mode should be used or which intra prediction mode is available for which part of the frame boundary, and different modes used or available for different parts of the frame boundary can be signaled, such as omnidirectional mode for some parts, such as the right and left boundaries, and vertical mode for other parts of the frame boundary, such as the top and bottom boundaries. The encoder and decoder simply need to know the part of a particular reference area of ​​the inter prediction block 80 that is beyond the frame boundary and ascertain which intra prediction mode it is associated with for padding according to the high level syntax in the data stream. The parameters listed above can be included in more than one high-level syntactic construct, for example a global parameter specified in the SPS can be overridden by a parameter specified in the PPS.

[0060] For some embodiments of the present application, this is required whenever the reference pixel area is partially or entirely outside the reference frame area. As a prerequisite, the reference pixel area is divided into a region outside the reference frame area and a region inside the reference frame area, as described above. All pixels within the reference frame area are directly used as input pixels for the motion compensation prediction of the current prediction block. All pixels outside the reference frame area are predicted with the help of intra prediction from adjacent boundary pixels of the reference frame, as described above.

[0061] In other words, the following steps again exemplarily explain how a preferred intra prediction angle or a preferred intra prediction mode to be used for padding can be selected from among multiple intra prediction angles available at the encoder and decoder. The selection process can be performed only once per luma (Y) color plane for each prediction block and each associated motion vector. For the chroma color planes (Cb and Cr), the same intra prediction angles can be selected as for the luma color plane: Determine the position of the reference area 100: Calculate the reference area 100 by adding the motion vector related to the current block to the area of ​​the current block (e.g., move the current block position by the related motion vector). Note that the motion vector may have a fractional part. Therefore, round the motion vector to the nearest integer part. Add the motion vector to the current block position. Increase the reference area 100 by one sample in each direction. That is, the area 100 may be one sample larger in all directions (left, right, top, bottom) than the inter-predicted block for which MMBP is performed. Determine the preferred IntraBP side: if the reference area 100 overlaps the reference frame 101 on one side only, use this side, otherwise the reference area 100 overlaps the reference frame 101 on two sides (see Fig. 5). In that case, use the default vertical padding on the side that overlaps the reference frame with fewer sample lines. Use IntraBP padding on the remaining sides. Determine the bottom and top lines: the bottom line is the first sample line of the reference area 100 parallel to the preferred IntraBP side and outside the reference frame 101 (110 in Fig. 8). The top line is the last sample line of the reference area parallel to the preferred IntraBP side and outside the reference frame (112 in Fig. 8), i.e. the one furthest from the reference frame boundary 106 is chosen. Constrain multiple available intra prediction modes: For the top line and each intra prediction mode, calculate the relevant / affected boundary pixels that need to be used for intra prediction by inverting the prediction direction. The maximum displacement of the boundary pixels from the vertical intra prediction mode is less than 1.25 * If it exceeds the width of the prediction block, skip / do not use this intra prediction angle. This step is optional: optionally, angular modes for which backprojecting the outermost line of samples 112 onto the boundary samples 110 causes the boundary samples 110 to be offset from the perpendicular backprojected footprint of the outermost sample line by more than a predefined threshold are excluded from the selectable list of modes. Calculate probe areas for bottom and top lines: For the bottom (top) line and for each intra prediction mode in the set of constrained intra prediction modes, reverse the prediction direction to determine the relevant / affected boundary pixels 110' that need to be used for intra prediction. Place the probe area 104 below the affected boundary pixels in the reference frame (104 in Fig. 7). Set the height of the probe area to two sample lines perpendicular to the preferred IntraBP side. Compute similarity measures: Predict probe area sample values ​​by inverted intra prediction from related / affected boundary pixels. Compute SAD measures between predicted probe area sample values ​​and reconstructed sample values ​​of reference frame at probe area location. Combine / add SAD measures from lower and upper probe areas. Select a preferred intra prediction angle: Among the constrained set of intra prediction angles, select the angle with the smallest similarity measure as the preferred intra prediction angle. If intra prediction angles have the same similarity measure, prioritize the angle that is closer to the vertical prediction angle.

[0062] If the selected preferred intra prediction angle is not equal to the vertical intra prediction angle, then the encoder and decoder perform IntraBP for each color plane using the selected intra prediction angle and the preferred IntraBP angle: Determine the location of the reference area: Calculate the reference area as described above, but increase the reference area by half the size of the interpolation filter taps in each direction instead of just one sample in each direction. Fill parts of the reference area that lie within the reference frame: Fills parts of the reference area that lie within the reference frame with the sample values ​​of the reference frame. Optionally fill non-preferred IntraBP sides: If the reference area overlaps the reference frame on two sides, default vertical padding is applied to the parts of the reference area that overlap the reference frame with fewer sample lines. · Perform IntraBP on the preferred IntraBP side: Using the preferred intra prediction angle, perform intra prediction on the boundary samples on the preferred IntraBP side to fill in the reference area 100' that is outside the reference frame.

[0063] As input values ​​for the motion compensated prediction of the current block, the sample values ​​of the reference area calculated as above are used for each color plane.

[0064] In addition, existing applications such as 360° video streaming services based on the MPEG OMAF standard rely heavily on spatial video partitioning or segmentation techniques. In such applications, spatial video segments are transmitted to clients and jointly decoded in an adaptive manner to the current client's viewing direction. Another related application that relies on spatial segments of video planes is the parallelization of encoding and decoding operations, for example to facilitate the multi-core and parallelization capabilities of modern computing platforms.

[0065] One such spatial segmentation technique is implemented in HEVC and is known as tiling, which divides the picture plane into segments that form a rectangular grid. The resulting spatial segments are independently coded for entropy coding and intra prediction. Furthermore, there are means to indicate that the spatial segments are also independently coded for inter prediction, which may also be combined with static spatial segmentation across pictures. Together, these constraints allow for truly independent coding of the spatial segmentation of an exemplary picture plane, with two such adjacent independently coded spatial segments.

[0066] More specifically, when encoding the current picture using the spatial segmentation shown in Figure 15, which is facilitated through inter-prediction of reference pictures that also use the spatial segmentation exemplified above, the motion vectors are constrained to avoid predicting intersecting spatial segment A through reference to samples of the reference picture that are present in spatial segment B, and vice versa.

[0067] Although independent coding of spatial segments enables existing applications as mentioned above, the associated coding constraints can impose significant coding costs. This bitrate overhead can result, for example, from signaling relatively larger motion vector differentials (MVDs) that modify available motion vector candidates, which would otherwise result in the use of sample positions outside a given spatial segment. Furthermore, to compensate for the degradation of the predictor for inter prediction, the amount of residual data signaled needs to be increased. It is therefore highly desirable to enhance the operation of the codec at the boundaries of such spatial segments.

[0068] One way to mitigate the coding penalty of the prior art solutions for such independent spatial segments is to use a scheme for padding intra-picture boundaries, i.e., segment boundaries inside a picture, in addition to outer frame boundaries, similar to or following the previously described MMBP concept based on picture boundaries with slight modifications and extensions described herein, or using a high-level syntax-based padding mode selection to signal which of two or more intra-prediction modes to use for a particular portion, such as a spatial segment boundary or a vertical portion of a spatial segment boundary. That is, instead of the encoder restricting the signaled motion vectors to not refer to reference portions that extend or lie beyond the boundary of the spatial segment, such portions are padded at the encoder and decoder, and the signaled intra-prediction mode is used for padding if such reference portions intersect or lie beyond a certain portion of the spatial segment boundary. The reduction of coding penalty can also be achieved without multi-mode selection by using any intra-coding mode, such as one of the vertical, omnidirectional, or angular intra-prediction modes, resulting in an encoder and decoder for block-based video coding configured to encode and decode in a data stream of an inter-predicted block a motion vector that references a reference portion 100 of a reference picture 101 that extends or resides beyond an intra-picture segment boundary of a given spatial segment in the reference picture in which the inter-predicted block resides, and to pad an area 100' of the reference portion 100 that extends or resides beyond an intra-picture segment boundary, i.e., outside the given spatial segment. The encoder does not need to redirect a motion vector predictor of a motion vector to force the motion vector to remain within a given spatial segment of the spatial segments in which the reference picture is partitioned in a manner in which the video picture, including the current picture, is constantly partitioned. The encoder does not need to signal a motion vector differential to redirect this vector.Signaling of motion vector differentials may lead to motion vectors referencing parts outside a given spatial segment, such as a motion vector differential of zero. In particular, the encoder may rely on a padding mechanism in the decoder that enforces independence of spatial segment coding of spatially offset spatial segments in the current picture and the reference picture, and the encoder is the same, i.e., padding to reconstruct inter-predicted blocks to fill the decoded picture buffer.

[0069] As a first aspect, high-level syntax elements of bitstream structures such as parameter sets, slice headers, etc., generally control the application of multi-mode boundary padding at spatial segment boundaries. One value of this syntax indicates to the decoder: Inter prediction of samples in a spatial segment is independent of samples belonging to other spatial segments. Sample positions outside spatial segment boundaries are used for inter prediction of samples within the spatial segment · The values ​​of samples at sample positions outside the spatial segment boundary are generated using the MMBP procedure at the spatial segment boundary from samples that belong to the spatial segment.

[0070] Although some aspects are described in the context of an apparatus, it is clear that these aspects also represent a description of a corresponding method, with blocks or devices corresponding to method steps or features of method steps. Similarly, aspects described in the context of a method step also represent a description of a corresponding block or item or feature of a corresponding apparatus. Some or all of the method steps may be performed by (or with) a hardware apparatus, such as, for example, a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, one or more of the most important method steps may be performed by such an apparatus.

[0071] The data stream of the present invention can be stored on a digital storage medium or can be transmitted over a transmission medium, such as a wireless or wired transmission medium, such as the Internet.

[0072] Depending on certain implementation requirements, embodiments of the present invention can be implemented in hardware or software. Implementation can be done using a digital storage medium, for example a floppy disk, DVD, Blu-ray, CD, ROM, PROM, EPROM, EEPROM or flash memory, on which electronically readable control signals are stored that cooperate (or can cooperate) with a programmable computer system such that the respective method is executed. The digital storage medium can thus be computer readable.

[0073] Some embodiments of the present invention include a data carrier having electronically readable control signals capable of cooperating with a programmable computer system to cause one of the methods described herein to be performed.

[0074] Generally, some embodiments of the present invention can be implemented as a computer program product having program code operable to perform one of the methods when the computer program product runs on a computer. The program code can be, for example, stored on a machine readable carrier.

[0075] Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

[0076] In other words, an embodiment of the inventive method is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

[0077] A further embodiment of the inventive method is therefore a data carrier (or digital storage medium or computer readable medium) having recorded thereon a computer program for performing one of the methods described herein. The data carrier, digital storage medium or recording medium is typically not tangible and / or non-transitory.

[0078] A further embodiment of the inventive method is therefore a data stream or a sequence of signals representing the computer program for performing one of the methods described herein, The data stream or the sequence of signals can for example be arranged to be transferred via a data communication connection, for example via the Internet.

[0079] A further embodiment comprises a processing means, for example a computer, or a programmable logic device, configured to or adapted to perform one of the methods described herein.

[0080] A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0081] Further embodiments according to the invention include an apparatus or system configured to transfer (e.g. electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a memory device, etc. The apparatus or system may, for example, comprise a file server for transferring the computer program to the receiver.

[0082] In some embodiments, a programmable logic device (e.g., a field programmable gate array) may be used to implement some or all of the functionality of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor to implement one of the methods described herein. In general, the methods are preferably implemented by any hardware apparatus.

[0083] The apparatus described herein may be implemented using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer.

[0084] The apparatus described herein, or any components of the apparatus described herein, may be implemented at least in part in hardware and / or in software.

[0085] The methods described herein may be performed using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer.

[0086] Any of the methods described herein or components of the apparatus described herein may be performed at least in part by hardware and / or by software.

[0087] The above-described embodiments merely exemplify the principles of the present invention. It is understood that modifications and variations of the configurations and details described herein are obvious to those skilled in the art. It is therefore intended to be limited only by the scope of the appended claims and not by the specific details presented as the description and explanation of the embodiments herein.

Claims

[Claim 1] A video encoder (10) having at least one processor and a memory, the memory being configured to, when executed by the at least one processor, performing padding of an area (100') of a reference portion (100) of a reference picture (101) that is referenced by an inter-predicted block of a current picture (12) different from the reference picture (100), the area (100') of the reference portion (100) extending or lying beyond a boundary (106) of the reference picture (101) by selecting one of a plurality of intra-prediction modes including an omnidirectional mode and a vertical mode, and padding the area (100') using the selected intra-prediction mode; predicting the inter-predicted block of the current picture using the reference portion; a video encoder (10) having instructions for causing the video encoder to execute:

Images