Apparatus for selecting intra prediction mode for padding

Abstract

To provide a video codec that performs boundary padding to improve the quality of temporal inter prediction.SOLUTION: A video decoder 20 for decoding a picture 12 ' from a data stream 14, wherein the video decoder 20 is configured to determine, for an inter-predicted block of the picture, a first reference area lying beyond a left vertical boundary or a right vertical boundary of a reference picture and a second reference area lying within said boundaries of the reference picture using motion vectors, select an omnidirectional padding mode for padding the first reference area based on an indication in the data stream, and generate padding for the first reference area by copying samples from an opposite vertical picture boundary; The padding and the second reference area are utilized to predict an inter-predicted block of the picture.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] This application relates to video coding / decoding. [Background technology]

[0002] All video coding standards since H.261, including the state-of-the-art ITU-T H.265 | MPEG H HEVC video coding standard (reference 1: ITU-T and ISO / IEC. High Efficiency Video Coding. ITU-T Recommendation H.265 | ISO / IEC 23008 10 (HEVC), edition 1, 2013; edition 2, 2014), rely on the basic concept of block-based hybrid video coding. Here, the input signal for each color component is segmented into disjoint blocks that are coded using a hybrid video coding scheme based on prediction, transformation, quantization, and entropy coding. Therefore, one of the first steps in encoding and decoding algorithms is the generation of a prediction signal for the current block, which is performed using either a spatial or temporal prediction scheme. In the case of temporal prediction, the prediction signal is derived from an already processed frame, referred to as a reference frame. For this purpose, a displacement vector specifies the offset of the prediction signal within the reference frame relative to the position of the current coded block. This temporal prediction scheme is also known as translational motion compensated prediction. In this sense, the displacement vector is referred to as a motion vector.

[0003] To enable motion vectors with sub-pixel accuracy, the prediction signal is calculated using an interpolation filter. Furthermore, the prediction signal can be constructed by superimposing multiple prediction signals, each specified by a different motion vector and / or a different reference frame. Regardless of the final calculation and construction of the prediction signal, the basic principle remains the same. The reference block area within the reference frame is determined by the position and size of the current coded block displaced by an additional motion vector. If an interpolation filter is used, this area must be expanded on each side depending on the number of filter taps used. The pixels within the reference block area are then used to generate a temporal prediction signal for the current coded block. Typically, the reference block area is within the reconstructed reference frame. Nevertheless, the reference block area may not completely overlap with the reconstructed frame area, which means that part of the reference pixel area may be outside the reference frame area.

[0004] In current video coding standards, the basic approach to handling a reference picture, i.e., a reference portion outside the reference frame area, i.e., a reference pixel area, is to constrain motion vectors or extend the reference frame area. For example, in H.261, motion vectors are constrained to picture boundaries so that all reference pixels are within the reference frame area, i.e., all samples of the sample portion of the reference picture are within the boundary of the reference frame. Alternatively, in H.264 / H.265, a process called boundary padding is used to extend an already reconstructed reference frame as shown in Figure 16. That is, boundary pixels (boundary samples 110) are copied to a padding area (an area outside the reference picture 101), which can be seen as intra-prediction of boundary pixels perpendicular to the frame boundary 106. Here, at the edge of the reference frame, the edge pixels are further copied to the remaining area outside the reference frame 101 area in the vicinity of the edge pixels, as shown in Figure 16.

[0005] Furthermore, known techniques for border padding are advantageous when dealing with certain types of video content known as omnidirectional or 360° video, which uses an omnidirectional projection to map the surrounding camera onto a rectangular video picture plane. One widely used format for such content is based on equirectangular projection (ERP). Omnidirectional content allows areas outside the picture boundary to be filled with the correct content, since the correct content is, by definition, depicted elsewhere within the same picture. The vertical picture boundary of ERP video content that depicts 360° horizontally can be padded by copying samples from the opposite vertical picture boundary. However, horizontal picture boundaries exhibit different characteristics, especially when the vertical viewing angle is not full coverage. Summary of the Invention [Problem to be solved by the invention]

[0006] 17 shows an exemplary ERP video frame with a horizontal coverage angle of 360° and a vertical coverage angle of less than 180°, i.e., only 30° below the equator at 0°. While optimal padding for vertical picture boundaries consists of using samples from the opposite vertical picture side, the top and bottom horizontal picture boundaries marked with dashed lines exhibit similar characteristics to picture boundaries in typical rectangular videos with limited field of view and therefore benefit from the multi-mode border padding described herein. Thus, one aspect of the present invention is to change the padding mode for each picture boundary, for example, depending on knowledge of content characteristics or as signaled in the bitstream.

[0007] In many cases, simple vertical extension of frame boundary pixels results in a splice that does not closely approximate the current predicted block. Better splicing may be achieved by using various intra prediction modes, such as angular prediction, DC prediction, or planar prediction, to extend the reference frame boundary. Exemplary intra prediction modes that can be used for boundary extension are those from H.265, described in Reference 1, or the Joint Exploration Test Model (JEM), described in Reference 2 (J. Chen, E. Alshina, G.J. Sullivan, J.-R. Ohm, J. Boyce, Algorithm Description of Joint Exploration Test Model 7 (JEM7), JVET, doc. JVET-G1001, July 2017). Furthermore, future embodiments of this method may also use future intra prediction modes to extend the reference boundary. The spatial extension mode of the frame boundary depends on the characteristics of the current predicted block. Therefore, a particular angular intra prediction mode needs to be chosen / selected from a set of multiple angular intra prediction modes to generate a prediction signal that provides the best approximation of the current prediction block. [Means for solving the problem]

[0008] It is therefore an object of the present invention to provide a video codec configured to perform boundary padding to improve the quality of temporal inter prediction.

[0009] This object is achieved by the subject matter of the claims of the present application.

[0010] One of the basic ideas of this application is to improve the quality of temporal inter prediction by applying an omnidirectional padding mode to vertical boundaries in ERP etc. based on bitstream instructions and performing padding by copying from the opposite boundary. In another embodiment, the padding quality can also be improved by selecting an optimal mode from multiple intra prediction modes (e.g., Angular) based on the characteristics of the current prediction block.

[0011] According to one embodiment, the angular intra prediction mode used for padding is selected based on an evaluation of the reference picture area at the portion intersecting the reference picture boundary with the reference portion. The evaluation is performed, for each of at least a subset of the plurality of angular intra prediction modes, based on boundary samples of the reference picture at the portion of the boundary, by filling a probe area farther from the boundary than the boundary samples along a direction opposite to the filling direction associated with the respective angular intra prediction mode to obtain a probe filling, and comparing the probe filling and the reconstructed probe area filling to measure the similarity between them. Thus, the selection is based on the characteristics of the current prediction block and can therefore provide the best approximation of the current prediction block, improving prediction quality.

[0012] According to some embodiments of the present application, the angular intra prediction mode used for padding is selected based on the prediction result of an area outside the reference picture of the reference portion. The reference portion is referenced by an inter-prediction block of the current picture. Therefore, the angular intra prediction mode is selected according to the characteristics of the area, and thus the most appropriate angular intra prediction mode for padding can be selected. In addition, multiple angular intra prediction modes used to predict the area can be signaled as a subset of supported angular intra prediction modes supported by the codec of the intra prediction block.

[0013] Further advantages are achieved by the claimed codec.Preferred embodiments of the present application are described below with reference to the drawings. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a block diagram of an apparatus for predictively coding video, as an example of a video decoder capable of implementing angular intra-prediction mode selection for padding concepts according to some embodiments of the present application. FIG. [Figure 2] 2 is a block diagram of an apparatus for predictively decoding video, compatible with the apparatus of FIG. 1, as an example of a video decoder capable of implementing angular intra-prediction mode selection for padding concepts according to some embodiments of the present application. [Figure 3] 1A and 1B are schematic diagrams showing examples of relationships between a prediction residual signal, a prediction signal and a reconstructed signal, respectively, to illustrate functions for setting subdivisions for defining a prediction signal, for handling a prediction residual signal, etc. [Figure 4] FIG. 1 is a schematic diagram illustrating an example of an angular intra-prediction mode used in some embodiments of the present application. [Figure 5] FIG. 10 is a schematic diagram illustrating an example of calculating a preferred border padding side according to an embodiment of the present application; [Figure 6] FIG. 10 is a schematic diagram illustrating a further example of calculating a preferential border padding side according to an embodiment of the present application; [Figure 7] FIG. 1 is a schematic diagram illustrating an example of a probe area according to an embodiment of the present application. [Figure 8] FIG. 1 is a schematic diagram showing an example used to explain the process of defining a probe area according to an embodiment of the present application. [Figure 9] FIG. 10 illustrates a sample table for deriving probe area coordinates according to an embodiment of the present application. [Figure 10] FIG. 10 is a schematic diagram illustrating an example of motion vector clipping of boundary-extended reference blocks according to an embodiment of the present application; [Figure 11]FIG. 10 is a schematic diagram illustrating an example of deriving a template area related to motion vector clipping according to an embodiment of the present application; [Figure 12] FIG. 1 is a schematic diagram illustrating an example of block size adaptive motion vector clipping according to an embodiment of the present application; [Figure 13] 1 is a schematic diagram illustrating an angular intra prediction mode used for padding, selected based on prediction results for areas outside the reference picture of the reference portion according to the present application. FIG. [Figure 14] 1 is a schematic diagram illustrating an angular intra prediction mode used for padding, selected based on prediction results for areas outside the reference picture of the reference portion according to the present application. FIG. [Figure 15] FIG. 2 is a schematic diagram illustrating an example of a horizontal boundary between two independent spatial segments according to an embodiment of the present application; [Figure 16] FIG. 1 is a schematic diagram showing vertical prediction to explain the state of the art. [Figure 17] FIG. 10 shows a sample image of a video frame with a horizontal coverage angle of 360° and a vertical coverage angle of less than 180°. DETAILED DESCRIPTION OF THE INVENTION

[0015] The following description of the figures begins with presenting a description of a video encoder and a video decoder of a block-based predictive codec for coding pictures of video to provide examples of coding frameworks in which embodiments of the configured predictive codec may be incorporated. The video encoder and video decoder are described in relation to Figures 1-3. Below, descriptions of embodiments of the configured prediction concepts of the present application are presented along with descriptions of how such concepts may be incorporated into the video encoder and decoder of Figures 1 and 2, respectively, although the embodiments described thereafter, beginning with Figure 4, may also be used to provide video encoders and video decoders that do not operate according to the coding framework underlying the video encoder and video decoder of Figures 1 and 2.

[0016] FIG. 1 illustrates an apparatus for predictively coding a video 11 consisting of a sequence of pictures 12 into a data stream 14. Block-based predictive coding is used for this purpose. Furthermore, transform-based residual coding is illustratively used. The apparatus, i.e., encoder, is indicated using the reference numeral 10. FIG. 2 illustrates a corresponding decoder 20, i.e., apparatus 20 configured to predictively decode a video 11′ consisting of pictures 12′ of picture blocks from data stream 14, again illustratively using transform-based residual decoding; an apostrophe is used to indicate that the pictures 12′ and video 11′ reconstructed by decoder 20 deviate from the picture 12 originally coded by apparatus 10 in terms of coding loss introduced by quantization of the predictive residual signal. While FIGS. 1 and 2 illustratively use transform-based predictive residual coding, embodiments of the present application are not limited to this type of predictive residual coding. This also applies to other details described in connection with FIGS. 1 and 2 outlined below.

[0017] The encoder 10 is configured to subject the prediction residual signal to a spatial-to-spectral transform and to encode the resulting prediction residual signal into a data stream 14. Similarly, the decoder 20 is configured to decode the prediction residual signal from the data stream 14 and to subject the resulting prediction residual signal to a spectral-to-spatial transform.

[0018] Internally, the encoder 10 may comprise a prediction residual signal generator 22 that generates a prediction residual 24 to measure the deviation of a prediction signal 26 from the original signal, i.e., the video 11 or the current picture 12. The prediction residual signal generator 22 may, for example, be a subtractor that subtracts the prediction signal from the original signal, i.e., the current picture 12. The encoder 10 then further comprises a transformer 28 that subjects the prediction residual signal 24 to a spatial-to-spectral transform to obtain a spectral-domain prediction residual signal 24′, which is then quantized by a quantizer 32 also comprised in the encoder 10. The prediction residual signal 24″ thus quantized is coded into the bitstream 14. For this purpose, the encoder 10 may optionally comprise an entropy coder 34 that entropy codes the transformed and quantized prediction residual signal into the data stream 14. Based on a prediction residual signal 24" decoded into and decodable from the data stream 14, a prediction residual 26 is generated by a prediction stage 36 of the encoder 10. To this end, the prediction stage 36 may internally comprise, as shown in FIG. 1, an inverse quantizer 38 that inversely quantizes the prediction residual signal 24" to obtain a spectral-domain prediction residual signal 24'" that corresponds to the signal 24' except for quantization losses, followed by an inverse transformer 40 that subjects the prediction residual signal 24'" to an inverse transform, i.e., a spectral-to-spatial transform, to obtain a prediction residual signal 24"" that corresponds to the original prediction residual signal 24 except for quantization losses. A combiner 42 of the prediction stage 36 then recombines, e.g., by addition, the prediction signal 26 and the prediction residual signal 24"" to obtain a reconstructed signal 46, i.e., a reconstruction of the original signal 12. The reconstructed signal 46 may correspond to the signal 12'.

[0019] A prediction module 44 of the prediction stage 36 then generates the predicted signal 26 based on the signal 46, for example by using spatial prediction, i.e., intra-prediction, and / or temporal prediction, i.e., inter-prediction, as will be described in more detail below.

[0020] Similarly, the decoder 20 is internally composed of corresponding components that may be interconnected in a manner corresponding to the prediction stage 36. In particular, an entropy decoder 50 of the decoder 20 is able to entropy decode a quantized spectral domain prediction residual signal 24″ from the data stream, followed by a dequantizer 52, an inverse transformer 54, a combiner 56, and a prediction module 58, which are interconnected and cooperate in the manner described above with respect to the modules of the prediction stage 36, to recover a reconstructed signal based on the prediction residual signal 24″; thus, as shown in FIG. 2, the output of the combiner 56 results in a reconstructed signal, i.e., the video 11′ or its current picture 12′.

[0021] Although not specifically mentioned above, it is clear that the encoder 10 can set certain coding parameters, including, for example, prediction modes, motion parameters, etc., according to an optimization scheme, such as, for example, optimizing certain rate-distortion-related criteria, i.e., coding cost, and / or using a certain rate control. As will be explained in more detail below, the encoder 10 and decoder 20 and corresponding modules 44, 58 support different prediction modes, such as intra-coding and inter-coding modes, respectively, which form a kind of set or pool of primitive prediction modes based on which the prediction of picture blocks is configured in a manner described in more detail below. The granularity at which the encoder and decoder switch between these prediction configurations may correspond to a subdivision of the pictures 12 and 12′, respectively, into blocks. Note that some of these blocks may be exclusively intra-coded, some may be exclusively inter-coded, and optionally, still other blocks may be blocks obtained using both intra- and inter-coding, with the details subsequently set. According to the intra-coding mode, the prediction signal of a block is obtained based on the spatially coded / decoded neighbors of the respective block. There can be several intra-coding sub-modes, the choice of which apparently represents a type of intra-prediction parameters. There can also be directional or angular intra-coding sub-modes, according to which the prediction signal of each block is filled by extrapolating the neighboring sample values ​​to the respective block along a specific direction specific to each directional intra-coding sub-mode.The intra-coding submodes may also include one or more further submodes, such as a DC coding mode, according to which the prediction signal of each block assigns a DC value to all samples within the respective block, and / or a planar intra-coding mode, according to which the prediction signal of each block is approximated or determined by deriving the slope and offset of a plane defined by a two-dimensional linear function based on neighboring samples, such that the spatial distribution of sample values ​​across the sample positions of the respective block is described by a two-dimensional linear function. In contrast, according to an inter-prediction mode, the prediction signal of a block is obtained, for example, by temporally predicting the interior of the block. To parameterize the inter-prediction mode, a motion vector is signaled in the data stream, indicating the spatial displacement of the portion of the previously coded picture of the video 11 from which the previously coded / decoded picture is sampled to obtain the prediction signal of the respective block. This means that in addition to the residual signal coding contained in the data stream 14, such as entropy-coded transform coefficient levels representing the quantized spectral domain prediction residual signal 24", the data stream 14 also encodes prediction-related parameters for assigning prediction modes to blocks, prediction parameters for the assigned prediction modes, such as motion parameters for inter-prediction modes, and, optionally, further parameters for controlling the construction of the final prediction signal for the blocks using the assigned prediction modes and prediction parameters, as outlined in more detail below. In addition, the data stream may contain parameters that control and signal the subdivision of the pictures 12 and 12' into blocks. The decoder 20 uses these parameters to subdivide the pictures in the same manner as the encoder did, assign the same prediction modes and parameters to the blocks, and perform the same predictions, resulting in the same prediction signals.

[0022] 3 illustrates the relationship between, on the one hand, the reconstructed signal, i.e., the reconstructed picture 12′, and, on the other hand, the combination of a prediction residual signal 24″″ and a prediction signal 26 signaled in a data stream. As already indicated above, the combination can be additive. Although the prediction signal 26 is illustrated in FIG. 3 as a subdivision of the picture area into blocks 80 of various sizes, this is merely an example. The subdivision can be any subdivision, such as a regular subdivision of the picture area into rows and columns of blocks, or a multi-tree subdivision into leaf blocks of various sizes, such as a quad-tree subdivision of the picture 12, a mixture of which is illustrated in FIG. 3, in which the picture area is first subdivided into rows and columns of tree-root blocks and then further subdivided according to a recursive multi-tree subdivision, resulting in the blocks 80.

[0023] The prediction residual signal 24"" in FIG. 3 is also illustrated as a subdivision of the picture area into blocks 84. These blocks may be called transform blocks to distinguish them from the coding blocks 80. In effect, FIG. 3 illustrates that the encoder 10 and the decoder 20 may use two different subdivisions of the picture 12 and the picture 12' into blocks, respectively: one subdivision into coding blocks 80 and the other subdivision into blocks 84. While both subdivisions may be the same, i.e., each block 80 simultaneously constitutes a transform block 84, and vice versa, FIG. 3 illustrates the case where the subdivision into transform blocks 84 constitutes an extension of the subdivision into blocks 80, for example, such that the boundary between two blocks 80 overlaps the boundary between two blocks 84, or in other words, such that each block 80 coincides with one of the transform blocks 84 or with a cluster of transform blocks 84. However, these subdivisions may also alternatively be determined or selected independently of each other, such that the transform blocks 84 may intersect the block boundaries between the blocks 80. Therefore, as far as the subdivision into transform blocks 84 is concerned, the same expressions as those mentioned above in relation to the subdivision into blocks 80 apply: the blocks 84 may be the result of a regular subdivision of a picture area into blocks, organized into rows and columns, a recursive multi-tree subdivision of a picture area, or a combination of these, or any other type of segmentation. As an aside, it should be noted that the blocks 80 and 84 are not limited to square, rectangular, or other shapes. Furthermore, the subdivision of the current picture 12 into blocks 80, from which a prediction signal is generated, and the subdivision of the current picture 12 into blocks 84, from which a prediction residual is coded, are not the only subdivisions used for coding / decoding. These subdivisions from the granularity at which the prediction signal is determined and the residual coding is performed differ in that, firstly, the residual coding may alternatively be performed without subdivision, and secondly, at other granularities than these subdivisions, the encoder and decoder may set certain coding parameters, which may include some of the parameters mentioned above, such as prediction parameters, prediction signal configuration control signals, etc.

[0024] 3 illustrates that the combination of the prediction signal 26 and the prediction residual signal 24"" directly results in the reconstructed signal 12'. However, it should be noted that, according to alternative embodiments, more than one prediction signal 26, such as prediction signals obtained from other views or from other coding layers that are coded / decoded in separate prediction loops with separate DPBs, can be combined with the prediction residual signal 24"" to result in the picture 12'.

[0025] 3, the transform block 84 has the following meaning: the transformer 28 and the inverse transformer 54 perform the transform in units of the transform block 84. For example, many codecs use some kind of DST or DCT for all transform blocks 84. Some codecs can skip the transform, so that for some of the transform blocks 84, the prediction residual signal is coded directly in the spatial domain. However, according to the embodiments described below, the encoder 10 and the decoder 20 are configured so that they support several transforms. For example, the transforms supported by the encoder 10 and the decoder 20 can include: DCT-II (or DCT-III), where DCT stands for Discrete Cosine Transform DST-IV, where DST stands for Discrete Sine Transform DCT-IV ○DST-VII Identity Transformation (IT)

[0026] Naturally, the transformer 28 will support the forward transform versions of all of these transforms, while the decoder 20 or inverse transformer 54 will support the corresponding backward or inverse versions: Inverse DCT-II (or inverse DCT-III) ○Reverse DST-IV ○Reverse DCT-IV ○Reverse DST-VII Identity transformation (IT)

[0027] In any case, it should be noted that the set of supported transforms may include only one transform, such as one spectral-to-spatial or spatial-to-spectral transform.

[0028] As already outlined above, Figures 1-3 are presented as examples in which the configured prediction concept, described further below, may be implemented, in order to give specific examples of video encoders and decoders according to the present application. Thus far, the video encoders and decoders of Figures 1 and 2 respectively represent possible implementations of the video encoders and decoders described later in this specification. As outlined in more detail below, when the video encoders and decoders of Figures 1 and 2 incorporate the below-described embodiments for configured prediction according to the present application, the video encoders of Figures 1 and 2 support, as at least one option, processing block 80 or even all blocks constituting the current picture 12 in a manner outlined in more detail below. Therefore, the embodiments described below refer, inter alia, to a video encoder equivalent to encoder 10 of Figure 1 that handles block 80 in a manner outlined in more detail below; the same is true for the decoder of Figure 2, which therefore represents an example of a video decoder according to one embodiment in which block 80 is handled in a manner outlined in more detail below. However, Figures 1 and 2 are merely illustrative examples. A video encoder according to some embodiments of the present application may perform block-based coding using concepts outlined in more detail below, but may differ from the encoder of Figure 1 in, for example, the subdivision into blocks 80 being performed differently than illustrated in Figure 3, or in that the encoder does not use transform prediction residual coding, but instead directly codes the prediction residual, for example, in the spatial domain. Similarly, a video decoder according to some embodiments of the present application may perform decoding from data stream 14 using configured predictive coding concepts outlined further below, but may differ from decoder 20 of Figure 2 in, for example, the subdivision into blocks of picture 12' being performed differently than described in connection with Figure 3, and / or in that the decoder derives the prediction residual from data stream 14 in, for example, the spatial domain rather than the transform domain.

[0029] It should be noted that, in particular, with regard to the block subdivision into blocks 80, the same can be done in the manner outlined in connection with FIG. 3 or in a different manner. The subdivision into transform blocks, if present, can also be done as described in connection with FIG. 3 or in a different manner. In particular, the subdivision into other blocks, such as into blocks on the one hand and into transform blocks on the other hand, can be done by subdividing picture 12 into these blocks separately, each independently or dependently. For example, one subdivision, such as into transform blocks, can be an extension of the other subdivision as described above, or both subdivisions can be separate extensions of a common primary subdivision, such as into a tree-root block array of the picture as described in connection with FIG. 3. The same also applies to other sub-picture granularities mentioned below, such as with regard to the definition of certain prediction parameters, prediction modes, contribution weights, etc. Different subdivisions can be used for different ones of these entities, and the same can be defined independently, partially independently, or as extensions of each other.

[0030] That said, the following description focuses on the situation where the motion vector points toward a portion in the reference picture plane that intersects a reference picture boundary beyond which the picture content is unavailable. To pad a sub-portion that extends beyond the boundary line or is completely outside the reference picture, a selection of one of multiple intra-prediction modes according to embodiments of the present invention is made. The aim is to improve temporal inter-prediction quality by selecting the most appropriate angular intra-prediction mode for padding based on the characteristics of the prediction block.

[0031] In the following description, the term "reference pixel area" corresponds to a "reference portion," "reference frame" corresponds to a "reference picture," "frame boundary" corresponds to a "reference picture boundary," "boundary pixel" corresponds to a "boundary sample," and "template area" corresponds to a "probe area." Corresponding terms may be used in the description. Additionally, "padding" has the same meaning as "boundary padding" in this specification. Furthermore, in the following embodiments, one of multiple angular intra prediction modes is selected as an example of an intra prediction mode for padding. However, the intra prediction mode of the present invention is not limited to the angular intra prediction mode, and the multiple intra prediction modes may include, for example, DC mode and / or planar mode, and / or a mode for padding other intra prediction modes. In the case of DC mode and planar mode, the vertical direction (perpendicular to the frame boundary) may be used for the boundary sample.

[0032] 4 illustrates 33 different angular intra-prediction modes used in some embodiments of the present application, where the prediction angle in a reference portion (reference pixel area) 100 of a reference picture (reference frame) 101 ranges between 45° and 135° relative to a frame boundary (reference picture boundary) 106 as shown in FIG.

[0033] The angular border padding implemented follows the angular intra prediction of the Joint Exploration Test Model (JEM) described in Reference 2. Note that the angular intra prediction algorithm of JEM assumes that the left and top boundary pixels of an area are always available for prediction. Nevertheless, the basic prediction algorithm assumes that all boundary pixels are straight lines, and therefore, JEM translates either the left or top boundary pixels. This translation step is omitted in the preferred embodiment of the present application. Furthermore, JEM uses 65 angular intra prediction modes, consisting of 33 vertical modes and 33 horizontal modes. In the preferred embodiment of the present application, only 33 vertical modes are used, as shown in FIG. 4. Furthermore, note that the vertical angular prediction method in JEM must be rotated clockwise by 0°, 90°, 180°, or 270° to perform boundary padding for the bottom, left, top, or right reference frame boundary 106, respectively.

[0034] If the reference pixel area 100 is partially or totally outside the reference frame 101 in the horizontal and vertical directions, for example at the edge of the reference frame (i.e., at the corner of the frame), a preferential border padding side is calculated. Hor and N Ver indicates the number of horizontal and vertical lines of the reference pixel area 101 that lie outside the reference frame area. Hor and N Ver Depending on the , the preferred border padding side can be calculated as follows: If edge == Top-Left-Edge then If N Ver > N Hor then Top-Side else Left-Side Else if edge == Top-Right-Edge then If N Ver > N Horthen Top-Side else Right-Side Else if edge == Bottom-Right-Edge then If N Ver > N Hor then Bottom-Side else Right-Side Else if edge == Bottom-Left-Edge then If N Ver > N Hor then Bottom-Side else Left-Side

[0035] For this calculated preferred border padding side, the claimed selected Angular intra prediction mode is applied, while for the remaining sides, the original vertical border padding process is applied as shown in Figure 5.

[0036] 5, at the frame boundary 106, the overlapping reference pixel area 100 is divided into an area where multi-mode border padding (MMBP) is used and the remaining area where default padding is used, which in this case is vertical padding.

[0037] Alternatively, a reference pixel area 100 outside a reference frame 101 is entirely padded with MMBP, as shown in Figure 6. In this case, not all angular intra-prediction modes can be used because, depending on the prediction angle, the MMBP padding may require boundary pixels 102 that are not available, as indicated by the dotted arrows in Figure 6. It should also be noted that the restriction of angular intra-prediction modes can be implicitly determined by the encoder (10) and decoder (20) without the need to signal the restricted subset. For this purpose, the prediction direction of corner pixels of the predicted portion of the reference pixel area is flipped by 180° to test whether all necessary boundary pixels are available.

[0038] In this embodiment, the most appropriate angular inter prediction mode for boundary padding is derived in the video encoder 10 and the video decoder 20 whenever the reference pixel area is partially or entirely outside the reference frame area. As a prerequisite, the reference pixel area 100 is divided into an area 100' outside the reference frame 101 and an area 100'' inside the reference frame 101, as shown in FIG. 7. All pixels within the reference frame 101 are directly used as input pixels for motion compensation prediction of the current prediction block. All pixels outside the reference frame 101 in the area 100' are predicted with the help of intra prediction from neighboring boundary pixels 102 of the reference frame 101, as will be described later. Note that angular intra prediction with a vertical angle of 90° corresponds to the state-of-the-art reference frame boundary padding used in H.265 and JEM.

[0039] To select one of multiple angular intra-prediction modes in the encoder 10 and decoder 20, the direction of each intra-prediction may be rotated 180° to face inward of the reference frame 101. Then, as shown in FIG. 7, a template area, i.e., a probe area 104, is defined based on the rotated intra-prediction within the reference frame 101. The size and shape of the template area 104 are not limited. For example, one possible shape would be a rectangular block. For each mode and associated template area 104, rotated intra-prediction is performed from adjacent boundary pixels 102 to predict all pixel values ​​within the template area 104. By evaluating the results of the above predictions, the most appropriate angular intra-prediction mode is selected.

[0040] For evaluation, predicted pixels are compared with already reconstructed pixels of the reference frame 101 within the template area 104 to derive the final angular intra prediction mode. According to the claim language, the probe area (template area) 104 is filled, and the probe filling is compared with the reconstructed probe area 104 filling to measure the similarity between them. Any measure of similarity between blocks can be used for this comparison, such as sum of absolute differences (SAD), sum of absolute difference transforms (SATD), sum of squared errors (SSE), or the Hadamard transform. To improve prediction, content-dependent transforms can be used to transform information into another, less sensitive domain, such as location. Such content-dependent transforms can also be trained. The angular intra prediction mode with the highest similarity measure between the predicted pixels and the reconstructed pixels within the template area 104 can be finally selected for use in padding the area 100′ of the reference portion 100 of the reference picture 101 that extends beyond the boundary 106 of the reference picture 101.

[0041] As a modification of the above embodiment, to further improve the selection accuracy, multiple (e.g., number N) different template areas (probe areas) can be used for each angular intra prediction mode, where N=2 in the following. For example, a first template area is defined depending on the outermost pixel line 112 of the reference pixel area 100 with respect to the frame boundary 106, as shown in Figure 8. A second template area is defined depending on a first pixel line 110 of the reference pixel area 100, which is within an area 100' of the reference frame 101, as shown in Figure 8.

[0042] For both of these pixel lines 110, 112 in the reference pixel area 100, the angular intra prediction mode is inverted to determine the leftmost and rightmost frame boundary pixels that need to be used for angular intra prediction. Without any kind of limitation, in the following it is assumed that the reference pixel area 100 overlaps with the reference frame 101 on the frame boundary 106 (i.e., the preferred padding side is on top). In this case, the leftmost and rightmost frame boundary pixels are located at coordinates {x left-most ;0} and {x right-most ;0}. Then the associated template area has {x left-most -1;1} and {x right-most 9 , the template area is defined by a rectangle with top-left and bottom-right coordinates of {+1;+3}. This means that the template area includes the next two pixel lines in the reference frame below the used frame boundary pixel 102, extended by one additional pixel in the left and right directions. If the reference pixel area 100 overlaps the reference frame 101 at the left, right, or bottom frame boundary (i.e., the preferred padding side is left, right, or bottom), the template area needs to be rotated accordingly and is defined by the table shown in FIG. 9 . That is, the area 100′ extends to the top side of the reference picture 101, i.e., above the boundary 106, and the template area is defined as described above. If the area 100′ extends to the left side of the reference picture 101, the template area is rotated with respect to the left boundary 101 of the reference picture. The coordinates of the defined template area are derived based on the table in FIG. 9 , and the height and width in the table in FIG. 9 are the size of the reference picture 101.

[0043] As mentioned above, a first template area based on the first pixel line 110 and a second template area based on the outermost pixel line 120 are defined by considering the table in FIG. 9 . Then, pixel values ​​within both of these template areas are predicted by rotating the original angular intra prediction by 180°, i.e., both template areas (probe areas) are filled based on boundary samples (boundary pixels) 102 of the reference picture 101 at the portion of the boundary 106 along the direction opposite to the filling direction associated with the respective angular intra prediction mode, as shown in FIG. 7 . Thus, the angular prediction scheme follows the Joint Exploration Test Model (JEM) angular intra prediction described in Reference 2. Then, predicted pixel values ​​within both template areas are compared with the reconstructed reference pixels using a sum of absolute difference (SAD) similarity measure. The final angular intra prediction mode can be selected as the mode that minimizes the sum of the SAD values ​​of both template areas for N=2, i.e., the angular intra prediction mode with the highest similarity between the probe filling and the reconstructed filling can be selected. It should be noted that other pixel lines in the reference pixel area can also be used to define the template area. Furthermore, for example, intermediate pixel lines between the outermost pixel lines and the first contour line can also be used to define the template area. In addition, the similarity measure can also include weighting of samples in each probe area for features such as luminance intensity or edge detection to differentiate the importance of the samples, thereby unequal weighting the similarity contribution of each sample in the template area to the similarity measure based on the psychovisual weighting performed.

[0044] In addition, high-level parameters controlling the behavior of boundary padding, which involves the derivation of an angular intra prediction mode for padding, may include the size and shape of the template area. This means that several additional pixel lines perpendicular to the frame boundary and optional extensions of the template area may be parallel to the left and right frame boundaries. Additionally, high-level parameters may be introduced to specify the lines within the reference pixel area used to define the template area, such as the outermost line, the first line in the reference frame, or an intermediate line between the outermost line and the first line. Furthermore, a reduction in the number of angular intra prediction modes used may be signaled. Additional high-level parameters may be introduced to specify minimum and / or maximum thresholds defining how far outside the reference frame the reference pixel area must be to select an angular intra prediction mode for padding. That is, the thresholds may define the amount of extension area of ​​the reference portion outside the reference picture. The high-level parameters may be signaled in a picture parameter set (PPS), a sequence parameter set (SPS), a slice header, or any other high-level syntax structure.

[0045] Furthermore, known codecs featuring vertical boundary extensions can be used with the process for selecting the probe area selection. In implementing vertical boundary extensions, a technique known as motion vector (MV) clipping is often used to reduce the memory cost associated with storing sample values ​​of boundary extension samples. Due to the repetitive nature of sample values ​​in vertical boundary extensions, it is wise to clip the motion vectors, resulting in a reference block containing only boundary extension samples. The content of the reference block resulting from the clipped MVs is similar to the original reference block. However, decoder implementations can conserve resources using such samples by generating boundary extensions of limited size. Typically, a fixed size constraint, such as the maximum coding unit size plus the size of the interpolation filter kernel, is used to ensure that all potential reference block sizes can fit into a limited amount of generated boundary extension samples, as illustrated in FIG. 10.

[0046] As illustrated in FIG. 11 , the distance of the reference pixel area from the frame boundary in multi-mode boundary padding (MMBP) affects the selection process of the probe area at the decoder 20. Therefore, to avoid inconsistencies due to the use of different probe areas, MV clipping needs to be performed in a coordinated manner at the encoder 10 and decoder 20. Therefore, one aspect of the present invention is a high-level syntax element in a bitstream structure, such as a parameter set or slice header, that enables the encoder 10 to indicate to the decoder 20 the MV clipping procedure to be used or the corresponding bitstream constraints, as described above. Such syntax elements include, for example, the maximum MV length or the distance from the reference block to the frame boundary at which MV clipping is performed, as well as the size to which the MV is clipped or the distance between the frame boundary and the reference pixel area due to the clipped motion vector. The included information is not limited to the above information and may also include further or other information.

[0047] Furthermore, adaptive MV clipping can be used to control probe area selection in a beneficial manner by adjusting the amount of clipping to the size of the reference pixel area, i.e., the reference block size. As illustrated in FIG. 12, the same MV is used for two blocks of different sizes B>A, and adaptive MV clipping results in two different MVs, e.g., colored red, that ensure the reference area is as close as possible to the frame boundary and minimizes the distance to the selected probe area. That is, for a smaller-sized reference portion 110′, i.e., block size A (smaller block), the amount of motion vectors is larger than for a larger-sized reference portion 110′, i.e., size B (larger block). Therefore, the motion vector coordinates of inter-predicted blocks of the current picture are clipped by an amount that depends on the size of the reference portion, with the amount being larger for smaller sizes. Therefore, another aspect of the present invention is a high-level syntax element in a bitstream structure, such as a parameter set or slice header, that enables the encoder 10 to indicate to the decoder 20 that MV clipping is performed adaptively to the reference block size, or to indicate corresponding bitstream constraints to the decoder 20.

[0048] The number of coding options evaluated at the encoder side is an important factor in the complexity of a codec. Therefore, one aspect of the present invention is a means for controlling the impact of the complexity of the boundary padding described herein. This means consists of performing motion estimation at the encoder, i.e., searching for various motion-compensated predictor samples for inter prediction and evaluating the coding cost by extending the known motion estimation through a second stage. The first motion compensation stage encompasses all steps of known motion compensation (e.g., generating known boundary padding samples and evaluating the coding cost of residual data for a given predictor), while the second stage is a modification to known motion compensation, where the best predictor is further enhanced through a more sophisticated boundary sample padding scheme, such as the scheme described herein.

[0049] If the reference portion is partially or entirely outside the reference picture, the following other embodiments also apply. As already mentioned above, the reference pixel area is divided into a region outside the reference frame area and a region inside the reference frame area as a prerequisite. All pixels within the reference frame area are directly used as input pixels for motion-compensated prediction of the current prediction block. As illustrated in Figures 13 and 14, all pixels outside the reference frame area are predicted with the help of angular intra prediction from adjacent boundary pixels of the reference frame. That is, the area 100' of the reference portion 100, i.e., the area 100' outside the boundary 106 of the reference picture 101 within the reference portion 100, is filled based on the boundary samples 110. The filling direction is selected from 33 different angular intra modes. In this case, it is not necessary to select from 33 angular intra modes, but rather it is possible to select from a limited number of angular intra prediction modes defined in advance as parameters. Figure 13 shows one angular intra prediction mode 120, and Figure 14 shows another angular intra prediction mode 122. 13 and 14, the boundary pixels 110′ used depend directly on the selected angular intra-prediction mode 120 and 122, respectively. It should also be noted that the angular intra-prediction mode with a vertical angle of 90° corresponds to the state-of-the-art reference frame boundary padding used in H.265 and JEM. The chosen, i.e., selected, angular intra-prediction mode is signaled to the decoder within the encoded bitstream.

[0050] Additionally, the selected angular intra prediction mode can be coded using context-based adaptive arithmetic coding (CABAC). Additionally, an enable flag is used to indicate whether default vertical border padding or border padding based on the selected angular intra prediction mode is applied. Because the default vertical mode is one of the 33 angular intra prediction modes, the remaining 32 angular intra prediction modes can be coded with five additional CABAC bins. The enable flag, as well as the remaining angular modes, are coded as a series of binary decisions with an adaptive binary probability model (a subset of these bins, e.g., angular intra prediction modes, can also be coded with a non-adaptive probability model). To model the enable flag, various context models can be used, which can be selected based on the amount by which the reference pixel area is outside the reference frame. For example, various context models can be used to match cases in which the reference pixel area is outside the reference frame by less than ¼, less than ½, less than ¾, or more than ¾.

[0051] Alternatively, a list of most probable modes (MPMs) is used to signal frequently occurring angular intra prediction modes for boundary padding. Possible MPM candidates may be angular intra prediction modes for boundary padding of previously coded neighboring blocks if angular intra prediction modes for boundary padding are available for those blocks. Other MPM candidates may be angular intra prediction modes of previously coded neighboring blocks, if available. Furthermore, angular intra prediction modes of coded blocks in reference frames co-located with the reference picture may be used as MPM candidates. In addition, angular intra prediction modes of coded blocks in associated random access point pictures co-located with the reference picture area may be used as MPM candidates. For the 33 angular intra prediction modes, the MPM candidate list contains one MPM candidate mode. A CABAC binary flag is used to signal whether the MPM candidate should be used. The remaining 32 angular modes can be signaled with five CABAC bins. The MPM flag, as well as the remaining angular intra prediction modes, are coded as a series of binary decisions with an adaptive binary probability model as described above.

[0052] The angular intra prediction mode for boundary mode is part of the motion information of the coded block. Therefore, when motion information is used / copied by a different coded block, for example, when merge mode or motion vector prediction is used, the angular intra prediction mode for boundary padding associated with the copied motion vector is also copied and may be applied when the reference pixel area is outside the reference frame area. In the case of motion vector prediction from a block that performs MMBP, if MMBP is used for the coded block, the selected angular intra prediction mode for padding may be the MPM for coding of the padding direction of the current coded block. Alternatively, the possible motion prediction candidates may provide a list of possible MPMs.

[0053] High-level parameters controlling the behavior of partial boundary padding involving signaling of a selected angular intra prediction mode for padding may include parameters for defining a subset of angular intra prediction modes to use for boundary padding. In this case, the number of supported angular intra prediction modes for padding can be limited to a smaller number, so that fewer CABAC bins need to be used to signal the selected angular intra prediction mode for boundary padding. For example, using only a subset of 18 angular modes in combination with an MPM candidate list of two modes leaves only 16 angular modes that can be coded in four CABAC bins (thus resulting in two bins for signaling MPM modes and five bins for signaling the remaining non-MPM modes). As mentioned above, additional high-level parameters may be introduced to specify minimum and / or maximum thresholds defining how far the reference pixel area must be outside the reference frame area to select the most appropriate angular intra prediction mode for padding. The high-level parameters may be signaled in a picture parameter set (PPS), a sequence parameter set (SPS), a slice header, or any other high-level syntax structure.

[0054] The above embodiments can be combined, for example, the template embodiment and the other embodiment that signal the selected angular intra-prediction mode as described above can be combined in a further embodiment, as described below.

[0055] The selection of the angular intra-prediction mode follows the embodiment described in FIG. 7. In this embodiment, an ordered list of angular intra-prediction modes for the padding is generated based on the aforementioned similarity measure. This generated ordered list of angular intra-prediction modes for the padding is truncated by a predefined value of the angular intra-prediction mode for the largest signaled padding. Finally, the angular intra-prediction mode for the selected padding in this truncated ordered list is signaled using context-based adaptive arithmetic coding (CABAC) as described above.

[0056] As already mentioned above, an enable flag can be used to indicate whether the default vertical border padding or a truncated ordered list of selected padding modes should be used. In addition, the number of padding modes signaled can be 2 N, which can be signaled in N CABAC bins. The selected angular intra prediction mode is coded as a series of binary decisions using an adaptive binary probability model (a subset of these bins can also be coded using a non-adaptive probability model). When a binary probability model is used to code the selected angular intra prediction mode, various context models can be used, as described above, that can be selected based on the amount of area of ​​the reference portion that is outside the reference frame area. Furthermore, a truncated ordered list of selected angular intra prediction modes for padding is used as an MPM candidate list. Here, the MPM candidate list and the remaining angular intra prediction modes for padding can be coded as described above. Furthermore, the same high-level parameters as described above can also be used, such as parameters for defining a subset of angular intra prediction modes, a threshold for applying the selected angular intra prediction mode, and parameters for defining the size and shape of the template area. Also, as described above, the high-level parameters can be signaled in a picture parameter set (PPS), a sequence parameter set (SPS), a slice header, or any other high-level syntax structure.

[0057] In a preferred embodiment of the present application, the high-level syntax includes syntax elements that control multi-mode boundary padding, which are conveyed in one or more high-level syntax structures, such as a sequence parameter set (SPS; valid for all pictures in a coded video sequence), a picture parameter set (PPS; valid for one or more pictures), or a slice header (valid for a slice).

[0058] The high-level syntax element that controls an application to select an Angular intra prediction mode can include, but is not limited to, one or more of the following parameters: A flag or IDC signaling which Angular inter prediction mode for padding (MMBP, vertical, or omnidirectional padding) should be applied Parameters for signaling or constraints indicating frame boundary unit combinations of padding modes. For example, whether MMBP or omnidirectional padding is used can be signaled in the data stream for parts along the frame boundary or outer picture boundary of the reference picture. As an option, it can be signaled that both options are available. Different available modes can be signaled so that the encoder and decoder can check which parts of the frame boundary of the reference picture overlap or intersect with the reference area to decide which intra prediction mode to use for different parts; Parameters for signaling the selection process of the above embodiment to be used Parameter for signaling angular intra-prediction mode that can be used for MMBP A parameter that specifies the exact angular intra prediction mode to be used for a particular reference frame boundary region. Minimum and / or maximum thresholds that define how far the reference pixel area must be outside the reference frame area for the Angular intra prediction mode selection to be applicable. Parameters for signaling the size and shape of the template area Parameters for signaling the set of lines within the reference pixel area used to determine the template area

[0059] Regarding the second option, please further note the following: while the embodiment outlined above uses the angular intra-prediction mode as an example, in which an intra-prediction mode for padding was ultimately selected, it is clear that padding in vertical padding mode, in which padding is performed by vertically extrapolating boundary samples to an area beyond the frame boundary, is also an intra-prediction mode, just as omnidirectional padding mode is an intra-prediction mode that fills / pads an area outside the frame boundary using samples in the same frame, i.e., the reference frame. Thus, the high-level syntax specifies which intra-prediction mode should be used or which intra-prediction mode is available for which portions of the frame boundary, and different modes used or available for different portions of the frame boundary may be signaled, such as omnidirectional mode for some portions, such as the right and left boundaries, and vertical mode for other portions, such as the top and bottom boundaries. The encoder and decoder simply need to recognize the portion of a particular reference area of ​​inter-prediction block 80 that exceeds the frame boundary and ascertain which intra-prediction mode it is associated with for padding according to the high-level syntax in the data stream. The parameters listed above can be included in more than one high-level syntactic construct, for example a global parameter specified in an SPS can be overridden by a parameter specified in a PPS.

[0060] For some embodiments of the present application, this is required whenever the reference pixel area is partially or entirely outside the reference frame area. As a prerequisite, the reference pixel area is divided into a region outside the reference frame area and a region inside the reference frame area, as described above. All pixels within the reference frame area are directly used as input pixels for motion compensation prediction of the current prediction block. All pixels outside the reference frame area are predicted with the help of intra prediction from adjacent boundary pixels of the reference frame, as described above.

[0061] In other words, the following steps again exemplarily explain how a preferred intra-prediction angle or a preferred intra-prediction mode to be used for padding can be selected from among multiple intra-prediction angles available at the encoder and decoder. The selection process can be performed only once per luma (Y) color plane for each prediction block and each associated motion vector. For the chrominance color planes (Cb and Cr), the same intra-prediction angles can be selected as for the luma color plane: Determine the position of the reference area 100: Calculate the reference area 100 by adding the motion vector associated with the current block to the area of ​​the current block (e.g., move the current block position by the associated motion vector). Note that motion vectors may have fractional parts. Therefore, round the motion vector to the nearest integer part. Add the motion vector to the current block position. Increase the reference area 100 by one sample in each direction. That is, the area 100 can be one sample larger in all directions (left, right, top, bottom) than the inter-predicted block for which MMBP is performed. Determine the preferred IntraBP side: if the reference area 100 overlaps the reference frame 101 on only one side, use this side, otherwise the reference area 100 overlaps the reference frame 101 on two sides (see Figure 5). In that case, use the default vertical padding on the side that overlaps the reference frame with fewer sample lines. Use IntraBP padding on the remaining sides. Determine the bottom and top lines: the bottom line is the first sample line of the reference area 100 parallel to the preferred IntraBP side and outside the reference frame 101 (110 in Fig. 8). The top line is the last sample line of the reference area parallel to the preferred IntraBP side and outside the reference frame (112 in Fig. 8), i.e. the one furthest from the reference frame boundary 106. Constraining multiple available intra prediction modes: For the top line and each intra prediction mode, calculate the relevant / affected boundary pixels that need to be used for intra prediction by reversing the prediction direction. The maximum displacement of boundary pixels from the vertical intra prediction mode is 1.25 * If it exceeds the width of the prediction block, skip / do not use this intra-prediction angle. This step is optional: optionally, angular modes in which backprojecting the outermost line of samples 112 onto the boundary samples 110 causes the boundary samples 110 to be offset from the perpendicular backprojected footprint of the outermost sample line by more than a predetermined threshold are excluded from the selectable list of modes. Calculate probe areas for bottom and top lines: For the bottom (top) line and for each intra prediction mode in the set of constrained intra prediction modes, reverse the prediction direction to determine the associated / affected boundary pixels 110' that need to be used for intra prediction. Place a probe area 104 below the affected boundary pixels in the reference frame (104 in Figure 7). Set the height of the probe area to two sample lines perpendicular to the preferred IntraBP side. Calculate similarity measures: Predict probe area sample values ​​by inverted intra prediction from related / affected boundary pixels. Calculate SAD measures between predicted probe area sample values ​​and reconstructed sample values ​​of reference frame at probe area location. Combine / add SAD measures from lower and upper probe areas. Select a preferred intra-prediction angle: From the set of constrained intra-prediction angles, select the angle with the smallest similarity measure as the preferred intra-prediction angle. If intra-prediction angles have the same similarity measure, prioritize the angle that is closer to the vertical prediction angle.

[0062] If the selected preferred intra prediction angle is not equal to the vertical intra prediction angle, the encoder and decoder perform IntraBP for each color plane using the selected intra prediction angle and the preferred IntraBP angle: Determine the location of the reference area: Calculate the reference area as described above, but increase the reference area by half the size of the interpolation filter taps in each direction instead of just one sample in each direction. Fill parts of the reference area that lie within the reference frame: Fills the parts of the reference area that lie within the reference frame with the sample values ​​of the reference frame. · Optionally fill non-preferred IntraBP side: If the reference area overlaps the reference frame on two sides, default vertical padding is applied to the part of the reference area that overlaps the reference frame with fewer sample lines. · Perform IntraBP on the preferred IntraBP side: Using the preferred intra prediction angle, perform intra prediction on the boundary samples on the preferred IntraBP side to fill in the reference area 100' outside the reference frame.

[0063] The sample values ​​of the reference area calculated as above for each color plane are used as input values ​​for the motion compensated prediction of the current block.

[0064] In addition, existing applications, such as 360° video streaming services based on the MPEG OMAF standard, rely heavily on spatial video partitioning or segmentation techniques. In such applications, spatial video segments are transmitted to clients and jointly decoded in an adaptive manner according to the current client's viewing direction. Another related application that relies on spatial segments of video planes is the parallelization of encoding and decoding operations, for example, to facilitate the multi-core and parallelization capabilities of modern computing platforms.

[0065] One such spatial segmentation technique, implemented in HEVC, is known as tiling, which divides a picture plane into segments that form a rectangular grid. The resulting spatial segments are independently coded for entropy coding and intra-prediction. Furthermore, means exist to indicate that spatial segments are also independently coded for inter-prediction, which may also be combined with static spatial segmentation across a picture. Together, these constraints enable truly independent coding of the spatial segmentation of an exemplary picture plane, involving two such adjacent, independently coded spatial segments.

[0066] More specifically, when encoding the current picture using the spatial segmentation shown in Figure 15, which is facilitated through inter-prediction of reference pictures that also use the spatial segmentation illustrated above, motion vectors are constrained to avoid predicting intersecting spatial segment A through reference to samples of reference pictures that are present in spatial segment B, and vice versa.

[0067] While independent coding of spatial segments enables existing applications as described above, the associated coding constraints can impose significant coding costs. This bitrate overhead can arise, for example, from signaling relatively large motion vector differentials (MVDs) that modify available motion vector candidates, which would otherwise result in the use of sample positions outside a given spatial segment. Furthermore, to compensate for the degradation of inter-prediction predictors, the amount of residual data signaled must be increased. Therefore, it is highly desirable to enhance the codec's behavior at such spatial segment boundaries.

[0068] One way to mitigate the coding penalty of prior art solutions for such independent spatial segments is to use a scheme for padding intra-picture boundaries, i.e., segment boundaries that are inside a picture, in addition to outer frame boundaries, similar to or in accordance with the above-mentioned MMBP concept based on picture boundaries with slight modifications and extensions described herein, or using a high-level syntax-based padding mode selection that signals which of two or more intra-prediction modes to use for a particular portion, such as a spatial segment boundary or a perpendicular portion of a spatial segment boundary. That is, instead of the encoder restricting signaled motion vectors to not refer to reference portions that extend or lie beyond the boundary of the spatial segment, such portions are padded at the encoder and decoder, and the signaled intra-prediction mode is used for padding if such reference portion intersects with or lies beyond a predetermined portion of the spatial segment boundary. Coding penalty reduction can also generally be achieved without multi-mode selection by using any intra-coding mode, such as one of vertical, omnidirectional, or angular intra-prediction modes, resulting in an encoder and decoder for block-based video coding configured to encode and decode, in a data stream for inter-predicted blocks, motion vectors that reference a reference portion 100 of a reference picture 101 that extends or resides beyond an intra-picture segment boundary of a predetermined spatial segment within the reference picture in which the inter-predicted block resides, and to pad an area 100' of the reference portion 100 that extends or resides beyond an intra-picture segment boundary, i.e., outside the predetermined spatial segment. The encoder does not need to redirect a motion vector predictor of a motion vector to force the signaled motion vector to remain within a predetermined spatial segment of the spatial segments into which the reference picture is partitioned in the manner in which video pictures, including the current picture, are constantly partitioned. The encoder does not need to signal a motion vector differential that redirects this vector.Signaling of motion vector differentials may lead to motion vectors referencing parts outside a given spatial segment, such as a motion vector differential of zero. In particular, the encoder may rely on a padding mechanism in the decoder that enforces independence of spatial segment coding of spatially offset spatial segments in the current picture and the reference picture, and the encoder is the same, i.e., padding to reconstruct inter-predicted blocks to fill the decoded picture buffer.

[0069] First, high-level syntax elements in bitstream structures such as parameter sets, slice headers, etc. generally control the application of multi-mode boundary padding at spatial segment boundaries. One value of this syntax indicates to the decoder: Inter-prediction of samples in a spatial segment is independent of samples belonging to other spatial segments. Sample positions outside spatial segment boundaries are used for inter-prediction of samples within the spatial segment. · The values ​​of samples at sample positions outside the spatial segment boundary are generated using the MMBP procedure at the spatial segment boundary from samples that belong to the spatial segment.

[0070] Although some aspects are described in the context of an apparatus, it is clear that these aspects also represent a description of a corresponding method, and that blocks or devices correspond to method steps or features of method steps. Similarly, aspects described in the context of a method step also represent a description of a corresponding block or item or feature of a corresponding apparatus. Some or all of the method steps may be performed by (or using) a hardware apparatus, such as, for example, a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, one or more of the most important method steps may be performed by such an apparatus.

[0071] The data stream of the present invention can be stored on a digital storage medium or can be transmitted over a transmission medium, such as a wireless or wired transmission medium, such as the Internet.

[0072] Depending on specific implementation requirements, embodiments of the present invention can be implemented in hardware or software. Implementation can be done using a digital storage medium, such as a floppy disk, DVD, Blu-ray, CD, ROM, PROM, EPROM, EEPROM, or flash memory, on which electronically readable control signals are stored that cooperate (or can cooperate) with a programmable computer system so that the respective methods are executed. The digital storage medium can therefore be computer-readable.

[0073] Some embodiments of the present invention include a data carrier having electronically readable control signals that can cooperate with a programmable computer system to perform one of the methods described herein.

[0074] Generally, some embodiments of the present invention can be implemented as a computer program product having program code operable to perform one of the methods when the computer program product runs on a computer, which program code can be stored on, for example, a machine-readable carrier.

[0075] Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

[0076] In other words, an embodiment of the inventive method is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

[0077] A further embodiment of the inventive method is, therefore, a data carrier (or digital storage medium or computer-readable medium) having recorded thereon a computer program for performing one of the methods described herein. The data carrier, digital storage medium or recording medium is typically not tangible and / or transitory.

[0078] A further embodiment of the inventive method is, therefore, a data stream or a sequence of signals representing the computer program for performing one of the methods described herein, The data stream or sequence of signals can for example be arranged to be transferred via a data communication connection, for example via the Internet.

[0079] A further embodiment comprises a processing means, for example a computer, or a programmable logic device, configured to or adapted to perform one of the methods described herein.

[0080] A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0081] Further embodiments according to the invention include an apparatus or system configured to transfer (e.g. electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a memory device, etc. The apparatus or system may, for example, comprise a file server for transferring the computer program to the receiver.

[0082] In some embodiments, a programmable logic device (e.g., a field programmable gate array) may be used to implement some or all of the functionality of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor to implement one of the methods described herein. In general, the methods are preferably implemented by any hardware apparatus.

[0083] The devices described herein may be implemented using a hardware device, or using a computer, or using a combination of a hardware device and a computer.

[0084] The devices described herein, or any components of the devices described herein, may be implemented at least in part in hardware and / or in software.

[0085] The methods described herein may be performed using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer.

[0086] Any component of the methods described herein or the apparatus described herein may be performed at least in part by hardware and / or by software.

[0087] The above-described embodiments merely illustrate the principles of the present invention. It is understood that modifications and variations of the arrangements and details described herein will be apparent to those skilled in the art. It is therefore intended to be limited only by the scope of the appended claims and not by the specific details presented by way of illustration and description of the embodiments herein.

Claims

1. 1. A video decoder for decoding pictures from a data stream, comprising: the video decoder having at least one processor; The at least one processor For an inter-predicted block of the picture, using a motion vector, identifying a first reference area that lies beyond a left vertical boundary or a right vertical boundary of a reference picture and a second reference area that lies within the boundary of the reference picture; selecting an omnidirectional padding mode for padding the first reference area based on an indication in the data stream; generating padding for said first reference area by copying samples from an opposite vertical picture boundary; configured to predict the inter-predicted block of the picture using the padding and the second reference area. Video decoder.

2. the indication is in at least one of a picture parameter set or a sequence parameter set.

2. The video decoder of claim 1.

3. the at least one processor is further configured to divide a picture plane of the video into spatial segments in a static manner across at least two pictures of the video, including the picture and the reference picture; the spatial segments are independently coded with respect to entropy coding; the boundary of the reference picture is a segment boundary of a predetermined spatial segment in the reference picture within which the inter-predicted block resides; 2. The video decoder of claim 1.

4. 1. A method for decoding pictures from a data stream, comprising: for an inter-predicted block of the picture, using a motion vector to identify a first reference area that lies beyond a left or right vertical boundary of a reference picture and a second reference area that lies within the boundary of the reference picture; selecting an omnidirectional padding mode for padding the first reference area based on an indication in the data stream; generating padding for said first reference area by copying samples from an opposite vertical picture boundary; predicting the inter-predicted block of the picture using the padding and the second reference area; method.

5. the indication is in at least one of a picture parameter set or a sequence parameter set. The method of claim 4.

6. The method further includes dividing a picture plane of the video into spatial segments in a static manner across at least two pictures of the video, including the picture and the reference picture; the spatial segments are independently coded with respect to entropy coding; the boundary of the reference picture is a segment boundary of a predetermined spatial segment in the reference picture within which the inter-predicted block resides; The method of claim 4.

7. 1. A video encoder for encoding pictures from a data stream, comprising: the video encoder having at least one processor; The at least one processor encoding an indication of omnidirectional padding mode into the data stream; For an inter-predicted block of the picture, using a motion vector, identifying a first reference area that lies beyond a left vertical boundary or a right vertical boundary of a reference picture and a second reference area that lies within the boundary of the reference picture; generating padding for said first reference area by copying samples from an opposite vertical picture boundary; configured to predict the inter-predicted block of the picture using the padding and the second reference area. Video encoder.

8. 1. A method for encoding a picture from a data stream, comprising: encoding an indication of omnidirectional padding mode into said data stream; for an inter-predicted block of the picture, using a motion vector to identify a first reference area that lies beyond a left or right vertical boundary of a reference picture and a second reference area that lies within the boundary of the reference picture; generating padding for said first reference area by copying samples from an opposite vertical picture boundary; predicting the inter-predicted block of the picture using the padding and the second reference area; method.

9. comprising instructions that, when executed, cause at least one processor to perform the method of claim 4 or 8, Non-transitory computer-readable medium.

Images