Apparatus for selecting intra prediction mode for padding

By dynamically selecting the intra prediction mode, the inter prediction of video encoding is improved, and the problem of insufficient prediction quality of pixel areas outside the image boundary is solved, and the efficiency and effect of video encoding are improved.

CN119031129APending Publication Date: 2024-11-26FRAUNHOFER GESELLSCHAFT ZUR FORDERUNG DER ANGEWANDTEN FORSCHUNG EV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411440186.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2018-03-29
Filing Date
2019-03-29
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

When existing video encoding standards process pixel areas outside the image boundary, the prediction quality is insufficient, especially in 360° video content. The traditional boundary filling method cannot effectively approximate the current prediction block, resulting in a decrease in inter prediction quality.

Method used

Improve the time-inter prediction quality by selecting the most appropriate intra prediction mode, the specific method is to evaluate multiple angle intra prediction modes based on the characteristics of the current prediction block, select a mode that can provide the best approximation, and be in the external area of ​​the reference image Make filling.

Benefits of technology

The quality of inter prediction is improved. By dynamically selecting appropriate intra prediction mode, the pixel areas outside the image boundary can be better approximated and processed, and the efficiency and effect of video encoding are improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119031129A_ABST
    Figure CN119031129A_ABST
Patent Text Reader

Abstract

A video codec for supporting temporal inter prediction is configured to perform padding of a region of a referenced portion of a reference picture that extends beyond a boundary of the reference picture, the referenced portion is referenced by an inter prediction block of the current image by selecting one of a plurality of intra prediction modes and filling the region with the selected intra prediction mode.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] This application is a divisional application of Chinese invention patent application 201980036448.X, filed on March 29, 2019, and entitled “Device for selecting an intra-frame prediction mode for padding”. Technical Field

[0002] The present application relates to video encoding / decoding. Background Art

[0003] Since H.261, all video coding standards, including the current state-of-the-art ITU T H.265 | MPEG HHEVC video coding standard (Reference 1: ITU-T and ISO / IEC. High Efficiency Video Coding. ITU-T Recommendation H.265 | ISO / IEC 23008 10 (HEVC), Version 1, 2013; Version 2, 2014), rely on the basic concept of block-based hybrid video coding. Here, the input signal for each color component is segmented into disjoint blocks, which are encoded using a hybrid video coding scheme based on prediction, transform, quantization and entropy coding. Therefore, one of the first steps in the encoding and decoding algorithm is to generate a prediction signal for the current block, which is performed by a spatial or temporal prediction scheme. In the case of temporal prediction, the prediction signal is derived from an already processed frame, which is denoted as a reference frame. For this purpose, a displacement vector specifies the offset of the prediction signal in the reference frame relative to the position of the current coded block. This temporal prediction scheme is also called translational motion compensated prediction. In this sense, the displacement vector is denoted as a motion vector.

[0004] In order to allow motion vectors with sub-pixel precision, an interpolation filter is used to calculate the prediction signal. In addition, the prediction signal can be combined by the superposition of multiple prediction signals, each of which is specified by a different motion vector and / or a different reference frame. Independent of the final calculation and composition of the prediction signal, the basic principle remains the same. The area of ​​the reference block within the reference frame is determined by the position and size of the current coding block after displacement according to the additional motion vector. In the case of using an interpolation filter, this area must be extended to each side depending on the number of filter taps used. Subsequently, the pixels within the area of ​​the reference block are used to generate the temporal prediction signal of the current coding block. Typically, the area of ​​the reference block is located within the reconstructed reference frame. However, the area of ​​the reference block may not completely overlap with the area of ​​the reconstructed frame, which means that part of the referenced pixel area may be located outside the reference frame area.

[0005] In current video coding standards, the basic approach for handling the referenced portion, i.e., the referenced pixel region located outside the reference image, i.e., the reference frame region, is to restrict the motion vector or to extend the reference frame region. For example, in H.261, the motion vector is restricted at the image boundary so that all pixels referenced are within the reference frame region, i.e., all samples of the sample portion of the reference image located within the borders of the reference frame. Alternatively, in H.264 / H.265, a process called boundary padding is used to extend the reconstructed reference frame, as in Fig.16 That is, the boundary pixels (boundary samples 110) are copied to the padding area (the area outside the reference image 101), which can be regarded as intra-frame prediction of the boundary pixels perpendicular to the frame boundary 106. Here, at the edge of the reference frame, the edge pixels are additionally copied to the remaining area outside the reference frame 101 area, which is adjacent to the area such as Fig.16 The edge pixels shown in .

[0006] Additionally, known methods of boundary filling are advantageous when processing certain video content, known as omnidirectional or 360° video, which uses omnidirectional projection to map the camera surroundings onto a rectangular video image plane. One of the widely used formats for such content is based on equirectangular projection (ERP). Omnidirectional content allows for filling areas with correct content outside of the image boundaries, since the correct content is by definition depicted elsewhere in the same image. Any vertical image boundary of ERP video content that depicts 360° horizontally can be filled by copying samples from the opposite vertical image boundary. However, horizontal image boundaries, especially in situations where vertical viewing angles provide incomplete coverage, exhibit different characteristics.

[0007] exist Fig.17 , an exemplary ERP video frame is shown with 360° horizontal coverage and less than 180° vertical coverage, i.e., only 30° of viewing angle down from the equator at 0°. While the best padding for any of the vertical image boundaries is constructed using samples from the opposite vertical image side, the top and bottom horizontal image boundaries marked with dashed lines exhibit similar characteristics to image boundaries of typical rectangular video with limited field of view, and therefore benefit from the multi-mode boundary padding described herein. Therefore, one aspect of the present invention is to change the padding mode for each image boundary, for example, depending on knowledge of content characteristics or as signaled in the bitstream.

[0008] In many cases, a simple vertical extension of frame boundary pixels results in a continuation that is not a good approximation of the current prediction block. Better continuations are achieved by extending the reference frame boundary using different intra prediction modes (e.g., angle, DC or plane prediction, etc.). Exemplary intra prediction modes that can be used for boundary extension are intra prediction modes from H.265 as described in reference 1 or from the Joint Exploration Test Model (JEM) as described in reference 2 (J. Chen, E. Alshina, GJ Sullivan, J.-R. Ohm, J. Boyce, Algorithm Description of Joint Exploration Test Model 7 (JEM 7), JVET, doc. JVET-G1001, July 2017). In addition, in future embodiments of this method, future intra prediction modes may also be used to extend the reference boundary. The pattern of spatial extension of the frame boundary depends on the characteristics of the current prediction block. Therefore, a certain angle intra prediction mode must be selected / selected from a set of multi-angle intra prediction modes to generate a prediction signal, which provides the best approximation of the current prediction block. Summary of the invention

[0009] It is therefore an object of the present invention to provide a video codec configured to perform boundary filling to improve the quality of temporal inter-frame prediction.

[0010] This object is achieved by the subject matter of the claims of the present application.

[0011] The basic idea underlying the present application is to achieve quality improvement of temporal inter prediction by selecting the most appropriate intra prediction mode for padding. This may be to select an angular intra prediction mode from among a plurality of such intra prediction modes based on the characteristics of the current prediction block.

[0012] According to an embodiment, an angular intra prediction mode to be used for filling is selected based on an evaluation of an area of ​​a reference image at a portion of a boundary of a reference image where the referenced portion intersects the boundary. The evaluation is performed by performing the following operations for each of at least one subset of a plurality of angular intra prediction modes: filling a detection area located farther from the boundary than the boundary samples based on boundary samples of the reference image at the portion of the boundary in a direction opposite to the filling direction associated with the corresponding angular intra prediction mode to obtain a detection filling; and comparing the detection filling with the reconstructed filling of the detection area to measure the similarity between them. Therefore, the selection is performed based on the characteristics of the current prediction block, and therefore, it is possible to provide the best approximation of the current prediction block and improve the prediction quality.

[0013] According to an embodiment of the present application, an angular intra prediction mode to be used for filling is selected based on a prediction result of an area outside a reference image of a referenced portion. The referenced portion is referenced by an inter prediction block of a current image. Therefore, the angular intra prediction mode is selected depending on the characteristics of the area, and therefore, it is possible to select the most appropriate angular intra prediction mode for filling. In addition, a plurality of used angular intra prediction modes for a prediction area can also be signaled as a subset of supported angular intra prediction modes supported by a codec for an intra prediction block. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Further advantages are achieved by the claimed codec. Preferred embodiments of the present application are described below with respect to the figures, in which:

[0015] Figure 1 A block diagram showing an apparatus for predictively encoding a video as an example for a video decoder in which selection of an angular intra prediction mode for the padding concept according to an embodiment of the present application may be implemented;

[0016] Figure 2 A block diagram showing an apparatus for predictively decoding a video as an example for a video decoder, the apparatus cooperating with Figure 1 A device, in which the selection of the angular intra prediction mode for the padding concept according to the embodiments of the present application can be implemented;

[0017] Figure 3 A schematic diagram illustrating an example of a relationship between a prediction residual signal, a prediction signal and a reconstruction signal is shown in order to illustrate the possibility of setting up subdivisions for defining the prediction signal, processing the prediction residual signal and the like, respectively;

[0018] Figure 4 A schematic diagram illustrating an example of an angular intra prediction mode used in an embodiment of the present application is shown;

[0019] Figure 5 A schematic diagram illustrating an example of calculating a preferred boundary filling side according to an embodiment of the present application is shown;

[0020] Figure 6 A schematic diagram illustrating another example of calculating a preferred boundary filling side according to an embodiment of the present application is shown;

[0021] Figure 7 A schematic diagram illustrating an example of a detection area according to an embodiment of the present application is shown;

[0022] Figure 8 A schematic diagram illustrating an example of a process for defining a detection area according to an embodiment of the present application is shown;

[0023] Fig. 9 A sample table showing the coordinates of the detection area according to an embodiment of the present application is obtained;

[0024] Fig.10 A schematic diagram illustrating an example of motion vector clipping for a boundary-extended reference block according to an embodiment of the present application is shown;

[0025] Fig.11 A schematic diagram illustrating an example of template region derivation in the context of motion vector clipping according to an embodiment of the present application is shown;

[0026] Fig.12 A schematic diagram illustrating an example of motion vector clipping of an adaptive block size according to an embodiment of the present application is shown;

[0027] Fig.13 and Fig.14 A schematic diagram illustrating an angular intra prediction mode to be used for filling selected based on a prediction result of an area of ​​a referenced portion outside a reference image according to the present application is shown;

[0028] Fig.15 shows a schematic diagram illustrating an example of a horizontal boundary between two independent space segments according to an embodiment of the present application;

[0029] Fig.16 A schematic diagram illustrating vertical projections to explain the current state of the art is shown;

[0030] Fig.17 Sample images of video frames having 360° horizontal coverage and less than 180° vertical coverage are shown. DETAILED DESCRIPTION

[0031] The following description of the drawings begins by providing a description of a video encoder and a video decoder for a block-based predictive codec for coding of video images to form an example of a coding framework in which an embodiment of a composed prediction codec may be built. Figures 1 to 3 The video encoder and the video decoder are described in detail below. Thereafter, a description of embodiments of the combined prediction concepts of the present application is provided, together with a description of how such concepts can be built into Figure 1 and Figure 2 The description in the video encoder and decoder, but using the subsequent Figure 4 The embodiments described in the following figures can also be used to form Figure 1 and Figure 2 The video encoder and video decoder are operated by the lower layer coding framework of the video encoder and video decoder.

[0032] Figure 1 An apparatus is shown for predictively encoding a video 11 consisting of a sequence of images 12 into a data stream 14. For this purpose, block-by-block predictive encoding is used. In addition, transform-based residual encoding is used exemplarily. The apparatus or encoder is indicated using reference numeral 10. Figure 2 A corresponding decoder 20 is shown, i.e. a device 20 configured to predictively decode a video 11' consisting of an image 12' in the form of image blocks from a data stream 14, whereby transform-based residual decoding is also exemplarily used, wherein a prime symbol has been used to indicate that the image 12' and video 11' reconstructed by the decoder 20 deviate from the image 12 originally encoded by the device 10 in terms of the coding loss introduced by the quantization of the prediction residual signal. Figure 1 and Figure 2 Transform-based prediction residual coding is used exemplarily, but the embodiments of the present application are not limited to such prediction residual coding. Figure 1 and Figure 2 The same is true for the other details of the description, as will be outlined below.

[0033] The encoder 10 is configured to subject the prediction residual signal to a spatial to spectral transform and to encode the prediction residual signal thus obtained into a data stream 14. Similarly, the decoder 20 is configured to decode the prediction residual signal from the data stream 14 and to subject the prediction residual signal thus obtained to a spectral to spatial transform.

[0034] Internally, the encoder 10 may include a prediction residual signal former 22, which generates a prediction residual 24 in order to measure the deviation of the prediction signal 26 relative to the original signal (i.e. the video 11 or the current image 12). The prediction residual signal former 22 may, for example, be a subtractor that subtracts the prediction signal from the original signal (i.e. the current image 12). The encoder 10 then further includes a transformer 28, which subjects the prediction residual signal 24 to a spatial to spectral transformation in order to obtain a spectral domain prediction residual signal 24', which is then subjected to quantization by a quantizer 32 also included in the encoder 10. The prediction residual signal 24" thus quantized is encoded into the bit stream 14. For this purpose, the encoder 10 optionally includes an entropy encoder 34, which entropy encodes the transformed and quantized prediction residual signal into the data stream 14. The prediction residual 26 is generated by a prediction stage 36 of the encoder 10 based on the prediction residual signal 24" that has been decoded into the data stream 14 and can be decoded from the data stream. For this purpose, as Figure 1As shown in , the prediction stage 36 may internally include: an inverse quantizer 38, which inverse quantizes the prediction residual signal 24" so as to obtain a spectral domain prediction residual signal 24"', which corresponds to the signal 24' except for the quantization loss; followed by an inverse transformer 40, which subjects the latter prediction residual signal 24"' to an inverse transform, that is, a spectrum to spatial transform, to obtain a prediction residual signal 24"" corresponding to the original prediction residual signal 24 except for the quantization loss. The combiner 42 of the prediction stage 36 then recombines the prediction signal 26 and the prediction residual signal 24"", such as by addition, so as to obtain a reconstructed signal 46, that is, a reconstruction of the original signal 12. The reconstructed signal 46 may correspond to the signal 12'.

[0035] Prediction module 44 of prediction stage 36 then generates prediction signal 26 based on signal 46 by using, for example, spatial prediction (ie, intra prediction) and / or temporal prediction (ie, inter prediction). Details regarding this are described below.

[0036] Similarly, the decoder 20 may internally include components corresponding to the prediction stage 36 and interconnected in a manner corresponding to the prediction stage 36. In particular, the entropy decoder 50 of the decoder 20 may entropy decode a quantized spectral domain prediction residual signal 24″ from the data stream, and then the inverse quantizer 52, the inverse transformer 54, the combiner 56 and the prediction module 58 interconnected and cooperating in the manner described above with respect to the module of the prediction stage 36 restore the reconstructed signal based on the prediction residual signal 24″ so that Figure 2 As shown in , the output of the combiner 56 results in a reconstructed signal, ie the video 11 ′ or its current image 12 ′.

[0037] Although not specifically described above, it is readily apparent that the encoder 10 may set some encoding parameters, including, for example, prediction modes, motion parameters and the like, according to an optimization scheme, such as in a manner that optimizes a certain rate and distortion related criterion (i.e., encoding cost) and / or uses a certain rate control. As described in more detail below, the encoder 10 and decoder 20 and corresponding modules 44, 58, respectively, support different prediction modes, such as intra-coding mode and inter-coding mode, which form a kind of set or pool of raw prediction modes based on which the prediction of the image block is combined in a manner described in more detail below. The granularity at which the encoder and decoder switch between these prediction components may correspond to the subdivision of the image 12 and 12', respectively, into blocks. It should be noted that some of these blocks may be only intra-coded blocks, and some blocks may be only inter-coded blocks, and optionally even other blocks may be blocks obtained using both intra-coding and inter-coding, but the details will be explained below. According to the intra-coding mode, the prediction signal for the block is obtained based on the spatial, coded / decoded neighboring blocks of the corresponding block. There may be several intra-coding sub-modes, the selection among which is similar to representing an intra-prediction parameter. There may be directional or angular intra-coding sub-modes, according to which the prediction signal for the respective block is filled by extrapolating sample values ​​of neighboring blocks into the respective block along a direction dedicated to the respective directional intra-coding sub-mode. The intra-coding sub-modes may, for example, also include one or more further sub-modes such as a DC coding mode, according to which the prediction signal for the respective block assigns a DC value to all samples within the respective block; and / or a planar intra-coding mode, according to which the prediction signal for the respective block is approximated or determined as a spatial distribution of sample values ​​at sample positions of the respective block described by a two-dimensional linear function, with a derived tilt and offset based on a plane defined by the two-dimensional linear function of the neighboring samples. In contrast, according to an inter-prediction mode, the prediction signal for the block may be obtained, for example, by predicting the interior of the block in time. For parameterization of the inter prediction mode, a motion vector may be signaled within the data stream, which indicates a spatial displacement of a portion of a previously encoded picture of the video 11 where the previously encoded / decoded picture was sampled in order to obtain a prediction signal for the corresponding block.This means that, in addition to the residual signal encoding comprised by the data stream 14, such as entropy encoded transform coefficient levels representing the quantized spectral domain prediction residual signal 24", the data stream 14 may also have encoded into the data stream prediction related parameters for assignment to block prediction modes, prediction parameters for the assigned prediction modes, such as motion parameters for an inter-frame prediction mode, and optionally further parameters controlling the composition of the final prediction signal for the block using the assigned prediction mode and prediction parameters, as will be outlined in more detail below. Additionally, the data stream may also include parameters for controlling and signaling the subdivision of the images 12 and 12', respectively, into blocks. The decoder 20 uses these parameters to subdivide the image in the same way as the encoder did, to assign the same prediction modes and parameters to the blocks, and to perform the same prediction to produce the same prediction signal.

[0038] Figure 3 The diagram shows the relationship between the reconstructed signal (ie the reconstructed image 12') on the one hand and the combination of the prediction residual signal 24"" and the prediction signal 26 as signaled in the data stream on the other hand. As already indicated above, the combination may be an addition. The prediction signal 26 is Figure 3 1 is illustrated as a subdivision of the image region into blocks 80 of varying sizes, but this is only an example. The subdivision may be any subdivision, such as a conventional subdivision of the image region into columns and rows of blocks, or a multi-tree subdivision of the image 12 into leaf blocks of varying sizes, such as a quadtree subdivision or the like, wherein Figure 3 A mixture of these subdivisions is illustrated in , where the image region is first subdivided into columns and rows of a root block, which is then further subdivided according to a recursive multi-tree subdivision to produce block 80.

[0039] Figure 3 The prediction residual signal 24″″ in is also illustrated as a subdivision of the image area into blocks 84. These blocks may be referred to as transform blocks in order to distinguish them from the coding blocks 80. In practice, Figure 3 It is shown that the encoder 10 and the decoder 20 can use two different subdivisions of the image 12 and the image 12', respectively, into blocks, namely one subdivision into coding blocks 80 and another subdivision into blocks 84. Both subdivisions may be identical, i.e. each block 80 may simultaneously form a transform block 84 and vice versa, but Figure 3The following situation is shown, in which, for example, the subdivision into transform blocks 84 forms an extension of the subdivision into blocks 80, so that any boundary between two blocks 80 overlaps with a boundary between two blocks 84, or alternatively, each block 80 coincides with one of the transform blocks 84, or with a cluster of transform blocks 84. However, the subdivision can also be determined or selected independently of each other, so that the transform blocks 84 can alternatively cross the block boundaries between blocks 80. As far as the subdivision into transform blocks 84 is concerned, similar statements as those made about the subdivision into blocks 80 also apply here, that is, the blocks 84 can be the result of a conventional subdivision of the image area into blocks arranged in columns and rows, the result of a recursive multi-tree subdivision of the image area, or a combination thereof, or any other kind of partitioning. By the way, it should be noted that the blocks 80 and 84 are not limited to square, rectangular or any other shape. In addition, the subdivision of the current image 12 into blocks 80 forming the prediction signal and the subdivision of the current image 12 into blocks 84 encoding the prediction residual may not be the only subdivision used for encoding / decoding. These subdivisions form the granularity at which prediction signal determination and residual encoding are performed, but firstly, residual encoding may alternatively be performed without subdivisions, and secondly, at a granularity different from these subdivisions, the encoder and decoder may set certain encoding parameters, which may include some of the aforementioned parameters, such as prediction parameters, prediction signal composition control signals and the like.

[0040] Figure 3 The illustrated combination of the prediction signal 26 and the prediction residual signal 24"" directly produces the reconstructed signal 12'. However, it should be noted that according to alternative embodiments, more than one prediction signal 26 may be combined with the prediction residual signal 24"" to produce the image 12', such as prediction signals obtained from other views or from other coding layers, for example encoded / decoded in a separate prediction loop with a separate DPB.

[0041] exist Figure 3 In the embodiment described below, the transform blocks 84 should have the following importance. The transformer 28 and the inverse transformer 54 perform their transforms in units of these transform blocks 84. For example, many codecs use some kind of DST or DCT for all transform blocks 84. Some codecs allow skipping of transforms so that for some of the transform blocks 84, the prediction residual signal is coded directly in the spatial domain. However, according to the embodiments described below, the encoder 10 and the decoder 20 are configured in a way that they support several transforms. For example, the transforms supported by the encoder 10 and the decoder 20 may include:

[0042] oDCT-II (or DCT-III), where DCT stands for Discrete Cosine Transform

[0043] oDST-IV, where DST stands for discrete sine transform

[0044] oDCT-IV

[0045] oDST-VII

[0046] oIdentity Transformation (IT)

[0047] Naturally, while the transformer 28 will support all forward transformed versions of these transforms, the decoder 20 or inverse transformer 54 will support their corresponding inverse or reverse versions:

[0048] oInverse DCT-II (or Inverse DCT-III)

[0049] oReverse DST-IV

[0050] oInverse DCT-IV

[0051] oInverse DST-VII

[0052] oIdentity Transformation (IT)

[0053] In any case, it should be noted that the set of supported transforms may include only one transform, such as a spectral to spatial or spatial to spectral transform.

[0054] As outlined above, it has been proposed Figures 1 to 3 As an example, the combined prediction concept described further below may be implemented in the example to form a specific example of a video encoder and decoder according to the present application. In this regard, Figure 1 and Figure 2 The video encoder and decoder described represent possible implementations of the video encoder and decoder described below, respectively. As will be outlined in more detail below, when having a built-in Figure 1 and Figure 2 The subsequently explained embodiments of the present application for combined prediction within a video encoder and decoder are, at least as an option, Figure 1 Video encoder and Figure 2 The video decoder supports processing of block 80 in a manner outlined in more detail below, or even all blocks of the current image 12. Therefore, the embodiments described below particularly relate to Figure 1 10 equivalent video encoder, which processes block 80 in a manner outlined in more detail below, and this also applies to Figure 2 The decoder, the Figure 2 The decoder of thus represents an example of a video decoder according to an embodiment in which block 80 is processed in the manner outlined in more detail below. However, Figure 1 and Figure 2 However, the video encoder according to the embodiment of the present application may use the video encoder described in more detail below and in conjunction with Figure 1The encoder performs block-based encoding differently than Figure 1 The encoders differ in that the subdivision into blocks 80 is different from that in Figure 3 , or in that the encoder does not use transform prediction residual coding, but instead encodes the prediction residual directly in the spatial domain, for example. Similarly, a video decoder according to an embodiment of the present application may use the combined predictive coding concept further outlined below to perform decoding from the data stream 14, but may be different from, for example, Figure 2 The decoder 20 is different in that Figure 3 The described different ways of dividing the image 12 ′ into blocks or in that the decoder does not derive the prediction residuals from the data stream 14 in the transform domain, but for example in the spatial domain.

[0055] In particular, with respect to the block subdivision into blocks 80, it should be noted that the block subdivision may be in accordance with Figure 3 The subdivision into transform blocks, if any, may also be as described for Figure 3 In particular, the subdivision into blocks on the one hand and the subdivision into other blocks such as transform blocks on the other hand may be performed independently of each other by subdividing the image 12 into these blocks separately, or in a dependent manner. For example, one subdivision such as the subdivision into transform blocks may form an extension of the other subdivision as described above, or both subdivisions may form separate extensions of a common original subdivision, such as the subdivision described with respect to FIG. Figure 3 The described subdivision of the image into an array of tree root blocks. And such possibilities also apply to other sub-image granularities to be mentioned below, such as mentioned with respect to the definition of certain prediction parameters, prediction modes, contribution weights or the like. Different subdivisions may be used for different of these entities and may be defined independently of each other, partially independently or as extensions of each other.

[0056] Although this has been described, the following description focuses on the case where the motion vector points to portions in the plane of the reference images, which cross the boundaries of these reference images, outside which no image content is available. In order to fill in sub-portions that extend beyond the boundaries or are completely outside the reference images, one of a plurality of intra-frame prediction modes is selected according to an embodiment of the invention. The purpose is to improve temporal inter-frame prediction by selecting the most appropriate angular intra-frame prediction mode for filling based on the characteristics of the prediction block.

[0057] In the following description, the term "referenced pixel area" corresponds to "referenced part", "reference frame" corresponds to "reference image", "frame boundary" corresponds to "boundary of reference image", "boundary pixel" corresponds to "boundary sample", and "template area" corresponds to "detection area". Each corresponding term may be used in the description. Additionally, in the present application document, "filling" has the same meaning as "boundary filling". In addition, in the following embodiments, one of multiple angular intra-frame prediction modes is selected as an example of an intra-frame prediction mode for filling. However, the intra-frame prediction mode of the present invention is not limited to the angular intra-frame prediction mode, and the multiple intra-frame prediction modes may include, for example, DC and / or planar modes, and / or modes for filling other intra-frame prediction modes. In the case of DC and planar modes, for boundary samples, a vertical direction (perpendicular to the frame boundary) can be used.

[0058] Figure 4 33 different angle intra prediction modes for the embodiment of the present application are shown. Here, the prediction angle in the referenced portion (referenced pixel region) 100 of the reference image (reference frame) 101 is located about Figure 4 The frame boundary (boundary of the reference image) 106 shown in FIG. 1 is within the range of 45° and 135°.

[0059] The implemented angular boundary filling follows the angular intra prediction of the Joint Exploration Test Model (JEM) as described in Reference 2. Please note that the angular intra prediction algorithm in JEM assumes that the upper left boundary pixel of the region is always available for prediction. However, the underlying prediction algorithm assumes that all boundary pixels are in a straight line, and therefore a translation of the left or top boundary pixels is performed in JEM. This translation step is omitted in the preferred embodiment of the present application. In addition, JEM uses 65 angular intra prediction modes consisting of 33 vertical modes and 33 horizontal modes. In the preferred embodiment of the present application, only 33 vertical modes are used, as in Figure 4 Note further that the method used for vertical angle prediction in JEM must be rotated clockwise by 0°, 90°, 180°, or 270° to perform line filling of the bottom, left, top, or right reference frame boundary 106, respectively.

[0060] In the case where the reference pixel area 100 is partially or completely outside the reference frame 101 in the horizontal direction and in the vertical direction, for example, at the reference frame edge (ie, frame corner), the preferred border filling side is calculated. Hor and N Ver Indicates the number of horizontal and vertical lines of the referenced pixel region 101 outside the reference frame region. Depends on the frame edge and N Hor and N Ver , the preferred boundary filling side can be calculated as follows:

[0061] If Margin == Top-Left-Margin Then

[0062] If N Ver >N Hor Then top-side else left-side

[0063] else if margin == top-right-margin then

[0064] If N Ver >N Hor Then top-side else right-side

[0065] Else if margin == bottom-right-margin then

[0066] If N Ver >N Hor Then bottom-side else right-side

[0067] else if edge == bottom-left-edge then

[0068] If N Ver >N Hor Then bottom-side else left-side

[0069] For the calculated preferred line filling side, the selected angle intra prediction mode according to the claim is applied. For the remaining sides, the original vertical line filling process is applied, as in Figure 5 as shown in .

[0070] exist Figure 5 In the embodiment of the present invention, at the frame boundary 106, the overlapped reference pixel area 100 is divided into an area where multi-mode boundary padding (MMBP) is used, and the remaining area where default padding is used. In this case, vertical padding is used as the default padding.

[0071] Alternatively, the reference pixel region 100 outside the reference frame 101 is filled entirely by MMBP, such as Figure 6 In this case, not all angular intra prediction modes can be used, because depending on the prediction angle, MMBP filling may require unavailable boundary pixels 102, for example, as in Figure 6 denoted by dashed arrows in FIG. 1 . Note further that the restriction of the angular intra prediction mode can be determined implicitly by the encoder (10) and decoder (20) without the need to signal the restricted subset. For this purpose, the prediction direction of the corner pixels of the predicted part of the reference pixel region is reversed by 180° to test whether all required boundary pixels are available.

[0072] In this embodiment, whenever the reference pixel region is partially or completely outside the reference frame region, the most appropriate angle inter-frame prediction mode for boundary filling is derived in the video encoder 10 and the video decoder 20. As a premise, the reference pixel region 100 is divided into a region 100' outside the reference frame 101 and a region 100" inside the reference frame 101, as shown in FIG. Figure 7 . All pixels inside the reference frame 101 are directly used as input pixels for motion compensated prediction of the current prediction block. All pixels outside the reference frame 101 in the region 100' are predicted with the help of intra prediction of neighboring boundary pixels 102 from the reference frame 101, as described below. Please note that the angular intra prediction with a vertical angle of 90° corresponds to the state of the art reference frame boundary padding used in H.265 and JEM.

[0073] In order to select one of the multiple angular intra prediction modes in the encoder 10 and the decoder 20, the direction of each intra prediction may be rotated 180° to point to the inside of the reference frame 101. Then a template region, i.e., a detection region 104, is defined based on the rotated intra prediction, which is located inside the reference frame 101, as shown in FIG. Figure 7 As shown in . The size and shape of the template area 104 are not limited. For example, one possible shape may be a rectangular block. For each mode and associated template area 104, a rotated intra prediction from the neighboring boundary pixels 102 is performed to predict all pixel values ​​within the template area 104. By evaluating the results of the above-mentioned predictions, the most appropriate angle intra prediction mode is selected.

[0074] For evaluation, the predicted pixels are compared with the reconstructed pixels of the reference frame 101 within the template area 104 to derive the final angular intra prediction mode. In the language of the claims, the probe area (template area) 104 is filled and the filled probe is compared with the reconstructed fill of the probe area 104 to measure the similarity between them. For this comparison, any measure of similarity between blocks can be used, such as the sum of absolute differences (SAD), the sum of absolute transform differences (SATD), the sum of squared errors (SSE), the Hadamard transform and others. In order to improve the prediction, a content-dependent transform can be used to transform the information into another domain that is, for example, less sensitive to position. Such content-dependent transforms can also be trained. The angular intra prediction mode with the highest similarity measure between the predicted pixels and the reconstructed pixels within the template area 104 can be finally selected for filling the area 100' of the referenced portion 100 of the reference image 101 that extends beyond the boundary 106 of the reference image 101.

[0075] As a modification of the above embodiment, multiple (for example, N number) different template regions (detection regions) may be used for each angular intra prediction mode, where N=2 is assumed in the following to further improve the selection accuracy. Figure 8 As shown in , a first template region is defined depending on the outermost pixel line 112 of the reference pixel region 100 relative to the frame boundary 106. Figure 8 As shown in , depending on the first pixel line 110 of the referenced pixel area 100 located in the area 100 ′ of the reference frame 101 , a second template area is defined.

[0076] For both of the two pixel lines 110, 112 in the referenced pixel region 100, the angular intra prediction mode is inverted to determine the leftmost and rightmost frame boundary pixels that must be used for angular intra prediction. Without being limited to any one kind, it is assumed below that the referenced pixel region 100 overlaps with the reference frame 101 at the top of the frame boundary 106 (or, preferably, the padded side is the top side). In this case, the leftmost and rightmost frame boundary pixels have coordinates {x left-most ;0} and {x right-most ; 0}, as in Fig. 9 The associated template region is then defined by a rectangular shape having {x left-most -1; 1} and {x right-most +1; 3} upper left and lower right coordinates. This means that the template area includes the next two pixel lines below the used frame boundary pixel 102 inside the reference frame, which extends one additional pixel to the left and right directions. In the case where the referenced pixel area 100 overlaps with the reference frame 101 at the left, right or bottom frame boundary (or the preferred padded side is the left side, right side or bottom side), the template area must be rotated accordingly and replaced by Fig. 9 That is, the region 100' extends to the top side of the reference image 101, i.e. above the border 106, the template region is defined as explained above. In the case where the region 100' extends to the left part of the reference image 101, the template region is rotated relative to the left border of the reference image 101. Based on Fig. 9 The table in gives the coordinates of the defined template area, where Fig. 9 The height and width in the table are the sizes of the reference image 101.

[0077] As mentioned above, consider Fig. 9 The first template region based on the first pixel line 110 and the second template region based on the outermost pixel line 120 are defined by using a table. Next, the pixel values ​​inside both template regions are predicted by rotating the original angle intra prediction by 180°, i.e., along the same direction as Figure 7The two template regions (detection regions) are filled based on the boundary samples (boundary pixels) 102 of the reference image 101 at the portion of the boundary 106 in the opposite direction of the filling direction associated with the corresponding angular intra prediction mode shown in . Therefore, the angular prediction scheme follows the angular intra prediction of the joint exploration test model (JEM) as described in reference 2. Next, the predicted pixel values ​​inside the two template regions are compared with the reconstructed reference pixels by using the sum of absolute difference (SAD) similarity measurement. The final angular intra prediction mode can be selected as the mode with the minimum sum of the SAD values ​​of the two template regions (in the case of N=2), that is, the angular intra prediction mode with the highest similarity between the fill detection and the reconstructed fill can be selected. Please note that other pixel lines of the referenced pixel area can also be used to define the template area. In addition, for example, the template area can also be defined using the intermediate pixel line between the outermost pixel line and the first contour line. Additionally, the similarity measure may also include weighting the samples within each detection region for characteristics such as brightness intensity or edge detection to differentiate the importance of the samples, whereby unequal weights are assigned to the similarity contribution to the similarity measure for each sample in the template region based on the psycho-visual weighting performed.

[0078] Further, the high-level parameters for controlling the behavior of boundary filling by deriving the angular intra-frame prediction mode for filling may include the size and shape of the template area. This means that multiple additional pixel lines perpendicular to the frame boundary and the optional extension of the template area may be parallel to the frame boundary to the left and right. Additionally, high-level parameters may be introduced to specify the lines in the referenced pixel area used to define the template area, for example, the outermost line, the first line in the reference frame, or the middle line between the outermost line and the first line. In addition, the reduction in the number of angular intra-frame prediction modes used may be signaled. Additional high-level parameters may be introduced to specify minimum and / or maximum thresholds, which define how much of the referenced pixel area must be located outside the reference frame to select the angular intra-frame prediction mode for filling. That is, the threshold may define the amount of the extended area of ​​the referenced portion outside the reference image. High-level parameters may be signaled in a picture parameter set (PPS), a sequence parameter set (SPS), a slice header, or any other high-level syntax structure.

[0079] Furthermore, with respect to the process for selecting a detection region selection, known codecs featuring vertical boundary extensions may be used. When implementing vertical boundary extensions, a technique known as motion vector (MV) clipping is often used to reduce the storage costs associated with storing sample values ​​for boundary extension samples. With respect to the repetitive nature of sample values ​​in vertical boundary extensions, it is desirable to clip the motion vectors that produce a reference block containing only boundary extension samples. The content of the reference block produced from the clipped MVs is similar to the original reference block. However, by using such samples, and by producing a boundary extension of limited size, decoder implementations may save resources. Typically, a fixed size limit, such as the size of the maximum decoding unit plus the size of the interpolation filter kernel, is used to be able to fit each potential reference block size to the limited amount of generated boundary extension samples, such as Fig.10 as shown in .

[0080] like Fig.11 As shown in , in the multi-mode boundary filling (MMBP), the distance between the reference pixel area and the frame boundary affects the selection process of the detection area on the decoder 20 side. Therefore, it is necessary to perform MV clipping on the encoder 10 and the decoder 20 side in a coordinated manner to avoid mismatches caused by using different detection areas. Therefore, one aspect of the present invention is a high-level syntax element in the bitstream structure as mentioned above, such as a parameter set, a slice header or other syntax elements that allow the encoder 10 to indicate the MV clipping process used or allow corresponding bitstream constraints to indicate the MV clipping process used to the decoder 20. Such syntax elements include, for example, the maximum MV length or the reference block to frame boundary distance (the MV is clipped from the maximum MV length or the reference block to frame boundary distance), and the size to which the MV is clipped or the distance between the frame boundary and the reference pixel area generated by the clipped motion vector. The information included is not limited to the information mentioned above, and may also include further information or other information.

[0081] Furthermore, adaptive MV cropping can be used to control the detection region selection in a beneficial manner by adjusting the amount of cropping to the size of the referenced pixel region (i.e., the referenced block size). Fig.12As illustrated in , the same MV is used for two blocks of different sizes B>A, and adaptive MV clipping produces two different MVs, colored in red, for example, which result in the referenced area being as close as possible to the frame boundary, which will also minimize the distance to the selected detection area. That is, the amount of the motion vector for the smaller-sized referenced portion 110', i.e., block size A (smaller block) is greater than the amount of the motion vector for the larger-sized referenced portion 110', i.e., size B (larger block) . Therefore, the coordinates of the motion vector of the inter-frame prediction block of the current image are clipped by an amount that depends on the amount of the size of the referenced portion, such that the amount is larger for smaller sizes. Therefore, another aspect of the present invention is a high-level syntax element in the bitstream structure, such as a parameter set, a slice header, or other syntax elements that allow the encoder 10 to indicate that MV clipping is performed in a manner adapted to the reference block size or that allow corresponding bitstream constraints to indicate to the decoder 20 that MV clipping is performed in a manner adapted to the reference block size.

[0082] The code amount options evaluated at the encoder side are an important factor for the complexity of the codec. Therefore, one aspect of the present invention is a measure to control the complexity impact of the boundary filling described herein. This measure includes performing motion estimation at the encoder, i.e., searching and coding cost evaluation of predictor samples after different motion compensation for inter-frame prediction, by extending known motion estimation with a second stage. The first motion compensation stage covers all steps of known motion compensation (e.g., generating known boundary filling samples, and evaluating the coding cost of residual data for a given predictor), while the second stage is modified to known motion compensation, where the best predictor is further enhanced by a more complex boundary sample filling scheme such as the scheme described herein.

[0083] In the case where the referenced portion is partially or completely outside the reference image, the other embodiments described below can also be applied. As mentioned above, as a premise, the referenced pixel area is divided into an area outside the reference frame area and an area inside the reference frame area. All pixels inside the reference frame area are directly used as input pixels for motion compensated prediction of the current prediction block. All pixels outside the reference frame area are predicted by means of angular intra-frame prediction from neighboring boundary pixels of the reference frame, such as in Fig.13 and Fig.14 . That is, based on the boundary sample 110, the area 100' of the referenced portion 100, that is, the area 100' outside the boundary 106 of the reference image 101 within the referenced portion 100 is filled. The filling direction is selected from 33 different angular intra modes. In this case, it is not required to select from 33 angular intra modes, and it is also possible to select from a limited number of angular intra prediction modes as defined in advance as a parameter. Fig.13An angular intra prediction mode 120 is shown, and Fig.14 Another angle intra prediction mode 122 is shown. Fig.13 and Fig.14 , the used boundary pixels 110' directly depend on the selected angular intra prediction modes 120 and 122, respectively. Note further that the angular intra prediction mode with a vertical angle of 90° corresponds to the state of the art reference frame boundary padding used in H.265 and JEM. The selected (i.e., selected) angular intra prediction mode is signaled to the decoder within the encoded bitstream.

[0084] In addition, the selected angular intra-frame prediction mode can be encoded using context-based adaptive arithmetic coding (CABAC). Additionally, an enable flag is used to indicate whether to apply default vertical boundary filling or boundary filling based on the selected angular intra-frame prediction mode. Due to the fact that the default vertical mode is one of the 33 angular intra-frame prediction modes, the remaining 32 angular intra-frame prediction modes can be encoded by five additional CABAC bins. The enable flag and the remaining angular modes are encoded as a series of binary decisions with an adaptive binary probability model (a subset of these bins, such as arithmetic angular intra-frame prediction modes, can also be encoded by a non-adaptive probability model). In order to model the enable flag, different context models can be used, and the different context models can be selected based on the amount by which the referenced pixel area is located outside the reference frame. For example, different context models can be used when less than 1 / 4 ,less than 1 / 2 ,less than 3 / 4 or more than 3 / 4 The case when the referenced pixel area is outside the reference frame.

[0085] Alternatively, a list of most probable modes (MPMs) is used to signal frequently occurring angular intra prediction modes for boundary filling. In the case where angular intra prediction modes for boundary filling for previously encoded neighboring blocks are available, possible MPM candidates may be angular intra prediction modes for boundary filling for these blocks. Where available, other MPM candidates may be angular intra prediction modes for adjacent previously encoded blocks. In addition, the angular intra prediction mode of a coding block in a reference frame juxtaposed with a referenced image may be used as an MPM candidate. Additionally, the angular intra prediction mode of a coding block in an associated random access point image juxtaposed with a referenced image area may be used as an MPM candidate. In the case of 33 angular intra prediction modes, the MPM candidate list contains one MPM candidate mode. A CABAC binary flag is used to signal whether the MPM candidate should be used. The remaining 32 angular modes may be signaled by 5 CABAC bits. The MPM flag and the remaining angular intra prediction modes are encoded as a series of binary decisions using an adaptive binary probability model as explained above.

[0086] The angular intra prediction mode for the boundary mode is part of the motion information of the coding block. Therefore, in the case where the motion information is used / copied by different coding blocks, for example, in the case of using merge mode or motion vector prediction, the angular intra prediction mode used for boundary filling associated with the copied motion vector can also be copied and applied in the case where the referenced pixel area is outside the reference frame area. In the case of motion vector prediction from a block for which MMBP has been activated, if MMBP is used for the coding block, the angular intra prediction mode selected for filling can become the MPM for encoding the filling direction of the current coding block. Alternatively, the possible motion prediction candidates can provide a list of possible MPMs.

[0087] Advanced parameters for controlling the behavior of partial boundary filling by signaling the selected angular intra-prediction mode for filling may include parameters for defining a subset of angular intra-prediction modes that should be used for boundary filling. In this case, the number of supported angular intra-prediction modes for filling may be limited to a smaller number, and fewer CABAC bits must be used to signal the selected angular intra-prediction mode for boundary filling. For example, by using only a subset of 18 angular modes in combination with an MPM candidate list of 2 modes, only 16 angular modes remain, which can be encoded by 4 CABAC bits (thus, resulting in 2 bits for signaling the MPM mode and 5 bits for signaling the remaining non-MPM modes). Additional advanced parameters may be introduced to specify minimum and / or maximum thresholds, which define how much of the referenced pixel area must be located outside the reference frame area to select the most appropriate angular intra-prediction mode for filling as described above. The high-level parameters may be signaled in a picture parameter set (PPS), a sequence parameter set (SPS), a slice header, or any other high-level syntax structure.

[0088] The above mentioned embodiments may be combined, for example, the embodiment using a template as described above may be combined with other embodiments signaling a selected angular intra prediction mode as further embodiments explained below.

[0089] The selection of the angular intra prediction mode follows as follows Figure 7 An embodiment of the invention is explained. In this embodiment, an ordered list of angular intra prediction modes for padding is generated based on the similarity measurement as described above. This generated ordered list of angular intra prediction modes for padding is truncated by the largest predefined value of the angular intra prediction mode for padding that is signaled. Finally, the selected angular intra prediction mode for padding within this truncated ordered list is signaled by using context-based adaptive arithmetic coding (CABAC) as described above.

[0090] As already mentioned above, an enable flag may be used to indicate whether the default vertical line padding or a truncated ordered list of selected padding patterns should be used. Additionally, the number of signaled padding patterns that can be signaled using N CABAC bits is set to 2. N. The selected angular intra prediction mode is encoded as a series of binary decisions using an adaptive binary probability model (a subset of these bits may also be encoded by a non-adaptive probability model). In the case where the selected angular intra prediction mode is encoded using a binary probability model, different context models may be used, which, as described above, can be selected based on the amount of area of ​​the referenced portion that is located outside the reference frame area. In addition, a truncated ordered list of the selected angular intra prediction modes for padding is used as an MPM candidate list. Here, the MPM candidate list for padding and the remaining angular intra prediction modes may be encoded as described above. In addition, the same advanced parameters as described above may also be used, such as parameters for defining a subset of angular intra prediction modes, thresholds for application of the selected angular intra prediction mode, and parameters for defining the size and shape of the template area. Similarly, as described above, the advanced parameters may be signaled in a picture parameter set (PPS), a sequence parameter set (SPS), a slice header, or any other high-level syntax structure.

[0091] In a preferred embodiment of the present application, the high-level syntax includes syntax elements for controlling multi-mode boundary filling. The high-level syntax elements are transmitted in one or more high-level syntax structures, such as sequence parameter sets (SPS: valid for all pictures within the coded video sequence), picture parameter sets (PPS: valid for one or more pictures), slice headers (valid for slices), etc.

[0092] The high-level syntax elements that control the application's selection of angular intra prediction mode may include, but are not limited to, one or more of the following parameters:

[0093] A flag or IDC that signals which angular inter prediction mode for padding should be applied (MMBP, vertical or omni padding)

[0094] Parameters for signaling a combination of padding modes on a per-frame-boundary basis or constraints indicating a combination of padding modes on a per-frame-boundary basis; for example, whether to use MMBP or to use omni-directional padding may be signaled in the data stream for a portion along a frame boundary or an outer picture border of a reference picture; as an option, both options may be signaled to be available; different available modes may be signaled for different portions, so that the encoder and decoder will check which portion of the frame boundary of the reference picture the referenced area overlaps with, or which portion the referenced area intersects with, in order to decide on which intra-prediction mode to use;

[0095] · Signaling parameters of which selection process to use in the above described embodiments

[0096] Parameters to signal which angular intra prediction modes can be used for MMBP

[0097] Parameters that specify which exact angular intra prediction modes to use for a particular reference frame bounding area

[0098] Minimum and / or maximum thresholds that define how much of the referenced pixel area must be outside the reference frame area to be used for selecting angular intra prediction mode

[0099] Signal the size and shape of the template area

[0100] Signaling the parameters of the set of lines in the reference pixel area used to determine the template area

[0101] With respect to the second alternative, additional attention is paid to the following: while the embodiments outlined above use angular intra prediction modes as examples, and ultimately select the intra prediction mode for padding from said angular intra prediction modes, it is clear that the vertical padding mode according to which padding is performed by vertically extrapolating boundary samples into areas beyond frame boundaries is also an intra prediction mode, just as the omni padding mode is also an intra prediction mode according to which areas located outside frame boundaries can be filled / padding with samples from within the same frame (i.e., the reference frame). Thus, the high-level syntax reveals information about which intra prediction mode should be used, or which intra prediction mode(s) can be used for which portion of frame boundaries, wherein different syntax can be signaled for or available for different portions of frame boundaries, such as an omni mode for one portion (such as the right and left boundaries) and a vertical mode for other portions of frame boundaries (such as the top and bottom boundaries). The encoder and decoder will only have to recognize the portion of the frame boundary that is exceeded by a certain referenced area of ​​the inter-prediction block 80 and see which intra-prediction mode it is associated with for padding purposes, according to the high-level syntax in the data stream. The parameters listed above may be included in more than one high-level syntax structure. For example, global parameters specified in the SPS may be overridden by parameters specified in the PPS.

[0102] For the embodiments of the present application, it is necessary as long as the referenced pixel area is partially or completely located outside the reference frame area. As a prerequisite, the referenced pixel area is divided into an area outside the reference frame area and an area inside the reference frame area as explained above. All pixels inside the reference frame area are directly used as input pixels for motion compensated prediction of the current prediction block. All pixels outside the reference frame area are predicted by means of intra-frame prediction from neighboring boundary pixels of the reference frame as explained above.

[0103] By using other words, the following steps again exemplarily describe how a preferred intra prediction angle or preferred intra prediction mode can be selected for passing among a plurality of available intra prediction angles in an encoder and a decoder. For the luma (Y) color plane, this selection process can be performed only once for each block to be predicted and each associated motion vector. For the chroma color planes (Cb and Cr), the same intra prediction angles as for the luma color plane can then be selected:

[0104] Determine the position of the referenced region 100: The referenced region 100 is calculated by adding the motion vector associated with the current block to the region of the current block (e.g., moving the current block position by the associated motion vector). Note that the motion vector may have a fractional portion. Therefore, the motion vector is rounded down to the nearest integer portion. The motion vector is added to the current block position. The referenced region 100 is increased by one sample in each direction. That is, the region 100 may be one sample larger in all directions (left, right, top, bottom) than the inter-predicted region for which the MMBP is performed.

[0105] Determine the preferred IntraBP side: If the referenced region 100 overlaps with the reference frame 101 on only one side, this side is used, otherwise, the referenced region 100 overlaps with the reference frame 101 on both sides (see Figure 5 ). Next, the default vertical padding is applied to the sides that overlap less sample lines with the reference frame. IntraBP padding is applied to the remaining sides.

[0106] Determine the bottom and top lines: The bottom line is the first sample line of the referenced region 100 outside the reference frame 101 parallel to the preferred IntraBP side ( Figure 8 The top line is the last sample line of the referenced region outside the reference frame parallel to the preferred IntraBP side ( Figure 8 That is, the sample line farthest from the boundary 106 of the reference frame is selected.

[0107] Constrain multiple available intra prediction modes: For the top line and for each intra prediction mode, calculate the relevant / affected boundary pixels that must be used for intra prediction by inverting the prediction direction. If the maximum displacement of boundary pixels from the vertical intra prediction mode is higher than 1.25*the width of the prediction block, skip / do not use this intra prediction angle. This step is optional. That is, optionally, exclude from the list of selectable modes those angle modes that, when back-projecting the outermost sample line 112 back onto the boundary sample 110, result in an area displacement of the boundary sample 110 relative to the vertical back-projection of the outermost sample line that is greater than a predetermined threshold.

[0108] Calculate the detection area for bottom and top lines: For the bottom (top) line and each intra prediction mode from the constrained set of intra prediction modes, invert the prediction direction to determine the relevant / affected boundary pixels 110' that must be used for intra prediction. Set the detection area 104 below the affected boundary pixels inside the reference frame ( Figure 7 104). The detection area height is set to two sample lines perpendicular to the preferred IntraBP side.

[0109] Calculate similarity measure: Predict the sample values ​​of the detection region by inverted intra prediction from the relevant / affected boundary pixels. Calculate the SAD measure between the predicted sample values ​​of the detection region and the reconstructed sample values ​​of the reference frame at that position of the detection region. Combine / add the SAD measures from the bottom and top detection regions.

[0110] Selecting the preferred intra prediction angle: Among the set of constrained intra prediction angles, the angle with the smallest similarity measure is selected as the preferred intra prediction angle. If the intra prediction angles have the same similarity measure, the angle closer to the vertical prediction angle is preferred.

[0111] If the selected preferred intra prediction angle is not equal to the vertical intra prediction angle, then IntraBP is performed for each color plane in the encoder and decoder using the selected intra prediction angle and the preferred IntraBP side:

[0112] • Determine the position of the referenced area: The referenced area is calculated similarly to the above description, but not only is the referenced area increased by one sample in each direction, but the referenced area is increased by half the size of the interpolation filter tap in each direction.

[0113] Filling the portion of the referenced area located inside the reference frame: Filling the portion of the referenced area located inside the reference frame with sample values ​​of the reference frame.

[0114] Optionally, fill non-preferred IntraBP sides: If the referenced region overlaps with the reference frame at both sides, a default vertical padding is performed for the portion of the referenced region that overlaps with the reference frame for fewer sample lines.

[0115] Performing IntraBP for the preferred IntraBP side: performing intra prediction on the boundary samples of the preferred IntraBP side using the preferred intra prediction angle to fill the referenced area 100 ′ located outside the reference frame.

[0116] The referenced region sample values ​​calculated for each color plane as described above are used as input values ​​for motion compensated prediction of the current block.

[0117] Additionally, existing applications based on the MPEG OMAF standard, such as 360° video streaming services, rely heavily on spatial video partitioning or segmentation techniques. In this application, spatial video segments are transmitted to the client and decoded jointly in a manner adapted to the current client viewing direction. Another related application that relies on spatial segmentation of video planes is the parallelization of encoding and decoding operations, for example, to exploit the multi-core and parallelization capabilities of modern computing platforms.

[0118] One such spatial segmentation technique is implemented in HEVC and is called tiles, which divide the image plane into segments to form a rectangular grid. The resulting spatial segments are encoded independently with respect to entropy coding and intra prediction. In addition, there are means for indicating that the spatial segments are also encoded independently with respect to inter prediction, which can also be combined with spatial segmentation being static across the image. These constraints taken together allow truly independent encoding of the spatial segmentation for an exemplary image plane with two such adjacent and independently coded spatial segments.

[0119] In more detail, when using Fig.15 When encoding the current image with the spatial partitioning shown in to facilitate inter-frame prediction by using a reference image that also uses the spatial partitioning described above, the motion vectors are constrained in the following way: avoiding prediction of crossing spatial segment A by referring to samples of the reference image located in spatial segment B, and vice versa.

[0120] While independent coding of spatial segments is an enabling approach for existing applications as described above, the associated coding constraints may be accompanied by significant coding costs. This bitrate overhead arises, for example, from signaling relatively large motion vector differences (MVDs) that correct available motion vector candidates that would otherwise result in the use of sample positions outside a given spatial segment in the absence of the MVD. In addition, an increased amount of residual data needs to be signaled to compensate for the degraded inter-prediction predictor. It is therefore highly desirable to enhance the codec operation at the boundaries of such spatial segments.

[0121] One way to mitigate the coding penalty of prior art solutions for such independent spatial segmentation is to employ a scheme for padding inner picture boundaries (i.e., segment boundaries located in the interior of the picture), such as in addition to outer frame boundaries, or in a manner similar to or in accordance with the MMBP concept described above based on picture boundaries with slight modifications and extensions described herein, or using a high-level syntax-based padding mode selection that signals which of two or more intra-frame prediction modes is to be used for a spatial segmentation boundary or a portion of a spatial segmentation boundary (such as a vertical boundary). That is, rather than the encoder restricting the signaled motion vectors to not reference referenced portions that extend beyond or are located outside of a spatial segmentation boundary, such portions are instead padded at the encoder and decoder, and if such referenced portions intersect or are located outside of a predetermined portion of a spatial segmentation boundary, the signaled intra-frame prediction mode is used for padding. In general, the reduction of coding penalties can also be achieved by using any intra-frame coding mode, i.e. without using one of the multiple mode selections, such as vertical, omni or angular intra-frame prediction modes, resulting in an encoder and decoder for block-based video coding, which are configured to encode into a data stream and decode from the data stream a motion vector referencing a referenced portion 100 of a reference image 101 for an inter-frame prediction block, the referenced portion extending beyond or outside an intra-image segmentation boundary of a predetermined spatial segmentation within which the inter-frame prediction block is located in the reference image, and to fill in an area 100' of the referenced portion 100 extending beyond or outside the intra-image segmentation boundary (i.e. outside the predetermined spatial segmentation). The encoder does not need to redirect the motion vector predictor of the motion vector to cause the signaled motion vector to remain within a predetermined spatial segmentation of the spatial segmentation into which the reference image (in the way in which the images of the video including the current image are always divided) is divided. The encoder does not need to signal the redirected motion vector difference for this vector. The signaled motion vector difference may result in a motion vector reference portion outside a predetermined spatial segment, such as a motion vector difference of zero. In particular, the encoder may rely on a padding mechanism of a decoder that enforces spatial segment coding independence of spatial segments of spatial offsets in the current picture and the reference picture, and the encoder performs the same, i.e., performs padding to fill a decoded picture buffer for reconstructing an inter-prediction block.

[0122] As a first aspect, a high-level syntax element in a bitstream structure such as a parameter set, slice header, or other structure generally controls the application of multi-mode boundary padding on spatial segment boundaries. A value of this syntax indicates to the decoder:

[0123] The inter prediction of samples of a spatial segment is independent of samples belonging to other spatial segments, and

[0124] Sample positions outside the spatial segment boundaries are used for inter prediction of samples within the spatial segment, and

[0125] - By using the MMBP process on the spatial segmentation boundaries from samples belonging to the spatial segment, the values ​​of samples at sample positions outside the spatial segmentation boundaries are generated.

[0126] Although some aspects have been described in the context of an apparatus, it is clear that these aspects also represent descriptions of corresponding methods, where blocks or devices correspond to method steps or features of method steps. Similarly, aspects described in the context of method steps also represent descriptions of features of corresponding blocks or items or corresponding apparatuses. Some or all of the method steps may be performed by (or using) hardware devices (e.g., microprocessors, programmable computers, or electronic circuits). In some embodiments, one or more of the most important method steps may be performed by such devices.

[0127] The data stream of the present invention may be stored on a digital storage medium or may be transmitted on a transmission medium such as a wireless transmission medium or a wired transmission medium such as the Internet.

[0128] Depending on certain implementation requirements, embodiments of the present invention may be implemented in hardware or in software. The implementation may be performed using a digital storage medium, such as a floppy disk, DVD, Blu-ray, CD, ROM, PROM, EPROM, EEPROM or flash memory, having electronically readable control signals stored thereon, which cooperate (or are capable of cooperating) with a programmable computer system to cause the execution of the corresponding method. Thus, the digital storage medium may be computer readable.

[0129] Some embodiments according to the invention comprise a data carrier having electronically readable control signals, which are capable of cooperating with a programmable computer system, such that one of the methods described herein is performed.

[0130] Generally, embodiments of the present invention can be implemented as a computer program product with a program code, the program code being operative for performing one of the methods when the computer program product runs on a computer.The program code may, for example, be stored on a machine readable carrier.

[0131] Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

[0132] In other words, an embodiment of the inventive method is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

[0133] A further embodiment of the inventive method is, therefore, a data carrier (or a digital storage medium, or a computer-readable medium) comprising, recorded thereon, the computer program for performing one of the methods described herein. The data carrier, the digital storage medium or the recorded medium are typically tangible and / or non-transitory.

[0134] Therefore, another embodiment of the method of the present invention is a data stream or a sequence of signals representing the computer program for performing one of the methods described herein. The data stream or the sequence of signals may, for example, be configured to be transmitted via a data communication connection (for example, via the Internet).

[0135] A further embodiment comprises a processing means, for example a computer or a programmable logic device, configured or adapted to perform one of the methods described herein.

[0136] A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[0137] A further embodiment according to the invention comprises an apparatus or system configured to transmit (e.g. electronically or optically) a computer program for performing one of the methods described herein to a receiver. For example, the receiver may be a computer, a mobile device, a storage device or the like. The apparatus or system may, for example, comprise a file server for transmitting the computer program to the receiver.

[0138] In some embodiments, a programmable logic device (e.g., a field programmable gate array) may be used to perform some or all of the functionality of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor to perform one of the methods described herein. Typically, the methods are preferably performed by any hardware device.

[0139] The devices described herein may be implemented using hardware devices or using computers or using a combination of hardware devices and computers.

[0140] The devices described herein or any component of a device described herein may be implemented at least partially in hardware and / or in software.

[0141] The methods described herein may be performed using a hardware device or using a computer or using a combination of a hardware device and a computer.

[0142] Any component of a method described herein or an apparatus described herein may be performed at least in part by hardware and / or by software.

[0143] The above embodiments are merely illustrative of the principles of the present invention. It should be understood that modifications and variations to the configurations and details described herein will be apparent to those skilled in the art. Therefore, it is intended that the present invention be limited only by the scope of the claims that follow, and not by the specific details presented by the description and explanation of the embodiments herein.

Claims

1. A video encoder (10) for supporting temporal inter-frame prediction, configured to Filling of an area (100') of a referenced portion (100) of a reference image (101) extending beyond a boundary (106) of the reference image (101) or located outside the boundary (106) of the reference image (101) is performed, wherein the referenced portion (100) is referenced by an inter-frame prediction block of a current image (12) by selecting one of a plurality of intra-frame prediction modes and filling the area (100') with the selected intra-frame prediction mode.

2. The video encoder (10) according to claim 1 is configured to perform the selection based on an evaluation of an area of ​​the reference image (101) at a portion where the referenced portion (100) in the boundary (106) of the reference image intersects the boundary (106).

3. The video encoder (10) of claim 2, configured to perform the evaluation by, for each of at least a subset of the plurality of intra prediction modes, Based on a boundary sample (102) of the reference image (101) at the portion of the boundary (106), filling a detection region (104) located farther from the boundary (106) than the boundary sample (102) in a direction opposite to a filling direction associated with a corresponding intra-frame prediction mode to obtain a detection filling; and The probe fill-in is compared to a reconstructed fill-in of the probe region (104) to measure similarity therebetween.

4. The video encoder (10) according to any one of claims 1 to 3, configured to signal the multiple intra-frame prediction modes as a subset of supported intra-frame prediction modes supported by the video encoder (10) for intra-frame prediction blocks.

5. A video encoder (10) according to any one of claims 1 to 4, configured to transmit in a signal a parameter defining the extent to which the referenced portion (100) needs to extend beyond the boundary (106) so as to initiate the execution of the padding by selecting one of the multiple intra-frame prediction modes, wherein the video encoder (10) is configured to pad the referenced portion (100) using a default padding mode when the referenced portion (100) extends beyond the boundary by less than the extent.

6. A video decoder (20) for supporting temporal inter-frame prediction, configured to Filling of an area (100') of a referenced portion (100) of a reference image (101) extending beyond a boundary (106) of the reference image (101) or located outside the boundary (106) of the reference image (101) is performed, wherein the referenced portion (100) is referenced by an inter-frame prediction block of a current image (12) by selecting one of a plurality of intra-frame prediction modes and filling the area (100') with the selected intra-frame prediction mode.

7. The video decoder (20) according to claim 8 is configured to perform the selection based on an evaluation of an area of ​​the reference image (101) at a portion of the referenced portion (100) in the boundary (106) of the reference image (100) where the boundary (106) intersects the referenced portion (100).

8. A coding method for supporting temporal inter-frame prediction, comprising: Filling of an area of ​​a referenced portion (100) of a reference image (101) extending beyond a boundary (106) of the reference image (101) is performed, wherein the referenced portion (100) is referenced by an inter-frame prediction block of a current image (12) by selecting one of a plurality of intra-frame prediction modes and filling the area with the selected intra-frame prediction mode.

9. A decoding method for supporting temporal inter-frame prediction, comprising: Filling of an area of ​​a referenced portion (100) of a reference image (101) extending beyond a boundary (106) of the reference image (101) is performed, wherein the referenced portion (100) is referenced by an inter-frame prediction block of a current image (12) by selecting one of a plurality of angular intra-frame prediction modes and filling the area with the selected angular intra-frame prediction mode. 10 . A computer program product having a program code for executing the method according to claim 8 when running on a computer.