A method, an apparatus and a computer program product for video coding and decoding

Abstract

The embodiments relate to method and technical equipment for decoding encoded samples of blocks of video sample data is provided, wherein for a block of video sample data the method comprises forming an intra prediction using an intra-prediction process, where the intra-prediction process comprises: extracting prediction information from at least one other block, where the at least one other block has been predicted using an inter-prediction process; deriving a histogram based on the extracted prediction information; determining one or more intra-prediction modes based on the derived histogram; forming one or more predictors based on the one or more determined intra-prediction modes; and forming a prediction for the block of video samples by means of blending the one or more predictors.

Description

[0001] A METHOD, AN APPARATUS AND A COMPUTER PROGRAM PRODUCT FOR VIDEO CODING AND DECODING

[0002] Technical Field

[0003] The present solution generally relates to a method and an apparatus and a computer program product for video coding and decoding.

[0004] Background

[0005] This section is intended to provide a background or context to the invention that is recited in the claims. The description herein may include concepts that could be pursued but are not necessarily ones that have been previously conceived or pursued. Therefore, unless otherwise indicated herein, what is described in this section is not prior art to the description and claims in this application and is not admitted to be prior art by inclusion in this section.

[0006] Decoder side intra mode derivation (DIMD) for video codecs uses a method to determine the Intra mode of the current block using directionality of the texture of the neighboring reconstructed samples located in a template region at top, top-left, and left sides of the current block. In DIMD, one (or more) intra modes and their corresponding weight factors are derived first. Then, predictors are generated for these intra modes, and also for the Planar mode. A final intra prediction for the current block is generated by combining these predictors by means of sample-wise or uniform weighting, using the derived weights.

[0007] Summary

[0008] The aim of the present solution is to provide an improved process for the DIMD to support also inter-predicted blocks.

[0009] The scope of protection sought for various embodiments of the invention is set out by the independent claims. The embodiments and features, if any, described in this specification that do not fall under the scope of the independent claims are to be interpreted as examples useful for understanding various embodiments of the invention.

[0010] Various aspects include a method, an apparatus and a computer readable medium comprising a computer program stored therein, which are characterized by what is stated in the independent claims. Various embodiments are disclosed in the dependent claims.

[0011] According to a first aspect, there is provided an apparatus for decoding encoded samples of blocks of video sample data, wherein for a block of video sample data the apparatus comprises means for forming an intra prediction by means for extracting prediction information from at least one other block, where the at least one other block has been predicted using an inter-prediction process; means for deriving a histogram based on the extracted prediction information; means for determining one or more intra-prediction modes based on the derived histogram; means for forming one or more predictors based on the one or more determined intra-prediction modes; and means for forming a prediction for the block of video samples by means of blending the one or more predictors.

[0012] According to a second aspect, there is provided a method for decoding encoded samples of blocks of video sample data, wherein for a block of video sample data the method comprises forming an intra prediction using an intra- prediction process, where the intra-prediction process comprises: extracting prediction information from at least one other block, where the at least one other block has been predicted using an inter-prediction process; deriving a histogram based on the extracted prediction information; determining one or more intra-prediction modes based on the derived histogram; forming one or more predictors based on the one or more determined intra-prediction modes; and forming a prediction for the block of video samples by means of blending the one or more predictors.

[0013] According to a third aspect, there is provided an apparatus for decoding encoded samples of blocks of video sample data, the apparatus comprising at least one processor, memory including computer program code, the memory and the computer program code configured to, with the at least one processor, cause the apparatus to form an intra prediction for a block of video sample data using an intra-prediction process, wherein for the intra-prediction process the apparatus is caused to extract prediction information from at least one other block, where the at least one other block has been predicted using an inter-prediction process; to derive a histogram based on the extracted prediction information; to determine one or more intra-prediction modes based on the derived histogram; to form one or more predictors based on the one or more determined intra-prediction modes; and to form a prediction for the block of video samples by means of blending the one or more predictors.

[0014] According to a fourth aspect, there is provided computer program product for decoding encoded samples of blocks of video sample data, the computer program product comprising computer program code configured to, when executed on at least one processor, cause an apparatus or a system to form an intra prediction for a block of video sample data using an intra-prediction process, wherein for the intra-prediction process the apparatus is caused to extract prediction information from at least one other block, where the at least one other block has been predicted using an inter-prediction process; to derive a histogram based on the extracted prediction information; to determine one or more intra-prediction modes based on the derived histogram; to form one or more predictors based on the one or more determined intra-prediction modes; and to form a prediction for the block of video samples by means of blending the one or more predictors.

[0015] According to an embodiment, the extracted prediction information comprises information extracted from an intra prediction mode buffer.

[0016] According to an embodiment, the intra prediction mode buffer comprises intra prediction information for a given location within a given frame.

[0017] According to an embodiment, the intra prediction information is determined for a location belonging to an inter-predicted block from the intra prediction mode buffer based on motion information of the inter-predicted block.

[0018] According to an embodiment, the motion information comprises a motion vector. According to an embodiment, the extracted prediction information comprises intra-prediction information extracted at a location that is derived based on characteristics of the current block.

[0019] According to an embodiment, the extracted prediction information comprises intra-prediction information extracted at a location that is derived based on characteristics of the at least one other block.

[0020] According to an embodiment, the histogram is formed by extracting intra prediction information from the intra prediction mode buffer at locations that are contained by said the at least one other block.

[0021] According to an embodiment, amplitudes in the histogram depend on the number of times a given intra prediction mode is extracted from the intra prediction mode buffer at locations that are contained by said at least one other block.

[0022] According to an embodiment, the histogram is derived based on information extracted from at least one other block predicted using intra-prediction.

[0023] According to an embodiment, the at least one other block is predicted using a decoder side intra mode derivation process.

[0024] According to an embodiment, amplitudes in the histogram are scaled based on a scaling factor.

[0025] According to an embodiment, the scaling factor depends on information extracted from the at least one other block predicted using a decoder side intra mode derivation process.

[0026] According to an embodiment, the histogram is formed based on information extracted from the intra prediction mode buffer.

[0027] According to an embodiment, information is extracted from a bitstream, wherein the information indicates usage of the intra-prediction process. According to an embodiment, the computer program product is embodied on a non-transitory computer readable medium.

[0028] Description of the Drawings

[0029] In the following, various embodiments will be described in more detail with reference to the appended drawings, in which

[0030] Fig. 1 shows a simplified example of an encoding process;

[0031] Fig. 2 shows a simplified example of a decoding process;

[0032] Fig. 3 shows a simplified example of forming Histogram of Gradients for a block;

[0033] Fig. 4 shows another simplified example for forming Histogram of Gradients for a block;

[0034] Fig. 5 is a flowchart illustrating a method according to an embodiment ; and

[0035] Fig. 6 shows an apparatus according to an embodiment.

[0036] Description of Example Embodiments

[0037] Certain abbreviations that may be found in the description are herewith defined as follows:

[0038] 2D Two-Dimensional

[0039] 3D Three-Dimensional

[0040] AMVP Advanced Motion Vector Prediction

[0041] AV1 AO Media Video 1

[0042] AVC Advanced Video Coding

[0043] CABAC Context-adaptive binary arithmetic coding

[0044] CDMA Code Division Multiple Access

[0045] CTU Coding Tree unit

[0046] CU Coding Unit

[0047] DC Direct Current DCT Discrete Cosine Transform

[0048] DIMD Decoder-side Intra Mode Derivation

[0049] DPB Decoder Picture Buffer

[0050] DSP Digital Signal Processor

[0051] ECM Enhanced Compression Model

[0052] FDMA Frequency Division Multiple Access

[0053] GSM Global System for Mobile communications

[0054] HEVC High Efficiency Video Coding

[0055] HoG Histogram of Gradients

[0056] IEC International Electrotechnical Commission

[0057] IMS Instant Messaging Service

[0058] IPM Intra Prediction Mode

[0059] ISO Internal Organization for Standardization

[0060] ISOBMFF ISO Base Media File Format

[0061] ITU-T Telecommunications Standardization Sector of International Telecommunication Union

[0062] JCT-VC Joint Collaborative Team - Video Coding

[0063] JVT Joint Video Team

[0064] LCU Largest Coding Unit

[0065] MCP Motion-Compensated Prediction

[0066] MHoG Merged Histogram of Gradients

[0067] MIMD Merged Intra Mode derivation

[0068] MMS Multimedia Messaging Service

[0069] MPEG Moving Picture Experts Group

[0070] MPG Most Probable Mode

[0071] MRL Multiple Reference Line

[0072] MVC Multiview Video Coding

[0073] MV Multiview

[0074] NAL Network Abstraction Layer

[0075] PC Personal Computer

[0076] PU Prediction Unit

[0077] REXT Range Extensions

[0078] SHVC Scalable High efficiency Video Coding

[0079] SMS Short Messaging Service

[0080] SNR Signal-to-Noise Ratio

[0081] SPGM Single-Path Global Modulation

[0082] SVC Scalable Video Coding TDMA Time Divisional Multiple Access

[0083] TCP-IP Transmission Control Protocol - Internet Protocol

[0084] TIMD Template-based Inter Mode Derivation

[0085] Til Transform Unit

[0086] TMRL Template-based Multiple Reference Line

[0087] TV Television

[0088] UMTS Universal Mobile Telecommunications System

[0089] VCEG Video Coding Experts Group

[0090] WC Versatile Video Coding

[0091] The following description and drawings are illustrative to discuss embodiments of the present solution with examples. The specific details are provided for understanding purposes. However, in certain instances, well-known or conventional details are not described in order to avoid obscuring the description. Reference in this specification to “one embodiment” or “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the disclosure.

[0092] The present embodiments relate to an Merged Intra Mode Derivation (MIMD) that is based on the computation of Merged Histogram of Gradients using information from neighboring blocks. MIMD has been developed from an idea originating in DIMD.

[0093] Before describing the embodiments further, a brief reference to evolution of video coding standardization is given. The present embodiments are suited within the context of next generation video coding standardization, e.g., H.267 video coding standard and the ECM (Enhanced Compression Model) exploration.

[0094] A video codec comprises an encoder and a decoder. The encoder transforms the input video into a compressed representation suited for storage / transmission. The decoder can un-compress the compressed video representation back into a viewable form. The encoder may discard some information in the original video sequence in order to represent the video in a more compact form (i.e. , at lower bitrate). Figure 1 shows a simple example of a principle of an encoding process for 2D pictures, and Figure 2 shows a simple example of a principle of a decoding process for 2D pictures. In Figure 1 , the following have been illustrated:

[0095] - an image to be encoded (ln);

[0096] - a predicted representation of an image block (P'n);

[0097] - a prediction error signal (Dn);

[0098] - a reconstructed prediction error signal (D'n);

[0099] - a preliminary reconstructed image (l'n);

[0100] - a final reconstructed image (R'n);

[0101] - a transform (T) and inverse transform (T-1);

[0102] - a quantization (Q) and inverse quantization (Q-1);

[0103] - entropy encoding (E);

[0104] - a reference frame memory (RFM);

[0105] - inter prediction (Pinter);

[0106] - intra prediction (Pintra);

[0107] - mode selection (MS), and

[0108] - filtering (F).

[0109] In Figure 2 the following have been illustrated:

[0110] - a predicted representation of an image block (P'n);

[0111] - a reconstructed prediction error signal (D'n);

[0112] - a preliminary reconstructed image (l'n);

[0113] - a final reconstructed image (R'n); an inverse transform (T1);

[0114] - an inverse quantization (Q-1);

[0115] - an entropy decoding (E-1);

[0116] - a reference frame memory (RFM);

[0117] - a prediction (either inter or intra) (P);

[0118] - and filtering (F).

[0119] An elementary unit for the input to an encoder and the output of a decoder, respectively, in most cases is a picture (also referred to as “an image”). A picture given as an input to an encoder may also be referred to as a source picture, and a picture decoded by a decoded may be referred to as a decoded picture or a reconstructed picture.

[0120] The source and decoded pictures are each comprised of one or more sample arrays, such as one of the following sets of sample arrays: - Luma (Y) only (monochrome).

[0121] - Luma and two chroma (YCbCr or YCgCo).

[0122] - Green, Blue and Red (GBR, also known as RGB).

[0123] - Arrays representing other unspecified monochrome or tristimulus color samplings (for example, YZX, also known as XYZ).

[0124] A picture may be defined to be either a frame or a field. A frame comprises a matrix of luma samples and possibly the corresponding chroma samples. A field is a set of alternate sample rows of a frame and may be used as encoder input, when the source signal is interlaced. Chroma sample arrays may be absent (and hence monochrome sampling may be in use) or chroma sample arrays may be subsampled when compared to luma sample arrays.

[0125] The Advanced Video Coding standard (which may be abbreviated AVC or H.264 / AVC) was developed by the Joint Video Team (JVT) of the Video Coding Experts Group (VCEG) of the Telecommunications Standardization Sector of International Telecommunication Union (ITU-T) and the Moving Picture Experts Group (MPEG) of International Organization for Standardization (ISO) I International Electrotechnical Commission (IEC). There have been multiple versions of the H.264 / AVC standard, each integrating several extensions or features to the specification. These extensions include Scalable Video Coding (SVC) and Multiview Video Coding (MVC).

[0126] The High Efficiency Video Coding standard (which may be abbreviated HEVC or H.265 / HEVC) was developed by the Joint Collaborative Team - Video Coding (JCT-VC) of VCEG and MPEG. Extensions to H.265 / HEVC include scalable, multiview, three-dimensional, and fidelity range extensions, which may be referred to as SHVC, MV-HEVC, 3D-HEVC, and REXT, respectively.

[0127] Versatile Video Coding (which may be abbreviated WC, H.266, or H.266A / VC) is a video compression standard developed as the successor to HEVC. WC is specified in ITU-T Recommendation H.266 and equivalently in ISO / IEC 23090-3, which is also referred to as MPEG-I Part 3. ECM was developed by JVET (Joint Video Experts Team) of ITU-T VCEG and ISO / IEC MPEG, to provide a future video coding technology the compression capability of which would exceed that of the VVC.

[0128] Some key definitions, bitstream and coding structures, and concepts of H.264 / AVC, HEVC, WC, and / or AV1 and some of their extensions are described in this section as an example of a video encoder, decoder, encoding method, decoding method, and a bitstream structure, wherein the embodiments may be implemented. The aspects of various embodiments are not limited to H.264 / AVC, HEVC, WC, and / or AV1 or their extensions, but rather the description is given for one possible basis on top of which the present embodiments may be partly or fully realized.

[0129] Hybrid video codecs, for example ITU-T H.263, H.264 / AVC, HEVC, and WC, may encode the video information in two phases. At first, pixel values in a certain picture area (or “block”) are predicted for example by motion compensation means or by spatial means.

[0130] In motion compensation based prediction (which may be referred to as inter prediction, temporal prediction or motion-compensated temporal prediction or motion-compensated prediction or MCP) an area in one of the previously coded frames that corresponds closely to the block being coded is found and used for prediction. Inter prediction may reduce temporal redundancy.

[0131] In spatial prediction pixel values around the block to be coded are used. In the first phase, predictive coding may be applied, for example, as so-called sample prediction and / or so-called syntax prediction. In the sample prediction, pixel or sample values in a certain picture area or "block" are predicted. These pixel or sample values can be predicted, for example, using one or more of motion compensation or intra prediction mechanisms.

[0132] Intra prediction, where pixel or sample values can be predicted by spatial mechanisms, involve finding and indicating a spatial region relationship. Intra prediction utilizes the fact that adjacent pixels within the same picture are likely to be correlated. Intra prediction can be performed in spatial or transform domain, i.e., either sample values or transform coefficients can be predicted. Intra prediction is typically exploited in intra coding, where no inter prediction is applied.

[0133] In motion vector prediction, motion vectors e.g., for inter and / or inter-view prediction may be coded differentially with respect to a block-specific predicted motion vector. In many video codecs, the predicted motion vectors are created in a predefined way, for example by calculating the median of the encoded or decoded motion vectors of the adjacent blocks. Another way to create motion vector predictions, sometimes referred to as advanced motion vector prediction (AMVP), is to generate a list of candidate predictions from adjacent blocks and / or co-located blocks in temporal reference pictures and signalling the chosen candidate as the motion vector predictor. In addition to predicting the motion vector values, the reference index of previously coded / decoded picture can be predicted. The reference index is typically predicted from adjacent blocks and / or co-located blocks in temporal reference picture. Differential coding of motion vectors is typically disabled across slice boundaries.

[0134] The block partitioning, e.g., from a coding tree unit (CTU) to coding units (CUs) and down to prediction units (PUs), may be predicted.

[0135] In filter parameter prediction, the filtering parameters e.g., for sample adaptive offset may be predicted. Prediction approaches using image information from a previously coded image can also be called as inter prediction methods which may also be referred to as temporal prediction and motion compensation. Prediction approaches using image information within the same image can also be called as intra prediction methods.

[0136] In the second phase of encoding, the prediction error, i.e., the difference between the predicted block of pixels and the original block of pixels, is coded. This may be done by transforming the difference in pixel values using a specified transform (e.g., Discrete Cosine Transform (DCT) or a variant of it), quantizing the coefficients and entropy coding the quantized coefficients. By varying the fidelity of the quantization process, encoder can control the balance between the accuracy of the pixel representation (picture quality) and size of the resulting coded video representation (file size of transmission bitrate). In some video codecs, such as H.265 / HEVC, video pictures are divided into coding units (CU) covering the area of the picture. A CU consists of one or more prediction units (Pll) defining the prediction process for the samples within the CU and one or more transform units (TU) defining the prediction error coding process for the samples in the said CU. A CU may consist of a square block of samples with a size selectable from a predefined set of possible CU sizes. A CU with the maximum allowed size is typically named as LCU (largest coding unit) or CTU (coding tree unit) and the video picture is divided into non-overlapping CTUs. A CTU can be further split into a combination of smaller CUs, e.g., by recursively splitting the CTU and resultant CUs. Each resulting CU typically has at least one PU and at least one TU associated with it. Each PU and TU can be further split into smaller PUs and TUs in order to increase granularity of the prediction and prediction error coding processes, respectively. Each PU has prediction information associated with it defining what kind of a prediction is to be applied for the pixels within that PU (e.g., motion vector information for inter predicted PUs and intra prediction directionality information for intra predicted PUs). Similarly, each TU is associated with information describing the prediction error decoding process for the samples within the said TU (including e.g., DCT coefficient information). It is typically signaled at CU level whether prediction error coding is applied or not for each CU. In the case there is no prediction error residual associated with the CU, it can be considered there are no TUs for the said CU. The division of the image into CUs, and division of CUs into PUs and TUs is typically signaled in the bitstream allowing the decoder to reproduce the intended structure of these units.

[0137] The decoder reconstructs the output video by applying prediction means similar to the encoder to form a predicted representation of the pixel blocks (using the motion or spatial information created by the encoder and stored in the compressed representation) and prediction error decoding (inverse operation of the prediction error coding recovering the quantized prediction error signal in spatial pixel domain). After applying prediction and prediction error decoding means the decoder sums up the prediction and prediction error signals (pixel values) to form the output video frame. The decoder (and encoder) can also apply additional filtering means to improve the quality of the output video before passing it for display and / or storing it as prediction reference for the forthcoming frames in the video sequence.

[0138] In many video codecs, including H.264 / AVC, HEVC, and WC, motion information is indicated by motion vectors associated with each motion compensated image block. Each of these motion vectors represents the displacement of the image block in the picture to be coded (in the encoder) or decoded (at the decoder) and the prediction source block in one of the previously coded or decoded images (or pictures). In order to represent motion vectors efficiently those are typically coded differentially with respect to block specific predicted motion vectors. In typical video codecs the predicted motion vectors are created in a predefined way, for example calculating the median of the encoded or decoded motion vectors of the adjacent blocks. Another way to create motion vector predictions is to generate a list of candidate predictions from adjacent blocks and / or co-located blocks in temporal reference pictures and signaling the chosen candidate as the motion vector predictor. In addition to predicting the motion vector values, the reference index of previously coded / decoded picture can be predicted. The reference index may be predicted from adjacent blocks and / or or co-located blocks in temporal reference picture. Moreover, high efficiency video codecs may employ an additional motion information coding / decoding mechanism, often called merging / merge mode, where all the motion field information, which includes motion vector and corresponding reference picture index for each available reference picture list, is predicted and used without any modification / correction. Similarly, predicting the motion field information is carried out using the motion field information of adjacent blocks and / or colocated blocks in temporal reference pictures and the used motion field information is signaled among a list of motion field candidate list filled with motion field information of available adjacent / co-located blocks.

[0139] Video codecs may support motion compensated prediction from one source image (uni-prediction) and two sources (bi-prediction). In the case of uniprediction a single motion vector is applied whereas in the case of bi-prediction two motion vectors are signaled and the motion compensated predictions from two sources are averaged to create the final sample prediction. In the case of weighted prediction, the relative weights of the two predictions can be adjusted, or a signaled offset can be added to the prediction signal. In addition to applying motion compensation for inter picture prediction, similar approach can be applied to intra picture prediction. In this case the displacement vector indicates where from the same picture a block of samples can be copied to form a prediction of the block to be coded or decoded. This kind of intra block copying methods can improve the coding efficiency substantially in presence of repeating structures within the frame - such as text or other graphics.

[0140] In video codecs the prediction residual after motion compensation or intra prediction may be first transformed with a transform kernel (like DCT) and then coded. The reason for this is that often there still exists some correlation among the residual and transform can in many cases help reduce this correlation and provide more efficient coding.

[0141] Many video encoders utilize Lagrangian cost functions to find optimal coding modes, e.g., the desired Macroblock mode and associated motion vectors. This kind of cost function uses a weighting factor A to tie together the (exact or estimated) image distortion due to lossy coding methods and the (exact or estimated) amount of information that is required to represent the pixel values in an image area:

[0142] C = D + AR (Eq. 1 )

[0143] Where C is the Lagrangian cost to be minimized, D is the image distortion (e.g., Mean Squared Error) with the mode and motion vectors considered, and R the number of bits needed to represent the required data to reconstruct the image block in the decoder (including the amount of data to represent the candidate motion vectors).

[0144] The phrase along the bitstream (e.g., indicating along the bitstream) may be defined to refer to out-of-band transmission, signaling, or storage in a manner that the out-of-band data is associated with the bitstream. The phrase decoding along the bitstream or alike may refer to decoding the referred out- of-band data (which may be obtained from out-of-band transmission, signaling, or storage) that is associated with the bitstream. For example, an indication along the bitstream may refer to metadata in a container file that encapsulates the bitstream.

[0145] There are number of intra-prediction modes available for existing video codecs. These intra-prediction modes comprise different directional intra- prediction modes, as well as prediction modes such as DC or Planar intra- prediction.

[0146] In Planar intra-prediction, interpolation processes in both the vertical and horizontal directions are performed to obtain vertical and horizontal predictors, respectively. Then, sample-by-sample mean of these two predictors can be computed. Each interpolation process may be carried out by computing weighted averages of the reference samples. As an example, when computing the vertical predictor, each predicted sample at a given horizontal coordinate is obtained as the weighted average between the two reference samples at the same horizontal coordinate extracted from the top row and bottom row of the current block, respectively. Due to the fact the samples located on the bottom row of the current block are not reconstructed when predicting the current block, these are padded using the sample on the bottom-left of the current block. Similarly, the horizontal predictor is computed as the weighted average of the reference samples extracted from the columns on the left and right of the current block.

[0147] ECM provides improved coding tools, such as a Decoder Side Intra Mode Derivation (DIMD). DIMD relies on an inference process operating on a region formed of already reconstructed samples in the surrounding of the current block. Gradient estimation techniques can be used to predict the directionality and strength of edges in the region. These are then used to infer intra- prediction directions and finally derive directional intra-prediction modes. These modes are used to predict the current block. The usage of DIMD can be signaled for a given block. When DIMD is signaled to be used, no further information is required to perform the intra prediction for the current block, and the intra prediction mode is instead inferred.

[0148] DIMD uses a method to determine the Intra prediction of the current block by using directionality of the texture of the neighboring reconstructed samples located in a template region at top, top-left, and left sides of the current block. In DIMD, one (or more) intra modes and their corresponding weight factors are derived at first. Then, predictors are generated for these intra modes, and also for the Planar mode. A final intra prediction for the current block is generated by combining these predictors by means of sample-wise or uniform weighting, using derived weights.

[0149] In DIMD, directionality of texture is derived for each neighboring sample in the template region, using 3x3 neighboring samples of that sample. This derivation may be performed in several steps. First, the horizontal and vertical direction strengths (Dx and Dy) may be calculated using 3x3 neighboring or different samples of the training sample. The corresponding region on the Intra prediction mode (angle) is determined using the sign of Dx and Dy. Then the ratio of Dx / Dy may be calculated, and the corresponding angle index may be determined using the ratio value and a table that maps the ratio to proper angle index. Then the final intra mode for that training sample may be determined using the region and derived angle. The amplitude (or importance) of this intra mode is derived as the sum of absolute values of Dx and Dy.

[0150] The intra mode (corresponding to a given directionality) and its corresponding amplitude are derived as above for each template sample. These may be then collected into a Histogram of Gradients, namely a histogram collecting for each intra mode the cumulative amplitude of all neighboring samples with that directionality. Finally, one or more intra modes are derived from the histogram of gradients, by finding the dominant intra modes, namely the modes with the highest amplitudes in the histogram. The weights of each extracted intra mode are determined based on these amplitudes, where higher amplitudes correspond to higher weights. A fixed weight is typically assigned to the planar mode. In case there are more than one intra mode that can be determined from the Histogram of Gradients, the Planar mode is always blended together with the determined intra modes to form the final DIMD prediction.

[0151] When using DIMD, a number of DIMD modes can be derived by analysing the directionality of the content in the surrounding of the current block. Figure 3 illustrates an example for this. Figure 3 shows a current CU, which is the block to be predicted. Figure 3 also shows a L-shaped template 305 formed of already reconstrued samples. A Histogram of Gradients (HoG) is computed on the L-shaped template 305. This can be obtained by using the Sobel filter, accumulating the magnitudes of all gradients at a given direction for all the samples within the gray area 310. The directions with highest cumulative magnitude, i.e. , the highest bars 350 of the DIMD HoG, can be selected as intra modes. These modes are used to compute different predictors. The predictors are blended to form the final DIMD prediction for the current block. The blending may be performed using weighted average. The weights of each DIMD intra mode can be determined based on the histogram amplitudes, where the higher amplitudes may correspond to higher weights. A fixed weight may be assigned to the planar mode, which is blended together with the determined intra modes to form the final DIMD prediction. The derived intra modes are included into a Most Probable Mode (MPM) list, which comprises a plurality of intra prediction modes.

[0152] MIMD refers to a Merged Intra Mode Derivation, where histograms defined for one or more neighboring blocks are merged into a histogram. Figure 4 shows an example of a MIMD, where the DIMD information 410, 420 extracted from neighboring blocks is used to compute the intra-prediction for the current block 400. Neighboring blocks not predicted using DIMD can also be used in MIMD. A synthetic histogram can be defined for such blocks, derived depending on the intra prediction mode or modes used to predict that block. In particular, a new Merged Histogram of Gradients (MHoG I MIMD HoG) 480 is computed for the current block based on the HoGs 420, 430 of neighboring blocks.

[0153] When a single neighboring block is available, its HoG can be used to form the MHoG for the current block. In case the block is MIMD or DIMD, the HoG can be used. In case the block is not predicted using MIMD or DIMD, then a synthetic histogram can be computed for the neighboring block. If more than one neighboring blocks are available, the corresponding histograms are combined by means of amplitude averaging to derive the MHoG.

[0154] The basic operation of the MIMD is discussed here. A number of N neighboring blocks are considered for a current block. A neighboring block is considered if it has been encoded with at least one directional intra-prediction mode. In case where the neighbouring block / has been encoded using DIMD or MIMD, then its HoG or MHoG is directly considered as Ht, where refers to the amplitude of directional mode m in the HoG, where m can take values from 0 to M where M is the maximum number of intra-prediction modes. A normalisation process can be used when considering Hi. In case where the neighbouring block / has been encoded using a non-DIMD intra-prediction directional mode m, then a synthetic is derived for that neighbouring block, the value A depends on the size of the current block. Then, the MHoG H can be computed using all the HoGs extracted from available neighbouring blocks as:

[0155] Finally, the MHoG is used to compute MIMD modes and weights. The directional modes and their weights corresponding to the highest amplitudes in the MHoG are selected as directional modes and weights for MIMD, as in conventional DIMD. Thus, the idea of MIMD is to merge histograms of neighboring blocks for the current block, where synthetic histograms are computed for intra-predicted neighboring blocks that are not predicted using DIMD or MIMD. This histogram is then used to determine one or more intra- prediction modes to compute predictors, as well as one or more blending weights used to blend the predictors to form the final prediction for the block.

[0156] MIMD is available as an option to be signaled. Signalling restrictions can be used. For instance, MIMD can be available only if the current block has at least one neighbor that is coded with DIMD or MIMD. Under these conditions, usage of MIMD can be signaled with a CABAC coded CU level flag.

[0157] It is realized that the basic operation of MIMD only works for intra-predicted blocks. Therefore, the present embodiments aim to improve the operation of MIMD to be applicable also with inter-predicted blocks.

[0158] It is known to use an Intra Prediction Mode (IPM) buffer in ECM during the construction of the Most Portable Mode (MPM) list for intra-predicted blocks extracted from inter frames. The MPM list is constructed using the intra- prediction modes of neighboring blocks. When coding an intra-block within an inter-predicted frame, it is possible that neighboring blocks may be encoded with inter-prediction. In that case, intra-prediction information stored in the IPM buffer may be used. The IPM buffer is stored for all blocks predicted with intra-prediction, and the information in the IPM buffer is propagated in inter-predicted blocks following the motion information. For instance, if a given block is inter-predicted with a given motion vector pointing to a given reference frame, then the information in the IPM buffer is extracted from that reference frame at the location pointed by the motion vector and copied in the IPM buffer for the current inter-predicted block. This allows a given location in an inter-predicted block to possibly have intra-prediction information. This information can then be accessed by neighboring blocks during the MPM list construction.

[0159] The idea of the present solution is to produce an intra-prediction for a given block using at least one intra-prediction process. The intra-prediction process comprises extracting prediction information from more than one neighboring blocks, where at least one of the neighboring blocks has been predicted using inter-prediction. The extracted prediction information is used to derive a histogram, and the derived histogram is used to determine one or more intra- prediction modes to compute predictors, as well as one or more blending weights used to blend the predictors to form the final prediction for the given block.

[0160] The extracted prediction information for at least one neighboring block may comprise at least one intra-prediction mode. For example, when the neighboring block has been predicted using inter-prediction, then an intra- prediction mode can be derived using the motion information of the interpredicted neighboring block. As an example, the prediction information extracted from the neighboring block may depend on a motion vector. As an example, in case the neighboring inter-predicted block is predicted using Geometric Partitioning Mode, then the intra-prediction mode can be derived using the split direction of the Geometric Partitioning Mode inter-predicted neighboring block.

[0161] The prediction information extracted from the neighboring block may comprise information extracted from an IPM buffer, where the IPM buffer for the neighboring block is computed by propagating intra-prediction mode information to the neighboring block from at least another block. The IPM buffer may contain intra-prediction mode information for given locations within a given frame. For example, if the neighboring block is inter-predicted with a given motion vector pointing to a reference frame, then the information in the I PM buffer is extracted from that reference frame at the location pointed by the motion vector and copied in the IPM buffer for the neighboring inter-predicted block.

[0162] The extracted prediction information is used for deriving a histogram. As an example, the histogram to be formed for the current block may be formed by combining different histograms determined for a number of neighboring blocks. As an example, a histogram is formed of a number of amplitudes corresponding to the number of possible intra-prediction modes. As an example, a histogram is formed of a number of amplitudes corresponding to the number of possible extended intra-prediction modes in the ECM.

[0163] As an example, if a neighboring block is coded using a DIMD process, the histogram for that neighboring block is determined by extracting the DIMD histogram of gradients for that block. As an example, if a neighboring block is coded using a given intra-prediction mode, then a histogram is considered where the amplitude corresponding to that intra-prediction mode is determined based on characteristics of the neighboring block. The characteristics may comprise e.g., a size of the neighboring block, or a distance of the neighboring block from the current block. As an example, if a neighboring block is coded using a given intra-prediction mode, then a histogram is considered where the amplitude corresponding to that intra-prediction mode is determined based on characteristics of the current block. The characteristics may comprise e.g., a size of the current block.

[0164] As an example, if a neighboring block has been coded using inter-prediction, then a synthetic histogram is computed for such block where the formation of the histogram depends on information extracted from the IPM buffer. For example, one or more intra-prediction modes may be extracted from the IPM buffer at a location dependent on the location of the neighboring block or at a location dependent on the location of the current block. As an example, the information extracted from the IPM buffer at the location corresponding to the top-left location of the neighboring block can be used. As an example, the information extracted from the IPM buffer at the location corresponding to the bottom-left location of the neighboring block can be used. As an example, the information extracted from the IPM buffer at the location corresponding to the top-right location of the neighboring block can be used. As an example, the location used to extract information from the IPM buffer may depend on the position of the neighboring block with respect to the current block. As an example, if the neighboring block is located above the current block, then a location within the bottom-most row of the neighboring block may be used. As an example, if the neighboring block is located on the left of the current block, then a location within the right-most column of the neighboring block may be used.

[0165] As an example, if a neighboring block has been coded using inter-prediction, then a synthetic histogram is computed for such block where the formation of the histogram depends on the intra prediction modes extracted from the IPM buffer at locations contained within the neighboring block. As an example, all locations available in the IPM buffer and contained within the neighboring block can be considered, where all the intra prediction modes extracted at such locations can be used to form the histogram. As an example, the amplitude of a given intra prediction mode in the histogram may be computed depending on the number of times such mode is extracted from the IPM buffer when considering all locations available in the IPM buffer within the neighboring block. As an example, the information extracted from the IPM buffer at the location corresponding to all 4x4 subblock positions of the neighboring block can be used. As an example, the information extracted from the IPM buffer at the location corresponding to spatially adjacent locations of the neighboring block can be used.

[0166] As an example, a neighboring block may be considered as valid only if the IPM buffer at a location within the neighboring block contains a valid intra-prediction mode. As an example, a valid intra-prediction mode corresponds to a directional intra-prediction mode.

[0167] The histograms computed in neighboring blocks or extracted from neighboring blocks are then used to determine one or more intra-prediction modes for the block. As an example, the histograms are combined together to form a merged histogram for the current block. As an example, histograms derived from interpredicted blocks are combined with histograms derived from intra-predicted blocks. As an example, averaging of the histograms can be performed to compute the merged histogram. As an example, the neighboring blocks can be sorted based on their distance with respect to the current block, and only a number of histograms may be considered corresponding to the blocks that are closest to the current block. As an example, the histograms determine in neighboring blocks can be scaled before being combined together. As an example, the scaling factor used may depend on the amplitudes determined in a neighboring block coded with DIMD. As an example, the scaling factor used may depend on the amplitudes determined in a neighboring block coded with MIMD. As an example, the maximum amplitude of all histograms determined in all available neighboring blocks coded with DIMD or MIMD may be used as a scaling factor.

[0168] The merged histogram that has been derived from a plurality of neighboring histograms for the current block is utilized to determine one or more intraprediction modes for the block. As an example, the modes having the highest peaks in the histogram are selected as the intra-prediction modes for the current block.

[0169] The one or more determined intra-prediction modes are used for forming one or more predictors, as a result of applying an intra prediction process.

[0170] Blending of the predictors may be performed in order to obtain a final prediction for the current block. Uniform blending may be used to blend the predictors. As an example, the weights used to blend the predictors may depend on the merged histogram computed for the current block. As an example, higher weights can be assigned to modes with higher amplitudes in the merged histogram, and correspondingly lower weights can be assigned to modes with lower amplitudes in the merged histogram. The process for determining a final prediction for the current block based on two predictors Poand Pi, may be as follows:

[0171] P(x,y) = wiPi(x,y) +woPo(x,y)

[0172] Sample-based blending may also be used to blend the predictors. The samplebased blending may operate based on determining specific weights for each predictor and for each sample within the current block. The weights may be determined in accordance with the location of each sample within the block, where different samples within the block may be blended using different weights.

[0173] Determining a type of blending to fuse the at least two predictors may be based on characteristics of the current block, and / or based on information extracted from neighboring blocks, and / or based on information extracted from a list of Most Probable Modes (MPM), and / or based on the directionality of the intra modes used to produce the predictors. As an example, if one of the modes used to produce the predictors has a directionality Above or Left, then samplebased blending may be inferred to be used. As another example, if the directionality of the modes used to produce the predictors is the same, then uniform blending may be inferred to be used.

[0174] Usage of this method may be determined based on signalling. As an example, a decoder may extract a flag from the bitstream to determine if the method is used to predict the current block or not. As an example, the presence of the flag may depend on characteristics of the current block. As an example, the presence of the flag may depend on characteristics of neighboring blocks. As an example, the flag may be present only in case at least one neighboring block is coded with DIMD or MIMD. As an example, the flag may be resent only in case at least one neighboring block is coded with intra-prediction.

[0175] The method according to an embodiment is shown in Figure 5. The method is aimed for decoding encoded samples of blocks of video sample data, wherein for a block of video sample data the method generally comprises forming an intra prediction using an intra-prediction process, where the intra-prediction process comprises: extracting 510 prediction information from at least one other block, where the at least one other block has been predicted using an inter-prediction process; deriving 520 a histogram based on the extracted prediction information; determining 530 one or more intra-prediction modes based on the derived histogram; forming 540 one or more predictors based on the one or more determined intra-prediction modes; and forming 550 a prediction for the block of video samples by means of blending the one or more predictors. Each of the steps can be implemented by a respective module of a computer system. An apparatus for decoding encoded samples of blocks of video sample data is provided, wherein for a block of video sample data the apparatus comprises means for forming an intra prediction using an intra-prediction process. The apparatus according to an embodiment comprises means for extracting prediction information from at least one other block, where the at least one other block has been predicted using an inter-prediction process; means for deriving a histogram based on the extracted prediction information; means for determining one or more intra-prediction modes based on the derived histogram; means for forming one or more predictors based on the one or more determined intra-prediction modes; and means for forming a prediction for the block of video samples by means of blending the one or more predictors. The means comprises at least one processor, and a memory including a computer program code, wherein the processor may further comprise processor circuitry. The memory and the computer program code are configured to, with the at least one processor, cause the apparatus to perform the method of Figure 5 according to various embodiments.

[0176] Figure 6 illustrates an example of an electronic apparatus 600, being an example of video coding system where the present embodiments can be implemented. In some embodiments, the apparatus may be a mobile terminal or a user equipment of a wireless communication system or a camera device. The apparatus 600 may also be comprised at a local or a remote server or a graphic processing unit of a computer. The apparatus may also be comprised as part of a head-mounted display device.

[0177] The apparatus may be configured to perform various functions, such as for example, gathering information by one or more sensors, encoding and / or decoding information, receiving and / or transmitting information, analyzing information gathered or received by the apparatus. An apparatus configured to encode a video scene may optionally comprise one or more microphones for capturing the scene and / or one or more cameras for capturing information about the physical environment in which the scene is captured. Alternatively, the apparatus configured for encoding may be configured to receive information about an environment in which a scene is captured and / or a simulated environment. An apparatus configured to decode and / or render the video scene may be configured to receive a bitstream comprising encoded video. An apparatus configured to decode and / or render the video scene may comprise one or more speakers / audio transducers and / or displays, and / or may be configured to transmit a decoded scene or signals to a device comprising one or more speakers / audio transducers and / or displays. An apparatus configured to decode and / or render the video scene may comprise a user equipment, a head-mounted display, or another device capable of rendering to a user an AR; VR and / or MR experience.

[0178] The apparatus 600 comprises one or more processors 610 and one or more memories 620 and one or more transceivers interconnected through one or more buses. The one or more memories 620 store computer instructions, for example in respective modules (Modulel , Module2, ModuleN). The one or more memories may store data in the form of image, video and / or audio data, and / or may also store instructions to be executed by the processors or the processor circuitry. The one or more processors may comprise a central processing unit (CPU) and / or a graphical processing unit (GPU). The one or more buses may be address, data or control buses, and may include interconnection mechanism, such as a series of lines on a motherboard or integrated circuit, fiber optics or other optical communication equipment. The apparatus also comprises a codec 630 that is configured to implement various embodiments relating to present solution. According to some embodiments, the apparatus may comprise an encoder or a decoder. The apparatus 600 also comprises a communication interface 640 which is suitable for generating wireless communication signals for example for communication with a cellular communications network, a wireless communications system, or a wireless local area network, and thus enabling data transfer over data transfer network 650.

[0179] The apparatus 600 may comprise a display in the form of a liquid crystal display. In other embodiments of the invention the display may be any suitable display technology suitable to display an image or video. The apparatus 600 may further comprise a keypad. In other embodiments of the invention any suitable data or user interface mechanism may be employed. For example, the user interface may be implemented as a virtual keyboard or data entry system as part of a touch-sensitive display. The apparatus 600 may comprise a microphone or any suitable audio input which may be a digital or analogue signal input. The apparatus 600 may further comprise an audio output device which in embodiments of the invention may be any one of: an earpiece, speaker, or an analogue audio or digital audio output connection. The apparatus 600 may also comprise a battery (or in other embodiments of the invention the device may be powered by any suitable mobile energy device such as solar cell, fuel cell or clockwork generator). The apparatus may further comprise a camera capable of recording or capturing images and / or video. The camera may be a multi-lens camera system having at least two camera sensors. The camera is capable of recording or detecting individual frames which are then passed to the codec 630 or to processor 610. The apparatus may receive the video and / or image data for processing from another device prior to transmission and / or storage.

[0180] The apparatus 600 may further comprise e.g., the other functional units disclosed in any of the Figures 1 - 2 for implementing any of the present embodiments.

[0181] The apparatus may operate in a system, comprising multiple communication devices, which can communicate through one or more networks. The system may comprise any combination of wired or wireless networks including, but not limited to a wireless cellular telephone network (such as a GSM, UMTS, CDMA network etc.), a wireless local area network (WLAN) such as defined by any of the IEEE 802.x standards, a Bluetooth personal area network, an Ethernet local area network, a token ring local area network, a wide area network, and the Internet.

[0182] For example, the system can be a mobile telephone network enabling a connection to the internet. The connection can form, but is not limited to, long range wireless connections, short range wireless connections, and various wired connections including, but not limited to, telephone lines, cable lines, power lines, and similar communication pathways.

[0183] The example communication devices operating in the system may include, but are not limited to, an electronic device or apparatus, a combination of a personal digital assistant (PDA) and a mobile telephone, a PDA, an integrated messaging device (IMD), a desktop computer, a notebook computer, each of which can be a representative of the apparatus according to present embodiments. The apparatus according to present embodiments may be stationary or mobile when carried by an individual who is moving. The apparatus may also be located in a mode of transport including, but not limited to, a car, a truck, a taxi, a bus, a train, a boat, an airplane, a bicycle, a motorcycle, or any similar suitable mode of transport.

[0184] The apparatus may also be a set-top box; i.e. a digital TV receiver, which may / may not have a display or wireless capabilities, a tablet or (laptop) a personal computer (PC), which have hardware or software or combination of the encoder / decoder implementations, in various operating systems, or a chipset, processor, DSP and / or embedded system offering hardware / software based coding.

[0185] The apparatus according to present embodiments may send and receive calls and messages and communicate with service providers through a wireless connection to a base station. The base station may be connected to a network server that allows communication between the mobile telephone network and the internet.

[0186] The apparatus may communicate using various transmission technologies including, but not limited to, code division multiple access (CDMA), global systems for mobile communications (GSM), universal mobile telecommunications system (UMTS), time divisional multiple access (TDMA), frequency division multiple access (FDMA), transmission control protocolinternet protocol (TCP-IP), short messaging service (SMS), multimedia messaging service (MMS), email, instant messaging service (IMS), Bluetooth, IEEE 802.11 and any similar wireless communication technology. A communications device involved in implementing various embodiments of the present invention may communicate using various media including, but not limited to, radio, infrared, laser, cable connections, and any suitable connection.

[0187] In telecommunications and data networks, a channel may refer either to a physical channel or to a logical channel. A physical channel may refer to a physical transmission medium such as a wire, whereas a logical channel may refer to a logical connection over a multiplexed medium, capable of conveying several logical channels. A channel may be used for conveying an information signal, for example a bitstream, from one or several senders (or transmitters) to one or several receivers. An MPEG-2 transport stream (TS), specified in ISO / IEC 13818-1 or equivalently in ITU-T Recommendation H.222.0, is a format for carrying audio, video, and other media as well as program metadata or other metadata, in a multiplexed stream. A packet identifier (PID) is used to identify an elementary stream (a.k.a. packetized elementary stream) within the TS. Hence, a logical channel within an MPEG-2 TS may be considered to correspond to a specific PID value.

[0188] Available media file format standards include ISO base media file format (ISO / IEC 14496-12, which may be abbreviated ISOBMFF) and file format for NAL unit structured video (ISO / IEC 14496-15), which derives from the ISOBMFF.

[0189] The various embodiments can be implemented with the help of computer program code that resides in a memory and causes the relevant apparatuses to carry out the method. For example, a device may comprise circuitry and electronics for handling, receiving, and transmitting data, computer program code in a memory, and a processor that, when running the computer program code, causes the device to carry out the features of an embodiment. Yet further, a network device like a server may comprise circuitry and electronics for handling, receiving, and transmitting data, computer program code in a memory, and a processor that, when running the computer program code, causes the network device to carry out the features of various embodiments.

[0190] If desired, the different functions discussed herein may be performed in a different order and / or concurrently with other. Furthermore, if desired, one or more of the above-described functions and embodiments may be optional or may be combined.

[0191] Although various aspects of the embodiments are set out in the independent claims, other aspects comprise other combinations of features from the described embodiments and / or the dependent claims with the features of the independent claims, and not solely the combinations explicitly set out in the claims. It is also noted herein that while the above describes example embodiments, these descriptions should not be viewed in a limiting sense. Rather, there are several variations and modifications, which may be made without departing from the scope of the present disclosure as, defined in the appended claims.

Claims

Claims:1 . An apparatus for decoding encoded samples of blocks of video sample data is provided, wherein for a block of video sample data the apparatus comprises means for forming an intra prediction by means for extracting prediction information from at least one other block, where the at least one other block has been predicted using an inter-prediction process; means for deriving a histogram based on the extracted prediction information; means for determining one or more intra-prediction modes based on the derived histogram; means for forming one or more predictors based on the one or more determined intra-prediction modes; and means for forming a prediction for the block of video samples by means of blending the one or more predictors.

2. The apparatus according to claim 1 , wherein the extracted prediction information comprises information extracted from an intra prediction mode buffer.

3. The apparatus according to claim 2, wherein the intra prediction mode buffer comprises intra prediction information for a given location within a given frame.

4. The apparatus according to claim 3, further comprising means for determining intra prediction information for a location belonging to an inter-predicted block from the intra prediction mode buffer based on motion information of the inter-predicted block.

5. The apparatus according to claim 4, wherein the motion information comprises a motion vector.

6. The apparatus according to claim 2, wherein the extracted prediction information comprises intra-prediction information extracted at a location that is derived based on characteristics of the current block.

7. The apparatus according to claim 2, wherein the extracted prediction information comprises intra-prediction information extracted at a location that is derived based on characteristics of the at least one other block.

8. The apparatus according to any of the claims 2 to 7, further comprising means for forming the histogram by extracting intra prediction information from the intra prediction mode buffer at locations that are contained by said at least one other block.

9. The apparatus according to claim 8, wherein amplitudes in the histogram depend on the number of times a given intra prediction mode is extracted from the intra prediction mode buffer at locations that are contained by said at least one other block.

10. The apparatus according to any of the claims 1 to 7, wherein the histogram is derived based on information extracted from at least one other block predicted using intra-prediction.

11. The apparatus according to claim 10, further comprising means for predicting the at least one other block using a decoder side intra mode derivation process.

12. The apparatus according to any of the claims 1 to 11 , further comprising means for scaling amplitudes in the histogram based on a scaling factor.

13. The apparatus according to claim 12, wherein the scaling factor depends on information extracted from the at least one other block predicted using a decoder side intra mode derivation process.

14. The apparatus according to any of the claims 1 to 13, wherein the histogram is formed based on information extracted from the intra prediction mode buffer.

15. The apparatus according to any of the claims 1 to 14, further comprising means for extracting information from a bitstream, wherein the information indicates usage of the intra-prediction process.

16. A method for decoding encoded samples of blocks of video sample data is provided, wherein for a block of video sample data the method comprises forming an intra prediction using an intra-prediction process, where the intra-prediction process comprises: extracting prediction information from at least one other block, where the at least one other block has been predicted using an inter-prediction process; deriving a histogram based on the extracted prediction information; determining one or more intra-prediction modes based on the derived histogram; forming one or more predictors based on the one or more determined intra-prediction modes; and forming a prediction for the block of video samples by means of blending the one or more predictors.

Images