Methods and equipment for data processing

Abstract

The invention relates to a device comprising means for processing images to predict blocks according to the prediction patterns of two prediction blocks; to determine when both prediction blocks for a block lie within the boundaries of an image, and to apply a double prediction based on the aforementioned two prediction blocks to the block resulting in the double prediction block; to determine when one of the two prediction blocks lies at least partially outside the boundaries of an image, and to apply a single prediction based on the prediction block within the image boundaries resulting in the single prediction block; to infer filter coefficients based on a selected set of samples of the double prediction block, wherein the selected set of samples includes samples that have been subsampled within the double prediction block by a subsampling method; and to filter at least partially of the single prediction block according to the filter coefficients.

Description

A METHOD, AN APPARATUS AND A COMPUTER PROGRAM PRODUCT FOR VIDEO ENCODING AND DECODING Technical Field

[0001] The present solution generally relates to video encoding and video decoding. Background

[0002] This section is intended to provide a background or context to the invention that is recited in the claims. The description herein may include concepts that could be pursued but are not necessarily ones that have been previously conceived or pursued. Therefore, unless otherwise indicated herein, what is described in this section is not prior art to the description and claims in this application and is not admitted to be prior art by inclusion in this section.

[0003] A video coding system may comprise an encoder that transforms an input video into a compressed representation suited for storage / transmission and a decoder that can uncompress the compressed video representation back into a viewable form. The encoder may discard some information in the original video sequence in order to represent the video in a more compact form, for example, to enable the storage / transmission of the video information at a lower bitrate than otherwise might be needed. Summary

[0004] The scope of protection sought for various embodiments of the invention is set out by the independent claims. The embodiments and features, if any, described in this specification that do not fall under the scope of the independent claims are to be interpreted as examples useful for understanding various embodiments of the invention.

[0005] Various aspects include a method, an apparatus and a computer readable medium comprising a computer program stored therein, which are characterized by what is stated in the independent claims. Various embodiments are disclosed in the dependent claims.

[0006] According to a first aspect, there is provided an apparatus comprising means for means for processing an image to predict a block according to prediction samples of two prediction blocks; means for determining when both of the two predictionblocks for the block are within boundaries of the image, and means for applying a bi- prediction based on said two prediction blocks for the block resulting in a bi-prediction block; means for determining when one of the two prediction blocks is at least partially out of the image’s boundaries, and means for applying a uni-prediction based on the prediction block within image boundaries resulting in a uni-prediction block; means for deriving filter coefficients based on selected set of samples of the bi-prediction block, wherein the selected set of samples comprises samples that have been subsampled in the bi-prediction block according to a subsampling method; and means for filtering at least a part of the uni-prediction block according to the filter coefficients to generate a filtered uni-prediction block.

[0007] According to a second aspect, there is provided a method, comprising: processing an image to predict a block according to prediction samples of two prediction blocks; determining when both of the two prediction blocks for the block are within boundaries of the image, and applying a bi-prediction based on said two prediction blocks for the block resulting in a bi-prediction block; determining when one of the two prediction blocks is at least partially out of the image’s boundaries, and applying a uni-prediction based on the prediction block within image boundaries resulting in a uni-prediction block; deriving filter coefficients based on selected set of samples of the bi-prediction block, wherein the selected set of samples comprises samples that have been subsampled in the bi-prediction block according to a subsampling method; and filtering at least a part of the uni-prediction block according to the filter coefficients to generate a filtered uni-prediction block.

[0008] According to a third aspect, there is provided an apparatus comprising at least one processor, memory including computer program code, the memory and the computer program code configured to, with the at least one processor, cause the apparatus to perform at least the following: process an image to predict a block according to prediction samples of two prediction blocks; determine when both of the two prediction blocks for the block are within boundaries of the image, and apply a bi- prediction based on said two prediction blocks for the block resulting in a bi-prediction block; determine when one of the two prediction blocks is at least partially out of the image’s boundaries, and apply a uni-prediction based on the prediction block within image boundaries resulting in a uni-prediction block; derive filter coefficients based on selected set of samples of the bi-prediction block, wherein the selected set of samples comprises samples that have been subsampled in the bi-prediction block according to a subsampling method; and filter at least a part of the uni-prediction block according to the filter coefficients to generate a filtered uni-prediction block.

[0009] According to a fourth aspect, there is provided computer program product comprising computer program code configured to, when executed on at least one processor, cause an apparatus or a system to: process an image to predict a block according to prediction samples of two prediction blocks; determine when both of the two prediction blocks for the block are within boundaries of the image, and apply a bi- prediction based on said two prediction blocks for the block resulting in a bi-prediction block; determine when one of the two prediction blocks is at least partially out of the image’s boundaries, and apply a uni-prediction based on the prediction block within image boundaries resulting in a uni-prediction block; derive filter coefficients based on selected set of samples of the bi-prediction block, wherein the selected set of samples comprises samples that have been subsampled in the bi-prediction block according to a subsampling method; and filter at least a part of the uni-prediction block according to the filter coefficients to generate a filtered uni-prediction block.

[0010] According to an embodiment, the subsampling method comprises selecting evenly spaced samples within the bi-prediction block with predefined intervals.

[0011] According to an embodiment, the subsampling method comprises selecting evenly spaced samples within the bi-prediction block so that the total number of the samples does not exceed a predefined threshold.

[0012] According to an embodiment, the subsampling method comprises dividing the bi-prediction block into a plurality of zones, and selecting samples with different intervals within each zone.

[0013] According to an embodiment, the subsampling method comprises selecting fully or partially samples included in prediction sample derivation.

[0014] According to an embodiment, the subsampling method comprises selecting samples based on subsampling matrices.

[0015] According to an embodiment, the subsampling method comprises selecting samples with different intervals in different directions.

[0016] According to an embodiment, a value of the predefined threshold depends on a prediction unit (PU), a coding unit (CU), a coding tree unit (CTU), a slice, a frame or sequence properties.

[0017] According to an embodiment, the number of zones is based on a prediction unit (PU), a coding unit (CU), a coding tree unit (CTU), a slice, a frame or sequence properties.

[0018] According to an embodiment, a geometry of a zone is derived as a function of height and width of the bi-prediction block.

[0019] According to an embodiment, overshoot parameters are defined to control the number of samples.

[0020] According to an embodiment, the overshoot parameters are based on a prediction unit (PU), a coding unit (CU), a coding tree unit (CTU), a slice, a frame or sequence properties.

[0021] According to an embodiment, the computer program product is embodied on a non-transitory computer readable medium. Description of the Drawings

[0022] In the following, various embodiments will be described in more detail with reference to the appended drawings, in which

[0023] Fig.1 shows an example of the location of the left and above samples of a current block involved in the CCLM mode.

[0024] Fig.2 a shows an example of derivation of chroma prediction mode from luma mode when CCLM is enabled.

[0025] Fig.2b shows an example of a unified binarization table for chroma prediction mode.

[0026] Fig.3 a shows two luma-to-chrome models obtained for luma Y threshold of 17.

[0027] Fig.3b shows an example of correspondence of each luma-to-chroma model to a spatial segmentation of the content.

[0028] Fig.4 shows an example of locations of the samples used for the derivation of CCCM filter.

[0029] Fig.5 shows examples of various filter kernels;

[0030] Fig.6 shows an example of four reference lines neighboring to a prediction block.

[0031] Fig.7 shows an example an example of matrix weighted intra prediction process.

[0032] Fig.8 shows an example of top and left neighboring blocks used in CIIP weight derivation.

[0033] Fig.9 shows an example of bounding box derived from OOB and non- OOB partitions of the block.

[0034] Fig.10 shows three zones used for ani-isotropic subsampling.

[0035] Fig.11 is a flowchart illustrating a method according to an embodiment;

[0036] Fig.12 shows an apparatus according to an embodiment.

[0037] Fig.13 shows an encoding process according to an embodiment.

[0038] Fig.14 shows a decoding process according to an embodiment.Description of Example Embodiments

[0039] The following description and drawings are illustrative and are not to be construed as unnecessarily limiting. The specific details are provided for a thorough understanding of the disclosure. However, in certain instances, well-known or conventional details are not described in order to avoid obscuring the description. References to one or an embodiment in the present disclosure can be, but not necessarily are, reference to the same embodiment and such references mean at least one of the embodiments.

[0040] Reference in this specification to “one embodiment” or “an embodiment” means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the disclosure.

[0041] In the following, several embodiments will be described in the context of one video coding arrangement. It is to be noted, however, that the present embodiments are not necessarily limited to this particular arrangement. The embodiments relate to reference sample selection in out-of-boundary sample filtering. Before describing the embodiments further, a short reference to evolution of video coding standardization is given. The present embodiments are suited within the context of next generation video coding standardization, e.g., H.267 video coding standard and the ECM (Enhanced Compression Model) exploration.

[0042] The Advanced Video Coding standard (which may be abbreviated AVC or H.264 / AVC) was developed by the Joint Video Team (JVT) of the Video Coding Experts Group (VCEG) of the Telecommunications Standardization Sector of International Telecommunication Union (ITU-T) and the Moving Picture Experts Group (MPEG) of International Organization for Standardization (ISO) / International Electrotechnical Commission (IEC). The H.264 / AVC standard is published by both parent standardization organizations, and it is referred to as ITU-T Recommendation H.264 and ISO / IEC International Standard 14496-10, also known as MPEG-4 Part 10 Advanced Video Coding (AVC). There have been multiple versions of the H.264 / AVC standard, each integrating new extensions or features to the specification. These extensions include Scalable Video Coding (SVC) and Multiview Video Coding (MVC).

[0043] The High Efficiency Video Coding standard (which may be abbreviated HEVC or H.265 / HEVC) was developed by the Joint Collaborative Team - Video Coding (JCT-VC) of VCEG and MPEG. The standard is published by both parent standardization organizations, and it is referred to as ITU-T Recommendation H.265 and ISO / IEC International Standard 23008-2, also known as MPEG-H Part 2 HighEfficiency Video Coding (HEVC). Extensions to H.265 / HEVC include scalable, multiview, three-dimensional, and fidelity range extensions, which may be referred to as SHVC, MV-HEVC, 3D-HEVC, and REXT, respectively. The references in this description to H.265 / HEVC, SHVC, MV-HEVC, 3D-HEVC and REXT that have been made for the purpose of understanding definitions, structures or concepts of these standard specifications are to be understood to be references to the latest versions of these standards that were available before the date of this application, unless otherwise indicated.

[0044] Versatile Video Coding (which may be abbreviated VVC, H.266, or H.266 / VVC) is a video compression standard developed as the successor to HEVC. VVC is specified in ITU-T Recommendation H.266 and equivalently in ISO / IEC 23090-3, which is also referred to as MPEG-I Part 3.

[0045] A specification of the AV1 bitstream format and decoding process were developed by the Alliance of Open Media (AOM). The AV1 specification was published in 2018. AOM is reportedly working on the AV2 specification.

[0046] Some key definitions, bitstream and coding structures, and concepts of H.264 / AVC, HEVC, VVC, and / or AV1 and some of their extensions are described in this section as an example of a video encoder, decoder, encoding method, decoding method, and a bitstream structure, wherein the embodiments may be implemented. The aspects of various embodiments are not limited to H.264 / AVC, HEVC, VVC, and / or AV1 or their extensions, but rather the description is given for one possible basis on top of which the present embodiments may be partly or fully realized.

[0047] A video codec may comprise an encoder that transforms the input video into a compressed representation suited for storage / transmission and a decoder that can uncompress the compressed video representation back into a viewable form. The compressed representation may be referred to as a bitstream or a video bitstream. A video encoder and / or a video decoder may also be separate from each other, i.e., need not form a codec. The encoder may discard some information in the original video sequence in order to represent the video in a more compact form (that is, at lower bitrate). The notation “(de)coder” means an encoder and / or a decoder.

[0048] In some video codecs, such as H.265 / HEVC, video pictures are divided into coding units (CU) covering the area of the picture. A CU consists of one or more prediction units (PU) defining the prediction process for the samples within the CU and one or more transform units (TU) defining the prediction error coding process for the samples in the said CU. CU may consist of a square block of samples with a size selectable from a predefined set of possible CU sizes. A CU with the maximum allowed size may be named as LCU (largest coding unit) or CTU (coding tree unit),and the video picture is divided into non-overlapping CTUs. A CTU can be further split into a combination of smaller CUs, e.g., by recursively splitting the CTU and resultant CUs. Each resulting CU may have at least one PU and at least one TU associated with it. Each PU and TU can be further split into smaller PUs and TUs in order to increase granularity of the prediction and prediction error coding processes, respectively. Each PU has prediction information associated with it defining what kind of a prediction is to be applied for the pixels within that PU (e.g., motion vector information for inter predicted PUs and intra prediction directionality information for intra predicted PUs). Similarly, each TU is associated with information describing the prediction error decoding process for the samples within the said TU (including e.g., DCT coefficient information). It may be signaled at CU level whether prediction error coding is applied or not for each CU. In the case there is no prediction error residual associated with the CU, it can be considered there are no TUs for the said CU. The division of the image into CUs, and division of CUs into PUs and TUs may be signaled in the bitstream allowing the decoder to reproduce the intended structure of these units.

[0049] Hybrid video codecs, for example ITU-T H.263, H.264 / AVC and HEVC, may encode the video information in two phases. At first, pixel values in a certain picture area (or “block”) are predicted for example by motion compensation means (finding and indicating an area in one of the previously coded video frames that corresponds closely to the block being coded) or by spatial means (using the pixel values around the block to be coded in a specified manner). In the first phase, predictive coding may be applied, for example, as so-called sample prediction and / or so-called syntax prediction.

[0050] In the sample prediction, pixel or sample values in a certain picture area or "block" are predicted. These pixel or sample values can be predicted, for example, using one or more of motion compensation or intra prediction mechanisms.

[0051] Motion compensation mechanisms (which may also be referred to as inter prediction, temporal prediction or motion-compensated temporal prediction or motion- compensated prediction or MCP) involve finding and indicating an area in one of the previously encoded video frames that corresponds closely to the block being coded. Inter prediction may reduce temporal redundancy.

[0052] Intra prediction, where pixel or sample values can be predicted by spatial mechanisms, involve finding and indicating a spatial region relationship. Intra prediction utilizes the fact that adjacent pixels within the same picture are likely to be correlated. Intra prediction can be performed in spatial or transform domain, i.e., either sample values or transform coefficients can be predicted. Intra prediction is typically exploited in intra coding, where no inter prediction is applied.

[0053] In the syntax prediction, which may also be referred to as parameter prediction, syntax elements and / or syntax element values and / or variables derived from syntax elements are predicted from syntax elements (de)coded earlier and / or variables derived earlier. Non-limiting examples of syntax prediction are provided below.

[0054] In motion vector prediction, motion vectors e.g., for inter and / or inter-view prediction may be coded differentially with respect to a block-specific predicted motion vector. In many video codecs, the predicted motion vectors are created in a predefined way, for example by calculating the median of the encoded or decoded motion vectors of the adjacent blocks. Another way to create motion vector predictions, sometimes referred to as advanced motion vector prediction (AMVP), is to generate a list of candidate predictions from adjacent blocks and / or co-located blocks in temporal reference pictures and signalling the chosen candidate as the motion vector predictor. In addition to predicting the motion vector values, the reference index of previously coded / decoded picture can be predicted. The reference index is typically predicted from adjacent blocks and / or co-located blocks in temporal reference picture. Differential coding of motion vectors is typically disabled across slice boundaries.

[0055] The block partitioning, e.g., from CTU to CUs and down to PUs, may be predicted.

[0056] In filter parameter prediction, the filtering parameters e.g., for sample adaptive offset may be predicted.

[0057] Prediction approaches using image information from a previously coded image can also be called as inter prediction methods which may also be referred to as temporal prediction and motion compensation.

[0058] Prediction approaches using image information within the same image can also be called as intra prediction methods.

[0059] Secondly, the prediction error, i.e., the difference between the predicted block of pixels and the original block of pixels, is coded. This may be done by transforming the difference in pixel values using a specified transform (e.g., Discrete Cosine Transform (DCT) or a variant of it), quantizing the coefficients and entropy coding the quantized coefficients. By varying the fidelity of the quantization process, encoder can control the balance between the accuracy of the pixel representation (picture quality) and size of the resulting coded video representation (file size of transmission bitrate).

[0060] An elementary unit for the input to an encoder and the output of a decoder, respectively, in most cases is a picture. A picture given as an input to an encoder mayalso be referred to as a source picture, and a picture decoded by a decoded may be referred to as a decoded picture or a reconstructed picture.

[0061] The source and decoded pictures are each comprised of one or more sample arrays, such as one of the following sets of sample arrays: − Luma (Y) only (monochrome). − Luma and two chroma (YCbCr or YCgCo). − Green, Blue and Red (GBR, also known as RGB). − Arrays representing other unspecified monochrome or tri-stimulus color samplings (for example, YZX, also known as XYZ).

[0062] In the following, these arrays may be referred to as luma (or L or Y) and chroma, where the two chroma arrays may be referred to as Cb and Cr; regardless of the actual color representation method in use. The actual color representation method in use can be indicated e.g., in a coded bitstream e.g., using the Video Usability Information (VUI) syntax of HEVC or alike. A component may be defined as an array or single sample from one of the three sample arrays (luma and two chroma) or the array or a single sample of the array that compose a picture in monochrome format.

[0063] A picture may be defined to be either a frame or a field. A frame comprises a matrix of luma samples and possibly the corresponding chroma samples. A field is a set of alternate sample rows of a frame and may be used as encoder input, when the source signal is interlaced. Chroma sample arrays may be absent (and hence monochrome sampling may be in use) or chroma sample arrays may be subsampled when compared to luma sample arrays.

[0064] The decoder reconstructs the output video by applying prediction means similar to the encoder to form a predicted representation of the pixel blocks (using the motion or spatial information created by the encoder and stored in the compressed representation) and prediction error decoding (inverse operation of the prediction error coding recovering the quantized prediction error signal in spatial pixel domain). After applying prediction and prediction error decoding means the decoder sums up the prediction and prediction error signals (pixel values) to form the output video frame. The decoder (and encoder) can also apply additional filtering means to improve the quality of the output video before passing it for display and / or storing it as prediction reference for the forthcoming frames in the video sequence.

[0065] The motion information may be indicated with motion vectors associated with each motion compensated image block in video codecs. Each of these motion vectors represents the displacement of the image block in the picture to be coded (in the encoder side) or decoded (in the decoder side) and the prediction source block in one of the previously coded or decoded pictures.H.264 / AVC and HEVC, as many other video compression standards, a picture is divided into a mesh of rectangles, for each of which a similar block in one of the reference pictures is indicated for inter prediction. The location of the prediction block is coded as a motion vector that indicates the position of the prediction block relative to the block being coded.

[0066] In order to represent motion vectors efficiently those may be coded differentially with respect to block specific predicted motion vectors. The predicted motion vectors may be created in a predefined way, for example calculating the median of the encoded or decoded motion vectors of the adjacent blocks. Another way to create motion vector predictions is to generate a list of candidate predictions from adjacent blocks and / or co-located blocks in temporal reference pictures and signaling the chosen candidate as the motion vector predictor. In addition to predicting the motion vector values, the reference index of previously coded / decoded picture can be predicted. The reference index may be predicted from adjacent blocks and / or or co- located blocks in temporal reference picture. Moreover, high efficiency video codecs may employ an additional motion information coding / decoding mechanism, often called merging / merge mode, where all the motion field information, which includes motion vector and corresponding reference picture index for each available reference picture list, is predicted and used without any modification / correction. Similarly, predicting the motion field information is carried out using the motion field information of adjacent blocks and / or co-located blocks in temporal reference pictures and the used motion field information is signaled among a list of motion field candidate list filled with motion field information of available adjacent / co-located blocks.

[0067] H.264 / AVC and HEVC, as many other video compression standards, a picture is divided into a mesh of rectangles, for each of which a similar block in one of the reference pictures is indicated for inter prediction. The location of the prediction block is coded as a motion vector that indicates the position of the prediction block relative to the block being coded.

[0068] A bitstream may be defined as a sequence of bits or a sequence of syntax structures. A bitstream format may constrain the order of syntax structures in the bitstream.

[0069] A syntax element may be defined as an element of data represented in the bitstream. A syntax structure may be defined as zero or more syntax elements present together in the bitstream in a specified order.

[0070] In some coding formats or standards, a bitstream may be in the form of a network abstraction layer (NAL) unit stream or a byte stream, which forms therepresentation of coded pictures and associated data forming one or more coded video sequences.

[0071] A NAL unit may be defined as a syntax structure containing an indication of the type of data to follow and bytes containing that data in the form of an RBSP interspersed as necessary with start code emulation prevention bytes. A raw byte sequence payload (RBSP) may be defined as a syntax structure containing an integer number of bytes that is encapsulated in a NAL unit. An RBSP is either empty or has the form of a string of data bits containing syntax elements followed by an RBSP stop bit and followed by zero or more subsequent bits equal to 0.

[0072] A NAL unit comprises a header and a payload. The NAL unit header indicates the type of the NAL unit among other things.

[0073] In some coding formats, such as AV1, a bitstream may comprise a sequence of open bitstream units (OBUs). An OBU comprises a header and a payload, wherein the header identifies a type of the OBU. Furthermore, the header may comprise a size of the payload in bytes.

[0074] The phrase along the bitstream (e.g., indicating along the bitstream) or along a coded unit of a bitstream (e.g., indicating along a coded tile) may be used in claims and described embodiments to refer to transmission, signaling, or storage in a manner that the "out-of-band" data is associated with but not included within the bitstream or the coded unit, respectively. The phrase decoding along the bitstream or along a coded unit of a bitstream or alike may refer to decoding the referred out-of-band data (which may be obtained from out-of-band transmission, signaling, or storage) that is associated with the bitstream or the coded unit, respectively. For example, the phrase along the bitstream may be used when the bitstream is contained in a container file, such as a file conforming to the ISO Base Media File Format, and certain file metadata is stored in the file in a manner that associates the metadata to the bitstream, such as boxes in the sample entry for a track containing the bitstream, a sample group for the track containing the bitstream, or a timed metadata track associated with the track containing the bitstream.

[0075] Video codecs may support motion compensated prediction from one source image (uni-prediction) and two sources (bi-prediction). In the case of uni-prediction a single motion vector is applied whereas in the case of bi-prediction two motion vectors are signaled and the motion compensated predictions from two sources are averaged to create the final sample prediction. In the case of weighted prediction, the relative weights of the two predictions can be adjusted, or a signaled offset can be added to the prediction signal.

[0076] In addition to applying motion compensation for inter picture prediction, similar approach can be applied to intra picture prediction. In this case the displacement vector indicates where from the same picture a block of samples can be copied to form a prediction of the block to be coded or decoded. This kind of intra block copying methods can improve the coding efficiency substantially in presence of repeating structures within the frame – such as text or other graphics.

[0077] The prediction residual after motion compensation or intra prediction may be first transformed with a transform kernel (like DCT) and then coded. The reason for this is that often there still exists some correlation among the residual and transform can in many cases help reduce this correlation and provide more efficient coding.

[0078] Video encoders may utilize Lagrangian cost functions to find optimal coding modes, e.g., the desired Macroblock mode and associated motion vectors. This kind of cost function uses a weighting factor λ to tie together the (exact or estimated) image distortion due to lossy coding methods and the (exact or estimated) amount of information that is required to represent the pixel values in an image area: C = D + λR (Eq.1) Where C is the Lagrangian cost to be minimized, D is the image distortion (e.g., Mean Squared Error) with the mode and motion vectors considered, and R the number of bits needed to represent the required data to reconstruct the image block in the decoder (including the amount of data to represent the candidate motion vectors).

[0079] Features and coding tools included in VVC include the following: ^ Intra prediction – 67 intra mode with wide angles mode extension – Block size and mode dependent 4 tap interpolation filter – Position dependent intra prediction combination (PDPC) – Cross component linear model intra prediction (CCLM) – Multi-reference line intra prediction – Intra sub-partitions – Weighted intra prediction with matrix multiplication ^ Inter-picture prediction – Block motion copy with spatial, temporal, history-based, and pairwise average merging candidates – Affine motion inter prediction – sub-block based temporal motion vector prediction – Adaptive motion vector resolution– 8x8 block-based motion compression for temporal motion prediction – High precision (1 / 16 pel) motion vector storage and motion compensation with 8-tap interpolation filter for luma component and 4- tap interpolation filter for chroma component – Triangular partitions – Combined intra and inter prediction – Merge with MVD (MMVD) – Symmetrical MVD coding – Bi-directional optical flow – Decoder side motion vector refinement – Bi-prediction with CU-level weight ^ Transform, quantization and coefficients coding – Multiple primary transform selection with DCT2, DST7 and DCT8 – Secondary transform for low frequency zone – Sub-block transform for inter predicted residual – Dependent quantization with max QP increased from 51 to 63 – Transform coefficient coding with sign data hiding – Transform skip residual coding ^ Entropy Coding – Arithmetic coding engine with adaptive double windows probability update ^ In loop filter – In-loop reshaping – Deblocking filter with strong longer filter – Sample adaptive offset – Adaptive Loop Filter ^ Screen content coding: – Current picture referencing with reference region restriction ^ 360-degree video coding – Horizontal wrap-around motion compensation ^ High-level syntax and parallel processing – Reference picture management with direct reference picture list signalling – Tile groups with rectangular shape tile groups

[0080] In H.266 / VVC, the following block partitioning applies. Pictures may be divided into coding tree units (CTUs). A picture may also be divided into slices, tiles, bricks, and sub-pictures. CTU may be split into smaller CUs using quaternary treestructure. Each CU may be divided using quad-tree and nested multi-type tree including ternary and binary split.

[0081] There are specific rules to infer partitioning in in picture boundaries.

[0082] The redundant split patterns are disallowed in nested multi-type partitioning.

[0083] Some video coding tools perform filtering operations which convolve a set of reference samples with a set of filter parameters to output, for example, a predicted value for a certain sample in a picture. In some cases, the filter parameters may be predetermined or signaled in the bitstream. In other cases, such as when cross- component linear model (CCLM) or cross-component convolutional model (CCCM) prediction is used, the parameters are calculated using a set of reference samples in both encoder and decoder. Generally, calculation of such filter parameters involves inversion of an auto-correlation matrix, which is a computationally challenging operation. Also, when there are many filter parameters to be determined, the size of the auto-correlation matrix becomes large, which can cause numerical stability issues (overflows or underflows).

[0084] To reduce the cross-component redundancy, a cross-component linear model (CCLM) prediction mode may be used in the VVC, for which the chroma samples are predicted based on the reconstructed luma samples of the same CU by using a linear model as follows: pred^(i, j)= α · rec^′(i, j)+ β (3-1) where pred^(i, j)represents the predicted chroma samples in a CU, and rec^'(i, j)represents the downsampled reconstructed luma samples of the same CU.

[0085] The CCLM parameters (α and β) are derived with at most four neighboring chroma samples and their corresponding down-sampled luma samples. Suppose the current chroma block dimensions are WxH, then W’ and H’ are set as − W’ = W, H’ = H when LM mode is applied; − W’ =W + H when LM-A mode is applied; − H’ = H + W when LM-L mode is applied.

[0086] The above neighboring positions are denoted as S[0, -1]…S[W’-1, -1] and the left neighboring positions are denoted as S[-1, 0]…S[-1, H’-1]. Then the four samples are selected as − S[W’ / 4, −1 ], S[ 3 * W’ / 4, −1 ], S[ −1, H’ / 4 ], S[ −1, 3 * H’ / 4 ] when LM mode is applied and both above and left neighboring samples are available;− S[ W’ / 8, −1 ], S[ 3 * W’ / 8, −1 ], S[ 5 * W’ / 8, −1 ], S[ 7 * W’ / 8, −1 ] when LM-A mode is applied or only the above neighboring samples are available; − S[ −1, H’ / 8 ], S[ −1, 3 * H’ / 8 ], S[ −1, 5 * H’ / 8 ], S[ −1, 7 * H’ / 8 ] when LM-L mode is applied or only the left neighboring samples are available.

[0087] The four neighboring luma samples at the selected positions are down- sampled and compared four times to find two smaller values: x0 A and x1 A, and two larger values: x0 B and x1 B. Their corresponding chroma sample values are denoted as Then xA, xB, yAand yBare derived as Xb=(x0 B + x1 B +1)>>1; Ya=(y0 A + y1 A +1)>>1; Yb=(y0 B + y1 B

[0088] Finally, the linear model parameters are obtained according to the following equations:

[0089] Figure 1 shows an example of the location of the left and above samples and the sample of the current block involved in the CCLM mode. The division operation to calculate parameter α is implemented with a look-up table. To reduce the memory required for storing the table, the diff value (difference between maximum and minimum values) and the parameter α are expressed by an exponential notation. For example, diff is approximated with a 4-bit significant part and an exponent. Consequently, the table for 1 / diff is reduced into 16 elements for 16 values of the significant as follows: DivTable [ ] = { 0, 7, 6, 5, 5, 4, 4, 3, 3, 2, 2, 1, 1, 1, 1, 0 }

[0090] This would have a benefit of both reducing the complexity of the calculation as well as the memory size required for storing the needed tables.

[0091] Besides the above template and left template can be used to calculate the linear model coefficients (i.e., parameters) together, they can also be used alternatively in the other 2LM modes, called LM_A, and LM_L modes.

[0092] In LM_A mode, only the above template is used to calculate the linear model coefficients (i.e., parameters). To get more samples, the above template is extended to (W+H). In LM_L mode, only left template is used to calculate the linear model coefficients (i.e., parameters). To get more samples, the left template is extended to (H+W).

[0093] For a non-square block, the above template is extended to W+W, the left template is extended to H+H.

[0094] To match the chroma sample locations for 4:2:0 video sequences, two types of downsampling filter are applied to luma samples to achieve 2 to 1 downsampling ratio in both horizontal and vertical directions. The selection of downsampling filter is specified by a SPS level flag. The two downsampling filters are as follows, which are corresponding to “type-0” and “type-2” content, respectively.rec^(2^, 2^ − 1) + rec^(2^ − 1,2^) + 4 ∙ rec^(2^, 2^) rec^′(^, ^)= ^ ^( )^(^ ≫ 3 +rec 2^ + 1,2^ + rec 2^, 2^ + 1)+ 4

[0095] It is appreciated that only one luma line (general line buffer in intra prediction) is used to make the down-sampled luma samples when the upper reference line is at the CTU boundary.

[0096] This parameter computation is performed as part of the decoding process and is not just as an encoder search operation. As a result, no syntax is used to convey the α and β values to the decoder.

[0097] For chroma intra mode coding, a total of 8 intra modes are allowed for chroma intra mode coding. Those modes include five traditional intra modes and three cross-component linear model modes (CCLM, LM_A, and LM_L). Chroma mode signalling and derivation process are shown in Table 1 in Figure 2a. Chroma mode coding directly depends on the intra prediction of the corresponding luma block. Since separate block partitioning structure for luma and chroma components is enabled in I slices, one chroma block may correspond to multiple luma blocks. Therefore, for Chroma DM mode, the intra prediction mode of the corresponding luma block covering the center position of the current chroma block is directly inherited.

[0098] A single binarization table is used regardless of the values of sps_cclm_enabled_flag as shown in Table 2 in Figure 2b. In Table 2, the first bin indicates whether it is regular (0) or LM modes (1). If it is LM mode, then the next bin indicates whether it is LM_CHROMA (0) or not. If it is not LM_CHROMA, next 1 bin indicates whether it is LM_L (0) or LM_A (1). For this case, when sps_cclm_enabled_flag is 0, the first bin of the binarization table for the corresponding intra_chroma_pred_mode can be discarded prior to the entropy coding. Or, in other words, the first bin is inferred to be 0 and hence not coded. This single binarization table is used for both sps_cclm_enabled_flag equal to 0 and 1 cases. The first two bines in Table 3-4 are context coded with its own context model, and the rest bins are bypass coded.

[0099] In addition, in order to reduce luma-chroma latency in dual tree, when the 64x64 luma coding tree node is partitioned with Not Split (and ISP is not used for the 64x64 CU) or QT, the chroma CUs in 32x32 / 32x16 chroma coding tree node are allowed to use CCLM in the following way: - If the 32x32 chroma node is not split or partitioned QT split, all chroma CUs in the 32x32 node can use CCLM - If the 32x32 chroma node is partitioned with Horizontal BT, and the 32x17 child node does not split or use Vertical BT split, all chroma CUs in the 32x16 chroma node can use CCLM. In all the other luma and chroma coding tree split conditions, CCLM is not allowed for chroma CU.

[0100] The CCLM included in VVC is extended by adding three Multi-model LM (MMLM) modes. In each MMLM mode, the reconstructed neighboring samples are classified into two classes using a threshold which is the average of the luma reconstructed neighboring samples. The linear model of each class is derived using the Least-Mean-Square (LMS) method. For the CCLM mode, the LMS method is also used to derive the linear model. Figure 3a illustrates two luma-to-chroma models obtained for luma (Y) threshold of 17. Each luma-to-chroma model has its own linear model parameters α and β. As can be seen from Figure 3b, each luma-to-chroma model corresponds to a spatial segmentation of the content (i.e., they correspond to different objects or textures in the scene).

[0101] An improved version of cross-component prediction, known as convolutional cross-component model (CCCM), uses 2D filter kernel to derive the luma-to-chroma model. The filter coefficients are derived decoder-side using reconstructed set of input data and chroma samples. For the filter coefficient derivation, co-located reference sample areas (consisting of reconstructed luma and chroma samples) are defined for both luma and chroma as shown in Figure 4, yet any number of reference lines (that can be realized by both the encoder and decoder) can be used. Generally, reference samples can contain any chroma and luma samples that have been reconstructed by both the encoder and decoder. Once the reference samples have been determined, the filter coefficients can be derived, for example, using different types of linear regression tools such as ordinary least-squares estimation, orthogonal matching pursuit, optimized orthogonal matching pursuit, ridge regression, or least absolute shrinkage and selection operator.

[0102] The dimensions of the filter kernel can be for example 1x3 (1D vertical), 3x1 (1D horizontal), 3x3, 7x7 or any dimensions, and can be shaped (by selecting only a subset of all possible kernel locations) as a cross or a diamond or as any given shape.When referring to the samples within the filter kernel, the following notation is used: north (above), east (right), south (below), west (left) and center, as illustrated in Figure 5 using the letters N, E, S, W, C. Figure 5 illustrate a 3-tap vertical kernel 501, 3-tap horizontal kernel 502, 5-tap cross kernel 503 and 25-tap diamond kernel 504.

[0103] The overall method of reconstructing chroma samples using convolution between a decoder-side obtained filter kernel and a set of input data is referred to as convolutional cross-component model (CCCM) here. The following steps can be applied to perform a CCCM operation: - Define co-located reference areas over the luma and chroma components; - Down-sample the luma samples to match the chroma grid (optional); - Scan the luma and chroma samples of the reference area and collet available statistics (such as auto-correlation matrix and cross-correlation vector) based on the filter shape; - Solve the filter coefficients by minimizing squared-error (or any other metric) based on the available statistics (such as the auto-correlation matrix and cross- correlation vector); - Calculate a predicted chroma block by convolving the down-sampled luma samples with the filter kernel.

[0104] In the following, the (possibly down-sampled) luma samples are defined as a 2D array Y(x, y) indexed using horizontal x-coordinate and vertical y-coordinate. Also, the co-located chroma samples are defined as a 2D array C(x, y) and the filter kernel (i.e., coefficients) as 3x3 array F(i, j). On a sample level, the convolution between Y and F is defined as

[0105] When using other data terms, such as the non-linear square-root term, the convolution becomeswhere ^^are filter coefficients that reside outside of the 2D filter kernel yet have been obtained as a part of the system of linear equations that were used to solve the 2D filter coefficients in Step 4 above. Similarly, the bias term can be added to the convolution with

[0106] Angular intra prediction (a.k.a. directional intra prediction) may be performed by extrapolating sample values from the reconstructed reference samples utilizing a given directionality. The reference samples may comprise the immediately neighboring sample row above and above-right of the current block (when available) and the immediately neighboring sample column on the left of the current block (when available), wherein availability may require decoding order earlier than that of the current block and presence in the same image segment, such as in the same tile. In order to simplify the process, all sample locations within one prediction block may be projected to a single reference row or column depending on the directionality of the selected prediction mode. A predicted sample within the block being encoded / decoded may be obtained by the following steps: - Projecting the location of a predicted sample to a location within a reference row or column by applying the selected prediction direction. The location within the reference row or column may have fractional sample accuracy, such as 1 / 32 pixel accuracy. - Interpolating a value for the sample location on the reference row or column from the reference samples at the reference row / column.

[0107] Multiple reference line (MRL) intra prediction uses more reference lines for intra prediction. In Figure 6, an example of four reference lines is depicted, where the samples of segments A and F are not fetched from reconstructed neighboring samples but padded with the closest samples from Segment B and E, respectively. HEVC intra- picture prediction uses the nearest reference line (i.e., reference line 0). In MRL, two additional lines (reference line 1 and reference line 3) are used.

[0108] The index of selected reference line (mrl_idx) is signaled and used to generate intra predictor. For reference line idx, which is greater than 0, only include additional reference line modes in MPM list and only signal mpm index without remaining mode. The reference line index is signaled before intra prediction modes, and Planar mode is excluded from intra prediction modes in case a nonzero reference line index is signaled.

[0109] MRL is disabled for the first line of blocks inside a CTU to prevent using extended reference samples outside the current CTU line. Also, PDPC is disabled when additional line is used. For MRL mode, the derivation of DC value in DC intra prediction mode for non-zero reference line indices are aligned with that of referenceline index 0. MRL requires the storage of 3 neighboring luma reference lines with a CTU to generate predictions. The Cross-Component Linear Model (CCLM) tool also requires 3 neighboring luma reference lines for its own down-sampling filters. The definition of MLR to use the same 3 lines is aligned as CCLM to reduce the storage requirements for decoders.

[0110] The intra sub-partitions (ISP) divides luma intra-predicted blocks vertically or horizontally into 2 or 4 sub-partitions depending on the block size. For example, minimum block size for ISP is 4x8 (or 8x4). If block size is greater than 4x8 (or 8x4), then the corresponding block is divided by four sub-partitions. It has been noticed that the Mx12 (with M ≤ 64) and 128xN (with N ≤ 64) ISP blocks could generate a potential issue with the 64x64 VDPU. For example, and Mx128 CU in the single tree ^ case has an M x 128 luma TB and two corresponding ^ × 64 chroma TBs. If the CU uses ISP, then the luma TB will be divided into four Mx32 TBs (only the horizontal split is possible), each of them smaller than a 64 x 64 block. However, in the current design of ISP, chroma blocks are not divided. Therefore, both chroma components will have a size greater than a 32 x 32 block.

[0111] Analogously, a similar situation could be created with a 128 x N CU using ISP. Hence, these two cases are an issue for the 64 x 64 decoder pipeline. For this reason, the CU sizes that can use ISP is restricted to a maximum 64 x 64. All sub- partitions fulfil the condition of having at least 16 samples.

[0112] Matrix weighted intra prediction (MIP) method is an intra prediction technique in VVC. For predicting the samples of a rectangular block of width W and height H, matrix weighted intra prediction (MIP) takes one line of H reconstructed neighboring boundary samples left of the block and one line of W reconstructed neighboring boundary samples above the block as input. If the reconstructed samples are unavailable, they are generated as it is done in the conventional intra prediction. The generation of the prediction signal is based on the following three steps, which are averaging, matrix vector multiplication and linear interpolation as shown in Figure 7.

[0113] Merged mode with motion vector difference (MMVD) is to signal MVDs and a resolution index after signalling merge candidate. The merge list may comprise the following candidates: 1) Spatial MVP from spatial neighbor CUs 2) Temporal MVP from collocated CUs 3) History-based MVP from a FIFO table 4) Pairwise average MVP (using the candidates already in the list) 5) Zero MVs.

[0114] In Symmetric MVD, motion information of list-1 is derived from motion information from list-0 in bi-prediction case.

[0115] In Affine prediction, several motion vectors are indicated / signaled for different corners of a block, which are used to derive the motion vectors of sub-block. In affine merge, affine motion information of a block is generated based on the normal or affine motion information of the neighboring blocks.

[0116] In Sub-block-based temporal motion vector prediction, motion vectors of sub-blocks of the current block are predicted from a proper subblocks in the reference frame which are indicated by the motion vector of a spatial neighboring vector (if available).

[0117] In Adaptive motion vector resolution (AMVR), precision of MVD is signaled for each CU.

[0118] In BI-prediction with CU-level weight, an index indicated the weight values for weighted average of two prediction block.

[0119] Bi-directional optical flow (BDOF) refines the motion vectors in bi- prediction case. BDOF generates two prediction blocks using the signaled motion vectors. Then a motion refinement is calculated to minimize the error between two prediction blocks using their gradient values. The final prediction blocks are refined using the motion refinement and gradient values.

[0120] In HEVC, the bi-prediction signal is generated by averaging two prediction signals obtained from two different reference pictures and / or using two different motion vectors. In VVC, the bi-prediction mode is extended beyond simple averaging to allow weighted averaging of the two prediction signals. This is called bi-prediction with CU-level weight (BCW): ^bi-pred= ^(8 − ^)∗ ^^+ ^ ∗ ^^+ 4^ ≫ 3

[0121] Five weights are allowed in the weighted averaging bi-prediction, ^ ∈ {−2, 3, 4, 5, 10}. For each bi-predicted CU, the weight w is determined in one of two ways: 1) for a non-merge CU, the weight index is signaled after the motion vector difference; 2) for a merge CU, the weight index is inferred from neighboring blocks based on the merge candidate index. BCW is only applied to CUs with 256 or more luma samples (i.e., CU width times CU height is greater than or equal to 256). For low-delay pictures, all 5 weights are used. For non-low-delay pictures, only 3 weights (w∈{3,4,5}) are used.

[0122] At the encoder, fast search algorithms are applied to find the weight index without significantly increasing the encoder complexity. These algorithms aresummarized as follows. For further details readers are referred to the VTM software and document JVET-L0646. When combined with AMVR, unequal weights are only conditionally checked for 1-pel and 4-pel motion vector precisions if the current picture is a low-delay picture. − When combined with affine, affine ME will be performed for unequal weights if and only if the affine mode is selected as the current best mode. − When the two reference pictures in bi-prediction are the same, unequal weights are only conditionally checked. − Unequal weights are not searched when certain conditions are met, depending on the POC distance between current picture and its reference pictures, the coding QP, and the temporal level.

[0123] The BCW weight index is coded using one context coded bin followed by bypass coded bins. The first context coded bin indicates if equal weight is used; and if unequal weight is used, additional bins are signalled using bypass coding to indicate which unequal weight is used.

[0124] Weighted prediction (WP) is a coding tool supported by the H.264 / AVC and HEVC standards to efficiently code video content with fading. Support for WP was also added into the VVC standard. WP allows weighting parameters (weight and offset) to be signaled for each reference picture in each of the reference picture lists L0 and L1. Then, during motion compensation, the weight(s) and offset(s) of the corresponding reference picture(s) are applied. WP and BCW are designed for different types of video content. In order to avoid interactions between WP and BCW, which will complicate VVC decoder design, if a CU uses WP, then the BCW weight index is not signaled, and w is inferred to be 4 (i.e., equal weight is applied). For a merge CU, the weight index is inferred from neighboring blocks based on the merge candidate index. This can be applied to both normal merge mode and inherited affine merge mode. For constructed affine merge mode, the affine motion information is constructed based on the motion information of up to 3 blocks. The BCW index for a CU using the constructed affine merge mode is simply set equal to the BCW index of the first control point MV.

[0125] In VVC, combined inter / intra prediction (CIIP) and bi-prediction with CU- level weight (BCW) cannot be jointly applied for a CU. When a CU is coded with CIIP mode, the BCW index of the current CU is set to 2, e.g., equal weight.

[0126] In VVC, when a CU is coded in merge mode, if the CU contains at least 64 luma samples (that is, CU width times CU height is equal to or larger than 64), and if both CU width and CU height are less than 128 luma samples, an additional flag is signaled to indicate if the combined inter / intra prediction (CIIP) mode is applied to thecurrent CU. As its name indicates, the CIIP prediction combines an inter prediction signal with an intra prediction signal. The inter prediction signal ^^^^^^in the CIIP mode is derived using the same inter prediction process applied to regular merge mode; and the intra prediction signal ^^^^^^is derived following the regular intra prediction process with the planar mode. Then, the intra and inter prediction signals ^^^^^^, ^^^^^^are combined using weighted averaging, where the weight value is calculated depending on the coding modes of the top and left neighboring blocks as follows: – If the top neighbor is available and intra coded, then set isIntraTop to 1, otherwise set isIntraTop to 0; – If the left neighbor is available and intra coded, then set isIntraLeft to 1, otherwise set isIntraLeft to 0; – If (isIntraLeft + isIntraTop) is equal to 2, then wt is set to 3; – Otherwise, if (isIntraLeft + isIntraTop) is equal to 1, then wt is set to 2; – Otherwise, set wt to 1. Figure 8 illustrates an example of top and left neighboring blocks used in CIIP weight derivation.

[0127] The CIIP prediction is formed as follows:

[0128] Local illumination compensation (LIC) is an inter prediction technique to model local illumination variation between current block and its prediction block as a function of that between current block template and reference block template. The parameters of the function can be denoted by a scale α and an offset β, which forms a linear equation, that is, α*p[x]+β to compensate illumination changes, where p[x] is a reference sample pointed to by MV at a location x on reference picture. Since α and β can be derived based on current block template and reference block template, no signaling overhead is required for them, except that an LIC flag is signaled for AMVP mode to indicate the use of LIC.

[0129] The local illumination compensation proposed in JVET-O0066 is used in ECM for uni-prediction inter CUs with the following modifications. ^ Intra neighbor samples can be used in LIC parameter derivation; ^ LIC is disabled for blocks with less than 32 luma samples; ^ For both non-subblock and affine modes, LIC parameter derivation is performed based on the template block samples corresponding to the currentCU, instead of partial template block samples corresponding to first top-left 16x16 unit; ^ Samples of the reference block template are generated by using MC with the block MV without rounding it to integer-pel precision.

[0130] In bi-directional motion compensation the out-of-boundary (OOB) prediction samples are discarded and only the non-OOB predictors, when available, are used to generate the final predictor. The OOB prediction samples are such samples whose reference sample is at least partly located outside the reference picture. Specifically, let ^^^_^^,^and ^^^_^^,^denote the position of one prediction sample in one current block, ^^_^^^ ^,^and ^^_^^^ ^,^(x = 0,1) denote the MV of the current block;and ^^^^^^^^^^^^^are the positions of four boundaries of the picture. One prediction sample is regarded as OOB when at least one of the following conditions is satisfied: (^^^_^ ^ ^,^+ ^^_^ ^ ^,^) > (^^^^^^^^^^^^+half_pixel), (^^^_^^,^+ ^^_^^^ ^,^) < (^^^^^^^^^^^- half_pixel), (^^^_^^,^+ ^^_^^^ ^,^) > (^^^^^^^^^^^^^+ half_pixel), (^^^_^^,^+ ^^_^^^ ^,^) < (^^^^^^^^^^- half_pixel) where half_pixel is equal to 8 that represents the half-pel sample distance in the 1 / 16- pel sample precision.

[0131] After examining the OOB condition for each sample, the final prediction samples of one bi-directional block are generated as shown in the following algorithm A: If ^^^ ^^ ^,^is OOB and ^^,^is non-OOB ^ ^^^^^ ^ ^,^ = ^ ^ ^,^else if ^^^ is non-OOB and ^^^ ^,^ ^,^is OOB ^ ^^^^^ ^,^ = ^^^ ^,^else

[0132] In the final prediction, as shown, above, the first part (“if… else if…”) is a uni-predictive part of the bi-directional prediction. The bi-predictive part of the bi-directional prediction is the second part (“else ^ ^^^^^ ^,^ = …”) in which two uni- predictions are (weighted) averaged to form a true bi-prediction.

[0133] OOB checking process is also applicable when BCW is enabled.

[0134] Finally, it is to be noticed that this sample-adaptive bi-prediction process may only apply to prediction units for which at least a reference block is first detected as partially or entirely out-of-boundary. Thus, a block-level OOB criteria is first checked. If both prediction blocks are non-OOB, then the usual bi-prediction takes place.

[0135] In OOB-aware bi-directional inter-prediction only samples that are available in both prediction blocks (i.e.., motion compensated blocks P0 and P1) can be merged into a bi-prediction. For areas, where only one of the predictors (either P0 or P1) is available (the other one being out of the image boundaries), only samples belonging to the available uni-predictor can be used. Therefore, the final prediction of the block is a mixture of uni-prediction samples and bi-prediction samples. This can lead to artefacts and abrupt edges when uni-predicted samples have different brightness or texture characteristics compared to the bi-predicted samples.

[0136] The out-of-boundary samples can be filtered to generate the final prediction samples for luma as follows: if ^^^ is OOB and ^^^ ^,^ ^,^is non-OOBelse if ^^^ ^^ ^,^is non-OOB and ^^,^is OOB ^ ^^^^^ = a ∗ ^^^ ^,^^ ^,^+ ^^else

[0137] The scales and offsets ^^, ^^, ^^, ^^are derived over the non-OOB parts of ^^^and ^^^using least squares minimization,and

[0138] where Ω^are the non-OOB samples 905 of ^^^and ^^^as illustrated in Figure 9 for the case of filtering the samples in ^^^. Scales and offsets may be derived using sample covariance and sample variance at integer precision. Either ^^^or ^^^, only, has OOB samples 905 and therefore only one set of parameters needs to be derived in most cases. The filtering method of this kind is also applicable when BCW is enabled and in such cases samples of ^^^^^^are derived accordingly using the BCW weights. The scales and offsets ^^, ^^, ^^, ^^may be collectively referred to as filter coefficients, or filtering coefficients, or filter parameters as those define a filter that is applied to generate an altered representation of the input samples.

[0139] In the filtering method, the filtering coefficients may be derived by minimizing squared error between samples of bi-prediction block and samples of uni- prediction block, and these filtering coefficients are used to filter at least part of the uni-prediction block.

[0140] The filtering scheme as discussed above may become computationally demanding for high resolution video, especially on the decoder side. This is because a large number of boundary samples are used in filter coefficient derivation. The problem becomes especially noticeable with state-of-the-art video coding systems, such as ECM-10.0 where maximum CU size is 256x256. Potentially tens of thousands of samples become involved in filter coefficient derivation thus rendering the naïve implementation (involving all available reference samples) relatively slow on the decoder-side. Thus, the reference samples selection must be performed in a subsampled and normative way.

[0141] The present embodiments are targeted to normative subsampling strategies to be applied for the filtering scheme to speed-up both encoders-side and decoder-side implementations.

[0142] Figure 9 illustrates geometrically the reference samples involved in filter coefficient derivation according to an embodiment. In particular, Figure 9 shows bounding box 910 of dimensions WxH derived from the OOB and non-OOB partitions of the block. Spatial subsampling strategies may be derived over the bounding box area 910. The sub-sampled samples within the bounding box 910 are used as coefficients for filtering derivation. The samples within the rectangular area 900 delimited by coordinates (x_min,y_min) and (x_max,y_max) which define an area 910 of WxH where W = x_max – x_min + 1 and H = y_max – y_min + 1. Samples within this area 910, or bounding box, are used for filter coefficient derivation involving sample variance and sample covariance.

[0143] When the bounding box dimensions become large, for example 100x100 = 10,000, samples become involved in the filter coefficient derivation and even forsimplistic coefficient derivation the computational complexity will increase linearly as a function of samples. However, often as little as 256 samples will do for robust coefficient derivation. Sub-sampling of reference samples will have to be performed identically on the encoder and decoder and therefore several normative sub-sampling techniques over the bounding box area are proposed.

[0144] Method 1

[0145] Method according to first embodiment comprises selecting evenly spaced samples within the bounding box 910 with intervals dx and dy. For WxH bounding box, this corresponds to selecting (WxH) / (dx*dy) number of reference samples. For example, if dx = 2 and dy = 2, every other sample is selected horizontally and vertically.

[0146] Method 2

[0147] Method according to second embodiment comprises selecting evenly spaced samples within the bounding box 910 so that the total number of reference samples never exceeds a threshold T. Therefore, for square bounding boxes there are dx >= HW / T / 2 and dy >= HW / T / 2. For example, if T = 256 and W = 32, H = 32 then WxH = 1023 = 4T then the following are set as dx = 4 / 2 = 2, dy = 4 / 2 = 2. For rectangular bounding boxes HW / T = R is set and two positive, larger than zero integers A,B are sought so that AxB >= R. After that there are two choices, either dx = A, dy = B, or dx = B and dy = A.

[0148] Method 3

[0149] Method according to third embodiment comprises sub-sampling anisotropically the samples within the bounding box 910. This means that different portions of the bounding box 910 use different level of subsampling as shown in Figure 10. In the examples of Figure 9 and Figure 10, Zone 11010 can avoid subsampling by setting dx=1, dy=1, while Zone 21020 uses dx = 2, dy = 2, and Zone 3 dx = 4, dy = 4. Therefore, the samples closest to the OOB area is best represented by the selection of references samples. The values of dx and dy can be set independently as in method according to second embodiment. The zones 1010, 1020, 1030 can have rectangular or non-rectangular shapes with varying sizes. Any number of zones can be used.

[0150] Method 4

[0151] Method according to fourth embodiment comprises selecting fully or partially samples included in the filter coefficient derivation based on values of the candidate samples. For example, the minimum and maximum sample values can be determined in the part of the prediction block (or a subset of samples is that block; or the set or a subset of reference samples used to generate the prediction block) that have only one prediction with in-picture reference samples. Then when selecting thesamples for parameter generation, the first N samples within that range in a determined scanning order can selected and the rest of the sample can be omitted. The scanning order can be advantageously selected in a way samples closer to the border between OOB samples and non-OOB samples can be scanned before samples further away from the border. The number of samples N can be a constant or can be a variable determined based on available information, for example, the size or dimensions of the prediction block.

[0152] Such value range can naturally be determined in different ways. For example, instead of using a minimum and maximum value directly, certain margin could be applied to those values. For example, a constant or a sample bitdepth dependent value could be added to the found maximum value and deducted from the found minimum value. Similarly, the found minimum and maximum values can be multiplied by different factors to substitute the found minimum and maximum values. Alternatively, an average value can be determined, and a range can be generated based on that, for example, by adding an offset to the average value to calculate the maximum value and deducting the same or a different offset to calculate the minimum value.

[0153] Method 5

[0154] In a method according to fifth embodiment, the subsampling method comprises selecting samples based on subsampling matrices. When using hard-coded subsampling matrices, that are currently used by InterCCCM in ECM-10.0, care must be taken to address the issue of H and W of the bounding box 910 not being integer powers of two. Such subsampling matrices guarantee that for any WxH block (for which W and H are integer powers of two) the subsampling factors dx and dy produce at most T reference samples.

[0155] For fixed T, the subsampling matrices Sx[9][9] and Sy[9][9] provide dx and dy values for any H, W in C-style pseudocode as, dx=Sx[floorLog2(W)][floorLog2(H)], dy=Sy[floorLog2(W)][floorLog2(H)].

[0156] Given the bounding box width W and height H, the subsampling factors dx and dy are retrieved from hard-coded subsampling matrices Sx, Sy as dx = Sx[xi][yi] and dy = Sy[xi][yi] where xi and yi are defined in C-style pseudocode as, xi = floorLog2( W ) xi = ((1<<xi) + (1<<(xi-2))) < W ? xi+1 : xi; yi = floorLog( H )yi = ((1<<yi) + (1<<(yi-2)) < H ? yi+1 : yi;

[0157] The above pseudo-code is designed to handle the situation when for non- integer power of two V we have (1<<floorLog2( V )) < V. For example, if W = 63, we have floorLog2(W) = 5, and without the above procedure too low values of dx and dy are retrieved and the threshold T is exceeded. With the above procedure we have xi = (32+8) < W and therefore xi becomes xi = xi + 1 = 6 and less than T samples are selected. Some overshoot in the number of selected samples happens when either for W and / or H, (1<<floorLog2(W)) < W < ((1<<floorLog2(W)) + (1<<floorLog2(W-2)) (1<<floorLog2(H)) < H < ((1<<floorLog2(H)) + (1<<floorLog2(H-2)) with the worst case being when above conditions apply for W and H both and the number of selected reference samples becomes 1.25*1.25*T ~= 1.5625T. In an implementation where overshoot is unacceptable, the following is used xi = floorLog2( W ) xi = (1<<xi) < W ? xi+1 : xi; yi = floorLog( H ) yi = (1<<yi) < H ? yi+1 : yi; However, this will produce undershoot in the number of reference samples for many non-power-of-two H and W. Overall parameters, i.e., overshoot parameters Px, Py can be defined to control the over- and undershoot behavior: xi = floorLog2( W ) xi = ((1<<xi) + Px) < W ? xi+1 : xi; yi = floorLog( H )´ yi = ((1<<yi) + Py) < H ? yi+1 : yi;

[0158] Method 6 Subsampling can be performed using increasing steps in one dimension while keeping the subsampling constant in the other dimension. This means that subsampling comprises selecting samples with different intervals in different directions. For example, if the OOB samples are located on the left part of the block as illustrated in Figure 9, it may be determined that there is no subsampling in vertical direction whenselecting the reference samples for filter coefficient calculation. However, when progressing the reference sample collection in the horizontal direction, increasing number of columns may be skipped to achieve sampling that includes substantial amount of reference samples close to the boundary while still also including some amount of further away samples which might be able to help build a more robust sample mapping model. The number of columns to skip may be determined exponentially or linearly or in another selected fashion. For example, each time a new column of samples in included as reference samples the gap between the columns may be doubled. Alternatively, a column index for columns included as source for reference samples may be calculated using an exponential or linear formula. For example, if the first column of samples used for collecting reference samples is having horizontal coordinate x_min, the columns with the following x coordinates may be included in the reference sample collection process in addition to the one with x = x_min: ^ = ^_^^^ + 2^, ^^^ ^ > 0 ^^^ ^ <= ^_^^^

[0159] In an embodiment, the threshold T can vary depending on the PU, CU, CTU, slice, frame or sequence properties.

[0160] In an embodiment the threshold T can be fixed, signaled or be inferred based on the PU, CU, CTU, slice, frame or sequence properties.

[0161] In an embodiment the number of zones in the method according to the third embodiment can be fixed, signaled or be inferred based on the PU, CU, CTU, slice, frame or sequence properties.

[0162] In an embodiment based on the method according to the third embodiment the number of zones can be derived as a function of H and W.

[0163] In an embodiment based on the method according to the third embodiment the geometry (i.e., shape) of zones can be derived as a function of H and W.

[0164] In an embodiment the overshoot parameters Px, Py of the method according to the fifth embodiment can be fixed, signaled or be inferred based on the PU, CU, CTU, slice, frame or sequence properties.

[0165] The method according to an embodiment is shown in Figure 11. The method generally comprises processing 1110 an image to predict a block according to prediction samples of two prediction blocks; determining 1120 when both of the two prediction blocks for the block are within boundaries of the image, and applying a bi- prediction based on said two prediction blocks for the block resulting in a bi-prediction block; determining 1130 when one of the two prediction blocks is at least partially out of the image’s boundaries, and applying a uni-prediction based on the prediction blockwithin image boundaries resulting in a uni-prediction block; deriving 1140 filter coefficients based on selected set of samples of the bi-prediction block, wherein the selected set of samples comprises samples that have been subsampled in the bi- prediction block according to a subsampling method; and filtering 1150 at least a part of the uni-prediction block according to the filter coefficients to generate a filtered uni- prediction block. Each of the steps can be implemented by a respective module of a computer system.

[0166] An apparatus according to an embodiment comprises means for processing an image to predict a block according to prediction samples of two prediction blocks; means for determining when both of the two prediction blocks for the block are within boundaries of the image, and means for applying a bi-prediction based on said two prediction blocks for the block resulting in a bi-prediction block; means for determining when one of the two prediction blocks is at least partially out of the image’s boundaries, and means for applying a uni-prediction based on the prediction block within image boundaries resulting in a uni-prediction block; means for deriving filter coefficients based on selected set of samples of the bi-prediction block, wherein the selected set of samples comprises samples that have been subsampled in the bi- prediction block according to a subsampling method; and means for filtering at least a part of the uni-prediction block according to the filter coefficients to generate a filtered uni-prediction block. The means comprises at least one processor, and a memory including a computer program code, wherein the processor may further comprise processor circuitry. The memory and the computer program code are configured to, with the at least one processor, cause the apparatus to perform the method of Figure 11 according to various embodiments.

[0167] An example of a data processing system for an apparatus is illustrated in Figure 12. Several functionalities can be carried out with a single physical device, e.g., all calculation procedures can be performed in a single processor if desired. The data processing system comprises a main processing unit 100, a memory 102, a storage device 104, an input device 106, an output device 108, and a graphics subsystem 110, which are all connected to each other via a data bus 112.

[0168] The main processing unit 100 is a conventional processing unit arranged to process data within the data processing system. The main processing unit 100 may comprise or be implemented as one or more processors or processor circuitry. The memory 102, the storage device 104, the input device 106, and the output device 108 may include conventional components as recognized by those skilled in the art. The memory 102 and storage device 104 store data in the data processing system 100.

[0169] Computer program code resides in the memory 102 for implementing, for example, a method as illustrated in a flowchart of Figure 14 according to various embodiments. The input device 106 inputs data into the system while the output device 108 receives data from the data processing system and forwards the data, for example to a display. The data bus 112 is a conventional data bus and while shown as a single line it may be any combination of the following: a processor bus, a PCI bus, a graphical bus, an ISA bus. Accordingly, a skilled person readily recognizes that the apparatus may be any data processing device, such as a computer device, a personal computer, a server computer, a mobile phone, a smart phone, or an Internet access device, for example Internet tablet computer.

[0170] Figure 13 illustrates an example of a video encoder, where In: Image to be encoded; P’n: Predicted representation of an image block; Dn: Prediction error signal; D’n: Reconstructed prediction error signal; I’n: Preliminary reconstructed image; R’n: Final reconstructed image ; T, T-1: Transform and inverse transform; Q, Q-1: Quantization and inverse quantization; E: Entropy encoding; RFM: Reference frame memory; Pinter: Inter prediction; Pintra: Intra prediction; MS: Mode selection; F: Filtering. Figure 14 illustrates a block diagram of a video decoder where P’n: Predicted representation of an image block; D’n: Reconstructed prediction error signal; I’n: Preliminary reconstructed image; R’ -1 n: Final reconstructed image; T : Inverse transform; Q-1: Inverse quantization; E-1: Entropy decoding; RFM: Reference frame memory; P: Prediction (either inter or intra); F: Filtering. An apparatus according to an embodiment may comprise only an encoder or a decoder, or both.

[0171] The various embodiments can be implemented with the help of computer program code that resides in a memory and causes the relevant apparatuses to carry out the method. For example, a device may comprise circuitry and electronics for handling, receiving, and transmitting data, computer program code in a memory, and a processor that, when running the computer program code, causes the device to carry out the features of an embodiment. Yet further, a network device like a server may comprise circuitry and electronics for handling, receiving, and transmitting data, computer program code in a memory, and a processor that, when running the computer program code, causes the network device to carry out the features of various embodiment.

[0172] If desired, the different functions discussed herein may be performed in a different order and / or concurrently with other. Furthermore, if desired, one or more of the above-described functions and embodiments may be optional or may be combined.

[0173] Although various aspects of the embodiments are set out in the independent claims, other aspects comprise other combinations of features from the describedembodiments and / or the dependent claims with the features of the independent claims, and not solely the combinations explicitly set out in the claims.

[0174] It is also noted herein that while the above describes example embodiments, these descriptions should not be viewed in a limiting sense. Rather, there are several variations and modifications, which may be made without departing from the scope of the present disclosure as, defined in the appended claims.

Claims

Claims:

1. An apparatus for encoding, comprising: means for processing an image to predict a block according to prediction samples of two prediction blocks; means for determining when both of the two prediction blocks for the block are within boundaries of the image, and means for applying a bi- prediction based on said two prediction blocks for the block resulting in a bi-prediction block; means for determining when one of the two prediction blocks is at least partially out of the image’s boundaries, and means for applying a uni- prediction based on the prediction block within image boundaries resulting in a uni-prediction block; means for deriving filter coefficients based on selected set of samples of the bi-prediction block, wherein the selected set of samples comprises samples that have been subsampled in the bi-prediction block according to a subsampling method; and means for filtering at least a part of the uni-prediction block according to the filter coefficients to generate a filtered uni-prediction block.

2. The apparatus according to claim 1, wherein the subsampling method comprises selecting evenly spaced samples within the bi-prediction block with predefined intervals.

3. The apparatus according to claim 1, wherein the subsampling method comprises selecting evenly spaced samples within the bi-prediction block so that the total number of the samples does not exceed a predefined threshold.

4. The apparatus according to claim 1, wherein the subsampling method comprises dividing the bi-prediction block into a plurality of zones, and selecting samples with different intervals within each zone.

5. The apparatus according to claim 1, wherein the subsampling method comprises selecting fully or partially samples included in prediction sample derivation.

6. The apparatus according to claim 1, wherein the subsampling method comprises selecting samples based on subsampling matrices.

7. The apparatus according to claim 1, wherein the subsampling method comprises selecting samples with different intervals in different directions.

8. The apparatus according to claim 3, wherein a value of the predefined threshold depends on a prediction unit (PU), a coding unit (CU), a coding tree unit (CTU), a slice, a frame or sequence properties.

9. The apparatus according to claim 4, wherein the number of zones is based on a prediction unit (PU), a coding unit (CU), a coding tree unit (CTU), a slice, a frame or sequence properties.

10. The apparatus according to claim 4 or 9, further comprising means for deriving a geometry of a zone as a function of height and width of the bi-prediction block.

11. The apparatus according to claim 6, further comprising means for defining overshoot parameters to control the number of samples.

12. The apparatus according to claim 11, wherein the overshoot parameters are based on a prediction unit (PU), a coding unit (CU), a coding tree unit (CTU), a slice, a frame or sequence properties.

13. A method comprising: processing an image to predict a block according to prediction samples of two prediction blocks; determining when both of the two prediction blocks for the block are within boundaries of the image, and applying a bi-prediction based on said two prediction blocks for the block resulting in a bi-prediction block; determining when one of the two prediction blocks is at least partially out of the image’s boundaries, and applying a uni-prediction based on the prediction block within image boundaries resulting in a uni- prediction block;deriving filter coefficients based on selected set of samples of the bi- prediction block, wherein the selected set of samples comprises samples that have been subsampled in the bi-prediction block according to a subsampling method; and filtering at least a part of the uni-prediction block according to the filter coefficients to generate a filtered uni-prediction block.

14. The method according to claim 13, wherein the subsampling method comprises selecting evenly spaced samples within the bi-prediction block with predefined intervals.

15. The method according to claim 13, wherein the subsampling method comprises selecting evenly spaced samples within the bi-prediction block so that the total number of the samples does not exceed a predefined threshold.

16. The method according to claim 13, wherein the subsampling method comprises dividing the bi-prediction block into a plurality of zones, and selecting samples with different intervals within each zone.

17. The method according to claim 13, wherein the subsampling method comprises selecting fully or partially samples included in prediction sample derivation.

18. The method according to claim 13, wherein the subsampling method comprises selecting samples based on subsampling matrices.

19. The method according to claim 13, wherein the subsampling method comprises selecting samples with different intervals in different directions.

20. The method according to claim 15, wherein a value of the predefined threshold depends on a prediction unit (PU), a coding unit (CU), a coding tree unit (CTU), a slice, a frame or sequence properties.

21. The method according to claim 16, wherein the number of zones is based on a prediction unit (PU), a coding unit (CU), a coding tree unit (CTU), a slice, a frame or sequence properties.

22. The method according to claim 16 or 21, further comprising deriving a geometry of a zone as a function of height and width of the bi-prediction block.

23. The method according to claim 18, further comprising defining overshoot parameters to control the number of samples.

24. The method according to claim 23, wherein the overshoot parameters are based on a prediction unit (PU), a coding unit (CU), a coding tree unit (CTU), a slice, a frame or sequence properties.

25. An apparatus comprising at least one processor, memory including computer program code, the memory and the computer program code configured to, with the at least one processor, cause the apparatus to perform at least the following: process an image to predict a block according to prediction samples of two prediction blocks; determine when both prediction blocks of the two prediction blocks for the block are within boundaries of the image, and apply a bi-prediction based on said two prediction blocks for the block resulting in a bi- prediction block; determine when one of the two prediction blocks is at least partially out of the image’s boundaries, and apply a uni-prediction based on the prediction block within image boundaries resulting in a uni-prediction block; derive filter coefficients based on selected set of samples of the bi- prediction block, wherein the selected set of samples comprises samples that have been subsampled in the bi-prediction block according to a subsampling method; and filter at least a part of the uni-prediction block according to the filter coefficients to generate a filtered uni-prediction block.

26. The apparatus according to claim 25, wherein the subsampling method comprises selecting evenly spaced samples within the bi-prediction block with predefined intervals.

27. The apparatus according to claim 25, wherein the subsampling method comprises selecting evenly spaced samples within the bi-prediction block so that the total number of the samples does not exceed a predefined threshold.

28. The apparatus according to claim 25, wherein the subsampling method comprises dividing the bi-prediction block into a plurality of zones, and selecting samples with different intervals within each zone.

29. The apparatus according to claim 25, wherein the subsampling method comprises selecting fully or partially samples included in prediction sample derivation.

30. The apparatus according to claim 25, wherein the subsampling method comprises selecting samples based on subsampling matrices.

31. The apparatus according to claim 25, wherein the subsampling method comprises selecting samples with different intervals in different directions.

32. The apparatus according to claim 27, wherein a value of the predefined threshold depends on a prediction unit (PU), a coding unit (CU), a coding tree unit (CTU), a slice, a frame or sequence properties.

33. The apparatus according to claim 28, wherein the number of zones is based on a prediction unit (PU), a coding unit (CU), a coding tree unit (CTU), a slice, a frame or sequence properties.

34. The apparatus according to claim 28 or 33, further comprising computer program code to cause the apparatus to derive a geometry of a zone as a function of height and width of the bi-prediction block.

35. The apparatus according to claim 30, further comprising computer program code to cause the apparatus to define overshoot parameters to control the number of samples.

36. The apparatus according to claim 35, wherein the overshoot parameters are based on a prediction unit (PU), a coding unit (CU), a coding tree unit (CTU), a slice, a frame or sequence properties.

Images