Method, apparatus and computer program product for video encoding and decoding

Abstract

The embodiment relates to a method and a technical device for implementing the method. The method comprises: processing (1010) a video frame to predict a block according to a prediction sample; determining (1020) when two prediction blocks (910, 920) for a block are within a boundary of an image, and generating a bi-prediction block (952) based on applying bi-prediction for the prediction blocks for a block; determining (1030) when one of the prediction blocks is at least partially outside the edge of the image, and generating a uni-prediction block (951) by applying uni-prediction based on the prediction block within the image boundary; deriving (1040) filter coefficients by minimizing a square error between the bidirectional prediction block and the unidirectional prediction block; at least a portion of the uni-prediction block (951) is filtered (1050) according to the filter coefficients to generate a filtered uni-prediction block. The embodiment also considers technical equipment for implementing the method.

Description

Technical Field

[0001] The embodiments of this disclosure generally relate to video encoding and video decoding. Specifically, this solution relates to methods, apparatus, and computer program products for video encoding and decoding. Background Technology

[0002] This section is intended to provide background or context for the invention as described in the claims. The description herein may include concepts that may be pursued, but not necessarily concepts that have been previously conceived or pursued. Therefore, unless otherwise stated herein, what is described in this section is not prior art to the description and claims of this application, and is not acknowledged as prior art by virtue of its inclusion in this section.

[0003] A video encoding system may include an encoder that converts input video into a compressed representation suitable for storage / transmission, and a decoder that can decompress the compressed video representation back into a visual form. The encoder may discard some information from the original video sequence to represent the video in a more compact form, for example, enabling the storage / transmission of video information at a lower bit rate than might otherwise be required. Summary of the Invention

[0004] The scope of protection sought by the various embodiments of the present invention is defined by the independent claims. Embodiments and features (if any) described in this specification that do not fall within the scope of the independent claims are to be construed as examples useful for understanding the various embodiments of the invention.

[0005] Various aspects include methods, apparatus, and computer-readable media including computer programs stored therein, characterized as set forth in the independent claims. Various embodiments are disclosed in the independent claims.

[0006] According to a first aspect, an apparatus is provided, comprising components for: processing video frames to predict blocks based on prediction samples; determining when two predicted blocks for a block are within the boundaries of an image, and generating a bidirectional predicted block by applying bidirectional prediction based on the predicted blocks for the block; determining when one of the predicted blocks is at least partially outside the boundaries of the image, and generating a unidirectional predicted block by applying unidirectional prediction based on the predicted blocks within the image boundaries; deriving filter coefficients by minimizing the squared error between the bidirectional and unidirectional predicted blocks; and filtering at least a portion of the unidirectional predicted blocks based on the filter coefficients to generate a filtered unidirectional predicted block.

[0007] According to a second aspect, a method is provided, comprising: processing a video frame to predict blocks based on prediction samples; determining when two predicted blocks for a block are within the boundaries of an image, and applying bidirectional prediction based on the predicted blocks for the block to generate a bidirectional predicted block; determining when at least one of the predicted blocks is at least partially outside the boundaries of the image, and applying unidirectional prediction based on the predicted blocks within the image boundaries to generate a unidirectional predicted block; deriving filter coefficients by minimizing the squared error between the bidirectional and unidirectional predicted blocks; and filtering at least a portion of the unidirectional predicted block based on the filter coefficients to generate a filtered unidirectional predicted block.

[0008] According to a third aspect, an apparatus is provided, comprising: at least one processor and a memory including computer program code configured, together with the at least one processor, such that the apparatus performs at least the following: processing video frames to predict blocks based on prediction samples; determining when two predicted blocks for a block are within the boundaries of an image, and applying bidirectional prediction based on the predicted blocks for the block to generate a bidirectional predicted block; determining when one of the predicted blocks is at least partially outside the boundaries of the image, and applying unidirectional prediction based on the predicted blocks within the image boundaries to generate a unidirectional predicted block; deriving filter coefficients by minimizing the squared error between the bidirectional and unidirectional predicted blocks; and filtering at least a portion of the unidirectional predicted block based on the filter coefficients to generate a filtered unidirectional predicted block.

[0009] According to the fourth aspect, a computer program product including computer program code is provided, the computer program code being configured to, when executed on at least one processor, cause an apparatus or system to: process video frames to predict blocks based on prediction samples; determine when two predicted blocks for a block are within the boundaries of an image, and apply bidirectional prediction based on the predicted blocks for the block to generate a bidirectional predicted block; determine when one of the predicted blocks is at least partially within the boundaries of an image, and apply unidirectional prediction based on the predicted block within the image boundaries to generate a unidirectional predicted block; derive filter coefficients by minimizing the squared error between the bidirectional and unidirectional predicted blocks; and filter at least a portion of the unidirectional predicted block according to the filter coefficients to generate a filtered unidirectional predicted block.

[0010] According to an embodiment, the use of filtering is indicated in the samples of a one-way prediction block or a two-way prediction block.

[0011] According to an embodiment, the filter includes a nonlinear term.

[0012] According to an embodiment, the filter coefficients are derived for a sample set that belongs to the overlapping region of the bidirectional prediction blocks.

[0013] According to an embodiment, more than one filter is derived, wherein each filter is derived using its own region and is applied to its own region on a unidirectional prediction block.

[0014] According to an embodiment, a region is indicated in which filter coefficients are derived in samples of a one-way prediction block or in a two-way prediction block.

[0015] According to an embodiment, a region is signaled, and the filter coefficients in that region are derived at the coding unit, prediction unit, coding tree unit, slice, and sequence level.

[0016] According to an embodiment, the computer program product is embodied on a non-transitory computer-readable medium. Attached Figure Description

[0017] In the following description, various embodiments will be depicted in more detail with reference to the accompanying drawings, in which:

[0018] Figure 1 This shows an example of the positions of the left and top samples of the current block involved in CCLM mode.

[0019] Figure 2a An example is shown of deriving the chromaticity prediction mode from the luminance mode when CCLM is enabled.

[0020] Figure 2b An example of a unified binary table for chromaticity prediction modes is shown.

[0021] Figure 3a Two luminance-to-chromaticity models are shown for a luminance Y threshold of 17.

[0022] Figure 3b An example is shown that corresponds to each luminance-to-chrominance model and the spatial segmentation of the content.

[0023] Figure 4 An example of the location of the sample used for the derivation of the CCCM filter is shown.

[0024] Figure 5 Examples of various filter kernels are shown;

[0025] Figure 6 An example of four reference lines adjacent to the prediction block is shown.

[0026] Figure 7 An example of a matrix-weighted intra-frame prediction process is shown.

[0027] Figure 8 An example of the top and left neighbor blocks used in the CIIP weight derivation is shown.

[0028] Figure 9 An example of a bidirectional prediction block is shown.

[0029] Figure 10 Examples of overlapping and non-overlapping portions are shown.

[0030] Figure 11 An example of filter coefficients is shown;

[0031] Figure 12 Examples of two filter coefficient sets are shown;

[0032] Figure 13 This is a flowchart illustrating a method according to an embodiment.

[0033] Figure 14 An apparatus according to an embodiment is shown.

[0034] Figure 15 The encoding process according to an embodiment is shown.

[0035] Figure 16 The decoding process according to an embodiment is shown. Detailed Implementation

[0036] The following description and accompanying drawings are illustrative and should not be construed as unnecessarily limiting. Specific details are provided for a thorough understanding of this disclosure. However, in some instances, well-known or conventional details have not been described to avoid obscuring the meaning. References to one embodiment or embodiments in this disclosure may be (but are not necessarily) references to the same embodiment, and such references imply at least one embodiment of the embodiment.

[0037] References to “an embodiment” or “an embodiment” in this application mean that a particular feature, structure, or characteristic described in connection with an embodiment is included in at least one embodiment of this disclosure.

[0038] In the following description, several embodiments will be presented in the context of a video coding arrangement. However, it should be noted that these embodiments are not necessarily limited to that particular arrangement. The embodiments relate to adjustments for bidirectional prediction based on error minimization in inter-frame prediction.

[0039] The Advanced Video Coding Standard (which may be abbreviated as AVE or H.265 / AVC) was developed by the Video Coding Experts Group (VCEG) of the Telecommunication Standardization Sector of the International Telecommunication Union (ITU-T) and the Joint Video Team (JVT) of the Moving Picture Experts Group (MPEG) of the International Organization for Standardization (ISO) / International Electrotechnical Commission (IEC). The H.264 / AVC standard was published by both parent standardization organizations and is known as ITU-T Recommendation H.265 and ISO / IEC International Standard 14496-10, also referred to as MPEG-4 Part 10 Advanced Video Coding (AVC). Multiple versions of the H.264 / AVC standard exist, each integrating new extensions or features into the specification. These extensions include Scalable Video Coding (SVC) and Multi-View Video Coding (MVC).

[0040] The High Efficiency Video Coding Standard (which may be abbreviated as HEVC or H.265 / HEVC) was developed by the joint collaborative team of VCEG and MPEG – Video Coding (JCT-C). This standard was published by both parent standards organizations and is known as ITU-T Recommendation H.265 and ISO / IEC International Standard 23008-2, also referred to as MPEG-H Part 2 High Efficiency Video Coding (HEVC). Extensions to H.265 / HEVC include Scalable, Multi-View, 3D, and Fidelity Range Extensions, which may be referred to as SHVC, MV-HEVC, 3D-HEVC, and REXT, respectively. References to H.265 / HEVC, SHVC, MV-HEVC, 3D-HEVC, and REXT in this description are made for the purpose of understanding the definitions, structure, or concepts of these standard specifications and, unless otherwise stated, should be construed as references to the most recent versions of these standards available prior to the date of this application.

[0041] Universal Video Coding (which may be abbreviated as VVC, H.266, or H.266 / VVC) is a video compression standard developed as the successor to HEVC. VVC is specified in ITU-T Recommendation H.266 and is equivalently specified in ISO / IEC 23090-3 (also known as MPEG-I Part 3).

[0042] The AV1 bitstream format and decoding process specification was developed by the Alliance for Open Media (AOM). The AV1 specification was released in 2018. According to reports, AOM is working on the AV2 specification.

[0043] Key definitions, bitstreams, and encoding structures of H.264 / AVC, HEVC, VVC, and / or AV1 and their extensions, as well as concepts described in this section, are examples of video encoders, decoders, encoding methods, decoding methods, and bitstream structures on which embodiments can be implemented. Aspects of the various embodiments are not limited to H.265 / AVC, HEVC, VVC, and / or AV1 and their extensions, but are described with respect to a possible basis on which these embodiments can be implemented partially or completely.

[0044] A video codec can include an encoder that converts input video into a compressed representation suitable for storage / transmission, and a decoder that decompresses the compressed video representation back into a visual form. The compressed representation can be referred to as a bitstream or video bitstream. Video encoders and / or video decoders can also be separate from each other; that is, they do not need to form a codec. An encoder can discard some information from the original video sequence to represent the video in a more compact form (i.e., at a lower bit rate). The symbol "(de)coder" means both encoder and / or decoder.

[0045] In some video codecs, such as H.265 / HEVC, video frames are divided into coding units (CUs) covering a region of the frame. A CU consists of one or more prediction units (PUs) and one or more transform units (TUs). The prediction units (PUs) define the prediction process for samples within the CU, and the transform units (TUs) define the prediction error coding process for samples within the CU. A CU can consist of square blocks of samples with sizes selectable from a predefined set of possible CU sizes. The CU with the largest allowed size can be referred to as an LCU (Maximum Coding Unit) or a CTU (Coding Tree Unit), and video frames are divided into non-overlapping CTUs. For example, a CTU can also be subdivided into combinations of smaller CUs by recursively segmenting the CTUs and the resulting CUs. Each resulting CU can have at least one associated PU and at least one associated TU. Each PU and TU can be further subdivided into smaller PUs and TUs to increase the granularity of the prediction and prediction error coding processes, respectively. Each PU has associated prediction information that defines which type of prediction is applied to the pixels within that PU (e.g., motion vector information for inter-frame prediction PUs and intra-frame prediction directionality information for intra-frame prediction PUs). Similarly, each TU is associated with information describing the prediction error decoding process for samples within that TU (including, for example, DCT coefficient information). Whether prediction error coding is applied for each CU can be signaled at the CU level. If no prediction error residual is associated with a CU, it can be assumed that no TU exists for that CU. The partitioning of the image to CUs, and the partitioning of CUs to PUs and PUs, can be signaled in a bitstream that allows the decoder to regenerate the expected structure of these units.

[0046] Hybrid video codecs (e.g., ITU-T H.263, H.264 / AVC, and HEVC) can encode video information in two stages. First, pixel values ​​in a picture region (or “block”) are predicted, for example, by a motion compensation component (which finds and indicates a region in a previously encoded video frame that closely corresponds to the block being encoded), or by a spatial component (using pixel values ​​around the block to be encoded in a specified manner). In this first stage, predictive coding can be applied, for example, so-called sample prediction and / or so-called syntax prediction.

[0047] In sample prediction, pixel or sample values ​​within a given image region or "block" are predicted. For example, these pixel or sample values ​​can be predicted using one or more of a motion compensation mechanism or an intra-frame prediction mechanism.

[0048] Motion compensation mechanisms (also known as inter-frame prediction, provisional prediction, or motion-compensated provisional prediction or motion-compensated prediction, or MCP) involve finding and indicating regions in a video frame that closely correspond to the previously encoded video frame of the block being encoded. Inter-frame prediction can reduce provisional redundancy.

[0049] Intra-frame prediction (where pixel or sample values ​​can be predicted by spatial mechanisms) involves finding and indicating spatial region relationships. Intra-frame prediction leverages the fact that neighboring pixels within the same image may be related. Intra-frame prediction can be performed in the spatial domain or the transform domain; that is, sample values ​​or transform coefficients can be predicted. Intra-frame prediction is typically used in intra-frame coding, where inter-frame prediction is not applied.

[0050] In syntax prediction (which may also be called parametric prediction), syntax elements and / or syntax element values ​​and / or variables derived from syntax elements are predicted based on earlier (de)encoded syntax elements and / or earlier derived variables. Non-restrictive examples of syntax prediction are provided below.

[0051] In motion vector prediction (e.g., for inter-frame and / or inter-view prediction), motion vectors can be differentially encoded relative to the predicted motion vectors of a specific block. In many video codecs, predicted motion vectors are created in a predefined manner, such as by calculating the median of the encoded or decoded motion vectors of neighboring blocks. Another way to create motion vector predictions (sometimes called Advanced Motion Vector Prediction (AMVP)) is to generate a list of candidate predictions from neighboring and / or co-occurring blocks in a temporary reference picture and signal the selected candidate as the motion vector predictor. In addition to the predicted motion vector values, reference indices of previously encoded / decoded pictures can be predicted. Reference indices are typically predicted based on neighboring and / or co-occurring blocks in a temporary reference picture. Differential encoding of motion vectors is typically disabled across slice boundaries.

[0052] Block partitioning can be predicted, for example, from CTU to CU and down to PU.

[0053] In filter parameter prediction, (e.g., for sample adaptive offset) filter parameters can be predicted.

[0054] Prediction schemes that use image information based on previously encoded images can also be called inter-frame prediction methods, or temporary prediction and motion compensation.

[0055] Prediction schemes that use information from the same image can also be called intra-frame prediction methods.

[0056] Secondly, the prediction error (i.e., the difference between the predicted block of a pixel and the original block of the pixel) is encoded. This can be done by transforming the differences in pixel values, the quantization coefficients, and the entropy-encoded quantized coefficients using a specific transform (e.g., Discrete Cosine Transform (DCT) or a variant thereof). By varying the fidelity of the quantization process, the encoder can control the balance between the accuracy of the pixel representation (image quality) and the size of the resulting encoded video representation (file size at the transmission bitrate).

[0057] The basic units used for the encoder's input and the decoder's output are, in most cases, images. The image given as the encoder's input can also be called the source image, and the image decoded by the decoder can be called the decoded image or the reconstructed image.

[0058] The source image and the decoded image each consist of one or more sample arrays, such as one of the following sample array sets: - Luminance (Y) only (monochrome). - Luminance and two chromaticities (YCbCR or YCgCO). - Green, blue, and red (GBR, also known as RGB). - Represents an array of other unspecified monochromatic or trichromatic stimulus color samples (e.g., YZX, also known as XYZ).

[0059] In the following text, these arrays may be referred to as luminance (or L or Y) and chrominance, where the two chrominance arrays may be referred to as Cb and Cr; regardless of the actual color representation used. The actual color representation used may be indicated, for example, in the encoded bitstream, using HEVC's Video Availability Information (VUI) or a similar method. A component may be defined as an array or a single sample of one of the three sample arrays (luminance and two chrominance), or an array or a single sample of an array constituting an image in monochrome format.

[0060] An image can be defined as a frame or a field. A frame consists of a matrix of luminance samples and possibly corresponding chrominance samples. A field is a set of alternating sample rows of a frame and can be used as encoder input when the source signal is interlaced. The chrominance sample array can be missing (and therefore monochrome sampling can be used), or the chrominance sample array can be subsampled compared to the luminance sample array.

[0061] The decoder reconstructs the output video by applying a prediction component, similar to that of an encoder, to form a predictive representation of pixel blocks (using motion or spatial information created by the encoder and stored in the compressed representation), and prediction error decoding (the inverse operation of prediction error encoding, which recovers the quantized prediction error signal in the spatial pixel domain). After applying the prediction and prediction error decoding components, the decoder adds the prediction and prediction error signals (pixel values) to form the output video frame. The decoder (and encoder) may also apply additional filtering components to improve the quality of the output video before transmitting it for display and / or storing it as a predictive reference for upcoming frames in a video sequence.

[0062] Motion information can be indicated using motion vectors associated with each motion-compensated image block in the video codec. Each of these motion vectors represents the displacement of an image block in the picture to be encoded (on the encoder side) or decoded (on the decoder side), as well as a predicted resource block in one of the previously encoded or decoded pictures. As with many other video compression standards, H.264 / AVC and HEVC divide the picture into a rectangular grid, and for each rectangular grid, a similar block in a reference picture is indicated for inter-frame prediction. The position of the predicted block is encoded as a motion vector that indicates the position of the predicted block relative to the block being encoded.

[0063] To efficiently represent motion vectors, these motion vectors can be differentially encoded relative to block-specific predicted motion vectors. Predicted motion vectors can be created in a predefined manner, for example, by calculating the median of the encoded or decoded motion vectors of neighboring blocks. Another way to create motion vector predictions is to generate a list of candidate predictions from neighboring and / or co-occurring blocks in a temporary reference image, and signal the selected candidate as the motion vector predictor. In addition to predicting motion vector values, reference indices of previously encoded / decoded images can be predicted. Reference indices can be predicted based on neighboring and / or co-occurring blocks in the temporary reference image. Furthermore, efficient video codecs can employ an additional motion information encoding / decoding mechanism, often referred to as a fusion / merging mode, where all motion field information, including motion vectors and corresponding reference image indices for each available list of reference images, is predicted and used without any modification / correction. Similarly, motion field information prediction is performed using motion field information from neighboring and / or co-occurring blocks in a temporary reference image, and the motion field information used is signaled within a list of motion field candidates that populates the list of available neighboring / co-occurring block motion field information.

[0064] Like many other video compression standards, H.264 / AVC and HEVC divide images into rectangular grids. For each rectangular grid, a similar block from a reference image is indicated for inter-frame prediction. The position of the predicted block is encoded as a motion vector, which indicates the position of the predicted block relative to the block being encoded.

[0065] A bitstream can be defined as a sequence of bits or a sequence of syntactic structures. The bitstream format can constrain the order of the syntactic structures in the bitstream.

[0066] Syntax elements can be defined as elements of data represented in a bitstream. Syntax structures can be defined as zero or more syntax elements that exist together in a bitstream in a specified order.

[0067] In some encoding formats or standards, bitstreams can be in the form of Network Abstraction Layer (NAL) unit streams or byte streams, which form a representation of encoded images and associated data that forms one or more encoded video sequences.

[0068] A NAL unit can be defined as a syntax structure containing: an indication of the data type to be followed, and bytes containing data in the form of an RSRP, which is distributed as needed with start code emulation stop bytes. The Raw Byte Sequence Payload (RSRP) can be defined as a syntax structure containing an integer number of bytes encapsulated within NAL units. An RBSP can be either empty or a string of data bits containing syntax elements, followed by RSRP stop bits and zero or more subsequent bits equal to 0.

[0069] A NAL cell includes a header and a payload. Among other things, the NAL cell header indicates the type of the NAL cell.

[0070] In some encoding formats (such as AV1), a bitstream may include a sequence of Open Bitstream Units (OBUs). An OBU includes a header and a payload, where the header identifies the type of OBU. Furthermore, the header may include the size of the payload in bytes.

[0071] The phrase "along the bitstream" (e.g., indicating along the bitstream) or "encoded unit along the bitstream" (e.g., indicating along an encoded tile) can be used in the claims and described embodiments to refer to transmission, signaling, or storage, respectively, in a manner that associates with, but is not included within, a unit of the bitstream or encoded data in the "out-of-band" data field. The phrase "decode along the bitstream," or "decode an encoded unit along the bitstream," or similar phrases can refer to decoding referenced out-of-band data (which may be obtained from out-of-band transmission, signaling, or storage), which is associated with a unit of the bitstream or encoded data, respectively. For example, the phrase "along the bitstream" can be used when the bitstream is contained in a container file (such as a file conforming to the ISO Basic Media File Format), and some file metadata is stored in the file in a manner that associates metadata with the bitstream, such as a box in a sample entry for a track containing the bitstream, a sample group for a track containing the bitstream, or a timing metadata track associated with a track containing the bitstream.

[0072] The video codec can support motion-compensated prediction from one source image (unidirectional prediction) and two sources (bidirectional prediction). In the case of unidirectional prediction, a single motion vector is applied, while in the case of bidirectional prediction, two motion vectors are signaled, and the motion-compensated predictions from the two sources are averaged to create the final sample prediction. In the case of weighted prediction, the correlation weights of the two predictions can be adjusted, or the offset signaled by the used signal can be added to the prediction signal.

[0073] Besides applying motion compensation for inter-frame image prediction, similar schemes can be applied to intra-frame image prediction. In this case, the displacement vector indicates where sample blocks can be copied from the same image to form the predicted blocks to be encoded or decoded. This type of intra-frame block copying method can significantly improve coding efficiency when there are repeating structures (such as text or other graphics) within the frame.

[0074] The prediction residuals after motion compensation or intra-frame prediction can first be transformed using a transform kernel (such as DCT) and then encoded. The reason for this is that there is usually still some correlation in the residuals, and the transform can help reduce this correlation and provide more efficient encoding in many cases.

[0075] Video encoders can use a Lagrange cost function to find the optimal encoding pattern (e.g., the desired macroblock pattern and associated motion vectors). This type of cost function uses a weighting factor λ to correlate the (exact or estimated) image distortion attributable to lossy coding methods with the (exact or estimated) amount of information needed to represent pixel values ​​in image regions. C = D + λR (Eq. 1) in C The Lagrange cost to be minimized. D It refers to image distortion (e.g., mean square error) under the consideration of the pattern and motion vectors, and R This represents the number of bits required for the data needed to reconstruct the image block in the decoder (including the amount of data representing candidate motion vectors).

[0076] The features and coding tools included in VVC include the following: Intra-frame prediction - 67-frame intra-mode with wide-angle mode extension - 4-tap interpolation filter that depends on block size and mode - Location-dependent intra-prediction combination (PDPC) - Cross-component linear model intra-frame prediction (CCLM) - Multi-reference line intra-frame prediction - Intra-frame sub-partition - Weighted intra-frame prediction with matrix multiplication Image prediction - Block motion replication with spatial, temporary, history-based, and distributed average fusion candidates - Affine motion inter-frame prediction - Sub-block-based temporary motion vector prediction - Adaptive motion vector resolution - 8x8 block-based motion compensation for temporary motion prediction - High-precision (1 / 16 pixel) motion vector storage and motion compensation with an 8-tap interpolation filter for the luma component and a 4-tap interpolation filter for the chroma component. - Triangular partition - Combine intra-frame and inter-frame prediction - Merged with MVD (MMVD) - Symmetric MVD encoding - Bidirectional optical flow - Decoder-side motion vector refinement - Bidirectional prediction with CU-level weights Transformation, quantization, and coefficient encoding - Multiple master transform selection with DCT2, DST7, and DCT8 - Auxiliary transform for low frequency regions - Sub-block transformation for inter-frame prediction residuals - Relevant quantization with a maximum QP increasing from 51 to 63 - Transform coefficient encoding with symbolic data hiding - Transform skip residual coding Entropy coding - Arithmetic coding engine with adaptive dual-window probability updates • In-loop filter - Intra-loop reshaping - Deblocking filter with strong, longer filter length - Sample adaptive offset - Adaptive ring green filter • Screen content encoding: - Current image reference with reference area limitations • 360-degree video encoding: - Horizontal orbital motion compensation • Advanced syntax and parallel processing: Reference image management with direct reference image list signaling - A set of tiles with rectangular shapes

[0077] In H.266 / VVC, the following block partitioning applies. Images can be divided into Coding Tree Units (CTUs). Images can also be divided into slices, tiles, bricks, and sub-images. CTUs can be partitioned into smaller CUs using a quadtree structure. Each CU can be partitioned using quadtrees, as well as nested multi-type trees including ternary and binary partitions.

[0078] There are specific rules for inferring the boundaries of an image.

[0079] Redundant partitioning patterns are not allowed in nested multi-type partitioning.

[0080] Some video coding tools perform filtering operations, convolving a set of reference samples with a set of filter parameters to output a predicted value, for example, for a sample in an image. In some cases, the filter parameters can be predetermined in the bitstream or signaled. In others, such as when cross-component linear model (CCLM) or cross-component convolutional model (CCCM) predictions are used, the parameters are computed using reference sets from both the encoder and decoder. Typically, the computation of such filter parameters involves inverting the autocorrelation matrix, which is computationally challenging. Furthermore, when there are many filter parameters to be determined, the size of the autocorrelation matrix becomes large, which can lead to numerical stability problems (overflow or underflow).

[0081] To reduce cross-component redundancy, a cross-component linear model (CCLM) prediction model can be used in VVC. For VVC, chroma samples are predicted using a linear model based on reconstructed luminance samples from the same CU: in This represents the predicted chromaticity samples in the CU, and This represents downsampled reconstructed luminance samples from the same CU.

[0082] The CCLM parameters (α and β) are derived using at least four neighboring chroma samples and their corresponding downsampled luminance samples. It is assumed that the current chroma block dimension is... ,but and Set as - When the LM pattern is applied ; - When LM-A mode is applied ; - When the LM-L pattern is applied .

[0083] The upper neighbor positions are denoted as S[0, -1]…S[W'-1, -1], and the left neighbor positions are denoted as S[-1, 0]…S[-1, H'-1]. These four samples are then selected as... - When LM mode is applied, and both the upper and left neighboring samples are available, ; - When LM-A mode is applied, or when only the upper neighboring sample is available. ; - When the LM-L mode is applied, or when only the left neighboring sample is available. ;

[0084] Four neighboring luminance samples at the selected location are downsampled and compared four times to find two smaller values: x0A and x1A, and two larger values: x0B and x1B. Their corresponding chromaticity samples are represented as y0A, y1A, y0B, and y1B. Then, xA, xB, yA, and yB are derived as follows:

[0085] Finally, the parameters of the linear model are obtained according to the following formula:

[0086] Figure 1Examples of samples from the left and top sides, as well as the current block, are shown in the CCLM model. The division operation for calculating the parameter α is implemented using a lookup table. To reduce the memory required to store the table, the diff value (the difference between the maximum and minimum values) and the parameter α are expressed using exponent notation. For example, the diff is approximated using a 4-bit significant part and an exponent. Therefore, the table for 1 / diff is reduced to 16 elements for the 16 significant values ​​as follows:

[0087] This will have the benefit of reducing computational complexity and the memory size required to store the necessary tables.

[0088] In addition to the top and left templates being used together to calculate the linear model coefficients (i.e., parameters), they can also be used alternatively for other 2LM modes, referred to as LM_A and LM_L modes.

[0089] In LM_A mode, only the top template is used to calculate the linear model coefficients (i.e., parameters). To obtain more samples, the top template is extended to (W+H). In LM_L mode, only the left template is used to calculate the linear model coefficients (i.e., parameters). To obtain more samples, the left template is extended to (H+W).

[0090] For non-square blocks, the top template is expanded to W+W, and the left template is expanded to H+H.

[0091] To match the chroma sample positions for a 4:2:0 video sequence, two types of downsampling filters are applied to the luminance samples to achieve a 2:1 downsampling ratio in both the horizontal and vertical directions. The selection of the downsampling filters is specified by the SPS level flag. The two downsampling filters are as follows, corresponding to "Type-0" and "Type-2" respectively:

[0092] It will be understood that when the top reference line is at the CTU boundary, only one luminance line (the universal line buffer in intra-frame prediction) is used to create downsampled luminance samples.

[0093] This parameter calculation is performed as part of the decoding process, and not just as part of the encoder's search operation. Therefore, no syntax is used to pass the α and β values ​​to the decoder.

[0094] For chroma intra-frame coding, a total of eight intra-frame modes are allowed. These modes include five traditional intra-frame modes and three cross-component linear model modes (CCLM, LM_A, and LM_L). The chroma mode signaling and derivation process is described in... Figure 2aThe results are shown in Table 1. Chroma mode coding directly depends on the intra-prediction of the corresponding luma block. Since separate block partitioning structures for luma and chroma components are enabled in the I-slice, one chroma block can correspond to multiple luma blocks. Therefore, for chroma DM mode, the intra-prediction mode of the corresponding luma block covering the center position of the current chroma block is directly inherited.

[0095] in spite of Figure 2b The values ​​of `sps_cclm_enabled_flag` shown in Table 2 are used by a single binary table. In Table 2, the first bit indicates whether it is in normal mode (0) or LM mode (1). If it is in LM mode, the next bit indicates whether it is LM_CHROMA (0). If it is not LM_CHROMA, one bit indicates whether it is LM_L (0) or LM_A (1). In this case, when `sps_cclm_enabled_flag` is 0, the first bit of the binary table for the corresponding `intra_chroma_pred_mode` can be discarded before entropy encoding. Or, in other words, the first bit is inferred to be 0 and therefore not encoded. This single binary table is used for both cases where `sps_cclm_enabled_flag` equals 0 and 1. The first two bits in Tables 3-4 are content-encoded using their own content model, and the remaining bits are bypass-encoded.

[0096] Furthermore, to reduce luma-chroma latency in binary trees, when 64x64 luma coding tree nodes are partitioned using no partitioning (and ISP is not used for 64x64 CUs) or QT, chroma CUs in 32x32 / 32x16 chroma coding tree nodes are allowed to use CCLM in the following manner: - If the 32x32 chroma node is not split or partitioned by QT, then all chroma CUs in the 32x32 node can be used with CCLM. - If a 32x32 chroma node is partitioned using horizontal BT, and the 32x17 child node is not partitioned or is partitioned using vertical BT, then all chroma CUs in a 32x16 chroma node can use CCLM. CCLM is not permitted for chroma CU under all other luma and chroma coding tree segmentation conditions.

[0097] The CCLM included in VVC is extended by adding three multi-model LM (MMLM) patterns. In each MMLM pattern, the reconstructed neighbor samples are classified into two classes using a threshold that is the average of the brightness reconstructed neighbor samples. The linear model for each class is derived using the least mean square (LMS) method. Figure 3aThe diagram illustrates two luminance-to-chrominance models obtained for a luminance (Y) threshold of 17. Each luminance-to-chrominance model has its own linear model parameters α and β. (As can be seen from...) Figure 3b It can be seen that each luminance-to-chrominance model corresponds to a spatial segment of the content (i.e., it can correspond to different objects or textures in the scene).

[0098] An improved version of cross-component prediction (called the Convolutional Cross-Component Model (CCCM)) uses a 2D filter kernel to derive the luma-to-chroma model. The filter coefficients are derived on the decoder side using the reconstructed set of input data and chroma samples. For the filter coefficient derivation, a co-located reference sample region (composed of reconstructed luma and chroma samples) is used. Figure 4 The luminance and chrominance shown are defined; however, any number of reference lines (which can be implemented by both the encoder and decoder) can be used. Typically, the reference samples can contain any luminance and chrominance samples that have already been reconstructed by both the encoder and decoder. Once the reference samples have been determined, the filter coefficients can be derived, for example, using different types of linear regression tools, such as ordinary least squares estimation, orthogonal matching pursuit, optimized orthogonal matching pursuit, ridge regression, or minimum absolute shrinkage and selection operators.

[0099] The dimensions of the filter kernel can be, for example, 1x3 (1D vertical), 3x1 (1D horizontal), 3x3, 7x7, or any other dimension, and can be shaped (by selecting only a subset of all possible kernel locations) into a cross, a rhombus, or any given shape. When referring to samples within the filter kernel, the following symbols are used: North (above), East (right), South (below), West (left), and Center, as shown in... Figure 5 The letters N, E, S, W, and C are used in the illustration. Figure 5 The diagram illustrates a 3-tap vertical core 501, a 3-tap horizontal core 502, a 5-tap cross core 503, and a 25-tap diamond core 504.

[0100] The overall method for reconstructing chroma samples using convolutions between the filter kernels obtained from the decoder side and the input dataset is referred to herein as the Convolutional Cross Component Model (CCCM). The following steps can be applied to perform the CCCM operation. - Define co-location reference regions for the luminance and chrominance components; - Downsample luminance samples to match the chromaticity grid (optional); - Scan the luminance and chromaticity samples of the reference region based on the filter shape and collect available statistics (such as autocorrelation matrix and cross-correlation vector). - The filter coefficients are solved by minimizing the squared error (or any other metric) based on available statistics (such as the autocorrelation matrix and cross-correlation vector); - Predicted chromaticity blocks are calculated by convolving downsampled luminance samples with a filter kernel.

[0101] In the following, (possibly downsampled) luminance samples are defined as a 2D array indexed using the horizontal x-coordinate and vertical y-coordinate. Furthermore, isotopic chromaticity samples are defined as 2D arrays. And the filter kernel (i.e., coefficients) is defined as a 3x3 array. At the sample level, the convolution between Y and F is defined as...

[0102] When using other data terms (such as non-linear square root terms), the attached convolution becomes: in These are the filter coefficients residing outside the 2D filter kernel; however, This has already been obtained as part of a system of linear equations used to solve the 2D filter coefficients in step 4 above. Similarly, the bias term can be added to the convolution using the following system of equations.

[0103] Angular intra-frame prediction (aka direct intra-frame prediction) can be performed by extrapolating sample values ​​from reconstructed reference samples using a given orientation. Reference samples can include the directly neighboring sample rows above and to the right of the current block (when available), and the directly neighboring sample column to the left of the current block (when available), where availability may require a decoding order earlier than the current block's decoding order and existing within the same image segment (e.g., within the same tile). To simplify the process, depending on the orientation of the chosen prediction mode, all sample locations within a prediction block can be projected onto a single reference row or column. Predicted samples within the block being encoded / decoded can be obtained through the following steps: - The predicted sample location is projected onto a location within a reference row or column by applying a selected prediction direction. The location within the reference row or column can have fractional sample accuracy, such as 1 / 32 pixel accuracy. - Interpolate the values ​​at the same position in the reference row or column from the reference sample.

[0104] Multiple Reference Lines (MRL) intra-frame prediction uses multiple reference lines for intra-frame prediction. Figure 6In the example, four reference lines are depicted, where the samples for segments A and F are not taken from the reconstructed neighboring samples, but are instead filled with the nearest samples from segments B and E, respectively. Inter-image prediction in HEVC uses the nearest reference line (i.e., reference line 0). In MRL, two additional lines (reference line 1 and reference line 3) are used.

[0105] The index of the selected reference line (mrl_idx) is signaled and used to generate the intra-predictor. For reference lines idx greater than 0, only additional reference line modes from the MPM list are included, and the mpm index is signaled only if no remaining modes are available. The reference line index is signaled before the intra-prediction modes, and Planar modes are excluded from the intra-prediction model if a non-zero reference line index is signaled.

[0106] For the first line of a block within a CTU, MRL is disabled to prevent the use of extended reference samples outside the current CTU line. Additionally, PDPC is disabled when additional lines are used. For MRL mode, the derivation of the DC value in the DC intra-prediction mode for non-zero reference line indices is aligned with reference line index 0. MRL requires storage of three neighboring luma reference lines with the CTU to generate predictions. The Cross Component Linear Model (CCLM) tool also requires three neighboring luma reference lines for its own downsampling filter. The definition of MLR using the same three lines is aligned with CCLM to reduce storage requirements for the decoder.

[0107] Depending on the block size, Intra-Frame Sub-Partitioning (ISP) divides the luma intra-prediction block vertically or horizontally into 2 or 4 sub-partitions. For example, the minimum block size for ISP is 4x8 (or 8x4). If the block size is larger than 4x8 (or 8x4), the corresponding block is divided into 4 sub-partitions. It has been noted that Mx12 (where M ≤ 64) and 128xN (where N ≤ 64) ISP blocks can generate potential problems with a 64x64 VDPU. For example, and in the case of a single tree, an Mx128 CU has M x 128 luma TB, and two corresponding... Chroma TB. If the CU uses a SIP, the luminance TB will be divided into four Mx32 TBs (horizontal division is possible only), each of which is smaller than a 64 x 64 block. However, in the current design of the ISP, the chroma blocks are not divided. Therefore, both chroma components will have a size larger than a 32 x 32 block.

[0108] Approximately, a similar case can be created using a 128 x N CU with an ISP. Therefore, both cases address the problem of a 64 x 64 decoder pipeline. Thus, the CU size that can be used with an ISP is limited to a maximum of 64 x 64. All sub-partitions satisfy the condition of having at least 16 samples.

[0109] Matrix-weighted intra-prediction (MIP) is an intra-prediction technique in VVC. To predict samples from a rectangular block of width W and height H, MIP takes H reconstructed neighbor boundary samples from the left row of the block and W reconstructed neighbor boundary samples from the top row of the block as input. If reconstructed samples are unavailable, they are generated as in convolutional intra-prediction. The generation of the predicted signal is based on three steps: averaging, matrix-vector multiplication, and linear interpolation, as follows: Figure 7 As shown in the image.

[0110] Merging mode with motion vector difference (MMVD) is initiated by signaling the merging candidates, followed by signaling the MVD and resolution index. The merging list can include the following candidates: 1) Spatial MVP from the spatially neighboring block CU 2) Temporary MVP from the same CU 3) Historical MVP from FIFO table 4) Pairwise average MVP (using candidates already in the list) 5) Zero music videos.

[0111] In symmetric MVD, in the case of bidirectional prediction, motion information from list 1 is derived from motion information from list 0.

[0112] In affine prediction, several motion vectors are indicated / signed for different angles of a block, and these motion vectors are used to derive the motion vectors of sub-blocks. In affine merging, the affine motion information of a block is generated based on the regular or affine motion information of neighboring blocks.

[0113] In sub-block-based provisional motion vector prediction, the motion vectors of the sub-blocks of the current block are predicted from the appropriate sub-blocks in the reference frame, which are indicated by the motion vectors of spatially neighboring vectors (if available).

[0114] In Adaptive Motion Vector Resolution (AMVR), the accuracy of MVD is communicated via signals for each CU.

[0115] In bidirectional forecasting with CU-level weights, the index indicates the weight value of the weighted average for the two forecast blocks.

[0116] Two-way optical flow (BDOF) refines motion vectors in the case of bidirectional prediction. BDOF uses the motion vectors fed by the signal to generate two prediction blocks. Motion refinement is then calculated to minimize the error between the two prediction blocks using its gradient value. The final prediction block is refined using the motion refinement and the gradient value.

[0117] In HEVC, bidirectional signals are generated by averaging two prediction signals acquired from two different reference images and / or using two different motion vectors. In VVC, the bidirectional prediction mode is extended beyond simple averaging to allow for a weighted average of the two prediction signals. This is known as bidirectional prediction with CU-level weights (BCW).

[0118] Five weights are allowed in weighted average two-way forecasting. For each bidirectional prediction CU, the weight w is determined in one of two ways: 1) for non-merged CUs, the weight index is signaled after the motion vector difference; 2) for merged CUs, the weight index is inferred from neighboring blocks based on the merge candidate index. BWC is only applied to CUs with 256 or more luminance samples (i.e., CU width multiplied by CU height is greater than or equal to 256). For low-latency images, all 5 weights are used. For non-low-latency images, only 3 weights are used ( w∈{3,4,5} ) is used.

[0119] At the encoder, block search algorithms are applied to find weight indices without significantly increasing encoder complexity. These algorithms are summarized below. For further details, the reader can refer to the VTM software and document JVET-L0646. When combined with AMVR, if the current image is a low-latency image, non-equal weights are conditionally checked only for 1-pixel and 4-pixel motion vector precision. - When combined with affine, affine ME will be executed for non-equal weights if and only if the affine mode is selected as the current best mode. - In bidirectional prediction, when the two reference images are the same, non-equal weights are only conditionally checked. - When certain conditions are met, depending on the POC distance between the current image and its reference images, the QP encoding, and the temporary level, non-equal weighted images will not be searched.

[0120] The BCW weight index is encoded using a content-encoded binary that follows a bypass-encoded binary. The first content-encoded binary indicates whether equal weights are used; and if unequal weights are used, an additional binary is signaled using bypass encoding to indicate which unequal weight was used.

[0121] Weighted Prediction (WP) is an encoding tool supported by the H.264 / AVC and HEVC standards for efficiently encoding video content with decay. WP support has also been added to the VVC standard. WP allows weighting parameters (weights and offsets) to be signaled for each reference picture in each of the reference picture lists L0 and L1. Then, during motion compensation, multiple weights and offsets of the corresponding reference pictures are applied. WP and BCW are designed for different types of video content. To avoid interaction between WP and BCW, which would complicate VVC encoder design, if the CU uses WP, the BCW weight index is not signaled, and w is inferred to be 4 (i.e., equal weights are applied). For merging CUs, the weight index is inferred from neighboring blocks based on merging candidate cells. This can be applied to both regular merging patterns and inherited affine merging patterns. For constructed affine merging patterns, affine pattern information is constructed based on motion information from up to 3 blocks. The BCW for a CU using constructed affine merging patterns is simply set to the BCW index of the first control point MV.

[0122] In VVC, combined inter-frame / intra-frame prediction (CIIP) and bidirectional prediction (BCW) with CU-level weights cannot be jointly applied to the CU. When a CU is encoded in CIIP mode, the BCW index of the current CU is set to 2, for example, with equal weights.

[0123] In VVC, when a CU is encoded in merged mode, if the CU contains at least 64 luma samples (i.e., the CU width multiplied by the CU height is equal to or greater than 64), and if both the CU width and CU height are less than 128 luma samples, an additional flag is signaled to indicate whether Combined Inter-Frame / Intra-Frame Prediction (CIIP) mode is applied to the current CU. As its name suggests, CIIP prediction combines inter-frame prediction signals with intra-frame prediction signals. The inter-frame prediction signal P in CIIP mode... inter The derivation is performed using the same inter-frame prediction process applied to the regular merging mode; and the intra-frame prediction signal P intra The derivation follows the conventional intra-frame prediction process with a planar pattern. Then, the intra-frame and inter-frame prediction signals P... intra P inter A weighted average is used for combining, where the weight values ​​are calculated based on the encoding patterns of the blocks above and to the left of the neighboring blocks, as follows: - If the adjacent block above is available and is intra-frame encoded, then set isIntraTop Set to 1, otherwise set to 1. isIntraTop =0; - If the left neighboring block is available and intra-frame encoded, then set isIntraLeft Set to 1, otherwise set to 1. isIntraLeft =0; - if( isIntraleft + isIntraTop If ) equals 2, then wt It was set to 3; - Otherwise, if ( isIntraleft + isIntraTop If ) equals 1, then wt It is set to 2; - Otherwise, set wt The value is 1. Figure 8 The illustration shows an example of the upper and left neighboring blocks used in the CIIP weight derivation.

[0124] The CIIP predictions are as follows:

[0125] Local Illumination Compensation (LIC) is an inter-frame prediction technique that models the local illumination variation between the current block and its prediction block as a function of the relationship between the current block template and the reference block template. The parameters of this function can be represented by scaling α and offset β, forming a linear equation, i.e., To compensate for changes in lighting, among which p[x [This refers to the location indicated by the MV on the reference image.] x The reference sample is located at [location]. Since α and β can be derived based on the current block template and the reference block template, they do not require signaling overhead, except that the LIC flag is signaled to indicate the use of LIC for AMVP mode.

[0126] The local illumination compensation proposed in JVET-O0066 is used in ECM for unidirectional prediction inter-frame CU with the following modifications. • Intra-frame neighbor samples can be used in LIC parameter derivation; • For blocks with fewer than 32 luminance samples, LIC is disabled; • For both non-subblock and affine modes, the LIC parameter derivation is based on the template block sample corresponding to the current CU being performed, rather than the partial template block sample corresponding to the first upper left 16x16 cell; • Samples of the reference block template are generated using a MC with a block MV, without rounding it to integer pixel precision.

[0127] In bidirectional motion compensation, out-of-bounds (OOB) prediction samples are discarded, and only non-OOB predictors are used to generate the final predictor when available. OOB prediction samples are those whose reference samples are at least partially located outside the reference image. Specifically, let... Pos_x i,j and Pos_y i,jThis indicates the location of a predicted sample within the current block. and (x = 0, 1) represents the MV of the current block; Pos LeftBdry , Pos RightBrdy , Pos TopBdry ,as well as Pos BottomBdry These are the positions of the four boundaries of the image. A predicted sample is considered out-of-bounds (OOB) if at least one of the following conditions is met: in half_pixel It equals 8, where 8 represents the half-pixel sample distance in 1 / 16 pixel sample precision.

[0128] After checking the OOB conditions for each sample, the final predicted sample for a bidirectional block is generated as shown in Algorithm A below: if It is OOB, and Yes or No OOB Otherwise It is not OOB, and It's OOB otherwise

[0129] As shown above, in the final prediction, the first part ("if...otherwise if...") is the unidirectional prediction part of the two-way prediction. The bidirectional prediction part of the two-way prediction is the second part ("otherwise..."). =…”, where two one-way forecasts are (weighted) averaged to form a true two-way forecast.

[0130] The OOB check process also applies when BCW is enabled.

[0131] Finally, it should be noted that this sample-adaptive bidirectional prediction process can be applied only to the prediction cell for which at least the reference block is first detected as partially or completely outside the boundary. Therefore, the block-level OOB criterion is checked first. If both prediction blocks are non-OBB, then typical bidirectional prediction occurs.

[0132] In OOB-aware bidirectional inter-frame prediction, samples available only in both prediction blocks (i.e., motion-compensated blocks P0 and P1) can be merged into the bidirectional prediction. For regions where only one predictor (P0 or P1) is available (the other is outside the image boundary), samples belonging only to the available single predictor can be used. Therefore, the final prediction of a block is a mixture of unidirectional and bidirectional prediction samples. This can lead to artifacts and steep edges when the unidirectional prediction samples have different brightness or texture features compared to the bidirectional prediction samples.

[0133] The current embodiment targets a linear filter for bidirectional prediction to adjust the unidirectional prediction portion of an OOB-aware bidirectional prediction block to better match the bidirectional prediction portion of the block. This means that the current embodiment aims to filter the unidirectional prediction portion based on how the bidirectional predictions appear. The filter coefficients are derived by minimizing the squared error between the bidirectional prediction portion and the unidirectional prediction sample set.

[0134] When OOB processing is enabled, inter-frame prediction blocks can become a mixture of bidirectional and unidirectional prediction signals. This means that in OOB-aware prediction, the final bidirectional prediction block comprises both unidirectional and bidirectional prediction components. The bidirectional prediction component is only used for regions where both blocks are within image edges. If a region in the block is outside image edges, unidirectional prediction is used for that region. Figure 9 Two unidirectional prediction blocks, P0 910 and P1 920, are shown, with the gray area of ​​P1 920 indicating the region outside the boundary of P1 920. When OOB processing is enabled, the final bidirectional prediction block is shown by reference numeral 952, and a portion 951 of the block is predicted using only the P0 unidirectional prediction block. This embodiment proposes filtering samples in the unidirectional prediction region based on how the bidirectional prediction portion (i.e., P0+P1) appears relative to the P0 (or P1) unidirectional prediction block.

[0135] Two-way prediction section P0+P1 The predicted sample values ​​in 900 can be calculated as follows: Or more generally calculated as ,in a It's the weight. And... P0P1 Overlapping P0 A portion can be defined as X ,in P0P1 Defined as .and P0 Overlapping P0P1 A portion is defined as Y ,like Figure 10 As shown in the image.

[0136] X and YIt can be vectorized into a column vector. Minimize coefficient c It has been confirmed. A It contains column vectors X、 And optionally include other columns in the data matrix, such as P0 The bias term or spatial shift. Matrix A can also contain samples from co-occurring blocks in another channel. For example, when a filter is applied to the chroma channel, matrix A... A One or more columns in the table can be filled with sample values ​​from the brightness patch. The minimization problem can be solved, for example, using simple linear regression, LDL, decomposition, or Gaussian elimination. In the coefficients... c After it has been estimated, P0 The predicted non-overlapping portion was adjusted to in A f Is Figure 10 In X f definition P0 The data matrix is ​​obtained from the non-overlapping regions.

[0137] Alternate partitions can be used for, for example Figure 11 The filter derivation shown in the figure uses only the edge regions of the overlapping regions (i.e., the dashed region 1110) for filter derivation; however, the filter is applied to the complete region. X f superior.

[0138] Furthermore, several filters can be derived on sub-partitions and applied to, for example... Figure 12 The corresponding sub-partitions are shown in the diagram. Figure 12 This shows the derivation of two filter coefficient sets over the overlapping region 0 / 1 and their application. X f Examples in the corresponding region 0 / 1 on the [the map].

[0139] When coefficients are used P0P1 / X / Y When the region is derived, the resulting filter can be closest to X f and X area At the edge, model the bidirectional prediction process, and further away X Block filtering accuracy may decrease. This is especially true for larger blocks, where the distance between some samples in the sample set increases, leading to a deterioration in the quality of the filtered samples. To mitigate this problem, the determined filter can be applied only to samples belonging to the block. X fA subset of the selected samples in the region. For example, only if... X f and X If the distance is less than N samples, then a filter can be applied, where N It can be a predetermined number, such as 4 or 8, or it can be determined in other ways, such as by using the dimension of the prediction block.

[0140] Alternate or additional sites, for the purpose of selection X f Filtering a subset of samples within a region, the effect of the filter can be based on... X f and X The distance between the edges is adjusted. This can be done, for example, by using a filter. A fC Output and X f This is accomplished by mixing input samples within a region. For example, if from... X f and X The distance between the edges is marked as d The output of the filtering operation can then be determined as follows: when d <N hour otherwise

[0141] As another example, P0P1 / X / Y Some samples within the region can also be filtered using the determined filter to smooth the transition between the bidirectional and unidirectional prediction regions. Similarly, in this case, samples closer to... X f and X The predicted samples of the edges between them can be determined as a weighted sum of the two samples, for example, P0P1 The bidirectional prediction samples in the data are obtained through filtering. X The samples are obtained by unidirectional prediction samples in the region.

[0142] According to the embodiments, the use of the described OOB filtering can be inferred from samples of a one-way predicted signal or from a combined two-way signal.

[0143] According to embodiments, the filter can have a simple model (such as...) a x+b Alternatively, it may follow a more complex 2D model in which additional data terms are acquired as spatial shifts of the unidirectional predictive signal (resulting in a 2D convolutional filter).

[0144] According to an embodiment, the filter may include nonlinear terms.

[0145] According to an embodiment, the filter coefficients can be derived for any set or configuration of samples belonging to the overlapping region of the OOB-sensing bidirectional prediction signal.

[0146] According to the embodiments, more than one filter can be derived and applied. Furthermore, each filter is derived using its own region, and each filter can be applied to its own region on the unidirectional prediction portion of the OOB sensing block.

[0147] According to an embodiment, the filter output is mixed with the unidirectional predicted sample value.

[0148] According to an embodiment, the filter output is mixed with the unidirectional prediction sample value based on the distance from the edge between the unidirectional prediction region and the bidirectional prediction region.

[0149] According to an embodiment, the filter coefficients are derived in this region, and these filter coefficients can be inferred from either a unidirectional prediction signal or a combined bidirectional prediction signal.

[0150] According to an embodiment, filter coefficients are derived over this region and can be signaled at the CU, PU, ​​CTU, slice, frame, or sequence level.

[0151] The method according to the embodiment is in Figure 13 The method is illustrated in the diagram. The method generally includes: processing 1010 video frames to predict blocks based on prediction samples; determining 1020 when two predicted blocks for a block are within the boundaries of an image, and applying bidirectional prediction based on the predicted blocks for the block to generate a bidirectional predicted block; determining 1031 when one of the predicted blocks is at least partially outside the boundaries of an image, and applying unidirectional prediction based on the predicted block within the image boundaries to generate a unidirectional predicted block; deriving 1040 filter coefficients by minimizing the squared error between the bidirectional and unidirectional predicted blocks; and filtering 1050 at least a portion of the unidirectional predicted block based on the filter coefficients to generate a filtered unidirectional predicted block. Each step in the process can be implemented by a corresponding module of a computer system.

[0152] An apparatus according to an embodiment includes: components for processing video frames to predict blocks based on prediction samples; components for determining when two predicted blocks for a block are within the boundaries of an image, and components for applying bidirectional prediction based on the predicted blocks for the block to generate a bidirectional predicted block; components for determining when one of the predicted blocks is at least partially outside the edges of the image, and components for applying unidirectional prediction based on the predicted blocks within the image boundaries to generate a unidirectional predicted block; components for deriving filter coefficients by minimizing the squared error between the bidirectional and unidirectional predicted blocks; and components for filtering at least a portion of the unidirectional predicted block based on the filter coefficients to generate a filtered unidirectional predicted block. The components include at least one processor and a memory including computer program code, wherein the processor may further include processor circuitry. The memory and computer program code are configured, together with the at least one processor, to cause the apparatus to perform actions according to various embodiments. Figure 13 The method.

[0153] An example of a data processing system for a device is shown in Figure 14 The diagram is shown in the figure. Several functions can be performed by a single physical device; for example, all computational processes can be performed in a single processor if desired. The data processing system includes a main processing unit 100, a memory 102, a storage device 104, an input device 106, an output device 108, and a graphics subsystem 110, all of which are interconnected via a data bus 112.

[0154] The main processing unit 100 is a conventional processing unit arranged to process data within a data processing system. The main processing unit 100 may include or be implemented as one or more processors or processor circuit systems. The processor 102, storage device 104, input device 106, and output device 108 may include conventional components recognized by those skilled in the art. The memory 102 and storage device 104 store data within the data processing system 100.

[0155] Computer program code resides in memory 102 for implementing, for example, according to various embodiments. Figure 14 The method is illustrated in the flowchart. Input device 106 inputs data into the system, while output device 108 receives data from the data processing system and forwards the data (e.g., to a display). Data bus 112 is a conventional data bus, although shown as a single line, it can be any combination of the following: processor bus, PCI bus, graphics bus, ISA bus. Therefore, those skilled in the art will readily recognize that the device can be any data processing device, such as a computer, personal computer, server computer, mobile phone, smartphone, or internet access device, such as an internet tablet.

[0156] Figure 15 An example of a video encoder is illustrated, where I n : The image to be encoded; P' n Predictive representation of image patches; D n : Prediction error signal; D' n : Reconstruct the prediction error signal; I' n Preliminary image reconstruction; R' n : Final reconstructed image; T, T -1 Transform and inverse transform; Q, Q -1 : Quantization and inverse quantization; E: Entropy coding; RFM: Reference frame memory; P inter Intra-frame prediction; P intra : Inter-frame prediction; MS: Mode selection; F: Filtering. Figure 16 The diagram illustrates the block diagram of a video decoder, where P' n : Predictive representation of image patches; D' n : Reconstruct the prediction error signal; I' n Preliminary image reconstruction; R' n : Final reconstructed image; T -1 Inverse transform; Q -1 E: Inverse quantization; E: Entropy decoding; RFM: Reference frame memory; P: Prediction (inter-frame or intra-frame); F: Filtering. The apparatus according to embodiments may include an encoder-only unit, or a decoder, or both.

[0157] Various embodiments can be implemented using computer program code residing in memory and causing the relevant means to perform methods. For example, the means may include circuitry and electronic devices for processing, receiving, and transmitting data, computer program code in memory, and a processor that, when running the computer program code, causes the means to perform the features of the embodiments. Furthermore, a network device, such as a server, may include circuitry and electronic devices for processing, receiving, and transmitting data, computer program code in memory, and a processor that, when running the computer program code, causes the network device to perform the features of various embodiments.

[0158] If desired, the different functions discussed herein may be performed in different orders and / or simultaneously with other functions. Furthermore, if desired, one or more of the functions and embodiments described above may be optional or may be combined.

[0159] Although various aspects of the embodiments are defined in the independent claims, other aspects include other combinations of features from the described embodiments and / or dependent claims with features of the independent claims, and not only combinations explicitly defined in the claims.

[0160] It should also be noted in this document that although exemplary embodiments have been described above, these descriptions should not be considered restrictive. Rather, several changes and modifications may be made without departing from the scope of this disclosure as defined in the appended claims.

Claims

1. An apparatus for encoding, comprising: A component used to process video frames to predict blocks based on prediction samples; The components are for determining when two predicted blocks for a block are within the boundaries of an image, and for generating bidirectional predicted blocks by applying bidirectional prediction based on the predicted blocks for the block. The component is used to determine when one of the prediction blocks is at least partially outside the boundary of the image, and the component is used to generate a unidirectional prediction block by applying a unidirectional prediction based on the prediction blocks within the image boundary. A component for deriving filter coefficients by minimizing the squared error between the bidirectional prediction block and the unidirectional prediction block; A component for filtering at least a portion of the one-way prediction block based on the filter coefficients to generate a filtered one-way prediction block.

2. The apparatus according to claim 1, further comprising: A component used to indicate the filtering used in the samples of the unidirectional or bidirectional prediction block.

3. The apparatus according to claim 1 or 2, wherein the filter comprises a nonlinear term.

4. The apparatus according to claim 1, 2, or 3, further comprising: The component is used to derive the filter coefficients for a sample set that belongs to the overlapping region of the bidirectional prediction block.

5. The apparatus according to any one of claims 1 to 4, further comprising: A component for deriving more than one filter, wherein each filter is derived using its own region and is applied to its own region on the unidirectional prediction block.

6. The apparatus according to any one of claims 1 to 5, further comprising: A component for indicating a region in which the filter coefficients are derived in the samples of the unidirectional prediction block or in the bidirectional prediction block.

7. The apparatus according to any one of claims 1 to 6, further comprising: Components for signaling a region, wherein the filter coefficients of the region are derived at the level of coding unit, prediction unit, coding tree unit, slice, frame, or sequence.

8. A method comprising: Process video frames to predict blocks based on prediction samples; Determine when two predicted blocks for a block are within the boundaries of the image, and generate bidirectional predicted blocks by applying bidirectional prediction based on the predicted blocks for the block. Determine when one of the prediction blocks is at least partially outside the image boundary, and generate a unidirectional prediction block by applying unidirectional prediction based on the prediction blocks inside the image boundary; The filter coefficients are derived by minimizing the squared error between the bidirectional prediction block and the unidirectional prediction block. At least a portion of the one-way prediction block is filtered according to the filter coefficients to generate a filtered one-way prediction block.

9. The method according to claim 8, further comprising: Indicates the use of filtering in the samples of the unidirectional or bidirectional prediction block.

10. The method of claim 8 or 9, wherein the filter comprises a nonlinear term.

11. The method according to claim 8, 9, or 10, further comprising: Derive the filter coefficients for a sample set that belongs to the overlapping region of the bidirectional prediction block.

12. The method according to any one of claims 8 to 11, further comprising: More than one filter is derived, wherein each filter is derived using its own region and is applied to its own region on the unidirectional prediction block.

13. The method according to any one of claims 8 to 12, further comprising: The region is indicated on which the filter coefficients are derived in the samples of the unidirectional prediction block or in the bidirectional prediction block.

14. The method according to any one of claims 8 to 13, further comprising: The region is notified by a signal, and the filter coefficients of the region are derived at the level of coding unit, prediction unit, coding tree unit, slice, frame, or sequence.

15. An apparatus comprising at least one processor and a memory including computer program code, the memory and the computer program code being configured together with the at least one processor such that the apparatus performs at least the following: Process video frames to predict blocks based on prediction samples; Determine when two predicted blocks for a block are within the boundaries of the image, and generate bidirectional predicted blocks by applying bidirectional prediction based on the predicted blocks for the block. Determine when at least one of the prediction blocks is at least partially outside the boundary of the image, and generate a unidirectional prediction block by applying unidirectional prediction based on the prediction block inside the image boundary; The filter coefficients are derived by minimizing the squared error between the bidirectional prediction block and the unidirectional prediction block. At least a portion of the one-way prediction block is filtered according to the filter coefficients to generate a filtered one-way prediction block.

Images