Method, apparatus and computer program product for video encoding and decoding

Abstract

Embodiments relate to an apparatus comprising means for processing an image to predict a block from prediction samples of two prediction blocks; determining when two of the two prediction blocks of the block are within the boundary of the image, and applying bidirectional prediction based on the two prediction blocks of the block to obtain a bidirectional prediction block; determining when one of the two prediction blocks is at least partially outside the boundary of the image, and applying unidirectional prediction based on the prediction blocks within the boundary of the image to obtain a unidirectional prediction block; deriving filter coefficients based on a selected sample set of the bidirectional prediction block, wherein the selected sample set includes samples that have been down-sampled in the bidirectional prediction block according to a down-sampling method; and filtering at least a portion of the uni-prediction block according to the filter coefficients.

Description

Technical Field

[0001] This technical solution generally involves video encoding and video decoding. Background Technology

[0002] This section is intended to provide background or context for the invention as described in the claims. The description herein may include concepts that may be pursued, but are not necessarily previously conceived or pursued. Therefore, unless otherwise stated herein, the content described in this section is not prior art to the specification and claims of this application, nor should it be considered prior art simply because it is included in this section.

[0003] A video encoding system may include an encoder and a decoder. The encoder transforms the input video into a compressed representation suitable for storage / transmission, and the decoder decompresses the compressed video representation back into a visual form. The encoder may discard some information from the original video sequence to represent the video in a more compact form, for example, to enable storage / transmission of video information at a lower bit rate than would otherwise be necessary. Summary of the Invention

[0004] The independent claims define the scope of protection sought for the various embodiments of the invention. Embodiments and features described in this specification that are not within the scope of the independent claims (if any) are to be interpreted as examples useful for understanding the various embodiments of the invention.

[0005] The various aspects include a method, an apparatus, and a computer-readable medium comprising a computer program stored therein, characterized by the contents described in the independent claim. Various embodiments are disclosed in the dependent claims.

[0006] According to a first aspect, an apparatus is provided, comprising: means for processing an image to predict a block based on prediction samples of two prediction blocks; means for determining when both of the two prediction blocks of the block are within a boundary of the image, and means for applying bidirectional prediction based on the two prediction blocks of the block to obtain a bidirectional prediction block; means for determining when one of the two prediction blocks is at least partially outside a boundary of the image, and means for applying unidirectional prediction based on the prediction block within the image boundary to obtain a unidirectional prediction block; means for deriving filter coefficients based on a selected sample set of the bidirectional prediction block, wherein the selected sample set includes samples that have been downsampled in the bidirectional prediction block according to a downsampling method; and means for filtering at least a portion of the unidirectional prediction block based on the filter coefficients to generate a filtered unidirectional prediction block.

[0007] According to a second aspect, a method is provided, the method comprising: processing an image to predict a block based on prediction samples of two prediction blocks; determining when both of the two prediction blocks of the block are within the boundaries of the image, and applying bidirectional prediction based on the two prediction blocks of the block to obtain a bidirectional prediction block; determining when one of the two prediction blocks is at least partially outside the boundaries of the image, and applying unidirectional prediction based on the prediction block within the image boundaries to obtain a unidirectional prediction block; deriving filter coefficients based on a selected sample set of the bidirectional prediction block, wherein the selected sample set includes samples that have been downsampled in the bidirectional prediction block according to a downsampling method; and filtering at least a portion of the unidirectional prediction block according to the filter coefficients to generate a filtered unidirectional prediction block.

[0008] According to a third aspect, an apparatus is provided comprising at least one processor and a memory including computer program code, the memory and the computer program code being configured together with the at least one processor such that the apparatus performs at least the following: processing an image to predict a block based on prediction samples of two prediction blocks; determining when both of the two prediction blocks of the block are within the boundaries of the image, and applying bidirectional prediction based on the two prediction blocks of the block to obtain a bidirectional prediction block; determining when one of the two prediction blocks is at least partially outside the boundaries of the image, and applying unidirectional prediction based on the prediction block within the image boundaries to obtain a unidirectional prediction block; deriving filter coefficients based on a selected sample set of the bidirectional prediction block, wherein the selected sample set includes samples that have been downsampled in the bidirectional prediction block according to a downsampling method; and filtering at least a portion of the unidirectional prediction block according to the filter coefficients to generate a filtered unidirectional prediction block.

[0009] According to a fourth aspect, a computer program product including computer program code is provided, the computer program code being configured to, when executed on at least one processor, cause an apparatus or system to: process an image to predict a block based on prediction samples of two prediction blocks; determine when both of the two prediction blocks of the block are within the boundaries of the image, and apply bidirectional prediction based on the two prediction blocks of the block to obtain a bidirectional prediction block; determine when one of the two prediction blocks is at least partially outside the boundaries of the image, and apply unidirectional prediction based on the prediction block within the image boundaries to obtain a unidirectional prediction block; derive filter coefficients based on a selected sample set of the bidirectional prediction block, wherein the selected sample set includes samples that have been downsampled in the bidirectional prediction block according to a downsampling method; and filter at least a portion of the unidirectional prediction block according to the filter coefficients to generate a filtered unidirectional prediction block.

[0010] According to one embodiment, the downsampling method includes: selecting uniformly spaced samples with predefined intervals within a bidirectional prediction block.

[0011] According to one embodiment, the downsampling method includes: selecting uniformly spaced samples within a bidirectional prediction block such that the total number of samples does not exceed a predefined threshold.

[0012] According to one embodiment, the downsampling method includes: dividing a bidirectional prediction block into multiple regions and selecting samples with different intervals within each region.

[0013] According to one embodiment, the downsampling method includes: selecting all or part of the samples included in the prediction sample derivation.

[0014] According to one embodiment, the downsampling method includes selecting samples based on a downsampling matrix.

[0015] According to one embodiment, the downsampling method includes selecting samples with different intervals in different directions.

[0016] According to one embodiment, the value of the predefined threshold depends on the prediction unit (PU), coding unit (CU), coding tree unit (CTU), slice, frame, or sequence attribute.

[0017] According to one embodiment, the number of regions is based on prediction units (PU), coding units (CU), coding tree units (CTU), slices, frames, or sequence attributes.

[0018] According to one embodiment, the geometry of the region is derived based on the height and width of the bidirectional prediction block.

[0019] According to one embodiment, the overshoot parameter is defined to control the number of samples.

[0020] According to one embodiment, the overshoot parameter is based on prediction unit (PU), coding unit (CU), coding tree unit (CTU), slice, frame, or sequence attribute.

[0021] According to one embodiment, the computer program product is embodied on a non-transitory computer-readable medium. Attached Figure Description

[0022] In the following description, various embodiments will be described in more detail with reference to the accompanying drawings, in which...

[0023] Figure 1 This shows an example of the positions of the samples to the left and above the current block involved in CCLM mode.

[0024] Figure 2a An example is shown of deriving the chromaticity prediction mode based on the luminance mode when CCLM is enabled.

[0025] Figure 2b An example of a unified binarization table for chromaticity prediction modes is shown.

[0026] Figure 3a Two luminance-to-chrominance models are shown when the luminance Y threshold is 17.

[0027] Figure 3b An example is shown showing the correspondence between each luminance-to-chrominance model and the spatial segmentation of the content.

[0028] Figure 4 An example is shown for deriving the location of samples for a CCCM filter.

[0029] Figure 5 Examples of various filter kernels are shown;

[0030] Figure 6 An example of four reference lines adjacent to the prediction block is shown.

[0031] Figure 7 An example of a matrix-weighted intra-frame prediction process is shown.

[0032] Figure 8 An example of the top and left adjacent blocks used in CIIP weight derivation is shown.

[0033] Figure 9 An example of a bounding box derived from the OOB and non-OOB partitions of a block is shown.

[0034] Figure 10 Three regions are shown for anisotropic downsampling.

[0035] Figure 11 This is a flowchart illustrating a method according to an embodiment;

[0036] Figure 12 An apparatus according to an embodiment is shown.

[0037] Figure 13 The encoding process according to an embodiment is shown.

[0038] Figure 14 The decoding process according to an embodiment is shown. Detailed Implementation

[0039] The following description and accompanying drawings are illustrative and should not be construed as unnecessarily limiting. Specific details are provided to provide a thorough understanding of this disclosure. However, in some cases, well-known or conventional details have not been described to avoid obscuring the description. In this disclosure, references to one or more embodiments may be, but are not necessarily, references to the same embodiment, and such references indicate at least one embodiment.

[0040] In this specification, references to "an embodiment" or "embodiment" indicate that a particular feature, structure, or characteristic described in connection with that embodiment is included in at least one embodiment of this disclosure.

[0041] In the following description, several embodiments will be presented in the context of a video coding arrangement. However, it should be noted that this embodiment is not necessarily limited to this particular arrangement. The embodiments relate to reference sample selection in out-of-bounds sample filtering. Before further describing the embodiments, brief reference is made to the evolution of video coding standardization. This embodiment is applicable in the context of next-generation video coding standardization, such as the H.267 video coding standard and ECM (Enhanced Compression Model) exploration.

[0042] The Advanced Video Coding Standard (which can be abbreviated as AVC or H.264 / AVC) was developed by the Video Coding Experts Group (VCEG) of the Telecommunication Standardization Sector of the International Telecommunication Union (ITU-T) and the Joint Video Team (JVT) of the Moving Picture Experts Group (MPEG) of the International Organization for Standardization (ISO) / International Electrotechnical Commission (IEC). The H.264 / AVC standard was published by these two higher-level standardization organizations and is known as ITU-T Recommendation H.264 and ISO / IEC International Standard 14496-10, also referred to as MPEG-4 Part 10 Advanced Video Coding (AVC). The H.264 / AVC standard has several versions, each integrating new extensions or features into the specification. These extensions include Scalable Video Coding (SVC) and Multi-View Video Coding (MVC).

[0043] The High Efficiency Video Coding Standard (which may be abbreviated as HEVC or H.265 / HEVC) was developed by the Joint Collaboration Team on Video Coding (JCT-VC) of VCEG and MPEG. This standard was published by the parent standardization organization and is known as ITU-T Recommendation H.265 and ISO / IEC International Standard 23008-2, also referred to as MPEG-H Part 2 High Efficiency Video Coding (HEVC). Extensions to H.265 / HEVC include scalable, multi-view, 3D, and fidelity range extensions, respectively referred to as SHVC, MV-HEVC, 3D-HEVC, and REXT. Unless otherwise stated, H.265 / HEVC, SHVC, MV-HEVC, 3D-HEVC, and REXT referenced in this specification for the purpose of understanding the definitions, structure, or concepts of these standard specifications should be understood as the latest versions of these standards available prior to the date of reference.

[0044] Multifunctional Video Coding (which can be abbreviated as VVC, H.266, or H.266 / VVC) is a video compression standard developed as the successor to HEVC. VVC is specified in ITU-T H.266 Recommendation and has an equivalent specification in ISO / IEC 23090-3 (also known as MPEG-I Part 3).

[0045] The Alliance for Open Media (AOM) developed the specifications for the AV1 bitstream format and decoding process. The AV1 specification was released in 2018. It has been reported that the AOM is working on the AV2 specification.

[0046] This section describes some key definitions, bitstream and encoding structures, and concepts of H.264 / AVC, HEVC, VVC, and / or AV1, as well as some extensions, using video encoders, decoders, encoding methods, decoding methods, and bitstream structures as examples, on which embodiments can be implemented. Aspects of the various embodiments are not limited to H.264 / AVC, HEVC, VVC, and / or AV1 or their extensions, but are described for a possible basis on which these embodiments can be partially or completely implemented.

[0047] A video codec can include an encoder and a decoder. The encoder transforms the input video into a compressed representation suitable for storage / transmission, and the decoder decompresses the compressed video representation back into a visual form. The compressed representation can be called a bitstream or video bitstream. Video encoders and / or video decoders can also be separate entities, i.e., they do not need to form a codec. The encoder can discard some information from the original video sequence to represent the video in a more compact form (i.e., at a lower bitrate). The symbol "(de)coder" refers to both the encoder and / or decoder.

[0048] In some video codecs, such as H.265 / HEVC, video frames are divided into coding units (CUs) covering regions of the frame. A CU consists of one or more prediction units (PUs) defining a prediction process for samples within the CU and one or more transform units (TUs) defining a prediction error coding process for samples within the CU. A CU can consist of a square block of samples, the size of which can be chosen from a predefined set of possible CU sizes. The CU with the largest allowed size can be named an LCU (Maximum Coding Unit) or a CTU (Coding Tree Unit), and video frames are divided into non-overlapping CTUs. A CTU can be further subdivided into combinations of smaller CUs, for example, by recursively splitting the CTUs and the resulting CUs. Each resulting CU can have at least one PU and at least one TU associated with it. Each PU and TU can be further subdivided into smaller PUs and TUs to improve the granularity of the prediction and prediction error coding processes, respectively. Each PU has prediction information associated with it that defines which prediction will be applied to the pixels within that PU (e.g., motion vector information for inter-frame prediction PUs and intra-frame prediction directionality information for intra-frame prediction PUs). Similarly, each TU is associated with information describing the prediction error decoding process of the samples within that TU (including, for example, DCT coefficient information). Prediction error coding can be signaled at the CU level. If there is no prediction error residual associated with a CU, that CU can be considered to have no TU. Signaling in the bitstream to divide the image into CUs and CUs into PUs and TUs can enable the decoder to reproduce the intended structure of these units.

[0049] Hybrid video codecs (e.g., ITU-T H.263, H.264 / AVC, and HEVC) can encode video information in two stages. First, pixel values ​​in a picture region (or “block”) are predicted, for example, through motion compensation (identifying and indicating a region in a previously encoded video frame that closely corresponds to the block being encoded) or through spatial means (using pixel values ​​around the block to be encoded in a specified manner). In this first stage, predictive coding can be applied, for example, as so-called sample prediction and / or so-called syntax prediction.

[0050] In sample prediction, pixel or sample values ​​are predicted for a given region or "block" of an image. These pixel or sample values ​​can be predicted using one or more motion compensation or intra-frame prediction mechanisms.

[0051] Motion compensation mechanisms (also known as inter-frame prediction, temporal prediction, motion-compensated temporal prediction, or motion-compensated prediction, or MCP) involve identifying and indicating regions in a previously coded video frame that closely correspond to the block being encoded. Inter-frame prediction can reduce temporal redundancy.

[0052] Intra-frame prediction (where pixel or sample values ​​can be predicted using spatial mechanisms) involves discovering and indicating spatial region relationships. Intra-frame prediction leverages the fact that neighboring pixels within the same image may be related. Intra-frame prediction can be performed in the spatial domain or the transform domain; that is, sample values ​​or transform coefficients can be predicted. Intra-frame prediction is typically used in intra-frame coding, where inter-frame prediction is not applied.

[0053] In syntax prediction (which may also be called parametric prediction), syntax elements, and / or syntax element values ​​and / or variables derived from syntax elements, are predicted based on earlier encoded / decoded syntax elements and / or earlier derived variables. A non-restrictive example of syntax prediction is provided below.

[0054] In motion vector prediction, motion vectors used for inter-frame and / or inter-view prediction can be differentially encoded relative to block-specific predicted motion vectors. In many video codecs, predicted motion vectors are created in a predefined manner, such as by calculating the median of encoded or decoded motion vectors from adjacent blocks. Another method for creating motion vector predictions (sometimes called Advanced Motion Vector Prediction (AMVP)) is to generate a list of candidate predictions based on adjacent and / or juxtaposed blocks in a time-referenced picture and signal the selected candidate as the motion vector predictor. In addition to predicting motion vector values, reference indices from previously encoded / decoded pictures can also be predicted. Reference indices are typically predicted based on adjacent and / or juxtaposed blocks in a time-referenced picture. Differential encoding of motion vectors is typically disabled across slice boundaries.

[0055] Block partitioning can be predicted, for example, from CTU to CU, and then to PU.

[0056] In filter parameter prediction, filter parameters, for example, used for sample adaptive offset, can be predicted.

[0057] Prediction methods that use image information from previously encoded images can also be called inter-frame prediction methods, or temporal prediction and motion compensation.

[0058] Prediction methods that use image information within the same image can also be called intra-frame prediction methods.

[0059] Secondly, the prediction error (i.e., the difference between the predicted pixel block and the original pixel block) is encoded. This can be achieved by transforming the differences in pixel values ​​using a specified transform (e.g., Discrete Cosine Transform (DCT) or a variant thereof), the quantization coefficients, and entropy encoding the quantization coefficients. By varying the fidelity of the quantization process, the encoder can control the balance between the precision of the pixel representation (image quality) and the size of the resulting encoded video representation (file size at the transmission bitrate).

[0060] In most cases, the basic units used as input to the encoder and output to the decoder are images. The image used as input to the encoder can also be called the source image, and the image decoded by the decoder can be called the decoded image or the reconstructed image.

[0061] Both the source image and the decoded image consist of one or more sample arrays, such as one of the following sample array sets: - Luminance (Y) only (monochrome). - Brightness and two chromaticities (YCbCr or YCgCo). - Green, blue, and red (GBR, also known as RGB). - Indicates an array of other unspecified monochromatic or tristimulus color samples (e.g., YZX, also known as XYZ).

[0062] In the following text, these arrays may be referred to as luminance (or L or Y) and chrominance, where the two chrominance arrays may be referred to as Cb and Cr; regardless of the actual color representation used. The actual color representation used may be indicated, for example, in the encoded bitstream, such as using HEVC's Video Availability Information (VUI) syntax. A component may be defined as an array or a single sample from one of the three sample arrays (luminance and two chrominance), or an array or a single sample of an array that makes up an image in monochrome format.

[0063] An image can be defined as a frame or a field. A frame consists of a luminance sample matrix and possible corresponding chrominance samples. A field is an alternating set of sample rows of frames and can be used as encoder input when the source signal is interleaved. The chrominance sample array may not exist (and therefore monochrome sampling can be used), or the chrominance sample array may be downsampled compared to the luminance sample array.

[0064] The decoder reconstructs the output video by applying a prediction component similar to that of an encoder, forming a predictive representation of pixel blocks (using motion or spatial information created by the encoder and stored in the compressed representation) and prediction error decoding (the inverse operation of prediction error encoding, which recovers the quantized prediction error signal in the spatial pixel domain). After applying the prediction and prediction error decoding components, the decoder adds the prediction and prediction error signals (pixel values) to form the output video frame. The decoder (and encoder) may also apply additional filtering components to improve the quality of the output video before passing it for display and / or storing it as a prediction reference for upcoming frames in a video sequence.

[0065] Motion information can be indicated using motion vectors associated with each motion-compensated image block in the video codec. Each of these motion vectors represents the displacement of an image block in the picture to be encoded (on the encoder side) or decoded (on the decoder side) and a predicted source block in one of the previously encoded or decoded pictures. Like many other video compression standards, H.264 / AVC and HEVC divide an image into a rectangular grid. For each rectangular grid, a similar block from a reference image is indicated for inter-frame prediction. The position of the predicted block is encoded as a motion vector, which indicates the position of the predicted block relative to the encoded block.

[0066] To efficiently represent motion vectors, these motion vectors can be differentially encoded relative to block-specific predicted motion vectors. Predicted motion vectors can be created in a predefined manner, such as by calculating the median of the encoded or decoded motion vectors of adjacent blocks. Another method for creating motion vector predictions is to generate a candidate prediction list based on adjacent and / or juxtaposed blocks in a time reference picture, and to signal the selected candidate as the motion vector predictor. In addition to predicting motion vector values, reference indices from previously encoded / decoded pictures can also be predicted. Reference indices can be predicted based on adjacent and / or juxtaposed blocks in a time reference picture. Furthermore, efficient video codecs can employ additional motion information encoding / decoding mechanisms, often referred to as merging / merging modes, where all motion field information, including motion vectors and corresponding reference picture indices for each available reference picture list, is predicted and used without any modification / correction. Similarly, motion field information prediction is performed using motion field information from adjacent and / or juxtaposed blocks in a time reference picture, and the motion field information used is signaled in a motion field candidate list populated with motion field information from available adjacent / juxtaposed blocks.

[0067] Like many other video compression standards, H.264 / AVC and HEVC divide an image into a rectangular grid. For each rectangular grid, a similar block from a reference image is indicated for inter-frame prediction. The position of the predicted block is encoded as a motion vector, which indicates the position of the predicted block relative to the encoded block.

[0068] A bitstream can be defined as a sequence of bits or a sequence of syntax structures. The bitstream format can constrain the order of the syntax structures in the bitstream.

[0069] Syntax elements can be defined as data elements represented in a bitstream. Syntax structures can be defined as zero or more syntax elements that appear together in a specified order in a bitstream.

[0070] In some encoding formats or standards, bitstreams can be in the form of Network Abstraction Layer (NAL) unit streams or byte streams, which form representations of encoded images and related data to create one or more encoded video sequences.

[0071] A NAL unit can be defined as a syntax structure containing an indication of the data type to be followed, and bytes containing that data in RBSP form, interspersed with start code emulation prevention bytes where necessary. A Raw Byte Sequence Payload (RBSP) can be defined as a syntax structure containing integer bytes encapsulated within a NAL unit. An RBSP can be empty or a string of data bits containing syntax elements, followed by RBSP stop bits, and then zero or more subsequent bits equal to 0.

[0072] NAL cells include a header and a payload. The NAL cell header indicates the type of NAL cell, etc.

[0073] In some encoding formats (such as AV1), a bitstream may include a sequence of Open Bitstream Units (OBUs). An OBU includes a header and a payload, where the header identifies the type of OBU. Additionally, the header may include the size of the payload in bytes.

[0074] In the claims and described embodiments, the phrases "along the bitstream" (e.g., "along the bitstream indicator") or "along the coding unit of the bitstream" (e.g., "along the coding block indicator") can be used to refer to transmissions, signaling, or storage that are associated with, but not included in, "out-of-band" data, respectively, with the bitstream or coding unit. Decoding the phrase "along the bitstream" or "along the coding unit of the bitstream," etc., can refer to decoding the referenced out-of-band data (which may be obtained from out-of-band transmission, signaling, or storage) associated with the bitstream or coding unit, respectively. For example, when the bitstream is contained in a container file, the phrase "along the bitstream" can be used, such as in a file conforming to the ISO Basic Media File Format, and some file metadata is stored in the file in a manner that associates metadata with the bitstream, such as boxes in sample entries for tracks containing the bitstream, sample groups for tracks containing the bitstream, or timing metadata tracks associated with tracks containing the bitstream.

[0075] The video codec can support motion-compensated prediction from one source image (unidirectional prediction) and two sources (bidirectional prediction). In the case of unidirectional prediction, a single motion vector is applied, while in the case of bidirectional prediction, two motion vectors are signaled, and the motion-compensated predictions from both sources are averaged to create the final sample prediction. In the case of weighted prediction, the relative weights of the two predictions can be adjusted, or the signaling offset can be added to the prediction signal.

[0076] Besides applying motion compensation for inter-image prediction, similar methods can be applied to intra-image prediction. In this case, the displacement vector indicates the predicted location of a block that can be copied from the same image to form the block to be encoded or decoded. This intra-block copying method can significantly improve coding efficiency when there are repetitive structures (such as text or other graphics) within the frame.

[0077] The prediction residuals after motion compensation or intra-frame prediction can be transformed using a transform kernel (such as DCT) before encoding. This is done because there is often some correlation between the residuals, and in many cases, the transform can help reduce this correlation and provide more efficient encoding.

[0078] Video encoders can use a Lagrange cost function to find the optimal coding pattern, such as the desired macroblock pattern and associated motion vectors. This cost function uses a weighting factor λ to correlate the (precise or estimated) image distortion caused by lossy coding methods with the (precise or estimated) amount of information needed to represent pixel values ​​in an image region: (Equation 1) in C It is the Lagrange cost that will be minimized.D It takes into account image distortion (e.g., mean square error) based on pattern and motion vectors, and R It represents the number of bits required for the data needed to reconstruct the image patch in the decoder (including the amount of data used to represent candidate motion vectors).

[0079] The features and coding tools included in VVC include the following: • Intra-frame prediction - 67-frame mode, using wide-angle mode extension - Block size and mode-dependent 4-tap interpolation filter - Location-dependent intra-frame prediction combination (PDPC) - Cross-component linear model intra-frame prediction (CCLM) - Multi-reference line intra-frame prediction - Intra-frame sub-partition - Weighted intra-frame prediction with matrix multiplication • Image prediction - Block motion replication using spatial, temporal, historical, and pairwise average merging candidates. - Affine motion inter-frame prediction - Sub-block-based temporal motion vector prediction - Adaptive motion vector resolution - 8×8 block-based motion compression for time motion prediction - High-precision (1 / 16 pixel) motion vector storage and motion compensation, using an 8-tap interpolation filter for the luminance component and a 4-tap interpolation filter for the chrominance component. - Triangular partition - Combining intra-frame and inter-frame prediction - Combined with MVD (MMVD) - Symmetric MVD encoding - Bidirectional optical flow - Decoder-side motion vector refinement - Bidirectional prediction using CU-level weights • Transformation, quantization, and coefficient encoding - Multiple master transform selection for DCT2, DST7, and DCT8 - Secondary transformation in the low-frequency region - Sub-block transformation of inter-frame prediction residuals - Due to quantization, the maximum QP increased from 51 to 63. - Transform coefficient encoding with symbolic data hiding - Transform skip residual coding • Entropy coding - Arithmetic coding engine with adaptive dual-window probability updates • Loop filter - Cyclic Reshaping - Deblocking filter with robust long filter - Sample adaptive offset - Adaptive Loop Filter • Screen content encoding: - The current image is for reference only and has a limited reference area. • 360-degree video encoding - Horizontal orbital motion compensation • Advanced syntax and parallel processing - Use direct reference image list signaling for reference image management - A tile group with rectangular tile groups

[0080] In H.266 / VVC, the following block partitioning applies. Images can be divided into Code Tree Units (CTUs). Images can also be divided into slices, tiles, bricks, and sub-images. CTUs can be split into smaller CUs using a quadtree structure. Each CU can be partitioned using quadtrees and nested multi-type trees, including ternary and binary partitions.

[0081] There are specific rules for inferring partitions within the boundaries of an image.

[0082] Redundant splitting patterns are not allowed in nested multi-type partitions.

[0083] Some video coding tools perform filtering operations, which convolve a set of reference samples with a set of filter parameters to output a predicted value for, for example, a sample in an image. In some cases, the filter parameters can be predetermined or signaled in the bitstream. In others, such as when using a cross-component linear model (CCLM) or cross-component convolutional model (CCCM) for prediction, these parameters are computed using the reference sample set in both the encoder and decoder. Typically, computing such filter parameters involves inverting the autocorrelation matrix, a computationally challenging operation. Furthermore, when many filter parameters need to be determined, the size of the autocorrelation matrix can become large, potentially leading to numerical stability issues (overflow or underflow).

[0084] To reduce cross-component redundancy, a cross-component linear model (CCLM) can be used to predict the mode in VVC. For this mode, chromaticity samples are predicted based on reconstructed luminance samples from the same CU using the following linear model: (3-1) in This represents the predicted chromaticity samples in the CU. This represents downsampled reconstructed luminance samples from the same CU.

[0085] The CCLM parameters (α and β) are derived from at most four adjacent chroma samples and their corresponding downsampled luminance samples. Assuming the current chroma block size is W×H, then W' and H' are set to... - When applying the LM pattern, W'=W, H'=H; - When applying the LM-A mode, W' = W + H; - When applying the LM-L pattern, H' = H + W.

[0086] The adjacent positions mentioned above are denoted as S[0, -1]……S[W'-1, -1], and the adjacent positions to the left are denoted as S[-1,0]……S[-1, H'-1]. Then, four samples are selected as follows. - When LM mode is applied and both the upper and left adjacent samples are available, S[W' / 4, -1], S[3] W' / 4, -1], S[-1, H' / 4], S[-1, 3 H' / 4]; - When applying LM-A mode or when only the upper adjacent sample is available, S[W' / 8, -1], S[3] W' / 8, -1]、S[5 W' / 8, -1]、S[7 W' / 8, -1]; - When applying the LM-L mode or when only the left adjacent sample is available, S[-1, H' / 8], S[-1, 3] H' / 8]、S[-1, 5 H' / 8]、S[-1, 7 H' / 8].

[0087] Four adjacent brightness samples at the selected location were downsampled and compared four times to find the two smaller values: x 0 A and x 1 A and two larger values: x 0 B and x 1 B Its corresponding chromaticity sample value is represented as y. 0 A y 1 A y 0 B and y 1 B Then, x A xB y A and y B The following derivation ; ; ;

[0088] Finally, the linear model parameters are obtained according to the following formula:

[0089] Figure 1 This example illustrates the positions of the left and top samples, as well as the sample in the current block, involved in the CCLM mode. The division operation for calculating the parameter α is implemented using a lookup table. To reduce the memory required to store the table, the diff value (the difference between the maximum and minimum values) and the parameter α are represented using exponential notation. For example, the diff is approximated using a 4-bit significant part and an exponent. Therefore, the 1 / diff table is reduced to 16 elements, corresponding to the 16 values ​​of the significant part, as shown below:

[0090] This will help reduce the complexity of the calculations and the amount of memory required to store the necessary tables.

[0091] In addition to the top and left templates being used together to calculate the coefficients (i.e., parameters) of the linear model, they can also be used alternately for other 2LM modes, known as LM_A and LM_L modes.

[0092] In LM_A mode, only the template above is used to calculate the linear model coefficients (i.e., parameters). To obtain more samples, the template above is expanded to... (W+H) In LM_L mode, only the left-hand template is used to compute the linear model coefficients (i.e., parameters). To obtain more samples, the left-hand template is expanded to... (H+W) .

[0093] For non-square blocks, the upper template is expanded to... W + W The template on the left is expanded to H + H .

[0094] To match the chroma sample positions of a 4:2:0 video sequence, two types of downsampling filters are applied to the luminance samples to achieve a 2:1 downsampling rate in both the horizontal and vertical directions. The selection of the downsampling filters is specified by the SPS level flag. The two downsampling filters are as follows, corresponding to "Type 0" and "Type 2" content, respectively.

[0095] It should be understood that when the upper reference line is located at the CTU boundary, only one luminance line (the universal line buffer in intra-frame prediction) is used to create downsampled luminance samples.

[0096] This parameter calculation is performed as part of the decoding process, not just as part of the encoder's search operation. Therefore, the α and β values ​​are not passed to the decoder using syntax.

[0097] For chroma intra-frame coding, a total of eight intra-frame modes are allowed. These modes include five traditional intra-frame modes and three cross-component linear model modes (CCLM, LM_A, and LM_L). The chroma mode signaling and derivation process are as follows: Figure 2a As shown in Table 1, chroma mode coding directly depends on the intra-prediction of the corresponding luma block. Because a separate block partitioning structure for luma and chroma components is enabled in the I-slice, one chroma block can correspond to multiple luma blocks. Therefore, for the chroma DM mode, the intra-prediction mode of the corresponding luma block covering the center position of the current chroma block is directly inherited.

[0098] Use a single binary table, regardless of the value of sps_cclm_enabled_flag, such as Figure 2b As shown in Table 2. In Table 2, the first binary bit indicates whether it is in normal mode (0) or LM mode (1). If it is in LM mode, the next binary bit indicates whether it is LM_CHROMA (0). If it is not LM_CHROMA, the next binary bit indicates whether it is LM_L (0) or LM_A (1). In this case, when sps_cclm_enabled_flag is 0, the first binary bit of the binarization table corresponding to intra_chroma_pred_mode can be discarded before entropy encoding. Alternatively, in other words, the first binary bit is inferred to be 0 and therefore not encoded. This single binarization table is used for both cases where sps_cclm_enabled_flag is equal to 0 and 1. The first two binary bits in Tables 3-4 are context-encoded with their own context model, and the remaining binary bits are bypass-encoded.

[0099] Furthermore, to reduce luma and chroma latency in dual-tree systems, when 64×64 luma coding tree nodes are partitioned using Not Split (and ISP is not used for 64×64 CUs) or QT, chroma CUs in 32×32 / 32×16 chroma coding tree nodes are allowed to use CCLM in the following manner: - If the 32×32 chroma node is not split or partitioned by QT, then all chroma CUs in the 32×32 node can use CCLM. - If a 32×32 chroma node is partitioned using horizontal BT, while the 32×16 child nodes are not split or are split using vertical BT, then all chroma CUs in the 32×16 chroma node can use CCLM. Under all other conditions of luminance and chrominance coding tree splitting, chrominance CU is not allowed to use CCLM.

[0100] The CCLM included in VVC is extended by adding three multi-model LM (MMLM) modes. In each MMLM mode, reconstructed neighboring samples are divided into two classes using a threshold that is the average of the brightness reconstructed neighboring samples. The linear model for each class is derived using the least mean square (LMS) method. For the CCLM mode, the LMS method is also used to derive the linear model. Figure 3a The diagram illustrates two luminance-to-chrominance models obtained when the luminance (Y) threshold is 17. Each luminance-to-chrominance model has its own linear model parameters α and β. For example... Figure 3b As shown, each luminance-to-chrominance model corresponds to a spatial segmentation of the content (i.e., they correspond to different objects or textures in the scene).

[0101] An improved version of cross-component prediction (called the Convolutional Cross-Component Model (CCCM)) uses 2D filter kernels to derive the luma-to-chroma model. The filter coefficients are derived on the decoder side using the reconstructed input dataset and chroma samples. For the filter coefficient derivation, juxtaposed reference sample regions (composed of reconstructed luma and chroma samples) are defined for both luma and chroma, such as... Figure 4 As shown, however, any number of reference lines can be used (which can be implemented by both the encoder and decoder). Typically, the reference samples can contain any chroma and luminance samples reconstructed by both the encoder and decoder. After the reference samples are determined, different types of linear regression tools can be used to derive the filter coefficients, such as ordinary least squares estimation, orthogonal matching pursuit, optimized orthogonal matching visit, ridge regression, or minimum absolute shrinkage and selection operators.

[0102] The dimensions of the filter kernel can be, for example, 1×3 (1D vertical), 3×1 (1D horizontal), 3×3, 7×7, or any other dimension, and can be shaped (by selecting only a subset of all possible kernel locations) into a cross, a rhombus, or any given shape. When referring to samples within the filter kernel, the following notations are used: North (above), East (right), South (below), West (left), and Center, as follows: Figure 5 As shown, the letters N, E, S, W, and C are used. Figure 5 The diagram illustrates a 3-tap vertical core 501, a 3-tap horizontal core 502, a 5-tap cross core 503, and a 25-tap diamond core 504.

[0103] The overall method for reconstructing chroma samples using convolutions between the filter kernels obtained from the decoder side and the input dataset is referred to here as the Convolutional Cross-Component Model (CCCM). The following steps can be used to perform CCCM operations: - Define juxtaposed reference regions for the luminance and chrominance components; - Downsample the luminance samples to match the chromaticity grid (optional); - Scan the luminance and chromaticity samples of the reference region and collect available statistics (such as autocorrelation matrix and cross-correlation vector) based on the filter shape. - Solve for the filter coefficients by minimizing the squared error (or any other metric) based on available statistical data (such as the autocorrelation matrix and cross-correlation vector); - Predicted chromaticity blocks are computed by convolving downsampled luminance samples with a filter kernel.

[0104] In the following text, (which may be downsampled) luminance samples are defined as a 2D array indexed using horizontal x-coordinates and vertical y-coordinates. Y(x, y) Furthermore, the juxtaposed chromaticity samples are defined as a 2D array. C(x, y) Furthermore, the filter kernel (i.e., the coefficients) is defined as a 3×3 array. F(i, j) At the sample level, Y and F The convolution between them is defined as .

[0105] When using other data items, such as non-linear square root terms, the additional convolution becomes , in These are filter coefficients located outside the 2D filter kernel, but have already been obtained as part of the system of linear equations used to solve for the 2D filter coefficients in step 4 above. Similarly, bias terms can be added to the convolution. .

[0106] Angular intra-frame prediction (also known as directional intra-frame prediction) can be performed by extrapolating sample values ​​from reconstructed reference samples using a given orientation. Reference samples can include the immediately adjacent sample rows above and to the right of the current block (when available) and the immediately adjacent sample column to the left of the current block (when available), where availability may require an earlier decoding order than the current block and exists within the same image segment, such as within the same tile. To simplify the process, depending on the orientation of the chosen prediction mode, all sample locations within a prediction block can be projected onto a single reference row or column. Predicted samples within the block being encoded / decoded can be obtained through the following steps: - By applying the selected prediction direction, the location of the predicted sample is projected onto a location within a reference row or column. The location within the reference row or column can have fractional sample precision, such as 1 / 32 pixel precision. - Interpolate the value at the sample position in the reference row or column from the reference sample in the reference row / column.

[0107] Multi-reference line (MRL) intra-prediction uses more reference lines for intra-prediction. Figure 6 The example depicts four reference lines, where samples from segments A and F are not retrieved from reconstructed neighboring samples, but are instead filled with the closest samples from segments B and E, respectively. Intra-image prediction in HEVC uses the nearest reference line (i.e., reference line 0). In MRL, two additional lines (reference line 1 and reference line 3) are used.

[0108] The index of the selected reference line (mrl_idx) is signaled and used to generate the intra-predictor. For reference line idx greater than 0, only the additional reference line mode is included in the MPM list, and only the MPM index is signaled without sending the remaining modes. The reference line index is signaled before the intra-predictor modes, and in the case of signaling a non-zero reference line index, the planar mode is excluded from the intra-predictor modes.

[0109] The MRL for the first row block within the CTU is disabled to prevent the use of extended reference samples outside the current CTU line. Additionally, PDPC is disabled when additional lines are used. For MRL mode, the derivation of the DC value in the DC intra-prediction mode with a non-zero reference line index is consistent with the derivation for reference line index 0. MRL requires the CTU to store three adjacent luma reference lines to generate predictions. The Cross-Component Linear Model (CCLM) tool also requires three adjacent luma reference lines for its own downsampling filter. The definition of MRL using the same three rows is consistent with CCLM to reduce the decoder's storage requirements.

[0110] Intra-frame sub-partitioning (ISP) divides the luma intra-prediction block vertically or horizontally into 2 or 4 sub-partitions based on the block size. For example, the minimum block size for ISP is 4×8 (or 8×4). If the block size is larger than 4×8 (or 8×4), the corresponding block is divided into four sub-partitions. It has been noted that M×12 (M≤64) and 128×N (N≤64) ISP blocks may pose potential problems for 64×64 VDPUs. For example, in a single-tree case, an M×128 CU has M×128 luma TBs and two corresponding... Chroma TB. If the CU uses an ISP, the luminance TB would be divided into four M×32 TBs (horizontal split only), each TB smaller than a 64×64 block. However, in the current ISP design, the chroma blocks are not divided. Therefore, both chroma components are larger than 32×32 blocks.

[0111] Similarly, using an ISP with a 128×N CU can produce similar results. Therefore, both of these situations pose a problem for a 64×64 decoder pipeline. Consequently, the CU size that can be used with an ISP is limited to a maximum of 64×64. All sub-partitions satisfy the condition of having at least 16 samples.

[0112] Matrix-weighted intra-prediction (MIP) is an intra-prediction technique in VVC. To predict samples from a rectangular block of width W and height H, MIP takes H reconstructed neighbor boundary samples from the left row of the block and W reconstructed neighbor boundary samples from the top row of the block as input. If reconstructed samples are unavailable, they are generated using the traditional intra-prediction method. The generation of the predicted signal is based on three steps: averaging, matrix-vector multiplication, and linear interpolation, as follows: Figure 7 As shown.

[0113] Merging modes with Motion Vector Difference (MMVD) involve signaling MVD and resolution index after signaling merge candidates. The merge list can include the following candidates: 1) Spatial MVP from spatially adjacent CUs 2) Time MVP from co-located CU 3) Historical MVP from FIFO table 4) Pairwise average MVP (using candidates already in the list) 5) Zero MV.

[0114] In symmetric MVD, the motion information of list 1 is derived from the motion information of list 0 in the case of bidirectional prediction.

[0115] In affine prediction, several motion vectors are indicated / signed for different angles of a block; these motion vectors are used to derive the motion vectors of sub-blocks. In affine merging, the affine motion information of a block is generated based on the ordinary or affine motion information of neighboring blocks.

[0116] In sub-block-based temporal motion vector prediction, the motion vectors of the sub-blocks of the current block are predicted based on the appropriate sub-blocks in the reference frame, which are indicated by the motion vectors of spatially adjacent vectors (if available).

[0117] In Adaptive Motion Vector Resolution (AMVR), the accuracy of the MVD for each CU is signaled.

[0118] In bidirectional forecasting with CU-level weights, the index indicates the value of the weighted average weight used for the two forecast blocks.

[0119] Two-way optical flow (BDOF) refines motion vectors in the case of bidirectional prediction. BDOF uses the signaled motion vectors to generate two prediction blocks. Then, the gradient values ​​of the two prediction blocks are used to compute motion refinement to minimize the error between the two prediction blocks. The motion refinement and gradient values ​​are then used to refine the final prediction block.

[0120] In HEVC, bidirectional prediction signals are generated by averaging two prediction signals acquired from two different reference images and / or using two different motion vectors. In VVC, the bidirectional prediction mode is extended beyond simple averaging to allow for a weighted average of the two prediction signals. This is known as bidirectional prediction with CU-level weights (BCW).

[0121] Five weights are allowed in weighted average two-way forecasting. w ∈{-2,3,4,5,10}. For each bidirectional prediction CU, the weights are... w The weights are determined using one of two methods: 1) For non-merged CUs, the weight index is signaled after the motion vector difference; 2) For merged CUs, the weight index is inferred from neighboring blocks based on the merge candidate index. BCW is only applicable to CUs with 256 or more luminance samples (i.e., CU width multiplied by CU height is greater than or equal to 256). For low-latency images, all 5 weights are used. For non-low-latency images, only 3 weights are used. w∈{3,4,5} ).

[0122] At the encoder, a fast search algorithm is applied to find the weight indices without significantly increasing the encoder's complexity. These algorithms are summarized below. For more details, please refer to the VTM software and documentation JVET-L0646. When used in conjunction with AMVR, if the current image is a low-latency image, a conditional unequal weight check is performed only for 1-pel and 4-pel motion vector precision. - When combined with affine, affine ME will only be performed on unequal weights if the affine mode is selected as the current best mode. - When the two reference images in bidirectional prediction are the same, unequal weights are only conditionally checked. - When certain conditions are met, unequal weights will not be searched based on the POC distance between the current image and its reference images, the QP encoding, and the time level.

[0123] The BCW weight index is encoded using a context-coded binary bit followed by a bypass-coded binary bit. The first context-coded binary bit indicates whether equal weights are used; if unequal weights are used, the bypass-coded bit is used to signal the additional binary bit to indicate which unequal weight is used.

[0124] Weighted Prediction (WP) is an encoding tool supported by the H.264 / AVC and HEVC standards for efficient encoding of fading video content. Support for WP has also been added to the VVC standard. WP allows sending weighted parameters (weights and offsets) for each reference picture in each of the reference picture lists L0 and L1. Then, during motion compensation, the weights and offsets of the corresponding reference pictures are applied. WP and BCW are designed for different types of video content. To avoid interaction between WP and BCW, which would complicate the VVC decoder design, if the CU uses WP, the BCW weight index is not signaled, and w is inferred as 4 (i.e., equal weights are applied). For merging CUs, the weight index is inferred from neighboring blocks based on the merge candidate index. This can be applied to both normal merge mode and inherited affine merge mode. For constructed affine merge mode, affine motion information is constructed based on motion information from up to 3 blocks. The BCW index of the CU using the constructed affine merge mode is simply set to be equal to the BCW index of the first control point MV.

[0125] In VVC, combined inter-frame / intra-frame prediction (CIIP) and bidirectional prediction with CU-level weights (BCW) cannot be jointly applied to the CU. When the CU is encoded in CIIP mode, the BCW index of the current CU is set to 2, for example, with equal weights.

[0126] In VVC, when a CU is encoded in merged mode, if the CU contains at least 64 luma samples (i.e., the CU width multiplied by the CU height is equal to or greater than 64), and if both the CU width and height are less than 128 luma samples, an additional flag is signaled to indicate whether Combined Inter-Frame / Intra-Frame Prediction (CIIP) mode is applied to the current CU. As the name suggests, CIIP prediction combines inter-frame prediction signals with intra-frame prediction signals. The inter-frame prediction signal in CIIP mode... The same inter-frame prediction process applied to the regular merging mode is used for derivation; and the intra-frame prediction signal... The derivation follows conventional intra-frame prediction processing with a planar pattern. Then, a weighted average is used to combine the intra-frame and inter-frame prediction signals. , The weight values ​​are calculated based on the encoding patterns of the top and left adjacent blocks, as shown below: - If the top neighbor is available and is intra-frame encoded, then isIntraTop Set to 1, otherwise isIntraTop Set to 0; - If the left neighbor is available and intra-frame encoded, then isIntraLeft Set to 1, otherwise isIntraLeft Set to 0; - if( isIntraLeft+isIntraTop If ) equals 2, then wt Set to 3; - Otherwise, if ( isIntraLeft+isIntraTop If ) equals 1, then wt Set to 2; Otherwise, wt Set to 1. Figure 8 The illustration shows an example of the top and left adjacent blocks used in CIIP weight derivation.

[0127] CIIP predictions are formed as follows:

[0128] Local illumination compensation (LIC) is an inter-frame prediction technique that models the local illumination variation between the current block and its predicted block as a function of the illumination variation between the current block template and the reference block template. The parameters of this function can be represented by scale α and offset β, forming a linear equation, i.e., α... p[x]+β is used to compensate for illumination changes, where p[x] is the reference sample pointed to by MV at position x on the reference image. Since α and β can be derived based on the current block template and the reference block template, α and β do not require signaling overhead except that the LIC flag in AMVP mode is signaled to indicate the use of LIC.

[0129] The local illumination compensation proposed in JVET-O0066 was used for unidirectional prediction of inter-frame CU in ECM, and the following modifications were made. • Intra-frame neighboring samples can be used to derive LIC parameters; • For blocks with fewer than 32 luminance samples, LIC is disabled; • For both non-subblock and affine modes, the LIC parameter derivation is performed based on the template block sample corresponding to the current CU, rather than on the partial template block sample corresponding to the top-left 16×16 cell; • Samples of the reference block template are generated using MC and block MV without rounding them to integer pixel precision.

[0130] In bidirectional motion compensation, out-of-bounds (OOB) prediction samples are discarded, and only non-OOB predictors (if available) are used to generate the final predictor. OOB prediction samples are those whose reference samples are at least partially located outside the reference image. Specifically, let... and This indicates the position of a predicted sample within the current block. and (x=0, 1) represents the MV of the current block; , , and These are the positions of the four boundaries of the image. A predicted sample is considered out-of-bounds (OOB) when at least one of the following conditions is met: , , , in half_pixel An 8 indicates the half-pixel sample distance at a precision of 1 / 16 of a pixel sample.

[0131] After checking the OOB conditions for each sample, a final predicted sample for a bidirectional block is generated, as shown in Algorithm A below: if It is OOB, and If it is not OOB, then Otherwise, if It is not OOB, and If it is OOB, then otherwise .

[0132] In the final prediction, as shown above, the first part ("If...otherwise...") is the unidirectional prediction part of the two-way prediction. The two-way prediction part of the two-way prediction is the second part ("Otherwise..."). =…), where two unidirectional forecasts are (weighted) averaged to form a true bidirectional forecast.

[0133] When BCW is enabled, the OOB check process also applies.

[0134] Finally, it should be noted that this sample-adaptive bidirectional prediction process may only apply to prediction units where at least the reference block is initially detected as partially or completely out of bounds. Therefore, block-level OOB criteria are checked first. If both prediction blocks are non-OOB, then normal bidirectional prediction is performed.

[0135] In OOB-aware bidirectional inter-frame prediction, only samples available in both prediction blocks (i.e., motion-compensated blocks P0 and P1) can be merged into the bidirectional prediction. For regions where only one predictor (P0 or P1) is available (the other being outside the image boundary), only samples belonging to the available unidirectional predictor can be used. Therefore, the final prediction for a block is a mixture of unidirectional and bidirectional prediction samples. This can lead to artifacts and abrupt edges when unidirectional prediction samples have different brightness or texture features compared to bidirectional prediction samples.

[0136] The samples outside the boundary can be filtered to generate the final predicted samples for brightness, as shown below: if It is OOB, and If it is not OOB, then Otherwise, if It is not OOB, and If it is OOB, then otherwise .

[0137] Using the least squares method and Derivation of scale and offset on the non-OOB portion a 0、 a 1. b 0、 b 1, and

[0138] in yes and Non-OOB sample 905, such as Figure 9 As shown, used for The samples are filtered. Scale and offset can be derived using integer-precision sample covariance and sample variance. Only... or With 905 OOB samples, and therefore in most cases only the parameter set needs to be derived. This filtering method also applies when BCW is enabled, and in this case, the parameters are derived accordingly using the BCW weights. The sample. Scale and offset. a 0、 a 1. b 0、 b 1 can be collectively referred to as filter coefficients, filter parameters, or filter parameters, because they define the filters used to generate a modified representation of the input samples.

[0139] In the filtering method, the filter coefficients can be derived by minimizing the squared error between bidirectional prediction block samples and unidirectional prediction block samples, and these filter coefficients are used to filter at least a portion of the unidirectional prediction block.

[0140] The filtering schemes described above can be computationally demanding for high-resolution video, especially on the decoder side. This is because a large number of boundary samples are used in the filter coefficient derivation. This problem becomes particularly pronounced in existing video coding systems, such as ECM-10.0, where the maximum CU size is 256×256. Tens of thousands of samples can be involved in the filter coefficient derivation, making the local implementation on the decoder side (involving all available reference samples) relatively slow. Therefore, the selection of reference samples must be done in a downsampling and normalization manner.

[0141] This embodiment aims to apply a standard downsampling strategy to the filtering scheme in order to accelerate the implementation on both the encoder and decoder sides.

[0142] Figure 9 The reference samples involved in the derivation of the filter coefficients according to the embodiment are illustrated geometrically. Specifically, Figure 9 The dimensions derived from the OOB and non-OOB partitions of the block are shown. W×H The bounding box 910. A spatial downsampling strategy can be derived over the bounding box region 910. Downsampled samples within the bounding box 910 are used as the derivation of filter coefficients. (Based on coordinates...) (x_min, y_min) and (x_max, y_max) Sample definition within a defined rectangular region of 900 W×H Area 910, of which W= x_max-x_min+1 and H = y_max - y_min + 1The samples within region 910 or the bounding box were used for the derivation of filter coefficients involving sample variance and sample covariance.

[0143] As the bounding box size increases, for example, 100 × 100 = 10,000 samples, the computational complexity increases linearly with the number of samples, even for simple coefficient derivations. However, robust coefficient derivations can typically be performed with only 256 samples. Downsampling of the reference samples must be performed in the same manner on both the encoder and decoder, leading to the development of several canonical downsampling techniques for the bounding box regions.

[0144] Method 1

[0145] The method according to the first embodiment includes selecting a bounding box 910 containing intervals. dx and dy Uniformly spaced samples. For W×H bounding box, which corresponds to selection (W×H) / (dx dy) A reference sample of the number. For example, if dx =2 and dy If the value is 2, then every other sample is selected horizontally and vertically.

[0146] Method 2

[0147] The method according to the second embodiment includes selecting uniformly spaced samples within the bounding box 910 such that the total number of reference samples never exceeds a threshold. T Therefore, for a square bounding box, there exists dx>=HW / T / 2 and dy>=HW / T / 2 For example, if T=256 , W=32 , H=32 ,but W × H = 1023 = 4T Then the following settings are: dx = 4 / 2 = 2 , dy = 4 / 2 = 2 For rectangular bounding boxes, set... HW / T=R And find two positive integers greater than zero. A , B , making A×B>=R Then there are two choices, either... dx =A , dy=B Or dx=B and dy=A .

[0148] Method 3

[0149] The method according to the third embodiment includes anisotropic downsampling of samples within the bounding box 910. This means that different portions of the bounding box 910 use different levels of downsampling, such as... Figure 10As shown. In Figure 9 and Figure 10 In the example, region 1 1010 can be set dx=1 and dy=1 To avoid downsampling, region 2 uses 1020. dx=2 and dy=2 Area 3 uses dx=4 and dy=4 Therefore, the sample closest to the OOB region is optimally represented by a reference sample selection. dx and dy The value can be set independently as in the method according to the second embodiment. Regions 1010, 1020, and 1030 can have rectangular or non-rectangular shapes of different sizes. Any number of regions can be used.

[0150] Method 4

[0151] The method according to the fourth embodiment includes selecting all or a subset of samples included in the filter coefficient derivation based on the values ​​of candidate samples. For example, the minimum and maximum sample values ​​can be determined in a portion of the prediction block (or a subset of samples in that block; or a reference sample set or subset used to generate the prediction block), which has only one prediction of the reference sample within the image. Then, when selecting samples for parameter generation, the first samples within that range can be selected according to the determined scan order. N The number of samples N can be selected, and the remaining samples can be omitted. The scanning order can be advantageously chosen so that samples closer to the boundary between OOB and non-OOB samples can be scanned before those further away from the boundary. The number of samples N can be a constant or a variable determined based on available information (e.g., the size or dimension of the prediction block).

[0152] Such a range of values ​​can naturally be determined in different ways. For example, instead of directly using the minimum and maximum values, a certain margin can be applied to these values. For instance, a constant or a value related to the sample bit depth can be added to the found maximum value and subtracted from the found minimum value. Similarly, the found minimum and maximum values ​​can be replaced by multiplying them by different factors. Alternatively, an average value can be determined, and a range can be generated based on this, for example, by adding an offset to the average value to calculate the maximum value and subtracting the same or different offset to calculate the minimum value.

[0153] Method 5

[0154] In the method according to the fifth embodiment, the downsampling method includes selecting samples based on a downsampling matrix. When using the hard-coded downsampling matrix currently used by InterCCCM in ECM-10.0, care must be taken to resolve the bounding box 910. H andW It's not a problem of integer powers of 2. This downsampling matrix guarantees that for any... W×H Block (of which) W and H (is an integer power of 2), downsampling factor dx and dy Produced the most T One reference sample.

[0155] For fixed T downsampling matrix Sx[9][9] and Sy[9][9] For any of the C-style pseudocode H , W supply dx and dy Values, as shown below: , .

[0156] Given bounding box width W and height H From the hard-coded downsampling matrix Sx , Sy Searching for downsampling factors dx and dy for dx=Sx[xi][yi] and dy=Sy[xi][yi] ,in xi and yi In C-style pseudocode, it is defined as follows: ; ;

[0157] The above pseudocode is designed to handle the following case: for non-integer powers of 2. V ,have (1< <floorLog2(V))<V For example, if W =63, then we have floorLog2(W)=5 Without the above process, the retrieved values ​​would be too low. dx and dy The value exceeds the threshold. T Through the above process, it is possible to achieve... xi = (32 + 8) <W And therefore xi Become xi = xi + 1 = 6 And select fewer than T samples. The number of samples selected may cause some overshoot in the following cases: for W and / or H , The worst-case scenario is that all of the above conditions apply simultaneously. W and H Both, and the number of the selected reference samples becomes 1.25 1.25 T~=1.5625T In implementations where overshoot is unacceptable, the following formula is used. ; ; However, for many non-power 2 exponents H and W This will produce an undershoot in the number of reference samples. A population parameter (i.e., an overshoot parameter) can be defined. Px , Py To control overshoot and undershoot behavior: ; ;

[0158] Method 6 Downsampling can be performed using an increasing step size in one dimension while keeping the downsampling constant in another dimension. This means that downsampling involves selecting samples with different intervals in different directions. For example, if the OOB sample is located in... Figure 9 On the left side of the block shown, when selecting reference samples for filter coefficient calculation, it can be determined that there is no downsampling in the vertical direction. However, when collecting reference samples in the horizontal direction, increasingly more columns can be skipped to achieve sampling that includes a large number of reference samples near the boundaries, while also including some further away samples that can help build a more robust sample mapping model. The number of columns to skip can be determined exponentially, linearly, or in another chosen manner. For example, the gap between columns may double each time a new column of samples is included as a reference sample. Alternatively, an exponential or linear formula can be used to calculate the column indices of the columns included as the source of reference samples. For example, if the first column of samples used to collect reference samples has a horizontal coordinate... x_min Then except x=x_min In addition to the columns, it has the following x The coordinate column can also be included in the reference sample collection process: ,in and

[0159] In one embodiment, the threshold T It can vary based on PU, CU, CTU, slice, frame, or sequence attributes.

[0160] In one embodiment, the threshold T It can be fixed, signaled, or inferred based on PU, CU, CTU, slice, frame, or sequence attributes.

[0161] In one embodiment, the number of regions in the method according to the third embodiment may be fixed, signaled, or inferred based on PU, CU, CTU, slice, frame, or sequence attributes.

[0162] In embodiments based on the method according to the third embodiment, the number of regions can be determined according to... H and W And thus, it was derived.

[0163] In embodiments based on the method according to the third embodiment, the geometry (i.e., shape) of the region can be determined according to... H and W And thus, it was derived.

[0164] In one embodiment, the overshoot parameter according to the method of the fifth embodiment Px , Py It can be fixed, signaled, or inferred based on PU, CU, CTU, slice, frame, or sequence attributes.

[0165] A method according to an embodiment, such as Figure 11 As shown in the diagram. The method generally includes: processing 1110 an image to predict a block based on prediction samples from two prediction blocks; determining 1120 when both of the two prediction blocks of the block are within the boundaries of the image, and applying bidirectional prediction based on the two prediction blocks of the block to obtain a bidirectional prediction block; determining 1130 when one of the two prediction blocks is at least partially outside the boundaries of the image, and applying unidirectional prediction based on the prediction block within the image boundaries to obtain a unidirectional prediction block; deriving 1140 filter coefficients based on a selected sample set of the bidirectional prediction block, wherein the selected sample set includes samples that have been downsampled in the bidirectional prediction block according to a downsampling method; and filtering 1150 at least a portion of the unidirectional prediction block according to the filter coefficients to generate a filtered unidirectional prediction block. Each step can be implemented by a corresponding module of a computer system.

[0166] An apparatus according to an embodiment includes: components for processing an image to predict blocks based on prediction samples of two prediction blocks; components for determining when both of the two prediction blocks of the block are within the boundaries of the image, and components for applying bidirectional prediction based on the two prediction blocks of the block to obtain a bidirectional prediction block; components for determining when one of the two prediction blocks is at least partially outside the boundaries of the image, and components for applying unidirectional prediction based on the prediction block within the image boundaries to obtain a unidirectional prediction block; components for deriving filter coefficients based on a selected sample set of the bidirectional prediction block, wherein the selected sample set includes samples that have been downsampled in the bidirectional prediction block according to a downsampling method; and components for filtering at least a portion of the unidirectional prediction block based on the filter coefficients to generate a filtered unidirectional prediction block. The apparatus includes at least one processor and a memory including computer program code, wherein the processor may further include processor circuitry. The memory and computer program code are configured, together with the at least one processor, to cause the apparatus to perform operations according to various embodiments. Figure 11 The method.

[0167] Figure 12 The diagram illustrates an example of a data processing system for a device. A single physical device can perform several functions; for example, all computational processes can be executed in a single processor if needed. The data processing system includes a main processing unit 100, a memory 102, a storage device 104, an input device 106, an output device 108, and a graphics subsystem 110, all of which are interconnected via a data bus 112.

[0168] The main processing unit 100 is a conventional processing unit arranged to process data within a data processing system. The main processing unit 100 may include or be implemented as one or more processors or processor circuit systems. The memory 102, storage device 104, input device 106, and output device 108 may include conventional components recognized by those skilled in the art. The memory 102 and storage device 104 store data within the data processing system 100.

[0169] Computer program code resides in memory 102 for implementing, for example, according to various embodiments. Figure 14The method is illustrated in the flowchart. Input device 106 inputs data into the system, while output device 108 receives data from the data processing system and forwards the data to, for example, a display. Data bus 112 is a conventional data bus, and although shown as a single line, it can be any combination of the following: processor bus, PCI bus, graphics bus, ISA bus. Therefore, those skilled in the art will readily recognize that the device can be any data processing device, such as a computer, personal computer, server computer, mobile phone, smartphone, or internet access device, such as an internet tablet.

[0170] Figure 13 An example of a video encoder is illustrated, where I n : The image to be encoded; P' n Predictive representation of image patches; D n : Prediction error signal; D' n : Reconstructed prediction error signal; I' n Preliminary image reconstruction; R' n : Final reconstructed image; T, T -1 Transform and inverse transform; Q, Q -1 : Quantization and inverse quantization; E: Entropy coding; RFM: Reference frame memory; P 帧间 Inter-frame prediction; P 帧内 : Intra-frame prediction; MS: Mode selection; F: Filtering. Figure 14 The diagram illustrates the block diagram of a video decoder, where P' n : Predictive representation of image patches; D' n : Reconstructed prediction error signal; I' n Preliminary image reconstruction; R' n : Final reconstructed image; T -1 Inverse transform; Q -1 Inverse quantization; E -1 Entropy decoding; RFM: Reference Frame Memory; P: Prediction (inter-frame or intra-frame); F: Filtering. The apparatus according to embodiments may include only an encoder or a decoder, or both.

[0171] Various embodiments can be implemented using computer program code residing in memory, causing the associated apparatus to perform the method. For example, the apparatus may include circuitry and electronics for processing, receiving, and transmitting data, computer program code in memory, and a processor that, when the computer program code is executed, causes the apparatus to perform the features of the embodiments. Furthermore, network devices such as servers may include circuitry and electronics for processing, receiving, and transmitting data, computer program code in memory, and a processor that, when the computer program code is executed, causes the network device to perform the features of various embodiments.

[0172] If necessary, the different functions discussed herein may be executed in different orders and / or concurrently with each other. Furthermore, if necessary, one or more of the functions and embodiments described above may be optional or may be combined.

[0173] Although various aspects of the embodiments are set forth in the independent claims, other aspects include other combinations of the features of the described embodiments and / or the features of the dependent claims with the features of the independent claims, and not just the combinations expressly set forth in the claims.

[0174] It should also be noted that while exemplary embodiments have been described above, these descriptions should not be considered limiting. Rather, several variations and modifications may be made without departing from the scope of this disclosure as defined in the appended claims.

Claims

1. An apparatus for encoding, comprising: A component used to process an image to predict blocks based on prediction samples from two prediction blocks; The components for determining when either of the two predicted blocks of the block is within the boundary of the image, and the components for applying bidirectional prediction based on the two predicted blocks of the block to obtain a bidirectional predicted block; The components are for determining when one of the two prediction blocks is at least partially outside the boundary of the image, and for applying unidirectional prediction based on the prediction block within the image boundary to obtain a unidirectional prediction block. A component for deriving filter coefficients based on a selected sample set of the bidirectional prediction block, wherein the selected sample set includes samples that have been downsampled in the bidirectional prediction block according to a downsampling method; as well as A component for filtering at least a portion of the one-way prediction block according to the filter coefficients to generate a filtered one-way prediction block.

2. The apparatus of claim 1, wherein the downsampling method comprises selecting uniformly spaced samples with predefined intervals within the bidirectional prediction block.

3. The apparatus of claim 1, wherein the downsampling method includes selecting uniformly spaced samples within the bidirectional prediction block such that the total number of samples does not exceed a predefined threshold.

4. The apparatus according to claim 1, wherein the downsampling method comprises: The bidirectional prediction block is divided into multiple regions, and samples with different intervals are selected in each region.

5. The apparatus of claim 1, wherein the downsampling method comprises selecting all or part of the samples included in the prediction sample derivation.

6. The apparatus of claim 1, wherein the downsampling method includes selecting samples based on a downsampling matrix.

7. The apparatus of claim 1, wherein the downsampling method comprises selecting samples with different intervals in different directions.

8. The apparatus of claim 3, wherein the value of the predefined threshold depends on the prediction unit (PU), coding unit (CU), coding tree unit (CTU), slice, frame, or sequence attribute.

9. The apparatus of claim 4, wherein the number of regions is based on prediction units (PU), coding units (CU), coding tree units (CTU), slices, frames, or sequence attributes.

10. The apparatus according to claim 4 or 9, further comprising: A component used to derive the geometry of a region based on the height and width of the bidirectional prediction block.

11. The apparatus according to claim 6, further comprising: A component used to define overshoot parameters to control the number of samples.

12. The apparatus of claim 11, wherein the overshoot parameter is based on a prediction unit (PU), a coding unit (CU), a coding tree unit (CTU), a slice, a frame, or a sequence attribute.

13. A method comprising: The image is processed to predict the blocks based on the prediction samples of the two prediction blocks; Determine when both of the two predicted blocks of the block are within the boundary of the image, and apply bidirectional prediction based on the two predicted blocks of the block to obtain a bidirectional predicted block; Determine when one of the two prediction blocks is at least partially outside the boundary of the image, and apply unidirectional prediction based on the prediction block inside the image boundary to obtain a unidirectional prediction block; The filter coefficients are derived based on the selected sample set of the bidirectional prediction block, wherein the selected sample set includes samples that have been downsampled in the bidirectional prediction block according to the downsampling method; as well as At least a portion of the one-way prediction block is filtered according to the filter coefficients to generate a filtered one-way prediction block.

14. The method of claim 13, wherein the downsampling method includes selecting uniformly spaced samples with predefined intervals within the bidirectional prediction block.

15. The method of claim 13, wherein the downsampling method includes selecting uniformly spaced samples within the bidirectional prediction block such that the total number of samples does not exceed a predefined threshold.

16. The method of claim 13, wherein the downsampling method comprises: The bidirectional prediction block is divided into multiple regions, and samples with different intervals are selected in each region.

17. The method of claim 13, wherein the downsampling method includes selecting all or part of the samples included in the prediction sample derivation.

18. The method of claim 13, wherein the downsampling method includes selecting samples based on a downsampling matrix.

19. The method of claim 13, wherein the downsampling method comprises selecting samples with different intervals in different directions.

20. The method of claim 15, wherein the value of the predefined threshold depends on the prediction unit (PU), coding unit (CU), coding tree unit (CTU), slice, frame, or sequence attribute.

21. The method of claim 16, wherein the number of regions is based on prediction units (PU), coding units (CU), coding tree units (CTU), slices, frames, or sequence attributes.

22. The method according to claim 16 or 21, further comprising: The geometry of the region is derived based on the height and width of the bidirectional prediction block.

23. The method of claim 18, further comprising: Define overshoot parameters to control the number of samples.

24. The method of claim 23, wherein the overshoot parameter is based on a prediction unit (PU), coding unit (CU), coding tree unit (CTU), slice, frame, or sequence attribute.

25. An apparatus comprising at least one processor and a memory including computer program code, the memory and the computer program code being configured, together with the at least one processor, such that the apparatus performs at least the following: The image is processed to predict the blocks based on the prediction samples of the two prediction blocks; Determine when both of the two predicted blocks of the block are within the boundary of the image, and apply bidirectional prediction based on the two predicted blocks of the block to obtain a bidirectional predicted block; Determine when one of the two prediction blocks is at least partially outside the boundary of the image, and apply unidirectional prediction based on the prediction block inside the image boundary to obtain a unidirectional prediction block; The filter coefficients are derived based on the selected sample set of the bidirectional prediction block, wherein the selected sample set includes samples that have been downsampled in the bidirectional prediction block according to the downsampling method; as well as At least a portion of the one-way prediction block is filtered according to the filter coefficients to generate a filtered one-way prediction block.

26. The apparatus of claim 25, wherein the downsampling method comprises selecting uniformly spaced samples with predefined intervals within the bidirectional prediction block.

27. The apparatus of claim 25, wherein the downsampling method includes selecting uniformly spaced samples within the bidirectional prediction block such that the total number of samples does not exceed a predefined threshold.

28. The apparatus of claim 25, wherein the downsampling method comprises: The bidirectional prediction block is divided into multiple regions, and samples with different intervals are selected in each region.

29. The apparatus of claim 25, wherein the downsampling method comprises selecting all or part of the samples included in the prediction sample derivation.

30. The apparatus of claim 25, wherein the downsampling method includes selecting samples based on a downsampling matrix.

31. The apparatus of claim 25, wherein the downsampling method comprises selecting samples with different intervals in different directions.

32. The apparatus of claim 27, wherein the value of the predefined threshold depends on the prediction unit (PU), coding unit (CU), coding tree unit (CTU), slice, frame, or sequence attribute.

33. The apparatus of claim 28, wherein the number of regions is based on prediction units (PU), coding units (CU), coding tree units (CTU), slices, frames, or sequence attributes.

34. The apparatus according to claim 28 or 33, further comprising: Computer program code that enables the device to derive the geometry of the region based on the height and width of the bidirectional prediction block.

35. The apparatus of claim 30, further comprising: Computer program code used to enable the device to define overshoot parameters to control the number of samples.

36. The apparatus of claim 35, wherein the overshoot parameter is based on a prediction unit (PU), a coding unit (CU), a coding tree unit (CTU), a slice, a frame, or a sequence attribute.

Images