Method, apparatus and computer program product for video encoding and decoding

Abstract

This embodiment relates to a method for encoding and a technical device for implementing the method. The method includes: determining a motion vector and a reference frame of a current block (1610); determining when the motion vector points to an area outside the reference frame (1620); using the following: directional intra prediction using border samples of the reference frame according to the direction of the motion vector; intra prediction using decoded samples in the current frame; generating an extended reference frame by filling the area outside the reference frame with one or more of the intra predicted samples of the current block (1630); predicting the current block by using motion compensation from the extended reference frame (1640); and encoding the current block into a bitstream (1650).

Description

[Technical Field]

[0001] The present solution relates generally to video encoding and decoding, and in particular to motion compensation in video encoding and decoding. [Background technology]

[0002] This section is intended to provide a background or context for the invention that is described in the claims. The description herein may include concepts that could be pursued, but not necessarily concepts that have been previously conceived or pursued. Thus, unless otherwise indicated herein, material in this section is not prior art to the present specification and claims and should not be admitted as prior art by inclusion in this section.

[0003] A video coding system may include an encoder that converts an input video into a compressed representation suitable for storage / transmission, and a decoder that can decompress the compressed video representation back into a viewable form. The encoder may discard some information in the original video sequence in order to represent the video in a more compact form, for example, to allow storage / transmission of the video information at a lower bitrate than might otherwise be required. Summary of the Invention

[0004] The scope of protection sought for various embodiments of the invention is defined by the independent claims. The embodiments and features described herein that are not included in the scope of the independent claims, if any, are to be interpreted as examples useful for understanding various embodiments of the invention.

[0005] Various aspects include a method, an apparatus and a computer readable medium storing a computer program as characterized by what is set forth in the independent claims. Various embodiments are disclosed in the dependent claims.

[0006] According to a first aspect, an apparatus is provided, comprising: means for determining a motion vector and a reference frame of a current block; means for determining when the motion vector points to an area outside the reference frame; means for generating an extended reference frame by filling the area outside the reference frame by one or more of the following means: directional intra prediction using border samples of the reference frame according to the direction of the motion vector; intra prediction using decoded samples in the current frame; intra predicted samples of the current block; means for predicting the current block by using motion compensation from the extended reference frame; and means for encoding the current block into a bitstream.

[0007] According to a second aspect, there is provided a method including: determining a motion vector and a reference frame for a current block; determining when the motion vector points to an area outside the reference frame; generating an extended reference frame by filling the area outside the reference frame with one or more of the following: directional intra prediction using border samples of the reference frame according to the direction of the motion vector; intra prediction using decoded samples in the current frame; intra predicted samples of the current block; predicting the current block by using motion compensation from the extended reference frame; and encoding the current block into a bitstream.

[0008] According to a third aspect, there is provided an apparatus comprising at least one processor and a memory containing computer program code, wherein the memory and the computer program code are configured to cause the apparatus, using the at least one processor, to perform at least: determining a motion vector and a reference frame for a current block; determining when the motion vector points to an area outside the reference frame; generating an extended reference frame by filling the area outside the reference frame with one or more of the following: directional intra prediction using border samples of the reference frame according to the direction of the motion vector; intra prediction using decoded samples in the current frame; intra predicted samples of the current block; predicting the current block by using motion compensation from the extended reference frame; and encoding the current block into a bitstream.

[0009] According to a fourth aspect, there is provided a computer program product comprising computer program code that, when executed on at least one processor, causes an apparatus or system to determine a motion vector and a reference frame for a current block; determine when the motion vector points to an area outside the reference frame; generate an extended reference frame by filling the area outside the reference frame with one or more of the following: directional intra prediction using border samples of the reference frame according to the direction of the motion vector; intra prediction using decoded samples in the current frame; intra predicted samples of the current block; predict the current block by using motion compensation from the extended reference frame; and encode the current block into a bitstream.

[0010] According to one embodiment, the intra prediction method is determined using texture analysis.

[0011] According to one embodiment, the texture analysis method is a decoder-side intra-mode derivation method.

[0012] According to one embodiment, the intra prediction method is determined using a template-based intra mode derivation method.

[0013] According to one embodiment, samples of regions outside the picture are predicted by cross-component prediction, where samples of regions outside the picture in the reference channel are used to predict samples of regions outside the picture in the current channel.

[0014] According to one embodiment, border samples between intra prediction and motion compensated prediction are filtered in the final prediction. According to one embodiment, a computer program product is embodied on a non-transitory computer-readable medium.

[0015] Various embodiments will now be described in more detail with reference to the accompanying drawings. [Brief explanation of the drawings]

[0016] [Figure 1] FIG. 10 is a diagram showing an example of the positions of the left and top samples of a current block involved in CCLM mode.

[0017] [Figure 2a] 10 illustrates an example of deriving a chroma prediction mode from a luma mode when CCLM is enabled.

[0018] [Figure 2b] 10 shows an example of a unified binarization table for saturation prediction modes.

[0019] [Figure 3a] Two luma-chroma models are shown, obtained with a luma Y threshold of 17.

[0020] [Figure 3b] FIG. 10 is a diagram illustrating an example of the correspondence of each luma-chroma model to spatial segmentation of content.

[0021] [Figure 4] FIG. 10 is a diagram showing an example of the positions of samples used to derive a CCCM filter.

[0022] [Figure 5] 1A and 1B are diagrams illustrating examples of various filter kernels.

[0023] [Figure 6] FIG. 10 is a diagram illustrating an example of four reference lines adjacent to a prediction block.

[0024] [Figure 7] FIG. 1 illustrates an example of a matrix weighted intra prediction process.

[0025] [Figure 8] FIG. 10 is a diagram illustrating an example of HoG computation from a template with a width of 3 pixels.

[0026] [Figure 9] FIG. 1 illustrates an example of a low frequency non-separable transform (LFNST) process.

[0027] [Figure 10] FIG. 10 is a diagram illustrating an example in which intra prediction is used to generate OOB area samples.

[0028] [Figure 11] FIG. 10 illustrates an example of predicting only a portion of the OOB area samples.

[0029] [Figure 12] FIG. 10 is a diagram showing an example of a motion vector of a block pointing to an OOB area in a reference frame.

[0030] [Figure 13] FIG. 10 is a diagram showing an example of combining motion compensated prediction and intra prediction from nearby reference samples when a motion vector points to an OOB area in a reference frame.

[0031] [Figure 14] FIG. 10 illustrates an example of using motion compensation samples together with reconstructed reference samples.

[0032] [Figure 15] A diagram showing an example of directional padding of OOB area samples in a reference picture.

[0033] [Figure 16] 1 is a flowchart illustrating a method according to one embodiment.

[0034] [Figure 17] FIG. 1 illustrates an apparatus according to one embodiment.

[0035] [Figure 18] FIG. 1 illustrates an encoding process according to one embodiment.

[0036] [Figure 19] FIG. 1 illustrates a decoding process according to one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0037] The following description and drawings are illustrative and should not be construed as unnecessarily limiting. Specific details are provided to provide a thorough understanding of the present disclosure. However, in certain instances, well-known or conventional details are not described to avoid obscuring the description. In this disclosure, reference to one embodiment or one embodiment may, but need not, refer to the same embodiment, and such reference means at least one of the embodiments.

[0038] References herein to "one embodiment" or "an embodiment" mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present disclosure.

[0039] In the following, some embodiments are described in the context of one video coding configuration. However, it should be noted that the embodiments are not necessarily limited to this particular configuration. The embodiments relate to border samples in motion compensation and methods for handling them.

[0040] The Advanced Video Coding standard (sometimes abbreviated as AVC or H.264 / AVC) was developed by the Joint Video Team (JVT) of the Video Coding Experts Group (VCEG) of the Telecommunications Standardization Sector of the International Telecommunication Union (ITU-T) and the Moving Picture Experts Group (MPEG) of the International Organization for Standardization (ISO) / International Electrotechnical Commission (IEC). The H.264 / AVC standard is published by both parent standards bodies and is known as ITU-T Recommendation H.264 and ISO / IEC International Standard 14496-10 (also known as MPEG-4 Part 10 Advanced Video Coding (AVC)). There are multiple versions of the H.264 / AVC standard, each incorporating new enhancements or features into the specification. These extensions include Scalable Video Coding (SVC) and Multiview Video Coding (MVC).

[0041] The High Efficiency Video Coding standard (sometimes abbreviated HEVC or H.265 / HEVC) was developed by the VCEG and MPEG's Joint Collaborative Team - Video Coding (JCT-VC). This standard, published by both parent standards organizations, is referred to as ITU-T Recommendation H.265 and ISO / IEC International Standard 23008-2, and is also known as MPEG-H Part 2 High Efficiency Video Coding (HEVC). Extensions to H.265 / HEVC include scalable, multiview, three-dimensional, and range-of-fidelity extensions, which are sometimes referred to as SHVC, MV-HEVC, 3D-HEVC, and REXT, respectively. References herein to H.265 / HEVC, SHVC, MV-HEVC, 3D-HEVC, and REXT for the purpose of understanding the definitions, structure, or concepts of these standards should be understood to refer to the most recent versions of these standards available prior to the filing date, unless otherwise indicated.

[0042] Versatile Video Coding (VVC, H.266, or sometimes abbreviated as H.266 / VVC) is a video compression standard developed as a successor to HEVC. VVC is specified in ITU-T Recommendation H.266 and ISO / IEC 23090-3, also known as MPEG-I Part 3.

[0043] The AV1 bitstream format and decoding process specifications were developed by the Alliance of Open Media (AOM). The AV1 specification was published in 2018. AOM is reportedly working on the AV2 specification.

[0044] Some key definitions, bitstream and coding structures, and concepts of H.264 / AVC, HEVC, VVC, and / or AV1, as well as some of their extensions, are described in this section as an example of a video encoder, decoder, encoding method, decoding method, and bitstream structure in which embodiments may be implemented. Aspects of various embodiments are not limited to H.264 / AVC, HEVC, VVC, and / or AV1, or their extensions, but rather are described on one possible basis in which the embodiments may be realized, partially or fully.

[0045] A video codec may comprise an encoder that converts an input video into a compressed representation suitable for storage / transmission, and a decoder that can decompress the compressed video representation back into a viewable form. The compressed representation is sometimes called a bitstream or video bitstream. The video encoder and / or video decoder may be separate from each other, i.e., may not be required to form a codec. The encoder may discard some information in the original video sequence in order to represent the video in a more compact form (i.e., at a lower bitrate). The term "(de)coder" refers to the encoder and / or decoder.

[0046] Hybrid video codecs, such as ITU-T H.263, H.264 / AVC, and HEVC, can encode video information in two stages. First, pixel values ​​of a certain picture region (or "block") are predicted, for example, by motion compensation means (locating and indicating a region of a previously encoded video frame that closely corresponds to the block being encoded) or spatial means (using pixel values ​​surrounding the block being encoded in a specified way). In the first phase, predictive coding can be applied, for example, as so-called sample prediction and / or so-called syntax prediction.

[0047] Sample prediction predicts pixel or sample values ​​within a picture region or "block." These pixel or sample values ​​can be predicted using, for example, one or more of motion compensation or intra-prediction mechanisms.

[0048] Motion compensation mechanisms (which may also be called inter-prediction, temporal prediction, or motion-compensated temporal prediction, or motion-compensated prediction, or MCP) involve finding and indicating regions in one of the previously encoded video frames that closely correspond to the block being encoded. Inter-prediction can reduce temporal redundancy.

[0049] Intra prediction involves finding and indicating spatial domain relationships, where sample-value pixels can be predicted by spatial mechanisms. Intra prediction takes advantage of the fact that adjacent pixels in the same picture are likely to be correlated. Intra prediction can be performed in the spatial domain or the transform domain, i.e., either sample values ​​or transform coefficients can be predicted. Intra prediction can be used in intra coding where inter-prediction is not applied.

[0050] In syntax prediction, sometimes also called parameter prediction, syntax elements and / or syntax element values ​​and / or variables derived from syntax elements are predicted from previously (de)coded syntax elements and / or previously derived variables. Non-limiting examples of syntax prediction are provided below.

[0051] In motion vector prediction, motion vectors, e.g., motion vectors for inter-prediction and / or inter-view prediction, may be differentially encoded with respect to block-specific predicted motion vectors. In many video codecs, predicted motion vectors are created in a predefined manner, for example, by calculating the median of coded or decoded motion vectors of neighboring blocks. Another method for creating motion vector predictions, sometimes called advanced motion vector prediction (AMVP), is to generate a list of prediction candidates from neighboring and / or co-located blocks in a temporal reference picture and signal a selected candidate as a motion vector predictor. In addition to predicting motion vector values, reference indices of previously coded / decoded pictures can be predicted. The reference indices can be predicted from neighboring and / or co-located blocks in the temporal reference picture. Differential coding of motion vectors can be disabled across slice boundaries. Block partitioning, e.g., from coding tree units (CTUs) to coding units (CUs) and prediction units (PUs), can be predicted.

[0052] In filter parameter prediction, filtering parameters, for example filtering parameters for sample adaptive offset, may be predicted.

[0053] Prediction approaches that use image information from previously encoded images may be called inter-prediction methods, which may also be called temporal prediction and motion compensation.

[0054] A prediction method that uses image information within the same image is also called an intra prediction method.

[0055] The prediction error, i.e., the difference between the predicted block of pixels and the original block of pixels, is then encoded. This can be done by transforming the pixel value differences using a predetermined transform (e.g., a discrete cosine transform (DCT) or a variant thereof), quantizing the coefficients, and entropy encoding the quantized coefficients. By varying the fidelity of the quantization process, the encoder can control the balance between the precision of the pixel representation (image quality) and the size of the resulting coded video representation (file size at transmission bit rate).

[0056] The basic unit of input to an encoder and output from a decoder is often a picture. A picture provided as input to an encoder is sometimes called a source picture, and a picture decoded by a decoder is sometimes called a decoded or reconstructed picture.

[0057] The source picture and the decoded picture each consist of one or more sample arrays, for example one of the following sets of sample arrays: Luma (Y) only (monochrome) Luma and two chromas (YCbCr or YCgCo) Green, Blue and Red (GBR, also known as RGB) Arrays representing other unspecified monochrome or tristimulus color sampling (e.g., also called YZX, XYZ).

[0058] In the following, these arrays may be referred to as luma (or L or Y) and chroma, and the two chroma arrays may be referred to as Cb and Cr. The actual color representation method used may be indicated in the coded bitstream, for example, using the HEVC Video Usability Information (VUI) syntax or a similar syntax. A component may be defined as an array or a single sample of one of the three sample arrays (luma and two chroma), or as an array or a single sample of an array that makes up a monochrome picture.

[0059] A picture can be defined as either a frame or a field. A frame comprises a matrix of luma samples and possibly corresponding chroma samples. A field is a set of alternating sample rows of a frame and can be used as encoder input if the source signal is interlaced. There can be no chroma sample array (and monochrome sampling is used), or the chroma sample array can be subsampled compared to the luma sample array.

[0060] The decoder applies similar prediction means as the encoder to form a predicted representation of the pixel block (using motion or spatial information created by the encoder and stored in the compressed representation), and reconstructs the output video by prediction error decoding (the inverse operation of prediction error encoding, which recovers the quantized prediction error signal in the spatial pixel domain). After applying prediction and prediction error decoding, the decoder sums the prediction and prediction error signals (pixel values) to form the output video frame. The decoder (and encoder) may also apply additional filtering means to improve the quality of the output video before passing it on for display and / or save it as a predictive reference for the next frame in the video sequence.

[0061] Motion information can be indicated by a motion vector associated with each motion-compensated image block in a video codec. Each of these motion vectors represents the displacement of an image block in a picture to be encoded (at the image capture device) or decoded (at the decoder device) and a source block in one of the previously encoded or decoded images (or pictures). H.264 / AVC and HEVC, like many other video compression standards, divide a picture into rectangular meshes, each of which indicates a similar block in one of the reference pictures for inter-prediction. The location of the predicted block is encoded as a motion vector by indicating the relative position of the predicted block with respect to the block to be encoded.

[0062] A bitstream can be defined as a sequence of bits or a sequence of syntactic structures. The bitstream format may constrain the ordering of syntactic structures within the bitstream.

[0063] A syntax element may be defined as an element of data represented in the bitstream. A syntax structure may be defined as zero or more syntax elements present together in the bitstream in a specified order.

[0064] Depending on the coding format or standard, the bitstream may be in the form of a Network Abstraction Layer (NAL) unit stream or byte stream, which forms a representation of coded pictures and associated data that form one or more coded video sequences.

[0065] An NAL unit can be defined as a syntax structure containing bytes indicating the type of data that follows and containing that data in the form of RBSP, optionally separated by start code emulation prevention bytes. A Raw Byte String Payload (RBSP) can be defined as a syntax structure containing an integer number of bytes encapsulated in an NAL unit. An RBSP is either empty or has the form: a string of data bits with syntax elements followed by an RBSP stop bit, followed by zero or more bits equal to 0.

[0066] A NAL unit comprises a header and a payload. The NAL unit header indicates, among other things, the type of the NAL unit.

[0067] In some coding formats, such as AV1, a bitstream may comprise a series of Open Bitstream Units (OBUs). An OBU comprises a header and a payload, where the header identifies the type of OBU. Furthermore, the header may comprise the size of the payload in bytes.

[0068] The phrases along the bitstream (e.g., indicating along the bitstream) or along a coding unit of the bitstream (e.g., indicating along a coding tile) may be used in the claims and described embodiments to refer to transmitting, signaling, or storing "out-of-band" data in a manner associated with, but not contained within, the bitstream or coding unit, respectively. The phrases decoding along the bitstream, or decoding along a coding unit of the bitstream, or similarly decoding, may refer to decoding referenced out-of-band data (which may be obtained from out-of-band transmission, signaling, or storage) associated with the bitstream or coding unit, respectively. For example, the phrase along the bitstream is used when the bitstream is contained in a container file, such as a file conforming to the ISO Base Media File Format, and specific file metadata is stored in the file in a manner that associates the metadata with the bitstream, such as a box of sample entries for the track that includes the bitstream, a sample group for the track that includes the bitstream, or a timed metadata track associated with the track that includes the bitstream.

[0069] The following describes in detail the division of a picture into sub-pictures, slices, and tiles according to H.266 / VVC, although similar concepts can be applied to other video coding specifications.

[0070] A picture is divided into one or more tile rows and one or more tile columns. A tile is a sequence of coding tree units (CTUs) that cover a rectangular area of ​​the picture. The CTUs within a tile are scanned in raster scan order within that tile.

[0071] A slice consists of an integer number of complete tiles or an integer number of contiguous complete CTU rows within a tile of a picture. As a result, each vertical slice boundary is always also a vertical tile boundary. It is possible for the horizontal boundary of a slice to consist of a horizontal CTU boundary within a tile rather than a tile boundary; this occurs when a tile is divided into multiple rectangular slices, each consisting of an integer number of contiguous complete CTU rows within the tile.

[0072] Two slice modes are supported: raster scan slice mode and rectangular slice mode. In raster scan slice mode, a slice contains a sequence of complete tiles in a tile raster scan of the picture. In rectangular slice mode, a slice contains a number of complete tiles that collectively form a rectangular area of ​​the picture, or a number of contiguous complete CTU rows of one tile that collectively form a rectangular area of ​​the picture. The tiles within a rectangular slice are scanned in tile raster scan order within the rectangular area corresponding to the slice.

[0073] A subpicture can be defined as a rectangular region of one or more slices within a picture, where one or more slices are complete. Thus, a subpicture consists of one or more slices that collectively cover a rectangular region of the picture. As a result, each subpicture boundary is always also a slice boundary, and each vertical subpicture boundary is always also a vertical tile boundary. Subpicture slices can be required to be rectangular slices.

[0074] For each subpicture and tile, one or both of the following conditions may be required to be met: i) All CTUs in a subpicture belong to the same tile. ii) All CTUs in a tile belong to the same subpicture.

[0075] In the following, the division of a picture into tiles and tile groups according to AV1 is explained in more detail, and similar concepts can be applied to other video coding specifications.

[0076] A tile consists of an integer number of complete superblocks that collectively form a complete rectangular region of the picture. Intra-picture prediction across tile boundaries is disabled. The minimum tile size is one superblock, and the maximum tile size for currently specified levels is 4096 x 2304 luma samples. A picture is divided into a tile grid, one or more tile rows and one or more tile columns. The tile grid can be signaled in the picture header to have uniform or non-uniform tile sizes, in which case the height of the tile rows and the width of the tile columns are signaled. Superblocks within a tile are scanned in raster scan order within that tile.

[0077] A tile group OBU holds one or more complete tiles. The first and last tiles in a tile group OBU can be indicated in the tile group OBU before the coded tile data. Tiles in a tile group OBU can appear in the tile raster scan of a picture.

[0078] The features and coding tools included in VVC are: Intra prediction 67 Intra modes with wide-angle mode extension Block-size and mode-dependent 4-tap interpolation filter Position-Dependent Intra-Prediction Combination (PDPC) Cross-Component Linear Model Intra Prediction (CCLM) Multi-baseline intra-prediction Intra-subpartition Weighted intra prediction with matrix multiplication Inter-picture prediction Block Motion Copying with Spatial, Temporal, History-Based, and Pairwise Average Merge Candidates Affine Motion Interpretation Subblock-based temporal motion vector prediction Adaptive Motion Vector Resolution 8x8 block-based motion compression for temporal motion estimation High-precision (1 / 16pel) motion vector preservation and motion compensation using an 8-tap interpolation filter for the luma component and a 4-tap interpolation filter for the chroma component triangulation Combining intra and inter prediction Merge with MVD (MMVD) Symmetric MVD coding Bidirectional Optical Flow Decoder-side motion vector refinement Bidirectional prediction with CU-level weights Transformation, Quantization and Coefficient Coding Multiple linear transform options with DCT2, DCT7 and DCT8 Second-order transformation for low frequency bands Sub-block transformation between prediction residuals Dependent Quantization with max QP increased from 51 to 63 Transform coefficient coding with code data hiding. Transform-skip residual coding Entropy Coding Adaptive double-window probability update arithmetic coding engine In-Loop Filter In-Loop Reshaping Powerful deblocking filter with long filter Sample Adaptive Offset Adaptive Loop Filter Screen Content Encoding Referencing the current image by limiting the reference area 360-degree video coding Horizontal wrap-around motion compensation High-level syntax and parallel processes Reference image management using direct reference image list signals Rectangular tile group

[0079] In H.266 / VVC, the following block division is applied: A picture is divided into CTUs. A picture can also be divided into slices, tiles, bricks, and subpictures. A CTU can be divided into smaller CUs using a quadtree structure. Each CU can be divided using nested multi-trees, including quadtrees, ternary trees, and binary trees.

[0080] There are certain rules for inferring partitioning at picture boundaries.

[0081] Redundant split patterns are prohibited.

[0082] To reduce cross-component redundancy, VVC uses the Cross-Component Linear Model (CCLM) prediction mode, where chroma samples are predicted based on the reconstructed luma samples of the same CU using a linear model such as: JPEG2025531486000002.jpg6150 where, pred c (i,j) represents the predicted chroma sample of the CU, and rec L '(i,j) represents the downsampled reconstructed luma sample of the same CU.

[0083] The CCLM parameters (α and β) are derived using up to four adjacent chroma samples and their corresponding downsampled luma samples. Given that the dimensions of the current chroma block are WxH, W' and H' are set as follows: When LM mode is applied, W'=W, H'=H; When LM-A mode is applied, W'=W+H; In the LM-L mode, H'=H+W.

[0084] The above neighboring positions are S[0,-1]...S[W'-1,-1], and the left neighboring positions are S[-1,0]...S[-1,H'-1]. As a result, the four samples are selected as follows: When LM mode is applied and both top and left neighboring samples are available, then S[W' / 4, -1], S[3*W' / 4, -1], S[-1, H' / 4], S[-1, 3*H' / 4]; When LM-A mode is applied or only the above neighboring samples are available: S[W' / 8,-1], S[3*W' / 8,-1], S[5*W' / 8,-1], S[7*W' / 8,-1] If LM-L mode is applied or only left neighboring samples are available, then S[-1,H' / 8], S[-1,3*H' / 8], S[-1,5*H' / 8], S[-1,7*H' / 8].

[0085] The four luma samples near the selected position are downsampled and compared four times to obtain two smaller values: x0A and x1A, and two larger values: x0B and x1B. The corresponding chroma sample values ​​are y0A, y1A, y0B, and y1B. Then, xA, xB, yA, and yB are derived as follows: Xa=(x 0 A+x 1 A+1)>>1, Xb=(x 0 B+x 1 B+1)>>1, Ya=(y 0 A+y 1 A+1)>>1, Yb=(y 0 B+y 1 B+1)>>1.

[0086] Finally, the parameters of the linear model are calculated according to the following formula: JPEG2025531486000003.jpg28150 JPEG2025531486000004.jpg6150

[0087] Figure 1 shows an example of the positions of the left and top samples involved in CCLM mode and the samples of the current block. The division operation to calculate the parameter α is implemented using a lookup table. To reduce the memory required to store the table, the diff value (the difference between the maximum and minimum values) and the parameter α are expressed in exponential notation. For example, the diff is approximated with a 4-bit significant part and an exponent. As a result, the table for 1 / diff is reduced to 16 elements for 16 values ​​of the significant part as follows: DivTable[]={0,7,6,5,4,4,3,3,2,2,1,1,1,0}

[0088] This has the advantage of not only reducing the computational complexity but also reducing the memory size required to store the necessary tables. The above template and the left template can be used together to calculate the linear model coefficients, as well as alternately in two other LM modes, called LM_A and LM_L modes.

[0089] In LM_A mode, only the upper template is used to calculate the linear model coefficients. To get more samples, the above template is extended to (W+H). In LM_L mode, only the left template is used to calculate the linear model coefficients. To get more samples, the left template is extended to (H+W).

[0090] For non-square blocks, the above template is expanded to W+W and the left template to H+H.

[0091] To match the chroma sample positions of a 4:2:0 video sequence, two types of downsampling filters are applied to the luma samples, resulting in a horizontal and vertical downsampling ratio of 2:1. The selection of the downsampling filter is specified by the SPS level flag. The two types of downsampling filters correspond to "type-0" and "type-2", respectively. JPEG2025531486000005.jpg13150 JPEG2025531486000006.jpg10150

[0092] It is understood that if the upper reference line is at a CTU boundary, only one luma line (general line buffer for intra prediction) is used to create the downsampled luma samples.

[0093] This parameter calculation is performed as part of the decoding process and is not simply a search operation of the encoder, so no syntax is used to communicate the α and β values ​​to the decoder.

[0094] A total of eight intra modes are allowed for chroma intra mode coding. These modes include five conventional intra modes and three cross-component linear model modes (CCLM, LM_A, and LM_L). The signaling and derivation of chroma modes are shown in Table 1 of Figure 2a. Chroma mode coding directly depends on the intra prediction of the corresponding luma block. In an I slice, separate block partitioning structures are enabled for the luma and chroma components, so one chroma block may correspond to multiple luma blocks. Therefore, in chroma DM mode, the intra prediction mode of the corresponding luma block covering the center position of the chroma block is inherited.

[0095] As shown in Table 2 in Figure 2b, a single binarization table is used regardless of the value of sps_cclm_enabled_flag. In Table 2, the first bin indicates normal mode (0) or LM mode (1). If LM mode, the next bin indicates whether LM_CHROMA (0) is selected. If not LM_CHROMA, the next bin indicates LM_L (0) or LM_A (1). In this case, if sps_cclm_enabled_flag is 0, the first bin of the corresponding intra_chroma_pred_mode binarization table can be discarded before entropy encoding. Or, in other words, the first bin is assumed to be 0 and therefore not encoded. This single binarization table is used both when sps_cclm_enabled_flag is 0 and when it is 1. The first two bins in Tables 3-4 are context coded with their own context model, and the remaining bins are bypass coded.

[0096] Additionally, if a 64x64 luma coding tree node is split with NotSplit (and does not use ISP for 64x64 CUs) or QT, then to reduce luma-chroma latency in the dual tree, the chroma CUs in the 32x32 / 32x16 chroma coding tree node can use CCLM as follows: If a 32x32 chroma node is not split or QT split, all chroma CUs within the 32x32 node can use CCLM. If a 32x32 chroma node is horizontally BT split and a 32x17 child node is not split or vertically BT split, all chroma CUs within the 32x16 chroma node can use CCLM. In other luma coding tree and chroma coding tree split conditions, the chroma CU cannot use CCLM.

[0097] The CCLM included in VVC is extended by adding three MMLM (Multi-model Linear Modeling) modes. In each MMLM mode, reconstructed neighboring samples are classified into two classes using a threshold that is the average of the luma-reconstructed neighboring samples. A linear model for each class is derived using the least-mean-square (LMS) method. In the CCLM mode, the LMS method is also used to derive the linear model. Figure 3a shows two luma-chroma models obtained when the luma (Y) threshold is 17. Each luma-chroma model has its own linear model parameters, α and β. As can be seen in Figure 3b, each luma-chroma model corresponds to a spatial segmentation of the content (i.e., they correspond to different objects or textures in the scene).

[0098] An improved version of cross-component prediction, known as the convolutional cross-component model (CCCM), uses a 2D filter kernel to derive a luma-chroma model. The filter coefficients are derived at the decoder side using the reconstructed input data set and chroma samples. For the derivation of the filter coefficients, a co-located reference sample region (consisting of reconstructed luma and chroma samples) is defined for both luma and chroma, as shown in FIG. 4, although any number of reference lines (realizable by both the encoder and the decoder) can be used. In general, the reference samples can include any chroma and luma samples reconstructed by both the encoder and the decoder. Once the reference samples are determined, the filter coefficients can be derived with different types of linear regression tools, such as ordinary least squares estimation, orthogonal matching pursuit, optimized orthogonal matching pursuit, ridge regression, or least absolute shrinkage and selection operator.

[0099] The dimensions of the filter kernel can be, for example, 1x3 (one-dimensional vertical), 3x1 (one-dimensional horizontal), 3x3, 7x7, or any other dimension, and can be cross-shaped, diamond-shaped, or any other shape (by selecting only a subset of all possible kernel positions). When referring to samples within a filter kernel, the notation North (top), East (right), South (bottom), West (left), and center is used, as shown by the letters N, E, S, W, and C in Figure 5. Figure 5 illustrates a 3-tap vertical kernel 501, a 3-tap horizontal kernel 502, a 5-tap cross kernel 503, and a 25-tap diamond kernel 504.

[0100] The overall method of reconstructing chroma samples using the convolution between the filter kernels obtained at the decoder side and the input data set is called here the Convolutional Cross-Component Model (CCCM). To implement CCCM, the following steps can be applied: Define co-located reference regions across the luma and chroma components; Optionally downsample luma samples to fit the chroma grid; Scan the luma and chroma samples of the reference region and collet available statistics (such as autocorrelation matrix and cross-correlation vector) based on the filter shape; Solving for the filter coefficients by minimizing the squared error (or other metric) based on available statistics (such as the autocorrelation matrix and cross-correlation vector); A prediction chroma block is computed by convolving the downsampled luma samples with a filter kernel.

[0101] In the following, we define the (possibly downsampled) luma samples as a 2D array Y(x,y) indexed by their horizontal x and vertical y coordinates, the co-located chroma samples as a 2D array C(x,y), and the filter kernel (i.e., coefficients) as a 3x3 array F(i,j). At the sample level, the convolution of Y and F is defined as follows: JPEG2025531486000007.jpg15150

[0102] If other data terms are used, such as nonlinear square root terms, the added convolution becomes: JPEG2025531486000008.jpg15150 where, JPEG2025531486000009.jpg6150 is a filter coefficient that lies outside the 2D filter kernel, but was obtained as part of the system of linear equations used to solve for the 2D filter coefficients in step 4 above. Similarly, a bias term can be added to the convolution as follows: JPEG2025531486000010.jpg15150

[0103] Angular intra-prediction (also known as directional intra-prediction) can be performed by extrapolating sample values ​​from reconstructed reference samples using a given directionality. The reference samples may comprise the immediately adjacent sample rows above and to the right of the current block (if available) and the immediately adjacent sample columns to the left of the current block (if available), and, if available, may require a decoding order earlier than that of the current block and within the same image segment, such as within the same tile. To simplify the process, all sample positions within one prediction block can be projected to a single reference row or column, depending on the directionality of the selected prediction mode. Prediction samples within a block to be coded / decoded can be obtained by the following steps: By applying the selected prediction direction, the predicted sample position is projected onto a position in a reference row or column, which may have fractional sample precision, such as 1 / 32 pixel precision. The value of the sample location on the reference row or column is interpolated from the reference sample of the reference row / column.

[0104] MRL (Multiple Reference Line) intra prediction uses more reference lines for intra prediction. In Figure 6, an example of four reference lines is depicted, where samples from segments A and F are padded with the nearest samples from segments B and E, respectively, rather than being obtained from reconstructed neighboring samples. In HEVC intra prediction, the nearest reference line (i.e., reference line 0) is used. In MRL, two more lines (reference line 1 and reference line 3) are used.

[0105] The index of the selected reference line (mrl_idx) is signaled and used to generate the intra predictor. For a reference line idx greater than 0, only the additional reference line modes are included in the MPM list, and only the mpm index is signaled without the remaining modes. The reference line index is signaled before the intra prediction modes, and if a reference line index other than 0 is signaled, planar modes are excluded from the intra prediction modes.

[0106] MRL is disabled for the first line of a block in a CTU to prevent the use of extended reference samples outside the current CTU line. PDPC is also disabled when additional lines are used. In MRL mode, the derivation of DC values ​​in DC intra prediction modes with non-zero reference line indexes is aligned with that for reference line index 0. MRL requires storage of three neighboring luma reference lines in the CTU to generate prediction values. The CCLM (Cross-Component Linear Model) tool also requires three neighboring luma reference lines for its downsampling filter. The definition of MLR, which uses the same three lines, is aligned with CCLM to reduce decoder storage requirements.

[0107] The intra subpartition (ISP) divides a luma intra-prediction block into two or four subpartitions vertically or horizontally, depending on the block size. For example, the minimum block size for ISP is 4x8 (or 8x4). If the block size is larger than 4x8 (or 8x4), the corresponding block is divided into four subpartitions. It has been noted that Mx12 (with M≦64) and 128xN (with N≦64) ISP blocks may cause potential problems with 64x64 VDPUs. For example, an Mx128 CU in the single-tree case has an M×128 luma TB and two corresponding M / 2×64 chroma TBs. When a CU uses ISP, the luma TB is divided into four M×32 TBs (only horizontal division is possible), each smaller than a 64×64 block. However, in current ISP designs, chroma blocks are not divided. Therefore, all chroma components have a size larger than a 32×32 block.

[0108] Similarly, a similar situation can occur with 128xN CUs using ISP. Therefore, these two cases are problematic for a 64x64 decoder pipeline. For this reason, the CU size that can use ISP is limited to a maximum of 64x64. All subpartitions must have at least 16 samples.

[0109] Matrix-weighted intra prediction (MIP) is an intra prediction technique in VVC. To predict samples of a rectangular block of width W and height H, MIP takes as input one line of H reconstructed adjacent border samples on the left side of the block and one line of W reconstructed adjacent border samples on the top side of the block. If reconstructed samples are not available, they are generated in the same way as conventional intra prediction. The generation of the prediction signal is based on three steps: averaging, matrix-vector multiplication, and linear interpolation, as shown in Figure 7.

[0110] When applying decoder-side intra-mode derivation (DIMD), two intra-modes are derived from the reconstructed neighboring samples, and their predictors are combined with the flattened mode predictor using gradient-derived weights. The division operations in the weight derivation are performed using the same look-up table (LUT)-based integerization scheme used in CCLM. For example, the division operations in the orientation calculation are JPEG2025531486000011.jpg6150 is calculated by the following LUT-based method: x=Floor( Log2( Gx ) ) normDiff=( ( Gx<< 4 ) >> x ) & 15 x +=( 3 + ( normDiff != 0 ) ? 1 : 0 ) Orient=(Gy* ( DivSigTable[normDiff] | 8 ) + ( 1<<( x-1 ) )) >> x where DivSigTable

[16] ={0,7,6,5 ,5,4,4,3,3,2,2,1,1,1,1,0}. The derived intra modes are included in the primary list of intra most probable modes (MPMs), so the DIMD process is performed before the MPM list is constructed. The primary derived intra modes of a DIMD block are stored with the block and are used to construct the MPM lists of neighboring blocks. Figure 8 shows an example of computing a Histogram of Oriented Gradients (HoG) from a 3-pixel wide template.

[0111] For each intra-prediction mode in the MPM, the sum of absolute transform differences (SATD) between the predicted sample and the reconstructed sample of the template is calculated. First, the two intra-prediction modes with the smallest SATD are selected as the TIMD mode. These two TIMD modes are fused with a weight after applying the PDPC process, and such weighted intra-prediction is used to encode the current CU. Position-dependent intra-prediction combining (PDPC) is included in the derivation of the TIMD mode. The costs of the two selected modes are compared with a threshold, and a test with a cost factor of 2 is applied as follows: costMode2 < 2*costMode1 If this condition is true, fusion is applied, otherwise only mode1 is used.

[0112] The weight of each mode is calculated from the SATD cost as follows: weight1=costMode2 / (costMode1+ costMode2) weight2=1-weight1 The division is performed using the same look-up table (LUT) based integerization scheme used in CCLM.

[0113] In VVC, as shown in Figure 9, a low-frequency non-separable transform (LFNST) is applied between the forward linear transform and quantization (encoder side) and between dequantization and the inverse linear transform (decoder side). LFNST applies a 4x4 non-separable transform or an 8x8 non-separable transform depending on the block size. For example, a 4x4 LFNST is applied to small blocks (i.e., min(width, height)<8), and an 8x8 LFNST is applied to large blocks (i.e., min(width, height)>4).

[0114] The application of the non-separable transform used in LFNST is explained using the input as an example. To apply 4x4 LFNST, we use a 4x4 input block X First, represent JPEG2025531486000012.jpg18150 as a vector Represent JPEG2025531486000013.jpg6150 as: JPEG2025531486000014.jpg11159

[0115] The non - separable transform is Calculated as JPEG2025531486000015.jpg6150, where JPEG2025531486000016.jpg6150 represents the transform coefficient vector, and T is a 16x16 transform matrix. The 16x1 coefficient vector is then reorganized into 4x4 blocks using the scan order (horizontal, vertical, or diagonal) of the block. Coefficients with smaller indices will be placed at the smaller scan indices of the 4x4 coefficient block.

[0116] Since LFNST is based on a direct matrix - multiplication approach for applying the non - separable transform, it is implemented in a single pass without repeated iterations. However, to minimize the computing complexity and the memory space for storing the transform coefficients, it is necessary to reduce the dimension of the non - separable transform matrix. Therefore, the reduced non - separable transform (or RST) method is used in LFNST. The main idea of the reduced non - separable transform is to map a vector of dimension N (N is generally equal to 64 in 8x8 NSST) to a vector of dimension R in another space, where N / R (R < N) is the reduction factor. Therefore, instead of an NxN matrix, the RST matrix becomes an RxN matrix as follows: JPEG2025531486000017.jpg2361 where the R rows of the transform are the R bases in N-dimensional space. The inverse transform matrix of RT is the transpose of its forward transform. For the 8x8 LFNST, a 4x reduction factor is applied, reducing the traditional 8x8 non-separable transform matrix size (64x64 direct matrix) to a 16x48 direct matrix. Therefore, a 48x16 inverse RST matrix is ​​used to generate the core (primary) transform coefficients of the 8x8 upper-left region at the decoder side. If the same transform set configuration were applied with a 16x48 matrix instead of a 16x64 matrix, each matrix would receive 48 input data from three 4x4 blocks in the upper-left 8x8 block, excluding the lower-right 4x4 block. With the help of dimensionality reduction, the memory usage for storing all LFNST matrices is reduced from 10KB to 8KB with a reasonable performance degradation. To reduce complexity, LFNST is restricted to be applied only when all coefficients outside the first coefficient subgroup are insignificant. Therefore, when LFNST is applied, all first-order transform coefficients must be zero-regularized. This conditions the LFNST index signal on the last significant position, thus avoiding the extra coefficient scan required in the current LFNST design to check only significant coefficients at specific positions. LFNST's worst-case handling (in terms of multiplications per pixel) limits non-separable transforms of 4x4 and 8x8 blocks to 8x16 and 8x48 transforms, respectively. In these cases, when LFNST is applied, the last-most significant scan position must be less than 8 for other sizes less than 16. For blocks with shapes of 4xN, Nx4, and N>8, the proposed restriction means that LFNST is applied only once, and only to the top-left 4x4 region. Because all first-order coefficients are zero-regularized when LFNST is applied, the number of operations required for the first-order transform is reduced in such cases. From the encoder's point of view, the quantization of coefficients is significantly simplified when the LFNST transform is tested: rate-distortion optimized quantization needs to be done on at most the first 16 coefficients (in scan order), while the remaining coefficients are forced to be zero-point corrected.

[0117] LFNST uses four transform sets and two non-separable transform matrices (kernels) per transform set. The mapping from intra prediction modes to transform sets is predefined as shown in the table below. If any of the three CCLM modes (INTRA_LT_CCLM, INTRA_T_CCLM, or INTRA_L_CCLM) is used for the current block (81≦predModeIntra≦83), transform set 0 is selected for the current chroma block. For each transform set, the selected non-separable secondary transform candidate is further specified by an explicitly singled LFNST index. This index is signaled in the bitstream once per Intra CU after the transform coefficients. TIFF2025531486000018.tif47127

[0118] Because LFNST is restricted to be applied only when all coefficients outside the first coefficient subgroup are non-significant, the coding of the LFNST index depends on the position of the last significant coefficient. In addition, the LFNST index is context coded, but is independent of the intra predicate mode; only the first bit is context coded. Furthermore, LFNST is applied to intra CUs in both intra and inter slices, and to both luma and chroma. When dual tree is enabled, the LFNST indicators for luma and chroma are signaled separately. For inter slices (dual tree is disabled), one LFNST index is signaled and used for both luma and chroma.

[0119] Due to the existing maximum transform size limitation (64x64), CUs larger than 64x64 are implicitly split (TU tiling), and LFNST index search can potentially increase data buffering by four times for a given number of decode pipeline stages. Therefore, the maximum size allowed by LFNST is limited to 64x64. LFNST is only enabled in DCT2. The LFNST index signal is placed before the MTS index signal.

[0120] The use of scaling matrices for perceptual quantization does not make it clear that the scaling matrices specified for the main matrix can be useful for LFNST coefficients. Therefore, the use of scaling matrices for LFNST coefficients is not permitted. In single-tree partition mode, chroma LFNST is not applied.

[0121] Scalable video coding refers to a coding structure in which one bitstream can contain multiple representations of content, e.g., at different bitrates, resolutions, or frame rates. In such cases, a receiver can extract the desired representation according to its characteristics (e.g., the resolution that best suits its display device). Alternatively, a server or network element can extract the portion of the bitstream to transmit to a receiver according to, e.g., the receiver's network characteristics or processing capabilities.

[0122] Scalable video coding can be achieved by layer coding. Layer coding is a concept in which an unencoded visual representation of a scene is mapped into multiple dependent or independent representations (called layers) by processes such as transformation and filtering. One or more encoders are used to encode the layered visual representations. When layers contain redundancy, a single encoder can be used to encode with significantly improved coding efficiency using inter-layer prediction techniques. Layered video coding is typically used to provide some kind of scalability (e.g., quality scalability, spatial scalability, temporal scalability, and view scalability) in services.

[0123] Temporal scalability may be treated differently compared to other types of scalability. A sub-layer, sub-layer, temporal sub-layer, or temporal sub-layer may be defined as a temporally scalable layer (or temporal layer, TL) of a temporally scalable bitstream. Each picture of a temporally scalable bitstream may be assigned a temporal identifier, which may be assigned, for example, to the variable TemporalId. The temporal identifier may be indicated, for example, in the NAL unit header or the OBU extension header. A TemporalId equal to 0 corresponds to the lowest temporal level. A bitstream created by excluding all coded pictures with a TemporalId equal to or greater than a selected value and including all other coded pictures maintains conformance. As a result, pictures with a TemporalId equal to tid_value do not use pictures with a TemporalId greater than tid_value as prediction references.

[0124] Inter prediction may involve referencing sample locations outside the picture boundary (also known as out-of-border (OOB)), i.e., motion vectors in inter prediction may point outside the picture boundary for at least, but not necessarily limited to, the following two reasons: First, the location of the predictive block corresponding to the motion vector used in inter prediction may be partially or entirely outside the picture boundary; Second, the location of the predictive block corresponding to the motion vector used in inter prediction may be within the picture boundary but include non-integer sample positions whose sample values ​​are interpolated using a filter that takes input samples from locations outside the picture boundary.

[0125] To allow motion vectors that point outside the picture boundary, existing codecs such as HEVC and VVC obtain samples outside the picture boundary by effectively duplicating samples on the picture boundary. Samples on the left and right picture boundaries are horizontally duplicated to OOB samples on the left and right sides of the picture, respectively. Samples on the top and bottom picture boundaries are vertically duplicated to OOB samples on the top and bottom sides of the picture, respectively. This duplication is sometimes called "padding." This reference picture extension allows the motion compensation process to point to the OOB area, improving prediction efficiency. However, because the extended area is effectively filled with padded samples, and these padded samples do not implement the actual texture behavior of the video, motion compensation performance is still suboptimal when using such samples for prediction.

[0126] Independent VVC subpictures are treated like pictures in the VVC decoding process. Motion vectors pointing outside the boundaries of the independent subpicture cause duplication of the subpicture's border samples. Furthermore, for independent VVC subpictures, it can be additionally requested that loop filtering across the boundaries of the independent VVC subpicture be disabled. A subpicture boundary is treated like a picture boundary in the VVC decoding process if sps_subpic_treated_as_pic_flag[i] is equal to 1 for the subpicture. Also, if sps_loop_filter_across_subpic_enabled_pic_flag[i] is 0, loop filtering across the boundaries of the subpicture is disabled in the VVC decoding process.

[0127] In the inter-prediction process, the mechanism for effectively duplicating border samples to OOB samples can be implemented in several ways. One method is to allocate a sample array that is larger than the decoded image size, i.e., with margins on the top, bottom, right, and left sides of the image. In addition to or instead of using such margins, the locations of samples used for prediction (either as inputs to fractional sample interpolation for the prediction block or as samples in the prediction block itself) can be saturated so that their locations do not cross picture boundaries (with margins, if used). Some video coding standards describe support for motion vectors that cross picture boundaries in this way.

[0128] In some projection formats for 360-degree panoramic and omnidirectional video, when horizontal samples outside the picture boundary are required in the inter-prediction process, instead of using replicated border samples, sample values ​​from the opposite side of the picture can be used, a prediction mode that is sometimes called wrap-round motion compensation.

[0129] The present embodiment relates to various examples for handling out of bounds (OOB) regions in samples in the motion compensation process when motion vectors point to such regions outside the picture.

[0130] In some embodiments, the out-of-bounds region is padded with samples that are replicated according to the motion vector direction instead of the traditional replication padding in the horizontal or vertical direction.

[0131] In some embodiments, samples in the out-of-bounds region are predicted or extrapolated using intra-prediction techniques using decoded samples within the picture. In such embodiments, the OOB area may be divided into MxN blocks, where M is the number of samples in the extended OOB area and N is the size of the other dimension of the block.

[0132] In some embodiments, intra prediction is used from the decoded samples of the current block, and the intra predicted samples are used as replacements for regions that fall in the OOB area in the motion compensation block.

[0133] FIG. 10 shows an exemplary process in which samples in an OOB area 1010 are predicted with intra prediction using decoded samples in a picture 1000.

[0134] According to one embodiment, the desired intra-prediction mode for predicting samples in the OOB area can be determined using texture analysis methods. For example, a texture analysis mechanism such as DIMD can be applied to the decoded samples in a picture to determine the intra-prediction mode.

[0135] According to one embodiment, the desired intra-prediction mode for predicting samples in the OOB area can be determined using a template-based intra derivation method. For example, a template-based method such as TIMD can be applied to the decoded samples in a picture to determine the intra-prediction mode.

[0136] According to one embodiment, the final prediction of the samples in the OOB area can be performed by combining two or more different intra predictions with specific weights. One or more of the following methods can be available and / or used for encoding and / or decoding: The weight value of each prediction can be determined in advance. The weight values ​​can be selected from among predefined values, the encoder can signal an associated index in or along the bitstream, and the decoder can decode the associated index from or along the bitstream, respectively. The weight values ​​may be signaled in or along the bitstream, for example by an encoder, and the weight values ​​may be decoded from or along the bitstream, for example by a decoder, respectively. Weight values ​​can be derived on the encoder and decoder side.

[0137] According to the embodiment shown in Figure 11, intra prediction can be used to predict samples for only a portion 1110 of the OOB area using decoded samples in picture 1100. The remaining portion 1120 may be padded horizontally or vertically with the last predicted sample, similar to how picture border samples are used for padding outside the picture boundary in published coding standards such as VVC. According to one embodiment, if a sample of the remaining portion (hereinafter a "corner sample") 1120 has both vertically and horizontally adjacent samples in the predicted OOB region 1110, the sample value of the corner sample is derived to be a weighted sum of the sample values ​​of the adjacent samples, where the weights may be inversely proportional to the spatial distance from the corner sample to the adjacent sample.

[0138] According to one embodiment, a cross-component prediction method such as CCLM or CCCM can be used to predict the samples of the OOB area. Model parameters can be derived, for example, using decoded samples of the current frame and the reference channel, and then the intra-predicted samples of the OOB area of ​​the reference channel can be used to predict the samples of the OOB area of ​​the current channel using the derived parameters.

[0139] According to one embodiment, in a motion compensation (MC) process, the region of a block that falls in an OOB area may be predicted using intra prediction from reference samples of a current block 1225 in a current frame 1220. Figure 12 shows an example where the motion vector of a block points to an OOB area in a reference frame 1210. Figure 13 shows an example where intra prediction samples 1320 from neighboring reference samples are used for the region of a block that falls in the OOB in the reference frame, and the remaining part of the block is filled with motion compensated prediction 1310.

[0140] According to one embodiment, the border samples between the intra prediction 1320X and the MC prediction 1310X (FIG. 13) in the final prediction can be filtered to generate a smoother prediction for the block, for example, a blending operation of the intra prediction samples and the inter prediction samples can be applied.

[0141] According to one embodiment, intra prediction can use motion-compensated samples together with reconstructed reference samples from neighboring blocks. Figure 14 shows an example in which motion-compensated samples together with reconstructed reference samples from neighboring blocks are used for bi-intra prediction of regions that fall under OOB in the motion compensation process.

[0142] In another embodiment, intra prediction may be obtained using only reconstructed reference samples from the neighborhood of the current block, and motion compensated samples may be used for filtering the intra predicted samples.

[0143] According to one embodiment, the intra prediction used to predict the regions of the block that fall within the OOB area may be a fixed mode, or may be determined using rate-distortion optimization involving signaling the most performing mode for each block, or alternatively may be determined using texture analysis methods (e.g., DIMD) or template-based methods (e.g., TIMD).

[0144] According to one embodiment, the intra prediction of the described region can be obtained by combining two or more intra prediction methods.

[0145] According to one embodiment, when bi-predictive motion compensation is used and only one of the motion vectors points to the OOB area, samples of the second motion compensated prediction can also be used in the intra prediction process.

[0146] According to one embodiment, the OOB area samples of a reference picture can be changed when a subsequent picture is coded or decoded, and when the subsequent picture is coded or decoded, the motion vector direction of the boundary block of the subsequent picture is used to derive the prediction direction for intra prediction or similarly directionally padding the reference picture.

[0147] According to one embodiment, the modification of the OOB area samples of a reference picture is invoked by a subsequent picture provided that the reference picture meets certain requirements, which may include, but are not necessarily limited to, one or more of the following requirements: A reference picture has a certain predefined picture type, for example, a reference picture is an IRAP (intra random access point) picture defined in VVC or a key frame defined in AV1. A reference picture has a particular picture type that is indicated in or along the bitstream, or is decoded from or along the bitstream. A reference picture has a certain predetermined coding type, for example, the reference picture is an intra-frame, or all slices of the reference picture are intra-slices. A reference picture has a particular coding type that is indicated in or along the bitstream or decoded from or along the bitstream.

[0148] According to one embodiment, the modification of the OOB area samples of the reference picture may be performed when encoding or decoding a single specific subsequent picture (hereinafter referred to as the current picture), provided that the current picture satisfies certain requirements, which may include, but are not limited to, one or more of the following: The current picture is the first subsequent picture in decoding order in the lowest temporal sublayer (denoted as TL0) following the reference picture, where the reference picture is also in the lowest temporal sublayer. The current picture is the first picture following the reference picture in decoding order and is in the same temporal sub-layer as the reference picture. The current picture uses the reference picture as a reference for inter-prediction.

[0149] According to one embodiment, modifying the OOB area samples of a reference picture may be performed iteratively when encoding or decoding a subsequent picture, provided that the subsequent picture meets certain requirements, which may include, but are not necessarily limited to, one or more of the following: A subsequent picture follows the reference picture in decoding order and has a lower temporal identifier value than any other picture that follows the reference picture in decoding order before this subsequent picture. Subsequent pictures use the reference picture as a reference for inter-prediction.

[0150] The above-described embodiments relating to performing modification of the OOB area samples of a reference picture based on modification of the reference picture or the subsequent picture can also be combined, i.e., the OOB area samples of the reference picture are modified when both the reference picture and the subsequent picture meet certain requirements as described in the above embodiments.

[0151] According to one embodiment, if a boundary block is encoded in intra-mode, the motion vector of a block neighboring the intra-encoded block may be used for directional padding. This may be a block immediately adjacent to the intra-encoded boundary block, or the first inter-encoded block at a certain distance and direction of the intra-encoded boundary block. The neighboring region may be defined by the coding standard and may be, for example, one or more CTUs, tiles, or slices. In another example, if a boundary block is encoded in intra-mode, the directional padding may follow the intra-prediction direction of the intra-encoded block.

[0152] The width or height of the directionally padded region can be selected according to the magnitude of the motion vector or can be hard coded (eg, to 64).

[0153] In some instances, samples that have already been directional padded may receive another directional padding. According to one embodiment, such samples are overwritten with new directional padding. According to one embodiment, the results of the directional padding may be blended to form sample values ​​for samples in the OOB area.

[0154] Figure 15 shows an example embodiment of directionally padding an OOB area with samples in a reference picture 1510. When a block at a picture boundary is encoded, the following steps are performed for each sub-block (e.g., 4x4 block) from which such a motion vector is derived that references a sample in the OOB area of ​​the reference picture 1510. In this process, a motion vector is copied for each block (e.g., 4x4 block) that references a sample outside the picture boundary. The motion vector is used to directionally pad the border samples in the reference picture. Motion compensation is performed from the padded reference picture.

[0155] Note that this process may be applied multiple times for candidate motion vectors selected by the encoder. Similarly, when a picture boundary block is decoded, the following steps are performed for each sub-block for which such a motion vector is derived that references a sample in the OOB area of ​​the reference picture 1510. 1. The motion vectors are used to derive a directional intra prediction mode or similar. 2. The derived intra prediction mode or the like is used to directionally pad the border samples of the reference picture relative to the OOB area of ​​the reference picture. 3. The prediction block for motion compensation of the sub-block being coded or decoded is formed using a reference picture with an OOB area with directionally padded samples.

[0156] According to one embodiment, when using intra prediction of OOB area samples, the motion vector direction of the boundary blocks may be taken into consideration so that an angular intra mode with the same or close direction is used for predicting the samples.

[0157] According to one embodiment, one or more of the described embodiments may be used individually or jointly to handle out-of-bounds samples in motion compensation.

[0158] According to one embodiment, the solution described in any other embodiment allows motion vectors to point to OOB areas, while their corresponding boundaries can be applied to tile, subpicture, and / or slice boundaries when configured so that they can be decoded independently. For example, if sps_subpic_treated_as_pic_flag[i] is indicated or inferred to be equal to 1 as in VVC, then subpicture boundaries are treated like picture boundaries in any described embodiment.

[0159] According to one embodiment, the number of padded samples (i.e., the width and height of the OOB area) is predefined. According to an embodiment, the width and / or height of the OOB area is indicated in or along the bitstream, e.g., at the channel, picture, or sequence level. In an embodiment, an encoder indicates the width and / or height of the OOB area in or along the bitstream, e.g., in a picture parameter set or sequence parameter set. The width and / or height can be indicated separately for all boundaries or jointly (i.e., a single value applied to both width and height), separately for horizontal and vertical boundaries, or separately for top, bottom, left, and right boundaries. The width and / or height can be indicated jointly for all independent regions to which OOB sample derivation applies, or individually for each independent region. Similarly, in an embodiment, a decoder decodes the width and / or height values ​​of the OOB area from or along the bitstream, such as from a picture parameter set or sequence parameter, and applies them accordingly during decoding. Controlling the width and / or height of the OOB area can be used to limit memory usage, which can be beneficial, for example, when the embodiment is applied to an independent region such as an independent sub-picture.

[0160] According to one embodiment, different methods for padding the OOB area are determined to be used for different borders. The padding method for a particular border can be signaled in the bitstream. For example, it can be signaled that out-painting based on texture analysis is used for the left and right borders, and padding based on the nearest available sample is used for the top and bottom borders.

[0161] According to one embodiment, the use of horizontal wrap-around motion compensation is enabled and indicated in the bitstream by an enabler (e.g., with pps_ref_wraparound_enabled_flag equal to 1, as in VVC, or similar), or decoded from the bitstream by a decoder. When horizontal wrap-around motion compensation is used, horizontal sample padding for the OOB area is turned off, and the sample padding described in this embodiment is only applied to the OOB area above or below the top and bottom boundaries. This can reduce memory usage in the implementation.

[0162] According to one embodiment, if a motion vector points to an area outside the first reference image, the motion vector is inverted by changing the signs of its horizontal and vertical motion vector components by a reference, and a second reference image is selected from a different temporal direction compared to the first reference image. In addition to inverting the motion vector, the motion vector may also be scaled depending on the temporal distance of the first and second reference images from the current image. The decision to invert the motion vector may be configured to depend on the size of the area falling outside the first reference image required to predict the block.

[0163] A method according to one embodiment is shown in Figure 16. The method generally includes determining a motion vector and reference frame for a current block (1610), determining when the motion vector points to an area outside the reference frame (1620), generating an extended reference frame by filling the area outside the reference frame with one or more of the following: directional intra prediction using border samples of the reference frame according to the direction of the motion vector; intra prediction using decoded samples in the current frame; or intra predicted samples of the current block (1630), predicting the current block using motion compensation from the extended reference frame by reference (1640), and encoding the current block into a bitstream (1650). Each of the steps can be implemented by a respective module of a computing system.

[0164] An apparatus according to one embodiment includes: means for determining a motion vector and a reference frame for a current block; means for determining when the motion vector points to an area outside the reference frame; means for generating an extended reference frame by filling the area outside the reference frame with one or more of the following: a directional intra-prediction method using border samples of the reference frame according to the direction of the motion vector, an intra-prediction method using decoded samples in the current frame; or an intra-predicted sample of the current block; means for predicting the current block by using motion compensation from the extended reference frame; and means for encoding the current block into a bitstream. The means include at least one processor and a memory containing computer program code, and the processor may further include a processor circuit. The memory and the computer program code, together with the at least one processor, are configured to cause the apparatus to perform the method of FIG. 16 in various embodiments.

[0165] An example of a data processing system for an apparatus is shown in Figure 17. Some functionality may be performed on a single physical device, for example, all computational procedures may be performed on a single processor if desired. The data processing system includes a main processing unit 100, memory 102, storage devices 104, input devices 106, output devices 108, and a graphics subsystem 110, all connected together via a data bus 112.

[0166] The main processing unit 100 is a conventional processing unit arranged to process data within a data processing system. The main processing unit 100 may comprise or be implemented as one or more processors or processing circuits. The memory 102, storage device 104, input device 106, and output device 108 may include conventional components as recognized by those skilled in the art. The memory 102 and storage device 104 store data within the data processing system 100.

[0167] Computer program code resides in memory 102 for implementing a method, such as that shown in the flowchart of FIG. 16 in various embodiments. Input device 106 inputs data into the system, while output device 108 receives data from the data processing system and transmits the data to, for example, a display. Data bus 112 is a conventional data bus and, although shown as a single line, may be any combination of a processor bus, PCI bus, graphics bus, or ISA bus. Accordingly, those skilled in the art will readily recognize that the present apparatus may be any data processing device, such as a computing device, personal computer, server computer, mobile phone, smartphone, or Internet access device, such as an Internet tablet computer.

[0168] Figure 18 shows an example of a video encoder, where In: image to be encoded; P'n: predictive representation of an image block; D'n: prediction error signal; D'n: reconstructed prediction error signal; I'n: preliminary reconstructed image; R'n: T, T-1: transform and inverse transform; Q, Q-1: quantization and inverse quantization; E: entropy coding; RFM: reference frame memory; Pinter: Pinter: inter prediction, Ptra: intra prediction, MS: mode selection, and F: filtering. Figure 19 shows a block diagram of a video decoder, where In: predictive representation of an image block; D'n: D'n: reconstructed prediction error signal; I'n: I'n: preliminary reconstructed image; R'n: final reconstructed image; T-1: inverse transform; Q-1: inverse quantization; E-1: entropy decoding; RFM: reference frame memory; P: prediction (either inter or intra); and F: filtering. An apparatus according to an embodiment may comprise only an encoder or only a decoder, or both.

[0169] Various embodiments may be implemented with the aid of computer program code residing in a memory and causing an associated device to perform a method. For example, a device may comprise circuitry and electronic devices for handling, receiving, and transmitting data, computer program code in a memory, and a processor that, when executing the computer program code, causes the device to perform the features in the embodiments. Furthermore, a network device, such as a server, may comprise circuitry and electronic devices for handling, receiving, and transmitting data, computer program code in a memory, and a processor that, when executing the computer program code, causes the network device to perform the features of the various embodiments.

[0170] This solution provides advantages: for example, in this embodiment, the motion compensation efficiency is improved when the motion vectors point to areas outside the picture boundaries in the reference frame.

[0171] If desired, different functions discussed herein may be performed in different orders and / or concurrently with others. Furthermore, if desired, one or more of the functions and embodiments described above may be optionally or combined.

[0172] While various aspects of the embodiments are defined in the independent claims, other aspects comprise other combinations of features from the described embodiments and / or dependent claims with features of the independent claims, not just the combinations explicitly set out in the claims.

[0173] It should also be noted that although exemplary embodiments have been described hereinabove, these descriptions should not be construed in a limiting sense. Rather, there are several variations and modifications that can be made without departing from the scope of the present disclosure, as defined in the appended claims. [Explanation of symbols]

[0174] 1610 Determine the motion vector and reference frame of the current block 1620 Determining when a motion vector points to an area outside the reference frame 1630 and following: Directional intra prediction using border samples of the reference frame according to the direction of the motion vector; Intra prediction method using decoded samples within the current frame; Generating an extended reference frame by filling the area outside the reference frame with one or more of the intra predicted samples of the current block. 1640 Predict the current block by using motion compensation from the extended reference frame 1650 Encode the current block into a bitstream

Claims

1. 1. An apparatus for encoding video frames, comprising: means for determining a motion vector and a reference frame for the current block; means for determining when the motion vector points to an area outside the reference frame; below; directional intra prediction using border samples of the reference frame according to the direction of the motion vector; Intra prediction methods using decoded samples within the current frame; Intra predicted samples of the current block; means for generating an extended reference frame by filling an area outside the reference frame with one or more of the following: means for predicting the current block by using motion compensation from the extended reference frame; means for encoding the current block into a bitstream; 1. An apparatus comprising:

2. The apparatus of claim 1 , further comprising: means for determining the intra-prediction method using texture analysis.

3. The apparatus of claim 2 , wherein the texture analysis method is a decoder-side intra-mode derivation method.

4. The apparatus according to claim 1 , further comprising means for determining the intra prediction method by using a template-based intra mode derivation method.

5. 2. The apparatus of claim 1, further comprising means for predicting samples of an area outside a picture cross-component prediction, wherein samples of an area outside the picture in the reference channel are used to predict samples of an area outside the picture in the current channel.

6. The device according to any one of claims 1 to 5, further comprising means for filtering border samples between the intra prediction and the motion compensated prediction in the final prediction.

7. 1. A method for encoding video frames, comprising: determining a motion vector and a reference frame for the current block; determining when the motion vector points to an area outside the reference frame; below: a directional intra prediction method using border samples of the reference frame according to the direction of the motion vector; Intra prediction methods using decoded samples within the current frame; Intra predicted samples of the current block; generating an extended reference frame by filling an area outside the reference frame with one or more of: predicting the current block by using motion compensation from the extended reference frame; encoding the current block into a bitstream; A method comprising:

8. The method of claim 7 , further comprising determining the intra-prediction method using texture analysis.

9. The method of claim 8 , wherein the texture analysis method is a decoder-side intra-mode derivation method.

10. The method according to any one of claims 7 to 9, further comprising determining the intra prediction method by using a template-based intra mode derivation method.

11. 8. The method of claim 7, further comprising predicting samples of an area outside the picture by cross-component prediction, wherein samples of the area outside the picture in the reference channel are used to predict samples of the area outside the picture in the current channel.

12. The method according to any of claims 7 to 11, further comprising filtering border samples between the intra prediction and the motion compensated prediction in the final prediction.

13. 1. An apparatus comprising at least one processor and a memory containing computer program code, The memory and the computer program code are configured to, using the at least one processor, perform at least: determining a motion vector and a reference frame for the current block; determining when a motion vector points to an area outside the reference frame; below: directional intra prediction using border samples of the reference frame according to the direction of the motion vector; Intra prediction methods using decoded samples within the current frame; Intra predicted samples of the current block; generating an extended reference frame by filling an area outside the reference frame with one or more of: predicting the current block by using motion compensation from the extended reference frame; and encoding the current block into a bitstream; The apparatus is configured to cause the apparatus to execute the

Images