Video encoding and video decoding methods, apparatus, and computer program products

Abstract

Embodiments relate to a method and technical equipment for carrying out the method. The method includes means for identifying one or more reference lines, extracting a reference sample for a block using reference lines not adjacent to the block, and means for performing an intra-prediction for the block using a planar prediction process and the extracted reference sample, wherein the planar prediction process determines a prediction sample according to the distance between the reference line and the block.

Description

[Technical Field]

[0001] This solution generally relates to video encoding and video decoding. [Background technology]

[0002] Video encoding is the process of converting input video into a compressed format suitable for storage or transmission. Video decoding does the reverse, that is, the compressed video is restored to a viewable format. The encoding process involves prediction, where the pixel values ​​of a certain picture section are predicted. Then, the prediction error, i.e., the difference between the predicted pixels and the original pixels, is coded. [Overview of the Initiative]

[0003] The embodiments described herein provide improved predictive solutions used in video encoding and decoding.

[0004] The scope of protection required for various embodiments of the present invention is described by the independent claims. Where there are embodiments and features described herein that are not included in the scope of the independent claims, they are to be construed as examples useful for understanding various embodiments of the present invention.

[0005] Various embodiments include methods, apparatus, and computer-readable media on which computer programs are stored, characterized by the content of the independent claims. Various embodiments are disclosed in the dependent claims.

[0006] According to a first embodiment, an apparatus is provided that includes means for identifying one or more reference lines, means for extracting a reference sample for a block using a reference line not adjacent to the block, and means for performing an intra-prediction for a block using a planar prediction process and the extracted reference sample, wherein the planar prediction process includes determining a prediction sample according to the distance between a reference line and a block.

[0007] A second aspect provides a method comprising identifying one or more reference lines, extracting a reference sample for a block from reference lines not adjacent to the block, and performing an intra-prediction for the block using a planar prediction process and the extracted reference sample, wherein the planar prediction process determines a prediction sample according to the distance between the reference line and the block.

[0008] According to a third aspect, a device is provided that includes at least one processor and a memory containing computer program code, wherein the memory and the computer program code are configured to cause the device to perform, by the at least one processor, at least one or more reference lines, extract reference samples for a block from reference lines not adjacent to the block, and perform an intra-prediction for the block using a planar prediction process and the extracted reference samples, the planar prediction process including determining a prediction sample according to the distance between the reference line and the block.

[0009] According to a fourth aspect, a computer program product is provided, which includes computer program code configured to cause an apparatus or system running on at least one processor to identify one or more reference lines, extract reference samples for a block from reference lines not adjacent to the block, and perform an intra-prediction for the block using a planar prediction process and the extracted reference samples, wherein the planar prediction process determines the prediction samples according to the distance between the reference lines and the block.

[0010] According to one embodiment, the reference line is identified by a multiple reference line index.

[0011] According to one embodiment, the planar prediction process includes horizontal and / or vertical interpolation.

[0012] According to one embodiment, horizontal and / or vertical interpolation includes defining an array of samples based on multiple reference line indices.

[0013] According to one embodiment, horizontal and / or vertical interpolation includes a weighted average based on the distance of each sample from a reference line.

[0014] According to one embodiment, the weighted average includes calculating a weight in which one of the weights is equal to the distance of each sample from a reference line.

[0015] According to one embodiment, the weighted average for vertical interpolation includes performing one of the following: division, multiplication, addition, or binary shift by the sum of the block height and multiple reference line indices.

[0016] According to one embodiment, the weighted average for horizontal interpolation includes performing one of the following: division, multiplication, addition, or binary shift by the sum of the block width and multiple reference line indices.

[0017] According to one embodiment, the planar prediction process is determined using a template matching multiple reference line method.

[0018] According to one embodiment, a reference sample is extracted by defining a horizontal and / or vertical array of the reference sample, and the apparatus includes means for forming an array of reconstructed samples using block locations and reference lines.

[0019] According to one embodiment, the planar prediction is formed using horizontal and / or vertical arrays.

[0020] According to one embodiment, the computer program product is embedded on a non-transient computer-readable medium.

[0021] In the following sections, various embodiments will be described in more detail with reference to the attached drawings. [Brief explanation of the drawing]

[0022] [Figure 1] This figure shows one example of the encoding process. [Figure 2] This figure shows one example of the decoding process. [Figure 3] This figure shows one example of a multiple reference line index. [Figure 4] This figure shows one example of determining vertical planar shape. [Figure 5] This figure shows an example of a two-stage planar interpolation operation. [Figure 6] This flowchart shows a method according to one embodiment. [Figure 7] This figure shows an apparatus according to one embodiment. [Modes for carrying out the invention]

[0023] The following descriptions and drawings are illustrative and should not be considered unnecessarily limiting. Certain details are given so as to enable a full understanding of this disclosure. However, in some cases, well-known or conventional details are omitted so as not to obscure the description. When the terms “Embodiment” or “Embodiment” are used herein, it means that certain features, structures or characteristics described in relation to that embodiment are included in at least one embodiment of this disclosure.

[0024] A video codec includes an encoder that converts input video into a compressed representation suitable for storage / transmission, and a decoder that can restore the compressed video representation into a viewable format. The encoder may discard some information from the original video sequence in order to represent the video in a more compact format (i.e., at a lower bitrate).

[0025] Figure 1 shows an example of the encoding process of a two-dimensional (2D) picture, and Figure 2 shows an example of the decoding process of a 2D picture. The following is shown in Figure 1. - An image (I n ) to be encoded, - A predicted representation (P’ n ) of an image block, - A prediction error signal (D n ), - A reconstructed prediction error signal (D’ n ), - A temporary reconstructed image (I’ n ), - A final reconstructed image (R’ n ), - Transformation (T) and inverse transformation (T -1 ), - Quantization (Q) and inverse quantization (Q -1 ), - Entropy encoding (E), - Reference frame memory (RFM), - Inter prediction (P inter ), - Intra prediction (P intra ), - Mode selection (MS), and - Filtering (F).

[0026] The following is shown in Figure 2. - A predicted representation (P’n) of an image block, - A reconstructed prediction error signal (D’n), - A temporary reconstructed image (I’n), - A final reconstructed image (R’n), inverse transformation (T-1), - Inverse quantization (Q-1), - Entropy decoding (E-1), - Reference frame memory (RFM), - Prediction (inter or intra) (P), - And filtering (F)

[0027] The basic unit of input to an encoder and output from a decoder is almost always a picture. The picture given as input to the encoder is sometimes called the source picture, and the picture decoded by the decoder is sometimes called the decoded picture or reconstructed picture. Both the source picture and the decoded picture consist of one or more sample sequences, such as one of the following sets of sample sequences. - Luma (Y) only (monochrome). - Luma and two chromosomes (YCbCr or YCgCo). - Green, blue, and red (also known as GBR or RGB). - An array representing other unspecified monochrome or tristimulus color samples (for example, YZX, also called XYZ).

[0028] A picture can be defined as a frame or a field. A frame contains a matrix of lumar samples and, potentially, corresponding chroma samples. A field is a set of alternating sample rows from a frame and can be used as an encoder input when the source signal is interlaced. There may be no chroma sample sequence (and therefore monochromatic sampling may be used), or the chroma sample sequence may be subsampled compared to the lumar sample sequence.

[0029] The Advanced Video Coding (AVC or sometimes abbreviated as H.264 / AVC) standard was developed by the Joint Video Team (JVT), a joint team of the Video Coding Experts Group (VCEG) of the Telecommunication Standardization Sector of the International Telecommunication Union (ITU-T) and the Moving Picture Experts Group (MPEG) of the International Organization for Standardization (ISO) / International Electrotechnical Commission (IEC). The H.264 / AVC standard is published by both parent standardization bodies and is known as ITU-T Recommendation H.264 and ISO / IEC International Standard 14496-10, and is also called MPEG-4 Part 10 Advanced Video Coding (AVC). There are multiple versions of the H.264 / AVC standard, each incorporating new extensions or features into the specification. These extensions include Scalable Video Coding (SVC) and Multiview Video Coding (MVC).

[0030] The High Efficiency Video Coding (HEVC or sometimes abbreviated as H.265 / HEVC) standard was developed by the Joint Collaborative Team-Video Coding (JCT-VC) of VCEG and MPEG. This standard was published by both parent standardization bodies and is known as ITU-T Recommendation H.265 and ISO / IEC International Standard 23008-2, and is also called MPEG-H Part 2 High Efficiency Video Coding (HEVC)). Extensions to H.265 / HEVC include scalable, multiview, 3D, and fidelity range extensions, which may be referred to as SHVC, MV-HEVC, 3D-HEVC, and REXT, respectively. In this specification, when H.265 / HEVC, SHVC, MV-HEVC, 3D-HEVC, and REXT are referred to for the purpose of understanding the definitions, structure, or concepts of these standards, unless otherwise specified, they should be understood to refer to the most recent editions of these standards that were available prior to the date of this application.

[0031] Versatile Video Coding (VVC, sometimes abbreviated as H.266 or H.266 / VVC) is a video compression standard developed as a successor to HEVC. VVC is defined in ITU-T Recommendation H2.66 and its equivalent, ISO / IEC 23090-3, also known as MPEG1 Part 3.

[0032] This section describes some key definitions, bitstream and coding structures, and concepts of H.264 / AVC, HEVC, VVC, and / or AV1 and their extensions, as examples of video encoders, decoders, encoding methods, decoding methods, and bitstream structures that embodiments can implement. The various embodiments are described not as being limited to H.264 / AVC, HEVC, VVC, and / or AV1 or their extensions, but rather as one possible basis upon which the presented embodiments can be realized, either partially or entirely.

[0033] Hybrid video codecs, such as ITU-T H.263, H.264 / AVC, HEVC, and VVC, can encode video information in two stages. First, pixel values ​​within a picture (or "block") are predicted, for example, by motion compensation or by spatial means. Motion-compensated prediction finds a section in one of the previously coded frames that closely corresponds to the block being coded and uses it for prediction. Spatial prediction uses pixel values ​​around the block being coded. In the first stage, predictive coding can be applied, for example, as so-called sample prediction and / or so-called syntactic prediction.

[0034] In sample prediction, pixel values ​​or sample values ​​within a picture parcel or "block" are predicted. These pixel values ​​or sample values ​​can be predicted using, for example, one or more motion compensation mechanisms or intra-prediction mechanisms.

[0035] Motion compensation mechanisms (sometimes called interpretation, time prediction, motion-compensated time prediction, motion-compensated prediction, or MCP) involve finding and indicating a segment in one of the previously encoded video frames that closely corresponds to the block being coded. Interpretation can reduce time redundancy.

[0036] Intra prediction, which can predict pixel values ​​or sample values ​​using spatial mechanisms, involves finding and indicating spatial domain relationships. Intra prediction leverages the fact that adjacent pixels within the same picture are likely to be correlated. Intra prediction can be performed in the spatial domain or the transformation domain, i.e., it can predict sample values ​​or transformation coefficients. Intra prediction is typically used in intra coding where inter-prediction is not applicable.

[0037] Syntax prediction, sometimes called parameter prediction, predicts syntactic elements and / or syntactic element values ​​and / or variables derived from syntactic elements based on previously (decoded) syntactic elements and / or previously derived variables. A non-restrictive example of syntax prediction is shown below.

[0038] In motion vector prediction, for example, motion vectors for inter-prediction and / or interview prediction can be coded to be relatively different from block-specific predicted motion vectors. In many video codecs, predicted motion vectors are created in a predefined manner, for example, by calculating the median of the encoded or decoded motion vectors of adjacent blocks. Another method of creating motion vector predictions, sometimes called advanced motion vector prediction (AMVP), is to generate a list of candidate predictions from adjacent blocks and / or co-location blocks in a temporal reference picture and signal the selected candidates as motion vector predictors. In addition to predicting motion vector values, it is possible to predict the reference index of a previously coded / decoded picture. The reference index is typically predicted from adjacent blocks and / or co-location blocks in the temporal reference picture. Difference coding of motion vectors across slice boundaries is typically disabled.

[0039] For example, it is possible to predict block division from coding tree units (CTUs) to coding units (CUs) and prediction units (PUs).

[0040] Filter parameter prediction can predict filtering parameters, for example, for sample-adaptive offsets. Prediction methods that use image information from previously coded images can also be called inter-prediction methods, sometimes referred to as time prediction and motion compensation. Prediction methods that use image information from the same image can also be called intra-prediction methods.

[0041] In the second stage of encoding, the prediction error, i.e., the difference between the predicted block of a pixel and the original block of the pixel, is coded. This can be done by transforming the difference in pixel values ​​using a specified transformation (e.g., the Discrete Cosine Transform (DCT) or a variation thereof), quantizing the coefficients, and entropy coding the quantized coefficients. By changing the fidelity of the quantization process, the encoder can control the balance between the accuracy of the pixel representation (picture quality) and the size of the resulting coded video representation (file size at the transmission bitrate).

[0042] In some video codecs, such as H.265 / HEVC, a video picture is divided into coding units (CUs) that cover the area of ​​the picture. A CU consists of one or more prediction units (PUs) that define the prediction process for the samples within the CU, and one or more transform units (TUs) that define the prediction error coding process for the samples within that CU. Typically, a CU consists of square blocks of samples with sizes selectable from a predefined set of possible CU sizes. A CU with the largest allowable size is typically called a LUC (largest coding unit) or CTU (coding tree unit), and the video picture is divided into non-overlapping CTUs. A CTU can be further divided into smaller combinations of CUs, for example, by recursively dividing a CTU and a resulting CU. Each resulting CU typically has at least one PU and at least one associated TU. Each PU and TU can be further divided into smaller PUs and TUs to increase the granularity of the prediction coding process and the prediction error coding process, respectively. Each PU has associated prediction information (e.g., motion vector information for interpredictive PUs and intrapredictive directionality information for intrapredictive PUs) that defines what type of prediction is applied to the pixels within that PU. Similarly, each TU is associated with information (e.g., including DCT coefficient information) that describes the prediction error decoding process for the samples within that TU. Typically, at the CU level, each CU is signaled as to whether or not prediction error coding is applied. If there are no prediction error residuals associated with a CU, that CU can be considered to have no TU. The division of an image into CUs and the division of CUs into PUs and TUs are typically signaled with a bitstream that allows the decoder to reconstruct the intended structure of these units.

[0043] The decoder reconstructs the output video by applying predictive means similar to those of the encoder to form a predictive representation of pixel blocks (using motion or spatial information created by the encoder and stored in a compressed representation), and predictive error decoding (the reverse operation of predictive error coding to recover quantization predictive error signals in the spatial pixel region). After applying the predictive and predictive error decoding means, the decoder sums the predictive and predictive error signals (pixel values) to form the output video frame. The decoder (and encoder) may also apply additional filtering means to improve the quality of the output video before passing it for display and / or storage as a predictive reference for future frames appearing in the video sequence.

[0044] Instead of, or in addition to, methods that use sample value prediction and transformation coding to indicate coded sample values, color palette-based coding can be used. Palette-based coding refers to a set of methods in which a palette, i.e., a set of colors and associated indices, is defined, and the value of each sample within a coding unit is represented by indicating its index in the palette. Palette-based coding can achieve good coding efficiency in coding units with a relatively small number of colors (such as image sections representing computer screen content like text or simple graphics). To improve the coding efficiency of palette coding, different types of palette index prediction methods can be used, or the palette index can be run-length coded to efficiently represent larger homogeneous image sections. Also, if a CU contains sample values ​​that do not recurse within the CU, escape coding can be used. Escaped coded samples are sent without referencing any palette index. Instead, their values ​​are individually indicated for each escaped coded sample.

[0045] In many video codecs, including H.264 / AVC, HEVC, and VVC, motion information is indicated by motion vectors associated with each motion-compensated image block. Each of these motion vectors represents the displacement of a predicted source block in one of the image blocks in the picture being coded (in the encoder) or decoded (in the decoder) and one of the previously coded or decoded images (or pictures). To efficiently represent motion vectors, they are typically differential-coded relative to a block-specific predicted motion vector. In typical video codecs, predicted motion vectors are created by a predefined scheme, such as calculating the median of the encoded or decoded motion vectors of adjacent blocks. Another scheme for creating motion vector predictions is to generate a list of candidate predictions from adjacent blocks and / or co-location blocks in the time-referenced picture and signal the selected candidates as motion vector predictors. In addition to predicting motion vector values, it is possible to predict the reference index of the previously coded / decoded picture. The reference index is typically predicted from adjacent blocks and / or co-location blocks in the time-referenced picture. Furthermore, typical high-efficiency video codecs employ an additional motion information coding / decoding mechanism, often called merging mode, in which all motion field information, including motion vectors and the corresponding reference picture index for each available reference picture list, is predicted and used without any modification / correction. Similarly, the prediction of motion field information is performed using motion field information from adjacent and / or same-location blocks within the time reference picture, and the motion field information used is signaled in a list of motion field candidate lists containing motion field information from available adjacent / same-location blocks.

[0046] The video codec can support motion-compensated prediction from a single source image (single prediction) and from two sources (dual prediction). In single prediction, a single motion vector is applied; in dual prediction, two motion vectors are signaled, and the motion-compensated predictions from the two sources are averaged to create the final sample prediction. In weighted prediction, the relative weights of the two predictions can be adjusted, or a signaled offset can be added to the prediction signal.

[0047] In addition to applying motion compensation for interpicture prediction, a similar technique can be applied to intrapicture prediction. In this case, the displacement vector indicates where sample blocks can be copied from the same picture to form predictions of the blocks being coded or decoded. This type of intrablock copying method can significantly improve coding efficiency when there are repetitive structures within a frame, such as text or other graphics.

[0048] In video codecs, the predicted residuals after motion compensation or intra-prediction can first be transformed by a transformation kernel (such as DCT) and then coded. This is because some correlation often still exists between the residuals, and the transformation can often reduce this correlation, making it easier to achieve more efficient coding.

[0049] Many video encoders use a Lagrangian cost function to find the optimal coding mode, such as the desired macroblock mode and associated motion vectors. This type of cost function uses a weighting coefficient λ, as follows, to couple the (exact or estimated) image distortion resulting from the lossy coding method with the (exact or estimated) amount of information required to represent the pixel values ​​within the image parcel: C=D+λR (formula 1)

[0050] Here, C is the Lagrangian cost to be minimized, D is the image distortion (e.g., mean squared error) taking into account the modes and motion vectors, and R is the number of bits required to represent the data needed to reconstruct the image block in the decoder (including the amount of data to represent the candidate motion vectors).

[0051] The phrase "associated with a bitstream" (for example, "indicated with a bitstream") can be defined as referring to an out-of-band transmission, signaling, or storage in which out-of-band data is associated with the bitstream. Phrases such as "decoded with a bitstream" may refer to the decoding of reference out-of-band data associated with the bitstream (which can be obtained from an out-of-band transmission, signaling, or storage). For example, an indication associated with a bitstream may refer to metadata within a container file that encapsulates the bitstream.

[0052] Scalable video coding refers to a coding structure in which a single bitstream can contain multiple representations of content at different bitrates, resolutions, or frame rates. In such cases, a receiver can extract the desired representation according to its characteristics (e.g., the resolution best suited to the display device). Alternatively, a server or network element can extract portions of the bitstream sent to the receiver, depending, for example, network characteristics or the receiver's processing power. A scalable bitstream may consist of a "base layer" that provides the lowest available video quality and one or more enhancement layers that improve video quality when received and decoded together with lower layers. To improve the coding efficiency of the enhancement layers, the coded representations of those layers typically depend on lower layers. For example, motion and mode information of the enhancement layers can be predicted from lower layers. Similarly, pixel data from lower layers can be used to create predictions for the enhancement layers.

[0053] Scalable video codecs for quality scalability (also known as signal-to-noise ratio or SNR) and / or spatial scalability can be implemented as follows: For the base layer, conventional non-scalable video encoders and decoders are used. The reconstructed / decoded picture from the base layer is placed in a reference picture buffer for the enhancement layer. In H.264 / AVC, HEVC, and similar codecs that use a reference picture list for inter-prediction, the decoded picture from the base layer can be inserted into the reference picture list for coding / decoding of the enhancement layer picture, similar to the decoded reference picture from the enhancement layer. Thus, the encoder can select the base layer reference picture as the inter-prediction reference and indicate its use, for example, using the reference picture index in the coded bitstream. The decoder decodes from the bitstream, for example from the reference picture index, that the base layer picture is to be used as the inter-prediction reference for the enhancement layer. When the decoded base layer picture is used as the prediction reference for the enhancement layer, that picture is called the inter-layer reference picture.

[0054] In addition to quality scalability, the following scalability modes exist: • Spatial scalability: Base layer pictures are coded at a lower resolution than reinforced layer pictures. • Bit depth scalability: Base layer pictures are coded with a lower bit depth (e.g., 8 bits) than reinforcement layer pictures (e.g., 10 bits or 12 bits). • Chroma format scalability: Enhanced layer pictures provide higher chroma fidelity (e.g., coded in 4:4:4:chroma format) than base layer pictures (e.g., 4:2:0 format). In all of the above scalability cases, base layer information can be used to code the reinforcement layer in order to minimize additional bitrate overhead.

[0055] Scalability can be achieved in two basic ways: by introducing a new coding mode to predict pixel values ​​or syntax from lower layers of the scalable representation, or by inserting lower-layer pictures into a reference picture buffer (decoded picture buffer, DPB) in a higher layer. The first method is more flexible and therefore, in most cases, can result in better coding efficiency. However, the second, reference frame-based scalability method can be implemented very efficiently with minimal modifications to a single-layer codec while achieving most of the coding efficiency gains. A reference frame-based scalability codec can be implemented using the same hardware or software embodiment for all layers, with only the need to address DPB management by external means.

[0056] There are several intra-predictive modes available for existing video codecs. These intra-predictive modes include different directional intra-predictive modes and predictive modes such as DC or planar intra-predictive modes.

[0057] In planar intraprediction, interpolation processes are performed in both the vertical and horizontal directions to obtain vertical and horizontal predictors, respectively. Next, the sample-by-sample average of these two predictors can be calculated. Each interpolation process can be performed by calculating a weighted average of reference samples. In one example, when calculating the vertical predictor, each predicted sample at a given horizontal coordinate is obtained as a weighted average of two reference samples at the same horizontal coordinate, extracted from the top and bottom rows of the current block, respectively. Since the sample located in the bottom row of the current block is not reconstructed when predicting the current block, it is padded using the bottom-left sample of the current block. Similarly, the horizontal predictor is calculated as a weighted average of reference samples extracted from the left and right columns of the current block.

[0058] In coding solutions prior to this embodiment, reference samples used in intra-prediction modes such as planar prediction were extracted from blocks directly adjacent to the current block. For example, samples located in the row directly above the current block and the column immediately to the left of the current block could be used.

[0059] However, in some cases, these reference samples may not be ideal, and other reference samples may be used instead. As one example, reference samples located in columns and / or rows further away from the current block can be used. However, when using such techniques involving the use of reference samples located further away from the current block, conventional planar intraprediction processes are not ideal because they do not take into account the spatial position of the reference samples relative to the current block, and therefore the resulting predicted block may not adequately represent the contents of the current block.

[0060] The use of intra-predictive reference lines located further away from the current block is well known in video coding. The use of these reference lines can be indicated using signaling. In VVC, the use of these reference lines is indicated using a Multiple Reference Line (MRL) index, where MRL index 0 corresponds to the use of conventional reference lines consisting of samples extracted directly above and to the left of the current block. As the MRL index increases, it corresponds to the use of samples located further away from the current block vertically and horizontally. The MRL index is then signaled in the bitstream.

[0061] Recently, more advanced techniques have been proposed for better signal utilization of MRL. One example is the Template-based Multiple Reference Line (TMRL). The essence of TMRL lies in computing a list of possible combinations of intra-predictive modes and MRL indices. This list is calculated based on the distortion cost obtained using a template, which consists of a reference sample located at MRL index 0. The index is then signaled with a bitstream that identifies the appropriate element in the TMRL list to be used for the current block. This TMRL index replaces the signaling of both its particular intra-mode and MRL index, thus improving coding efficiency.

[0062] Herein, the presented embodiment improves upon known intra-predictive solutions.

[0063] Therefore, the objective is to generate an intra-prediction for a block using a planar intra-prediction process that uses reference samples extracted from reference lines not adjacent to the block, and the planar intra-prediction process is performed according to the distance of the reference sample from the current block. Specifically, the planar intra-prediction process is calculated by performing an interpolation process in the vertical and / or horizontal directions, and the predicted samples calculated by the interpolation process are obtained according to the distance of the reference row from the current block. In one embodiment, each reference row can be identified by a multi-reference row index. In one embodiment, the planar intra-prediction process is performed according to the multi-reference row index. In one embodiment, the horizontal and / or vertical interpolation process required to obtain the planar intra-prediction is obtained according to the multi-reference row index.

[0064] According to one embodiment, horizontal and / or vertical interpolation processes can be performed to obtain a planar intraprediction. Horizontal and / or vertical interpolation includes calculating a weighted average that depends on the distance of each sample from the reference row. For example, in the case of vertical interpolation, the weighted average includes dividing the block height by the sum of the multiple reference row indices. Similarly, in the case of horizontal interpolation, the weighted average includes dividing the block width by the sum of the multiple reference row indices.

[0065] In the embodiments described above, division is used. However, according to other embodiments, other operations can be used instead. Examples of other operations include multiplication, addition, or binary shift. For example, in the case of vertical interpolation, the weighted average may involve multiplying the block height by the sum of the multiple reference row indices. Similarly, in the case of horizontal interpolation, the weighted average may involve multiplying the block width by the sum of the multiple reference row indices. In another embodiment, division can be approximated by multiplication and binary shift. The multipliers and shift parameters for approximate division can be determined in different ways, for example, by using a combination of normalization (binary shift based on the size of the denominator) and table loop-up operations.

[0066] In the embodiments described above, the current intra-prediction can be identified using a template matching multiple reference row method, which includes computing a planar intra-prediction process using reference samples extracted from reference rows that do not necessarily have to be adjacent to a block.

[0067] In this regard, an encoding apparatus according to one embodiment includes means for identifying one or more reference rows, means for extracting reference samples for a block from reference lines not adjacent to the block, and means for performing an intra-prediction for a block using a planar prediction process and the reference samples, wherein the planar prediction process includes determining a prediction sample according to the distance between a reference row and a block.

[0068] One embodiment of plana-intra prediction for a block uses reference samples extracted from reference rows that do not necessarily have to be adjacent to the block. The plana-intra prediction process is performed according to the distance of the reference samples from the current block. In one embodiment, each reference row is identified by a multiple reference row (MRL) index, and as shown in Figure 3, a larger MRL index corresponds to a reference row further away from the current block. In one embodiment, considering the current block, the location of the current block in the frame is indicated by x, which shows the location of the top-left sample within the block. curr and y curr This is defined as follows. Then, as one embodiment, the reference sample used to form the planar intraprediction is the vertical coordinate y ref ga y ref <y curr It may include a sample extracted from a location within the current block such that the horizontal coordinate x is -1, and / or the horizontal coordinate x ref is x ref <x curr It can include samples extracted from locations within the current block such that the value is -1.

[0069] According to one embodiment, the planar intraprediction process is calculated by performing an interpolation process in the vertical and / or horizontal directions, and the predicted samples calculated by the interpolation process are obtained based on the distance of the reference row from the current block. In one embodiment, each reference row can be identified by a multi-reference row index. In one embodiment, the planar intraprediction process can be performed according to a multi-industry row index. In one embodiment, the horizontal and / or vertical interpolation process required to obtain the planar intraprediction can be obtained according to a multi-reference row index.

[0070] As one embodiment, consider a current block having size W×H and MRL index m. Also, consider a given set of reference samples consisting of already reconstructed samples around the current block. Samples that have not yet been reconstructed can be replaced with other already reconstructed samples, or computed using other padding techniques. Reference samples can be extracted by defining a horizontal and vertical array of reference samples, where the horizontal array R H This consists of reconstruction samples from the following locations: R H (i) = (x curr -1-m+i,y curr -1-m), i=0,1,2... Vertical array R V This similarly consists of reconstruction samples from the following locations: R V (j)=(x curr -1m, y curr (-1-m+j), j=0,1,2...

[0071] According to one embodiment, two arrays R H and R V The data may be filtered and / or processed before being used for prediction. In one embodiment, a padding operation may be used to fill the sequence.

[0072] These two arrays of the reference sample can then be used to form a planar prediction for the current block. According to one embodiment, the vertical planar prediction can be calculated as follows: - Define the sequence topRow based on the MRL index m. In one example, topRow is defined as follows: topRow(k)=R H (k+m), k=0,1,...,W-1 - Define the sample value bottomLeft based on the MRL index m. In one example, bottomLeft is defined as follows: bottomLeft=R V (H+m)

[0073] In another example, bottomLeft can be defined as follows: bottomLeft=R V (H+m+1) - Vertical planar prediction P based on MRL index m v The (x,y) coordinates are calculated. In one embodiment, the vertical planar prediction is calculated based on the vertical distance of a given pixel from a reference sample. Specifically, the distance of a given pixel at a given vertical coordinate y can be defined as follows: D V (y=y curr +m+1 - Two weights W BL (y=D V (y) W TR (y) = H + mD V (y) This is how it is defined. - The vertical planar prediction is calculated as follows.

[0074]

number

[0075] In another embodiment, the distance of a given pixel at a given vertical coordinate y can be defined as follows: D V (y=y- ycurr +m - Two weights W BL (y=D V (y) W TR (y) = H + mD V (y) This is how it is defined. - The vertical planar prediction is calculated as follows.

[0076]

number

[0077] In another embodiment, the distance of a given pixel at a given vertical coordinate y can be defined as follows: D V (y) = yy curr +m+1 - Two weights W BL (y=D V (y) W TR (y) = H + 1 + mD V (y) This is how it is defined. - The vertical planar prediction is calculated as follows.

[0078]

number

[0079] Figure 4 shows the process for calculating the vertical planar prediction.

[0080] Similarly, horizontal planar predictions can be calculated as follows. - Define the array leftColumn based on the MRL index m. In one example, leftColumn is defined as follows: leftColumn(k)=R V (k+m), k=0,1,...,H-1 - Define the sample value rightTop based on the MRL index m. In one example, rightTop is defined as follows: rightTop=R H (W+m) As another example, the rightTop sample value can be defined as follows: rightTop=R H (W+m+1) - Horizontal planar prediction P based on MRLindexm HDefine (x,y). In one embodiment, horizontal planar prediction is determined based on the horizontal distance of a given pixel from a reference sample. Specifically, the distance of a given pixel at a given horizontal coordinate x can be defined as follows: D H (x=xx) curr +m+1 - Two weights W RT (x=D) H (x) W LC (x)W+mD H (x) This is how it is defined. - The horizontal planar prediction is calculated as follows.

[0081]

number

[0082] In another embodiment, the distance of a given pixel at a given horizontal coordinate x can be defined as follows: D H (x=xx) curr +m - Two weights W RT (x=D) H (x) W LC (x) = W + mD H (x) This is how it is defined. - The horizontal planar prediction is calculated as follows.

[0083]

number

[0084]

number

[0085] Planar prediction using two predictors P H (x,y) and P V The average of (x,y) is calculated as follows.

[0086]

number

[0087] P H (x,y) and P V The denominators needed to calculate (x,y) can be equal to the following: W BL (y) + W TR (y) = H + m W RT (x) + W LC (x) = W + m

[0088] As another example, P H (x,y) and P V The denominators needed to calculate (x,y) can be equal to the following: W BL (y) + W TR (y) = H + m + 1 W RT (x) + W LC (x) = W + m + 1

[0089] According to one embodiment, division by (H+m) and (W+m) can be replaced with binary shifts, multiplication, and addition to make the calculation of horizontal and vertical planar predictions easier to perform.

[0090] According to one embodiment, distance DH (x) and D V (y) can be calculated as follows. D V (y) = y - y curr D H (x) = x - x curr These are W BL (y) + W TR (y) = H W RT (x) + W LC (x) = W corresponding to, when H and W can be accurately represented as powers of 2, this is equivalent to replacing division with binary shift.

[0091] As yet another example, by performing a two - stage prediction, the number of division or multiplication - shift - addition operations that replace division can be reduced. In the first stage, the samples of topRow and leftColumn are projected to the locations corresponding to multiple reference line indices 0. This can be done by using the vertical and horizontal plane predictions described above, for example, by obtaining temporary reference arrays topRowTemp and leftColumnTemp as follows. topRowTemp(x) = P v (x, y curr - 1), where x = 0...W - 1 leftColumnTemp(y) = P h (x curr - 1, y), where y = 0...H - 1

[0092] In the second stage, the topRowTemp and leftColumnTemp arrays can be used instead of topRow and leftColumn to determine the final planar prediction. In this case, if the width W and height H of the prediction block are powers of 2, the denominator of the planar equation will also be a power of 2, allowing for the advantageous use of bitwise shift operations instead of division, which leads to a reduction in complexity in a typical implementation environment. Figure 5 shows the two-stage planar interpolation operation.

[0093] The planar prediction process described herein can also be used to compute predictions for template regions of already reconstructed samples around the current block. In one embodiment, the method operating according to the present invention can create an intra-prediction for a given block, and the current intra-prediction mode used can be identified using a template matching multiple reference line method, which includes computing a planar intra-prediction process using reference samples extracted from reference lines that do not necessarily have to be adjacent to the block. Specifically, assuming an L-shaped template region around the current block consisting of T reference lines, reference samples can be extracted at multiple reference line index m=T according to the process described above.

[0094] The planar prediction process described herein can also be used to determine the optimal intra-prediction mode to use for a given block using template-based intra-mode derivation (TIMD). In one embodiment, the method operating according to the present invention can create an intra-prediction for a given block, and the current intra-prediction mode to be used can be identified using TIMD, the TIMD process comprising computing a planar intra-prediction process using reference samples extracted from reference lines that do not necessarily have to be adjacent to the block, and the MRL index m may depend on the size of the TIMD template.

[0095] A method according to one embodiment is shown in Figure 6. The method generally includes identifying one or more reference lines 610, extracting reference samples for a block from reference lines not adjacent to the block 620, and performing an intra-prediction for the block using a planar prediction process and the extracted reference samples 630, the planar prediction process including determining the prediction samples according to the distance between the reference lines and the block. Each of these steps can be performed by the respective modules of a computer system.

[0096] An apparatus according to one embodiment includes means for identifying one or more reference lines, means for extracting a reference sample for a block from a reference line not adjacent to the block, and means for a planar prediction process and performing an intra-prediction for the block using the extracted reference sample, wherein the planar prediction process includes determining a prediction sample according to the distance between the reference line and the block. The means includes at least one processor and a memory containing computer program code, the processor may further include processor circuitry. The memory and computer program code are configured, by at least one processor, to cause the apparatus to perform the method of Figure 6 according to various embodiments.

[0097] Figure 7 shows one embodiment of an electronic device 1200, which is an embodiment of a video coding system. In one embodiment, the device may be a mobile terminal or user equipment of a wireless communication system, or a camera device. The device 1200 may be included in a local or remote server or a computer's graphics processing unit. The device may be included as part of a head-mounted display device.

[0098] The device 1200 includes one or more processors 1210 and a memory 1220, the memory 1220 storing computer instructions in, for example, each module (module 1, module 2, module N). The memory can store data in the form of images, video and / or audio data, and / or instructions executed by the processor or the processor's circuitry. The device also includes a codec 1230 configured to implement various embodiments relating to this solution. According to one embodiment, the device may include an encoder or decoder. The device 1200 also includes a communication interface 1240 suitable for generating radio communication signals for communication with, for example, a cellular communication network, a radio communication system, or a radio local area network, and thus enabling data transfer over a data transfer network 1250.

[0099] The device 1200 may include a display in the form of a liquid crystal display. In other embodiments of the present invention, the display may be any suitable display technology suitable for displaying images or videos. The device 1200 may further include a keyboard. In other embodiments of the present invention, any suitable data or user interface mechanism may be employed. For example, the user interface can be implemented as a virtual keyboard or data input system as part of a touch-sensitive display. The device 1200 may include a microphone or any suitable audio input, which may be a digital or analog signal input. In embodiments of the present invention, the device 1200 may further include an audio output device, which may be an earphone, a speaker, or one of analog audio or digital audio output connections. The device 1200 may also include a battery (or, in other embodiments of the present invention, the device may be powered by any suitable mobile energy device, such as a solar cell, a fuel cell, or an auto-winding power generator). The device may further include a camera capable of recording or capturing images and / or videos. The camera may be a multi-lens camera system having at least two camera sensors. The camera can record or detect individual frames, which are then passed to the codec 1230 or processor 1210. The device can receive video and / or image data from another device for processing before transmission and / or storage.

[0100] The apparatus 1200 may further include other functional units disclosed in any of Figures 1 to 2, for example, to carry out any of the presented embodiments.

[0101] Various embodiments can be implemented using computer program code that resides in memory and causes a suitable device to perform the method. For example, a device may include circuits and electronic components for processing, receiving, and transmitting data, computer program code in memory, and a processor that, when the computer program code is executed, causes the device to perform the functions of the embodiment. Furthermore, a network device such as a server may include circuits and electronic components for processing, receiving, and transmitting data, computer program code in memory, and a processor that, when the computer program code is executed, causes the network device to perform the functions of various embodiments.

[0102] If necessary, the different functions described herein may be performed in different orders and / or in parallel with other functions. Also, if necessary, one or more of the functions and embodiments described above may be optional or in combination.

[0103] While various aspects of the embodiment are described in the independent claims, other aspects include, but are not limited to, the combinations expressed in the claims, other combinations of functions from the described embodiments and / or dependent claims and functions from the independent claims.

[0104] Furthermore, while the above describes illustrative embodiments, it should be noted that these descriptions should not be taken as limiting. On the contrary, there are several variations and modifications that can be made without departing from the scope of this disclosure as defined in the attached claims.

Claims

1. Means for identifying one or more reference lines, Extracting a reference sample for a block based on a reference line that is not adjacent to the block, An apparatus comprising means for performing an intra-prediction for the block using a planar prediction process and the extracted reference sample, The aforementioned planar prediction process, An apparatus that includes determining a predicted sample according to the distance between a reference line and the block.

2. The apparatus according to claim 1, wherein the reference line is identified by a plurality of reference line indices.

3. The apparatus according to claim 2, wherein the planar prediction process includes horizontal and / or vertical interpolation.

4. The apparatus according to claim 3, further comprising means for defining the sequence of samples used in the horizontal and / or vertical interpolation based on the multiple reference line indices.

5. The apparatus according to claim 3, wherein the horizontal and / or vertical interpolation includes a weighted average based on the distance of each sample from the reference line.

6. The apparatus according to claim 5, further comprising means for calculating the weights of the weighted average, wherein one of the weights is equal to the distance of each sample from the reference line.

7. The apparatus according to claim 5, further comprising means for performing one of division, multiplication, addition, or binary shift by the sum of the block height and the multiple reference line indices for the weighted average for the vertical interpolation.

8. The apparatus according to claim 5, further comprising means for performing one of division, multiplication, addition, or binary shift by the sum of the width of the block and the multiple reference line indices for the weighted average for the horizontal interpolation.

9. The apparatus according to claim 1, further comprising means for determining the planar prediction process using a template matching multiple reference line method.

10. The apparatus according to claim 1, wherein the reference sample is extracted by defining a horizontal and / or vertical arrangement of the reference sample, and the apparatus further includes means for forming an arrangement of reconstructed samples based on the location of the block and the reference line.

11. The apparatus according to claim 10, further comprising means for forming a planar prediction using the horizontal and / or vertical arrangement.

12. Identifying one or more reference lines, Extracting a reference sample for a block from a reference line not adjacent to the block, The planar prediction process includes performing an intra-prediction for the block using the extracted reference sample, wherein the planar prediction process is A method comprising determining a predicted sample according to the distance between a reference line and the block.

13. The method according to claim 12, wherein the reference line is identified by a multiple reference line index.

14. The method according to claim 12, wherein the planar prediction process includes horizontal and / or vertical interpolation.

15. The method according to claim 14, further comprising defining the sequence of samples used in the horizontal and / or vertical interpolation based on the multiple reference line indices.

Images