Motion information parameter propagation based on intra prediction direction

By copying the motion information of adjacent blocks to the buffer in video encoding technology and selecting intra mode for encoding or decoding, the problem of intra coding efficiency in intra prediction mode is solved, and more efficient motion information propagation and frames are achieved Intra-prediction improves video encoding performance.

CN118975232APending Publication Date: 2024-11-15INTERDIGITAL CE PATENT HOLDINGS SAS
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202380032735.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-04-07
Filing Date
2023-03-22
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

The existing video encoding technology has the problem of inefficient encoding efficiency in intra prediction mode, especially the lack of effective mechanisms in the propagation of motion information, which affects the encoding efficiency.

Method used

By copying the motion information of adjacent blocks into the buffer, selecting motion information from the buffer based on the intra mode, encoding or decoding the video block, and using the intra mode propagation mechanism to fill the motion information, improving encoding efficiency.

Benefits of technology

It improves the encoding efficiency of video encoding, reduces the storage amount, and optimizes the propagation of motion information during intra prediction, and improves the compression performance of video data.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118975232A_ABST
    Figure CN118975232A_ABST
Patent Text Reader

Abstract

A method and apparatus for filling a motion information buffer with a coding unit or sub-blocks of a coding unit encoded in intra mode includes selecting neighboring inter prediction parameters into the motion information buffer using an intra prediction direction. In one embodiment, re-scaling of inter prediction parameters is performed. In a variant, motion information, such as motion vectors, may be averaged, or an affine model may be applied prior to storing the parameters in a motion information buffer.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of EPO application serial number 22305471.9, filed on April 7, 2022, which is incorporated herein by reference in its entirety. Technical Field

[0002] At least one of the present embodiments generally relates to a method or apparatus for video encoding or decoding, compression or decompression. Background Art

[0003] In order to achieve high compression efficiency, image and video coding schemes usually use prediction (including motion vector prediction) and transformation to exploit spatial and temporal redundancy in video content. In general, intra-frame or inter-frame prediction is used to exploit frame correlation within or between frames, and then the difference between the original image and the predicted image (usually expressed as a prediction error or prediction residual) is transformed, quantized, and entropy encoded. In order to reconstruct the video, the compressed data is decoded by the inverse process corresponding to entropy coding, quantization, transformation, and prediction. Summary of the invention

[0004] At least one of the present embodiments generally relates to a method or apparatus for video encoding or decoding, and more specifically to a method or apparatus for improving the encoding efficiency of an intra-frame prediction mode.

[0005] According to a first aspect, a method is provided. The method comprises the steps of: copying available motion information from neighboring blocks into a buffer; selecting motion information from the buffer based on an intra mode of a video block; and encoding at least a portion of the video block using the selected motion information.

[0006] According to a second aspect, a method is provided. The method comprises the steps of: copying available motion information from neighboring blocks into a buffer; selecting motion information from the buffer based on an intra mode of a video block; and decoding at least a portion of the video block using the selected motion information.

[0007] According to another aspect, a device is provided. The device includes a processor. The processor can be configured to encode a video block or decode a bit stream by executing any one of the above methods.

[0008] According to another general aspect of at least one embodiment, a device is provided, comprising an apparatus according to any one of the decoding embodiments; and at least one of the following: (i) an antenna configured to receive a signal, the signal comprising a video block; (ii) a band limiter configured to limit the received signal to a frequency band comprising the video block; or (iii) a display configured to display an output representing the video block.

[0009] According to another general aspect of at least one embodiment, there is provided a non-transitory computer-readable medium containing data content generated according to any of the described encoding embodiments or variations.

[0010] According to another general aspect of at least one embodiment, there is provided a signal comprising video data generated according to any of the described encoding embodiments or variants.

[0011] According to another general aspect of at least one embodiment, a bitstream is formatted to include data content generated according to any of the described encoding embodiments or variations.

[0012] According to another general aspect of at least one embodiment, there is provided a computer program product comprising instructions which, when executed by a computer, cause the computer to perform any of the described decoding embodiments or variants.

[0013] These and other aspects, features and advantages of the general aspects will become apparent from the following detailed description of exemplary embodiments, which is to be read in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 A general video encoding or compression system is illustrated.

[0015] Figure 2 A general video decoding or decompression system is illustrated.

[0016] Figure 3 An example of a GPM prediction is shown.

[0017] Figure 4 An example of two GPM predictions and coarser stored parameters (dashed sub-blocks) in the motion information buffer is illustrated.

[0018] Figure 5 The diagram illustrates the storage of the current CU encoding intra mode information in the IPM buffer with sub-block accuracy.

[0019] Figure 6 Reference samples used for intra prediction are illustrated.

[0020] Figure 7The intra prediction direction in HEVC (left) and the intra prediction direction in VVC (right) are illustrated.

[0021] Figure 8 Wide-angle intra prediction is illustrated.

[0022] Fig. 9 An example of a planar prediction mode is illustrated.

[0023] Fig.10 An example of motion information (MI) propagation direction originating from the intra mode direction is illustrated.

[0024] Fig.11 Examples of (a) a 4-parameter affine model and (b) a 6-parameter affine model are illustrated.

[0025] Fig.12 A general overview of the described aspects is illustrated.

[0026] Fig.13 One embodiment of a method for performing the described aspects is illustrated.

[0027] Fig.14 Another embodiment of a method for performing the described aspects is illustrated.

[0028] Fig.15 One embodiment of an apparatus for implementing the described aspects is illustrated.

[0029] Fig.16 A processor-based system for implementing the described aspects is illustrated.

[0030] Fig.17 Another embodiment of a method for performing the described aspects is illustrated.

[0031] Fig.18 Another embodiment of a method for performing the described aspects is illustrated. DETAILED DESCRIPTION

[0032] Intra prediction is a fundamental coding tool in video compression. The encoder selects the best prediction mode and signals its index to the decoder to perform the same prediction. Intra prediction is performed using reference samples, which are samples around the current block that are used to decode already decoded data. Currently, there are several modes available in the VVC standard: -DC: Use reference samples to create uniform predictions - Planar mode: uses reference samples to create a smooth prediction of the block - Directional mode: Uses reference samples to "fill in" these samples along a specific direction to create a prediction - MIP (Matrix-based Intra Prediction): Uses linear combinations of reference samples to create a prediction All of these modes use spatial information to create predictions.

[0033] In inter-coding, samples from one or more reference frames are used to create a prediction.

[0034] Inter-frame mode storage In the ECM codec, the motion information (MV value and reference index) used to encode the CU in inter-frame mode is stored in a buffer (MI buffer) associated with the current frame. It is stored with the reference picture, which is similar to the decoder picture buffer (DPB). Then, the motion information used to decode some samples in the reference picture is made available to the current picture. It can be used to build a MV neighbor candidate list using the MI buffer of the current frame, or to obtain a co-located MV using the MI buffer of a reference picture used as a co-located reference (signaled in a slice or picture header).

[0035] Intra-mode propagation In the ECM codec, the intra-mode used to encode a CU in intra-mode is stored in a buffer associated with the current frame (IPM buffer). It is stored with the reference picture, which is similar to the decoder picture buffer (DPB). Then, the intra-mode used to decode some samples in the reference picture is made available for the current picture. Both the IPM buffer and the MI buffer can be combined (the buffer contains both intra- and inter-frame information).

[0036] To reduce the amount of storage, information can be stored at a coarser resolution than the current image (e.g. 4×4 resolution, Figure 4 ). For example, if the ratio between the current picture size and the motion information is si=4, the "MI" parameter of the block (or sub-block) located at (x, y) is stored in the motion information buffer as mi(x / 4; y / 4).

[0037] In the case where the current CU is coded in inter unidirectional mode, the intra mode associated with the reference block (available via the IPM buffer associated with the reference picture) is copied into the IPM buffer of the current CU. In the case where the current CU is coded in inter bi-prediction mode, some rules (500) allow the intra mode to be selected from the IPM buffer associated with the reference frame of list 0 (ref_0 from ipm0) or the reference frame of list 1 (ref_1 from ipm1): –if ref_0 is encoded in intra mode and ref_1 is encoded in inter mode: ipm0(510) –else if ref_0 is coded in inter mode and ref_1 is coded in intra mode: ipm1(520) –else if(ipm0>DC_IDX&&ipm1<=DC_IDX):ipm0(530) –else if(ipm0<=DC_IDX&&ipm1>DC_IDX):ipm1(540) –else if(pocDiff0 <pocDiff1):ipm0(550) –else if(pocDiff1 <pocDiff0):ipm1(560) –else if(QP-ref0>QP-ref1):ipm1(570) –else:ipm0 Where: pocDiffX = pocRefX – pocCur, DC_IDX is the intra-frame mode DC Therefore, intra mode information can be propagated temporarily and made available to all reconstructed CUs. For the current CU coded in intra mode, this information is further used to build a list of most probable modes (MPMs) with neighboring reconstructed CUs, even if they are coded in inter mode: - If the CU neighbors are coded in intra mode, use intra mode to build the MPM list, - If the CU neighbor is coded in inter mode, the MPM list is constructed using the intra modes stored in the IPM buffer of the CU neighbor.

[0038] Intra-mode propagation using GPM with inter-prediction and intra-prediction The geometric partition mode (GPM) allows predicting a CU using two non-rectangular partitions, such as Figure 3 . Each partition may be an inter partition (275) or an intra partition (260). Samples of the inter partition are predicted using a conventional inter prediction process using motion compensated reference samples selected from a (uni-predicted) reference picture. Samples of the intra partition are predicted using a conventional intra prediction mode (IPM) and prediction process, where the available IPM candidates are parallel angle mode (parallel mode) for GPM block boundaries, vertical angle mode (vertical mode) for GPM block boundaries, and planar mode, respectively. Figure 3 a to Figure 3 As shown in c.

[0039] The motion vectors used to reconstruct the inter-prediction part and the IPMs used to reconstruct the intra-partitions are stored in the MI and IPM buffers, respectively. In the case of GPM, since the partitions can be non-rectangular, for sub-blocks shared by two partitions, the stored information corresponds to the partition occupying the larger area, such as Figure 4 and Figure 5 ——described in 620(620).

[0040] For example, in Figure 4 In (a), the CU is predicted using an inter partition and an intra partition. The corresponding inter and intra parameters are stored in the MI and IPM buffers with a 4×4 resolution, as shown in Figure 4 (b).

[0041] This information can be used later to encode subsequent CUs in the current picture and CUs in subsequent pictures (in coding order), for example to predict the IPM or motion to use. For a given sub-block (x, y), if more samples are predicted using intra mode than inter mode (620), the intra parameters are stored in the MI buffer (630). If more samples are predicted using inter mode than intra mode (620), the intra parameters are retrieved from the reference MI buffer, which is the MI buffer associated with the reference picture used for inter prediction. The location of the intra parameters is mi((x+mv X ) / si,(y+mv Y ) / si)(640), where (mv X ,mv Y ) is the motion vector for inter prediction, and si is the ratio between the current picture size and the motion information buffer size.

[0042] Intra prediction The intra prediction process in HEVC and VVC consists of three steps: Reference sample generation Intra-sample prediction, and Post-processing of predicted samples.

[0043] The reference sample generation process is as follows: Figure 6 The reference sample ref[] is also called L-shaped. For a prediction unit (PU) of size N×N, the top row (2N+refIdx) of decoded samples is formed from the top and upper right pixels previously reconstructed to the current PU. Similarly, the left column (2N+refIdx) of samples is formed from the reconstructed left and lower left pixels. In VVC, the distance (d=refIdx) between the reference row and column of a sample and the current block can be more than one sample, such as Figure 6 Described in.

[0044] The corner pixel at the top left position is also used to fill the gap between the top row and left column references. If some samples at the top or left are not available because the corresponding CU is not in the same slice, or the current CU is at a frame boundary, etc., a method called reference sample replacement is performed, in which the missing samples are copied from the available samples in a clockwise direction. Then, the reference samples are filtered using the specified filter depending on the current CU size and prediction mode.

[0045] Intra-sample prediction consists of predicting the pixels of the target CU based on reference samples. There are different prediction modes: planar and DC prediction modes are used to predict smooth and gradually changing areas, while angular (angles defined from 45 degrees to -135 degrees in a clockwise direction) prediction modes are used to capture different directional structures. For square blocks, HEVC supports 33 directional prediction modes, which are indexed from 2 to 34. These prediction modes correspond to different prediction directions, such as Figure 7 In VVC, there are 65 angular prediction modes, corresponding to the 33 angular directions defined in HEVC and an additional 32 directions, each corresponding to the middle direction between adjacent pairs ( Figure 7 on the right side of the ).

[0046] In VVC, for non-square blocks, conventional directional intra prediction is not allowed (see Figure 8 ) is replaced by an additional wide-angle intra prediction mode.

[0047] For a given angular prediction mode, the predictor samples on the reference array are copied along the corresponding direction inside the target PU. Some predictor samples may have an integral location, in which case they match the corresponding reference samples; other predictor locations will have a fractional part, indicating that their location will fall between two reference samples. In the latter case, the predictor samples are interpolated using the closest reference samples. In HEVC, linear interpolation of the two closest reference samples is performed to calculate the predictor sample values; in VVC, to interpolate the predictor samples, a 4-tap filter fT[] is used that depends on the selected intra mode direction.

[0048] In addition to directional mode, DC mode fills the prediction with the average of samples in the L shape (except for rectangular CUs which use the average of the reference samples of the longer sides), and planar mode spatially interpolates the reference samples, as Fig. 9 Described in.

[0049] The IPM buffer implements a propagation process, allowing information to be filled in for all CUs that can be coded in intra or inter mode. However, this mechanism does not exist for motion information. In the case of coding modes that use motion information, this lack of data may negatively affect coding efficiency.

[0050] A motion information propagation process is proposed to fill the MI buffer in case of a CU (or sub-CU) coded in intra mode.

[0051] Some embodiments are proposed to propagate motion information when the current CU is encoded in intra mode. A general outline of the method is given in Fig.12 This method is applicable to the sub-block parts coded in intra mode in the current CU. The sub-block parts coded in inter mode use the conventional method (copying the inter parameters in the current co-located MI buffer) ( Fig.12 830 in ). Fig.12 Some of the logic blocks in the overview may not be present, depending on the embodiment. Their functions may vary depending on the embodiment / variant.

[0052] Example 1 - Propagating motion information of adjacent blocks The motion information (MI) of the neighboring blocks is propagated into the MI buffer at the location of the current CU (current MI). A list of predetermined neighboring block locations can be constructed. The first item of the list where the MI is available is selected (850) and copied into the current MI (840). For example, the list is all reconstructed CUs from left to right on top of the current CU, followed by all CUs from top to bottom on the left. In a variant, the neighboring CU with the highest number of samples adjacent to the current CU is placed at the top of the list. If two CUs have the same MI, they are considered to be the same CU (the number of adjacent samples is added).

[0053] The expression "MI unavailable" can be defined as follows (these can be combined): If the current CU is at a picture boundary (or a slice boundary, or a tile boundary) or if the current CU is at the left boundary of the picture, the left CU is not available.

[0054] • If the neighboring CU does not have MI data stored in its buffer, it is not available.

[0055] In a variation, if the first two MIs available in the list have the same reference, the motion vector values ​​may be averaged (870).

[0056] Example 2 - Propagating motion information along with intra-frame direction If the current intra mode (predMode) is directional, the direction is used to select the location of the MI to propagate (850). For example, if intra prediction uses samples from the top, the MI candidates to propagate (miRef) are picked only from the top CU.

[0057] In another example: if(predMode <HOR_IDX(18)) ○miRef=(left-bottom available)? mi(left-bottom):mi(top-left) else if(predMode <DIA_IDX(34)) ○miRef=(left-top available)? mi(left-top):mi(top-left) else if(predMode <VER_IDX(50)) ○miRef=(up-left available)? mi(up-left):mi(left-top) ·else / / if(predMode <VDIA_IDX) ○miRef=(up-right available)? mi(up-right):mi(left-top) In a variant, if several MIs have the same reference, their MV values ​​may be averaged (870). For example: if(predMode <HOR_IDX(18)) ○if (left-bottom & left-top are available and have the same reference ■miRef=average{mi(left-bottom);mi(left-top)} ○else ■miRef=(left-bottom available)? mi(left-bottom):mi(top-left) else if(predMode <DIA_IDX(34)) ○if (left-top & top-left are available and have the same reference ■miRef=average{mi(left-top);mi(top-left)} ○else ■miRef=(left-top available)? mi(left-top):mi(top-left) else if(predMode <VER_IDX(50)) ○if(top-left & top-right are available and have the same reference ■miRef=average{mi(up-left);mi(up-right)} ○else ■miRef=(up-left available)? mi(up-left):mi(left-top) ·else / / if(predMode <VDIA_IDX) ○if(top-left & top-right are available and have the same reference ■miRef=average{mi(up-left);mi(up-right)} ○else ■miRef=(up-right available)? mi(up-right):mi(left-top) In a variation, if reference MIs are available, but do not have the same reference, they may be rescaled (860) using the POC of the reference so that they have the same reference. The POC of the reference closest to the current frame (pocRef) may be selected as the reference POC (pocCur).

[0058] miRescaled=miRef×(pocCur-pocRef) / (poc-pocRef) Where: miRescaled, miRef are the values ​​of the motion vector component X or Y, and poc is the POC of miRef to be rescaled.

[0059] Example 3 - Using an affine model to propagate motion information If bottom-left, top-right, and left-top are available, an affine 6-parameter model can be used to fill the MI buffer of the current CU with spatially varying values ​​(870). If they do not have the same reference, MI rescaling can be applied (see Embodiment 2).

[0060] In a variation, an affine 4-parameter model may be used in the same manner as in Embodiment 2 by selecting 2 reference MIs using the intra-frame direction.

[0061] For the 4-parameter affine motion model, the motion vector at a sample position (x,y) in a block is derived as: For the 6-parameter affine motion model, the motion vector at a sample position (x,y) in a block is derived as: Example 4 - Using intra-frame reconstruction samples to estimate motion parameters to be propagated According to this embodiment, the reconstructed samples of the current intra CU are used to estimate the best motion vector candidate to be propagated. The list of motion vector candidates for the current CU is constructed using a conventional list merge candidate construction in the same way as for a CU encoded in inter-frame merge mode. The cost of each candidate is evaluated using SATD using the reconstructed samples of the current coding unit. In a variant, in order to simplify the amount of calculation, a reduced number of reconstructed samples (a sub-portion of the reconstructed CU samples) can be used to calculate the cost. The candidate with the smallest cost is selected for propagation for the current CU.

[0062] In a variant, affine merge candidates are also evaluated.

[0063] Example 5 - Propagated motion information not used for spatial candidates Embodiment 4 introduces additional latency, since the propagation of motion parameters should be done after the samples of the current CU have been reconstructed. The propagated motion parameters cannot then be used to construct motion candidates for the next CU, unless waiting for their availability, which may introduce additional latency. In this embodiment, the propagated motion information of the intra CU is not used for spatial candidates, but only for temporal candidates. Advantageously (in a variant), the propagated motion information of the intra CU is not used for spatial candidates only in the case of embodiment 4.

[0064] Example 6 - Sub-block partition propagation The MI propagation in the current intra block can be performed per sub-block. For example, if the neighboring CU consists of sub-partitions, the MI parameters of each partition can be propagated in the current block, such as Fig.17 In another variant, in the case of GPM, the neighboring MI can be selected from the current CU, from a sub-partition coded in inter mode, such as Fig.18 as depicted in the example of .

[0065] Example 7 - Back Propagation In general, the prediction signals of the top and left samples of a block encoded in intra mode are well correlated with the neighboring CU reconstructed samples. This is why these neighboring CU reconstructed samples are used to construct the intra prediction of the current CU. However, other samples of the current CU (e.g., right and bottom samples) may not be very correlated with the neighboring (left and / or top) CU reconstructed samples. In this embodiment, if the current CU is encoded in inter mode and a left (or top) neighboring CU has been encoded in intra mode, the MI parameters of the current CU can be propagated to the MI buffer associated with the neighboring (left or top) CU encoded in intra mode.

[0066] One embodiment of method 1300 under the general aspects described herein is Fig.13 . The method starts at start block 1301, and control proceeds to block 1310 for copying available motion information from neighboring blocks into a buffer. Control proceeds from block 1310 to block 1320 for selecting motion information from the buffer based on an intra mode of the video block. Control proceeds from block 1320 to block 1330 for encoding at least a portion of the video block using the selected motion information.

[0067] One embodiment of method 1400 under the general aspects described herein is Fig.14 . The method starts at start block 1401, and control proceeds to block 1410 for copying available motion information from neighboring blocks into a buffer. Control proceeds from block 1410 to block 1420 for selecting motion information from the buffer based on an intra mode of the video block. Control proceeds from block 1420 to block 1430 for decoding at least a portion of the video block using the selected motion information.

[0068] Fig.15 An embodiment of an apparatus 1500 for encoding, decoding, compressing or decompressing video data using any of the above methods or variations is shown. The apparatus includes a processor 1510 and may be interconnected to a memory 1520 via at least one port. Both the processor 1510 and the memory 1520 may also have one or more additional interconnections to external connections.

[0069] Processor 1510 is also configured to insert or receive information in a bitstream and compress, encode, or decode using any of the described aspects.

[0070] The embodiments described herein include various aspects, including tools, features, embodiments, models, methods, etc. Many of these aspects are described as having specificity, and are usually described in a manner that may sound restrictive, at least to illustrate individual characteristics. However, this is for the purpose of clarity of the aspects, and does not limit the application or scope of these aspects. In fact, all different aspects can be combined and exchanged to provide additional aspects. In addition, these aspects can also be combined and exchanged with the aspects described in the previous files.

[0071] The aspects described and contemplated in this application can be implemented in many different forms. Figure 1 , Figure 2 and Fig.16 Some embodiments are provided, but other embodiments are contemplated, and Figure 1 , Figure 2 and Fig.16 The discussion does not limit the breadth of implementations. At least one of these aspects generally relates to video encoding and decoding, and at least another aspect generally relates to transmitting a generated or encoded bitstream. These and other aspects can be implemented as methods, apparatus, a computer-readable storage medium having stored thereon instructions for encoding or decoding video data according to any of the described methods, and / or a computer-readable storage medium having stored thereon a bitstream generated according to any of the described methods.

[0072] In this application, the terms "reconstruction" and "decoding" can be used interchangeably, the terms "pixel" and "sample" can be used interchangeably, and the terms "image", "picture" and "frame" can be used interchangeably. Usually (but not necessarily), the term "reconstruction" is used on the encoder side, while "decoding" is used on the decoder side.

[0073] Various methods are described herein, and each method includes one or more steps or actions for implementing the described method. Unless a specific order of steps or actions is required for the correct operation of the method, the order and / or use of specific steps and / or actions may be modified or combined.

[0074] The various methods and other aspects described in this application can be used to modify a module, for example, Figure 1 and Figure 2 Intra-frame prediction, entropy encoding and / or decoding modules (160, 360, 145, 330) of the video encoder 100 and decoder 200 shown. In addition, the present aspects are not limited to VVC or HEVC, and can be applied to, for example, other standards and recommendations (whether pre-existing or developed in the future) and extensions of any such standards and recommendations (including VVC and HEVC). Unless otherwise indicated or technically excluded, the aspects described in this application can be used alone or in combination.

[0075] Various numerical values ​​are used in this application. Specific values ​​are used for illustrative purposes, and the described aspects are not limited to these specific values.

[0076] Figure 1 An encoder 100 is illustrated. Variations of the encoder 100 are conceivable, but for the sake of clarity, the encoder 100 is described below without describing all contemplated variations.

[0077] Before being encoded, the video sequence may undergo pre-encoding processing (101), for example, applying a color transform to the input color picture (e.g., conversion from RGB 4:4:4 to YCbCr 4:2:0), or performing a remapping of the input picture components in order to obtain a signal distribution that is more resilient to compression (e.g., using histogram equalization of one of the color components). Metadata may be associated with the pre-processing and appended to the bitstream.

[0078] In encoder 100, a picture is encoded by encoder elements as described below. The picture to be encoded is partitioned (102) and processed in units such as CUs. Each unit is encoded using, for example, intra mode or inter mode. When a unit is encoded in intra mode, it performs intra prediction (160). In inter mode, motion estimation (175) and compensation (170) are performed. The encoder decides (105) which of intra mode or inter mode to use to encode the unit, and indicates the intra / inter decision by, for example, a prediction mode flag. A prediction residual is calculated, for example, by subtracting (110) the predicted block from the original image block.

[0079] The prediction residual is then transformed (125) and quantized (130). The quantized transform coefficients, along with motion vectors and other syntax elements, are entropy encoded (145) to output a bitstream. The encoder may skip the transform and directly quantize the untransformed residual signal. The encoder may bypass both the transform and quantization, i.e., directly encode the residual without applying the transform or quantization process.

[0080] The encoder decodes the encoded block to provide a reference for further prediction. The quantized transform coefficients are dequantized (140) and inverse transformed (150) to decode the prediction residual. The decoded prediction residual and the prediction block are combined (155) to reconstruct the image block. An in-loop filter (165) is applied to the reconstructed picture to perform, for example, deblocking / SAO (sample adaptive offset) / ALF (adaptive loop filtering) filtering to reduce coding artifacts. The filtered image is stored in a reference picture buffer (180).

[0081] Figure 2 2 shows a block diagram of a video decoder 200. In the decoder 200, the bitstream is decoded by the decoder elements as described below. The video decoder 200 generally performs the same Figure 1 The encoding pass described in is the reciprocal decoding pass of .Encoder 100 also typically performs video decoding as part of encoding the video data.

[0082] In particular, the input to the decoder includes a video bitstream, which may be generated by the video encoder 100. The bitstream is first entropy encoded (230) to obtain transform coefficients, motion vectors, and other encoding information. The picture partition information indicates how the picture is partitioned. Therefore, the decoder can divide the picture according to the decoded picture partition information (235). The transform coefficients are dequantized (240) and inverse transformed (250) to decode the prediction residual. The decoded prediction residual and the prediction block are combined (255) to reconstruct the image block. The prediction block can be obtained (270) from intra-frame prediction (260) or motion compensated prediction (i.e., inter-frame prediction) (275). An in-loop filter (265) is applied to the reconstructed image. The filtered image is stored in a reference picture buffer (280).

[0083] The decoded picture may be further subjected to post-decoding processing (285), such as an inverse color transform (e.g., a transform from YCbCr 4:2:0 to RGB 4:4:4) or an inverse remapping that performs the inverse of the remapping process performed in the pre-encoding processing (101). The post-decoding processing may use metadata derived in the pre-encoding processing and signaled in the bitstream.

[0084] Fig.16 A block diagram of an example of a system in which various aspects and embodiments are implemented is illustrated. System 1000 may be embodied as a device including the following various components, and is configured to perform one or more aspects described in this document. Examples of such devices include, but are not limited to, various electronic devices, such as personal computers, laptop computers, smart phones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected household appliances, and servers. The elements of system 1000 may be embodied in a single integrated circuit (IC), multiple ICs, and / or discrete components, either individually or in combination. For example, in at least one embodiment, the processing and encoder / decoder elements of system 1000 are distributed across multiple ICs and / or discrete components. In various embodiments, system 1000 is coupled to one or more other systems or other electronic devices via, for example, a communication bus or by a dedicated input and / or output port communication. In various embodiments, system 1000 is configured to implement one or more aspects described in this document.

[0085] The system 1000 includes at least one processor 1010 configured to execute instructions loaded therein to implement, for example, various aspects described in this document. The processor 1010 may include embedded memory, input-output interfaces, and various other circuits known in the art. The system 1000 includes at least one memory 1020 (e.g., a volatile memory device and / or a non-volatile memory device). The system 1000 includes a storage device 1040, which may include a non-volatile memory and / or a volatile memory, including but not limited to an electrically erasable programmable read-only memory (EEPROM), a read-only memory (ROM), a programmable read-only memory (PROM), a random access memory (RAM), a dynamic random access memory (DRAM), a static random access memory (SRAM), a flash memory, a magnetic disk drive, and / or an optical disk drive. The storage device 1040 may include an internal storage device, an attached storage device (including a removable and non-removable storage device), and / or a network-accessible storage device, as a non-limiting example.

[0086] The system 1000 includes an encoder / decoder module 1030, which is configured to process data to provide encoded video or decoded video, for example, and the encoder / decoder module 1030 may include its own processor and memory. The encoder / decoder module 1030 represents a (multiple) module that can be included in a device to perform encoding and / or decoding functions. As is known, a device may include one or both of the encoding and decoding modules. Additionally, the encoder / decoder module 1030 may be implemented as a separate element of the system 1000, or may be incorporated into the processor 1010 as a combination of hardware and software known to those skilled in the art.

[0087] Program code to be loaded onto the processor 1010 or the encoder / decoder 1030 to perform various aspects described in this document may be stored in the storage device 1040 and subsequently loaded onto the memory 1020 for execution by the processor 1010. According to various embodiments, one or more of the processor 1010, the memory 1020, the storage device 1040, and the encoder / decoder module 1030 may store one or more of various items during the execution of the processes described in this document. Such stored items may include, but are not limited to, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and operation logic.

[0088] In some embodiments, memory internal to the processor 1010 and / or encoder / decoder module 1030 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, memory external to the processing device (e.g., the processing device may be the processor 1010 or the encoder / decoder module 1030) is used for one or more of these functions. The external memory may be memory 1020 and / or storage device 1040, such as dynamic volatile memory and / or non-volatile memory. In several embodiments, external non-volatile flash memory is used to store, for example, an operating system for a television. In at least one embodiment, a fast external dynamic volatile memory (such as RAM) is used as working memory for video encoding and decoding operations, such as for MPEG-2 (MPEG refers to Moving Picture Experts Group, MPEG-2 is also known as ISO / IEC 13818, and 13818-1 is also known as H.222, and 13818-2 is also known as H.262), HEVC (HEVC refers to High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2), or VVC (Versatile Video Coding, a new standard developed by JVET (Joint Video Experts Team)).

[0089] Inputs to the elements of system 1000 may be provided through various input devices indicated in box 1130. Such input devices include, but are not limited to: (i) a radio frequency (RF) section that receives an RF signal transmitted over the air, such as by a broadcaster; (ii) a component (COMP) input terminal (or a set of COMP input terminals); (iii) a universal serial bus (USB) input terminal; and / or (iv) a high-definition multimedia interface (HDMI) input terminal. Fig.16 Other examples not shown include composite video.

[0090] In various embodiments, the input device of block 1130 has associated corresponding input processing elements known in the art. For example, the RF section may be associated with an element adapted to: (i) select a desired frequency (also referred to as selecting a signal, or band limiting a signal to a certain frequency band); (ii) down-convert the selected signal; (iii) again band-limit the frequency band to a narrower frequency band to select (for example) a signal frequency band, which may be referred to as a channel in some embodiments; (iv) demodulate the down-converted and band-limited signal; (v) perform error correction; and (vi) demultiplex to select a desired data packet stream. The RF section of various embodiments includes one or more elements for performing these functions, such as a frequency selector, a signal selector, a band limiter, a channel selector, a filter, a down-converter, a demodulator, an error corrector, and a demultiplexer. The RF section may include a tuner that performs various of these functions, including, for example, down-converting a received signal to a lower frequency (e.g., an intermediate frequency or a near-baseband frequency) or baseband. In a set-top box embodiment, the RF part and its associated input processing element receive the RF signal transmitted by wired (for example, cable) medium, and filter to the desired frequency band by filtering, down-conversion and again to perform frequency selection.Various embodiments rearrange the order of above-mentioned (and other) elements, remove some elements in these elements and / or add other elements that perform similar or different functions.Adding element can include inserting element between existing element, such as, for example, inserting amplifier and analog-to-digital converter.In various embodiments, the RF part comprises antenna.

[0091] Additionally, the USB and / or HDMI terminals may include corresponding interface processors for connecting the system 1000 to other electronic devices across the USB and / or HDMI connections. It should be understood that various aspects of input processing (e.g., Reed-Solomon error correction) may be implemented as desired, for example, within a separate input processing IC or within the processor 1010. Similarly, aspects of USB or HDMI interface processing may be implemented as desired within a separate interface IC or within the processor 1010. The demodulated, error corrected, and demultiplexed streams are provided to various processing elements, including, for example, the processor 1010 and the encoder / decoder 1030, which operate in combination with memory and storage elements to process the data streams as desired for presentation on an output device.

[0092] The various elements of system 1000 may be provided within an integrated housing within which the various elements may be interconnected and data transmitted therebetween using a suitable connection arrangement, such as an internal bus known in the art, including an Inter-IC (I2C) bus, wiring, and printed circuit boards.

[0093] The system 1000 includes a communication interface 1050 that enables communication with other devices via a communication channel 1060. The communication interface 1050 may include, but is not limited to, a transceiver configured to transmit and receive data over the communication channel 1060. The communication interface 1050 may include, but is not limited to, a modem or a network card, and the communication channel 1060 may be implemented, for example, within a wired and / or wireless medium.

[0094] In various embodiments, a wireless network, such as a Wi-Fi network, for example IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers), is used to stream data to the system 1000 or otherwise provide it to the system 1000. The Wi-Fi signal of these embodiments is received by a communication channel 1060 and a communication interface 1050 adapted to Wi-Fi communication. The communication channel 1060 of these embodiments is usually connected to an access point or router, which provides access to an external network including the Internet to allow streaming applications and other over-the-top communications. Other embodiments use a set-top box to provide streaming data to the system 1000, which delivers data through the HDMI connection of the input box 1130. Still other embodiments use the RF connection of the input box 1130 to provide streaming data to the system 1000. As indicated above, various embodiments provide data in a non-streaming manner. Additionally, various embodiments use wireless networks other than Wi-Fi, such as cellular networks or Bluetooth networks.

[0095] The system 1000 can provide output signals to various output devices, including a display 1100, a speaker 1110, and other peripherals 1120. The display 1100 of various embodiments may include, for example, one or more of a touch screen display, an organic light emitting diode (OLED) display, a curved display, and / or a foldable display. The display 1100 can be used for a television, a tablet computer, a cellular phone (mobile phone), or another device. The display 1100 can also be integrated with other components (e.g., as in a smart phone), or be separate (e.g., an external display of a laptop). In various examples of embodiments, other peripherals 1120 include one or more of a stand-alone digital video disc (or digital versatile disc) (DVR, for both terms), a disc player, a stereo system, and / or a lighting system. Various embodiments use one or more peripherals 1120 that provide functions based on the output of the system 1000. For example, a disc player performs the function of playing the output of the system 1000.

[0096] In various embodiments, control signals are transmitted between the system 1000 and the display 1100, speaker 1110, or other peripheral device 1120 using signaling such as AV.Link, Consumer Electronics Control (CEC), or other communication protocols that enable device-to-device control with or without user intervention. Output devices may be communicatively coupled to the system 1000 via dedicated connections through respective interfaces 1070, 1080, and 1090. Alternatively, the output devices may be connected to the system 1000 using a communication channel 1060 via a communication interface 1050. The display 1100 and speaker 1110 may be integrated into a single unit with other components of the system 1000 in an electronic device such as, for example, a television. In various embodiments, the display interface 1070 includes a display driver such as, for example, a timing controller (T Con) chip.

[0097] Display 1100 and speaker 1110 may alternatively be separate from one or more of the other components, for example if the RF portion of input 1130 is part of a separate set-top box. In various embodiments where display 1100 and speaker 1110 are external components, output signals may be provided via dedicated output connections including, for example, an HDMI port, a USB port, or a COMP output.

[0098] The embodiments may be performed by computer software implemented by the processor 1010, or by hardware, or by a combination of hardware and software. As a non-limiting example, the embodiments may be implemented by one or more integrated circuits. The memory 1020 may be of any type suitable for the technical environment and may be implemented using any appropriate data storage technology, such as optical memory devices, magnetic memory, semiconductor-based memory devices, fixed memory, and removable memory, as non-limiting examples. The processor 1010 may be of any type suitable for the technical environment and may include one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a processor based on a multi-core architecture, as non-limiting examples.

[0099] Various implementations involve decoding. "Decoding" as used herein may encompass, for example, all or part of a process performed on a received coded sequence to produce a final output suitable for display. In various embodiments, such a process includes one or more of the processes typically performed by a decoder, such as entropy coding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such a process also or alternatively includes a process performed by a decoder of the various implementations described herein.

[0100] As a further example, in one embodiment, "decoding" refers only to entropy decoding; in another embodiment, "decoding" refers only to differential decoding; and in another embodiment, "decoding" refers to a combination of entropy decoding and differential decoding. Whether the phrase "decoding process" is intended to refer specifically to a subset of operations or generally to a broader decoding process will be clear based on the context of the specific description and is believed to be well understood by those skilled in the art.

[0101] Various implementations involve encoding. In a manner similar to the above discussion of "decoding", "encoding" as used in this application may include, for example, all or part of a process performed on an input video sequence to produce an encoded bitstream. In various embodiments, such processes include one or more of the processes typically performed by an encoder, such as partitioning, differential encoding, transforms, quantization, and entropy encoding. In various embodiments, such processes also or alternatively include processes performed by an encoder of the various implementations described in this application.

[0102] As a further example, in one embodiment, "encoding" refers only to entropy encoding; in another embodiment, "decoding" refers only to differential encoding; and in another embodiment, "decoding" refers to a combination of entropy encoding and differential encoding. Whether the phrase "encoding process" is intended to specifically refer to a subset of operations or to generally refer to a broader encoding process will be clear based on the context of the specific description and is believed to be well understood by those skilled in the art.

[0103] Note that the syntax elements used in this article are descriptive terms. Therefore, they do not exclude the use of other syntax element names.

[0104] When a figure is presented as a flow chart, it should be understood that it also provides a block diagram of the corresponding apparatus. Similarly, when a figure is presented as a block diagram, it should be understood that it also provides a flow chart of the corresponding method / process.

[0105] Various embodiments may relate to parameter models or rate-distortion optimization. In particular, in the encoding process, a balance or trade-off between rate and distortion is generally considered, given a computational complexity constraint. It can be measured by a rate-distortion optimization (RDO) metric, or by a least mean square (LMS), mean absolute error (MAE), or other such measurements. Rate-distortion optimization is generally formulated as minimizing a rate-distortion function, which is a weighted sum of rate and distortion. There are different approaches to solving the rate-distortion optimization problem. For example, an approach may be based on extensive testing of all coding options (including all considered modes or coding parameter values), using a complete evaluation of their coding costs and the associated distortion of the reconstructed signal after encoding and decoding. A faster approach may also be used to save coding complexity, in particular by calculating approximate distortion based on a prediction or prediction residual signal rather than a reconstructed signal. A mixture of these two approaches may also be used, such as by using approximate distortion only for some of the possible coding options and using full distortion for other coding options. Other approaches evaluate only a subset of possible coding options. More generally, many methods employ any of a variety of techniques to perform the optimization, but the optimization is not necessarily a complete assessment of both the coding cost and the associated distortion.

[0106] The implementations and aspects described herein may be implemented in, for example, methods or processes, devices, software programs, data streams, or signals. Even if only discussed in the context of a single form of implementation (e.g., discussed only as a method), the implementation of the features discussed may also be implemented in other forms (e.g., devices or programs). Devices may be implemented in, for example, appropriate hardware, software, and firmware. Methods may be implemented in, for example, a processor, which generally refers to a processing device, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include communication devices, such as, for example, computers, cellular phones, portable / personal digital assistants ("PDAs"), and other devices that facilitate information communication between end users.

[0107] References to "one embodiment" or "an embodiment" or "one implementation" or "an implementation" and other variations thereof mean that a particular feature, structure, characteristic, etc. described in connection with the embodiment is included in at least one embodiment. Thus, the phrases "in one embodiment" or "in an embodiment" or "in an implementation" or "in an implementation" and any other variations appearing in various places throughout this application are not necessarily all referring to the same embodiment.

[0108] Additionally, the present application may refer to "determining" various pieces of information. Determining information may include one or more of the following: for example, estimating information, calculating information, predicting information, or retrieving information from a memory.

[0109] Further, the present application may involve "accessing" various pieces of information. Accessing information may include one or more of: for example, receiving information, retrieving information (e.g., from a memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0110] Additionally, the present application may refer to "receiving" various pieces of information. Like "accessing," receiving is intended to be a broad term. Receiving information may include one or more of: for example, accessing information or retrieving information (e.g., from a memory). Further, "receiving" generally refers in one way or another during an operation such as, for example, storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0111] It is to be appreciated that the use of any of " / ", "and / or", and "at least one of" below - for example, in the case of "A / B", "A and / or B", and "at least one of A and B" - is intended to cover selecting only the first listed option (A), or only the second listed option (B), or both options (A and B). As a further example, in the case of "A, B, and / or C" and "at least one of A, B, and C", such wording is intended to cover selecting only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or all three options (A and B and C). It will be apparent to one of ordinary skill in this and related arts that this can be extended for as many items as listed.

[0112] Likewise, as used herein, among other aspects, the word "signal" is to indicate something to the corresponding decoder. For example, in some embodiments, the decoder sends a specific one of multiple transforms, coding modes or flags with a signal. In this way, in one embodiment, the same transform, parameter or mode is used on the encoder side and the decoder side. Therefore, for example, the encoder can transmit specific parameters to the decoder (explicit signaling) so that the decoder can use the same specific parameters. On the contrary, if the decoder already has specific parameters and other parameters, signaling can be used without transmission (implicit signaling) to allow only the decoder to know and select specific parameters. By avoiding the transmission of any actual function, bit saving is achieved in various embodiments. It is to be appreciated that signaling can be completed in a variety of ways. For example, in various embodiments, one or more grammatical elements, flags, etc. are used to send information to the corresponding decoder with a signal. Although the verb form of the word "send with a signal" is related to the above, the word "signal" can also be used as a noun in this article.

[0113] It will be apparent to one of ordinary skill in the art that implementations may generate various signals formatted to carry information that may, for example, be stored or transmitted. For example, the information may include instructions for executing the method or data generated by one of the described implementations. For example, a signal may be formatted to carry a bitstream of the described embodiments. Such a signal may be formatted, for example, as an electromagnetic wave (e.g., using a radio frequency portion of a spectrum) or as a baseband signal. For example, formatting may include encoding a data stream and modulating a carrier with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. As is known, the signal may be transmitted over a variety of different wired or wireless links. The signal may be stored on a processor readable medium.

[0114] The preceding sections describe multiple embodiments across various claim categories and types. Features of these embodiments may be provided individually or in any combination. Further, embodiments may include one or more of the following features, devices, or aspects, individually or in any combination across various claim categories and types: One embodiment includes copying motion information from neighboring blocks to a buffer for use in intra encoding / decoding the current video block.

[0115] One embodiment includes encoding / decoding video blocks using the buffer information described above.

[0116] One embodiment includes averaging the motion vectors when motion vectors from two or more motion information have the same reference.

[0117] Other embodiments include any of the methods above, wherein the affine model includes the motion information.

[0118] Other embodiments include any of the methods above, wherein reconstructed samples of a coding unit coded in the current intra frame are used to estimate a motion vector candidate for propagation to a buffer.

[0119] One embodiment includes a bitstream or signal including one or more syntax elements or variations thereof for performing the functions described above.

[0120] One embodiment includes a bitstream or signal including syntax conveying information generated according to any of the described embodiments.

[0121] An embodiment comprises creating and / or transmitting and / or receiving and / or decoding according to any of the described embodiments.

[0122] One embodiment includes a method, process, apparatus, medium storing instructions, medium storing data, or signal according to any of the described embodiments.

[0123] One embodiment comprises inserting syntax elements in the signaling that enable the decoder to determine decoding information in a manner corresponding to that used by the encoder.

[0124] One embodiment comprises creating and / or transmitting and / or receiving and / or decoding a bitstream or signal comprising one or more of the described syntax elements or variants thereof.

[0125] One embodiment includes a TV, set-top box, cell phone, tablet or other electronic device that performs the transformation method(s) according to any of the described embodiments.

[0126] One embodiment includes a TV, set-top box, cell phone, tablet, or other electronic device that performs the transformation method(s) determined according to any of the described embodiments and displays (e.g., using a monitor, screen, or other type of display) the resulting image.

[0127] One embodiment includes a TV, set-top box, cellular phone, tablet computer, or other electronic device that selects, band-limits, or tunes (e.g., using a tuner) a channel to receive a signal including an encoded image and performs (multiple) transformation methods according to any of the described embodiments.

[0128] One embodiment includes a TV, set-top box, cell phone, tablet, or other electronic device that receives over-the-air (eg, using an antenna) a signal including an encoded image, and performs the transform method(s).

Claims

1. A method for encoding a video block, the method comprising: copying available motion information from the block into a buffer; selecting motion information from the buffer based on an intra mode of the video block; as well as At least a portion of the video block is encoded using the selected motion information.

2. A device, comprising: Memory, and A processor configured to execute: copying available motion information from the block into a buffer; selecting motion information from the buffer based on an intra mode of the video block; as well as At least a portion of the video block is encoded using the selected motion information.

3. A method comprising: copying available motion information from the block into a buffer; selecting motion information from the buffer based on an intra mode of the video block; as well as At least a portion of the video block is decoded using the selected motion information.

4. A device, comprising: Memory, and A processor configured to execute: copying available motion information from the block into a buffer; selecting motion information from the buffer based on an intra mode of the video block; as well as At least a portion of the video block is decoded using the selected motion information.

5. The method according to claim 1 or 3 or the device according to claim 2 or 4, wherein motion information of a neighboring coding unit with a highest number of samples adjacent to a current coding unit including the video block is placed in the buffer.

6. The method of claim 1, 3 or 5 or the apparatus of claim 2, 4 or 5, wherein when two highest available motion information comprise the same reference, the motion vector values ​​of the two highest available motion information are averaged.

7. The method or apparatus of claim 6, wherein the direction of the current intra mode is used to select the location of the motion information to be added to the buffer.

8. The method according to claim 1 or 3 or 5 to 7 or the device according to claim 2 or 4 or 5 to 7, wherein when several available motion information include the same reference, the motion vector values ​​of the several available motion information are averaged.

9. The method of claim 1 or 3 or 5 to 8 or the apparatus of claim 2 or 4 or 5 to 8, wherein the motion information of a reference is rescaled using a picture order count of another reference.

10. The method of claim 1 or 3 or 5 to 9 or the apparatus of claim 2 or 4 or 5 to 9, wherein an affine parameter model is used to fill the buffer of the current coding unit with spatially varying values.

11. The method of claim 1 or 3 or 5 to 10 or the apparatus of claim 2 or 4 or 5 to 10, wherein the motion vector candidates to be added to the buffer are estimated using reconstructed samples of a coding unit encoded in the current intra frame.

12. A device, comprising: The device according to claim 4; as well as At least one of: (i) an antenna configured to receive a signal, the signal comprising a video block; (ii) a band limiter configured to limit the received signal to a frequency band comprising the video block; and (iii) a display configured to display an output representing the video block.

13. A non-transitory computer readable medium containing data content generated by the method according to any one of claims 1 and 5 to 11 or by the apparatus according to any one of claims 2 and 5 to 11 for playback using a processor.

14. A signal comprising video data generated by the method of any one of claims 1 and 5 to 11 or by the apparatus of any one of claims 2 and 5 to 11 for playback using a processor.

15. A computer program product comprising instructions which, when said program is executed by a computer, cause said computer to perform the method according to any one of claims 1, 3 and 5 to 11.