Motion information parameter propagation based on intra prediction direction

Abstract

A method and apparatus for filling a motion information buffer with coding units or sub-blocks of coding units coded in intra mode includes using an intra prediction direction to select adjacent inter prediction parameters into the motion information buffer. In one embodiment, a rescaling of the inter prediction parameters is performed. In a variant, before storing the parameters in the motion information buffer, motion information such as motion vectors can be averaged or an affine model can be applied.

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of EPO Application No. 22305471.9, filed April 7, 2022, which is incorporated by reference in its entirety.

[0002] FIELD OF THEINVENTION At least one of the present embodiments generally relates to a method or apparatus for video encoding or decoding, compression, or decompression. [Background technology]

[0003] To achieve high compression efficiency, image and video coding schemes usually employ prediction, including motion vector prediction, and transformation to exploit spatial and temporal redundancy in the video content. Intra- or inter-prediction is generally used to exploit correlation within or between frames, whereby the difference between an original image and a predicted image, often referred to as a prediction error or prediction residual, is transformed, quantized, and entropy coded. To reconstruct the video, the compressed data is decoded by an inverse process corresponding to entropy coding, quantization, transformation, and prediction. Summary of the Invention

[0004] At least one of the present embodiments relates generally to a method or apparatus for video encoding or decoding, and more particularly to a method or apparatus for improving coding efficiency of intra-prediction modes.

[0005] According to a first aspect, there is provided a method that includes copying available motion information from a neighboring block to a buffer, selecting motion information from the buffer based on an intra mode for the video block, and encoding at least a portion of the video block using the selected motion information.

[0006] According to a second aspect, another method is provided that includes copying available motion information from a neighboring block to a buffer, selecting motion information from the buffer based on an intra mode for the video block, and decoding at least a portion of the video block using the selected motion information.

[0007] According to another aspect, there is provided an apparatus comprising a processor, the processor may be configured to encode a block of a video or decode a bitstream by performing any of the methods described above.

[0008] According to another general aspect of at least one embodiment, there is provided a device comprising an apparatus according to any of the decoding embodiments and at least one of: (i) an antenna configured to receive a signal, the signal including a video block; (ii) a band limiter configured to limit the received signal to a band of frequencies including the video block; or (iii) a display configured to display an output representative of the video block.

[0009] According to another general aspect of at least one embodiment, a non-transitory computer-readable medium is provided that includes data content generated according to any of the described encoding embodiments or variations.

[0010] According to another general aspect of at least one embodiment there is provided a signal including video data generated according to any of the described encoding embodiments or variations.

[0011] According to another general aspect of at least one embodiment, a bitstream is formatted to include data content generated according to any of the described encoding embodiments or variations.

[0012] According to another general aspect of at least one embodiment, there is provided a computer program product including instructions that, when executed by a computer, cause the computer to perform any of the described decoding embodiments or variations.

[0013] These and other aspects, features, and advantages of the general aspects will become apparent from the following detailed description of exemplary embodiments, which is to be read in conjunction with the accompanying drawings. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1 illustrates a general purpose video encoding or compression system. [Diagram 2] FIG. 1 illustrates a general purpose video decoding or decompression system. [Diagram 3] FIG. 1 illustrates an example GPM forecast. [Figure 4] FIG. 1 shows an example of two GPM predictions and coarser storage parameters (dashed sub-blocks) in the motion information buffer. [Diagram 5] FIG. 13 illustrates storage of coded intra-mode information of a current CU at sub-block granularity in an IPM buffer. [Figure 6] FIG. 2 illustrates reference samples for intra prediction. [Figure 7] FIG. 1 is a diagram showing the intra prediction direction of HEVC (left) and the intra prediction direction of VVC (right). [Figure 8] FIG. 1 illustrates wide-angle intra prediction. [Figure 9] FIG. 13 is a diagram illustrating an example of a plane prediction mode. [Figure 10] FIG. 2 is a diagram showing an example of a motion information (MI) propagation direction derived from an intra-mode direction. [Figure 11] FIG. 1A is a diagram showing an example of a four-parameter affine model, and FIG. 1B is a diagram showing an example of a six-parameter affine model. [Figure 12] FIG. 1 shows a general overview of the described embodiment. [Figure 13] FIG. 1 illustrates one embodiment of a method for carrying out the described aspects. [Figure 14] FIG. 13 illustrates another embodiment of a method for carrying out the described aspects. [Figure 15] FIG. 1 illustrates one embodiment of an apparatus for implementing the described aspects. [Figure 16] FIG. 1 illustrates a processor-based system for implementing the described aspects. [Figure 17] FIG. 13 illustrates another embodiment of a method for carrying out the described aspects. [Figure 18] FIG. 13 illustrates another embodiment of a method for carrying out the described aspects. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Intra prediction is a basic coding tool in video compression. The encoder selects the best prediction mode and signals its index to the decoder to perform the same prediction. Intra prediction is performed using reference samples, which are samples around the current block that have already been decoded. Currently, several modes are available in the VVC standard: -DC: Use a reference sample to create uniform forecasts - Planar mode: Uses reference samples to create a smooth prediction of the block -Directional mode: Uses reference samples to "pad" these samples along a specific direction to create a prediction - MIP (Matrix-based Intra Prediction): Uses a linear combination of reference samples to create a prediction

[0016] All these modes use spatial information to make predictions.

[0017] In intercoding, a prediction is made using samples from one or more reference frames.

[0018] Intermode Memory In ECM codecs, the motion information (MV values ​​and reference indices) used in inter-mode to code a CU is stored in a buffer (MI buffer) associated with the current frame. It is stored together with the reference picture, similar to the decoder picture buffer (DPB). The motion information used to decode some samples in the reference picture is then available for the current picture. This can be used to build a list of MV neighbor candidates by the MI buffer of the current frame, or to obtain co-located MVs by the MI buffer of the reference picture used as the co-located reference (signaled in the slice or picture header).

[0019] Intramode Propagation In ECM codecs, the intra mode used to code a CU intra is stored in a buffer (IPM buffer) associated with the current frame, which is stored together with the reference picture, similar to the Decoder Picture Buffer (DPB). The intra mode used to decode some samples in the reference picture is then made available for the current picture. Both IPM and MI buffers can be combined (a buffer stores both intra and inter information).

[0020] To reduce the amount of storage, the information can be stored at a coarser resolution than the current picture (e.g., 4x4 resolution in Fig. 4). For example, if the ratio between the current picture size and the motion information is si=4, the "MI" parameters of a block (or sub-block) located at (x, y) are stored in mi(x / 4;y / 4) of the motion information buffer.

[0021] If the current CU is coded in inter unidirectional mode, the intra mode associated with the reference block (available through the IPM buffer associated with the reference picture) is copied to the IPM buffer of the current CU. If the current CU is coded in inter bidirectional predictive mode, the following few rules (500) allow to select the intra mode from the IPM buffer associated with the reference frame of list 0 (ref_0 from ipm0) or the reference frame of list 1 (ref_1 from ipm1).

[0022] [Table 1] where pocDiffX=pocRefX-preSevent and DC_IDX is the intra-mode DC.

[0023] This allows the intra mode information to be propagated in time and available to all reconstructed CUs. For a current CU that is intra coded, this information is further used to build a list of most probable modes (MPMs) with neighboring reconstructed CUs, even if these CUs are inter-mode coded. - If the CU neighborhood is intra coded, use intra mode to build the MPM list. - If the CU neighbourhood is inter coded, build the MPM list using the intra modes stored in the IPM buffer of the CU neighbourhood.

[0024] Intramode propagation via GPM with inter- and intra-prediction. As illustrated in some embodiments of FIG. 3, the geometric partition mode (GPM) allows predicting one CU using two non-rectangular partitions. Each partition can be inter (275) or intra (260). Samples of the inter partition are predicted with a normal inter prediction process using motion compensation reference samples chosen from one (unidirectional prediction) reference picture. Samples of the intra partition are predicted with a normal intra prediction mode (IPM) and prediction process, and the available IPM candidates are the angular mode parallel to the GPM block boundary (parallel mode), the angular mode perpendicular to the GPM block boundary (vertical mode), and the planar mode, as shown in FIG. 3a-c, respectively.

[0025] The motion vectors used to reconstruct the inter-prediction unit and the IPM used to reconstruct the intra-partition are stored in the MI buffer and the IPM buffer, respectively. In the case of GPM, the partitions may be non-rectangular, so for a sub-block shared by two partitions, as shown in Figure 4 and Figure 5-620 (620), the stored information corresponds to the partition that occupies the larger area.

[0026] For example, in Fig. 4(a), a CU is predicted with one inter partition and one intra partition, and the corresponding inter and intra parameters are stored in the MI and IPM buffers with a 4x4 resolution in Fig. 4(b).

[0027] This information can be used later to code subsequent CUs in the current picture and CUs in subsequent pictures (in coding order), for example to predict the motion or IPM to be used. For sub-block (x,y), if more samples are predicted intra than inter (620), the intra parameters are stored in the MI buffer (630). If more samples are predicted inter than intra (620), the intra parameters are obtained from the reference MI buffer, which is the MI buffer associated with the reference picture used for inter prediction. The location of the intra parameters is mi((x+mv X / si, (y+mv Y ) / si)(640), where (mv X , mv Y ) is the motion vector for inter prediction, and si is the ratio between the current picture size and the motion information buffer size.

[0028] Intra prediction The intra prediction process in HEVC and VVC consists of the following three steps: Reference sample generation Intra-sample prediction, and Post-processing of prediction samples.

[0029] The reference sample generation process is shown in Figure 6. The reference sample ref[] is also known as L-shaped. For a prediction unit (PU) of size NxN, a top (2N+refIdx) row of decoded samples is formed in the current PU from the previously reconstructed top and top-right pixels. Similarly, a left (2N+refIdx) column of samples is formed from the reconstructed left and bottom-left pixels. As shown in Figure 6, in VVC, the reference row and column of samples may be more than one sample (d=refIdx) away for the current block.

[0030] In addition, the corner pixels of the top left position are used to fill the gap between the top row and the left column references. If some of the top or left samples are unavailable because the corresponding CU is not in the same slice or the current CU is at a frame boundary, we perform a method called reference sample replacement, which copies the missing samples from the available samples in a clockwise direction. Then, we filter the reference samples using a specified filter depending on the current CU size and prediction mode.

[0031] Intra-sample prediction consists of predicting pixels of a target CU based on a reference sample. Different prediction modes exist. For example, planar and DC prediction modes are used to predict smooth and gradually changing regions, whereas angular (angle defined in the clockwise direction ranging from 45 degrees to -135 degrees) prediction modes are used to capture structures in different directions. For square blocks, HEVC supports 33 directional prediction modes, indexed from 2 to 34. These prediction modes correspond to different prediction directions, as shown on the left side of Figure 7. In VVC, there are 65 angular prediction modes corresponding to the 33 angular directions defined in HEVC, and an additional 32 directions, each corresponding to an intermediate direction between adjacent pairs (right side of Figure 7).

[0032] In VVC, for non-square blocks, the usual directional intra-prediction (see FIG. 8), which is not allowed, is replaced with an additional wide-angle intra-prediction mode.

[0033] For a given angular prediction mode, the predictor samples on the reference array are copied along the corresponding direction in the target PU. Some predictor samples have integral positions, in which case they match the corresponding reference samples. Other predictor positions have fractional parts indicating that their positions fall between two reference samples. In the latter case, the predictor samples are interpolated using the nearest reference samples. In HEVC, a linear interpolation of the two nearest reference samples is performed to calculate the predictor sample value. In VVC, a 4-tap filter fT[], selected by the intra-mode direction, is used to interpolate the predictor samples.

[0034] In addition to the directional modes, as shown in Figure 9, the DC mode fills the prediction with the average of L-shaped samples (except for rectangular CUs, which use the average of the reference samples on the long side), and the planar mode spatially interpolates the reference samples.

[0035] The IPM buffer implements a propagation process that allows filling in information about all CUs that may be coded in intra or inter mode. However, no such mechanism exists for motion information. For coding modes that use motion information, the lack of such data may have a negative impact on coding efficiency.

[0036] For CUs (or sub-CUs) coded in intra mode, we propose a motion information propagation process to fill the MI buffer.

[0037] In some embodiments, we propose to propagate motion information if the current CU is coded in intra mode. A general overview of this method is shown in Fig. 12. This method is applied to the intra-coded sub-block parts in the current CU. The inter-coded sub-block parts use the normal method (copying inter parameters in the current co-located MI buffer) (830 in Fig. 12). Depending on the embodiment, some logical blocks of the overview in Fig. 12 may not be present. Their functionality may vary depending on the embodiment / variant.

[0038] Embodiment 1 - Propagate motion information of neighboring blocks. The motion information (MI) of the neighboring blocks is propagated to the MI buffer at the position of the current CU (current MI). A list of predefined neighboring block positions can be constructed. The first item of the list containing an available MI is selected (850) and copied to the current MI (840). For example, the list is all reconstructed CUs, ordered from left to right on top of the current CU, then from left to top to bottom. In a variant, the neighboring CU that contains the largest number of samples adjacent to the current CU is placed at the top of the list. If the MI of two CUs is the same, they are considered to be the same CU (adding multiple consecutive samples).

[0039] The expression "MI is not available" can be defined as follows (several combinations may be used): When the current CU is on a picture boundary (or a slice boundary, or a tile boundary). If the current CU is located on the left boundary of a picture, the CU to the left cannot be used. If a neighboring CU does not have MI data stored in its buffer, it cannot use it.

[0040] In a variant, if the first two MIs available in the list have the same reference, the motion vector values ​​can be averaged (870).

[0041] Embodiment 2 - Propagating motion information along with intra direction If the current intra mode (predMode) is directional, the direction is used to select the location of the MI to be propagated (850). For example, if intra prediction uses samples from the top, MI candidates to be propagated (mIRef) are picked up only from the top CU.

[0042] In another example,

[0043] [Table 2] In a variant, the MV values ​​of several MIs may be averaged (870) if they have the same reference. For example:

[0044] [Table 3] In a variant, if reference MIs are available but do not have the same reference, they can be rescaled using the POC of the reference so that they have the same reference (860). The closest POC of the reference (pocRef) can be selected, taking the current frame as the reference POC (pocCur). ·miRescaled=miRef x (pocCur-pocRef) / (poc-pocRef) ·miRescaled, mIRef is the value of the motion vector component X or Y. poc is the POC of mIRef to be rescaled.

[0045] Embodiment 3 - Propagating motion information with an affine model If Bottom-Left, Above-Right, and Left-Above are available, an affine six-parameter model can be used to fill the MI buffer of the current CU with spatially varying values ​​(870). If they do not have the same reference, MI rescaling may be applied (see embodiment 2).

[0046] In a variant, similar to embodiment 2, an affine four-parameter model can be used by using the intra direction to select the two reference MIs.

[0047] For the four-parameter affine motion model, the motion vector at a sample position (x,y) within a block is derived as follows:

[0048]

number

[0049] For a six-parameter affine motion model, the motion vector at a sample position (x,y) within a block is derived as follows:

[0050]

number

[0051] Embodiment 4 - Using intra-reconstructed samples to estimate propagated motion parameters According to this embodiment, the reconstructed samples of the current intra CU are used to estimate the best motion vector candidate to propagate. The list of motion vector candidates for the current CU is constructed using the normal list-merge candidate construction in the same way as for CUs coded in inter-merge mode. The cost of each candidate is evaluated using the SATD with the reconstructed samples of the current coding unit. In a variant, to simplify the computational effort, the cost can be calculated over a reduced number of reconstructed samples (a sub-part of the reconstructed CU samples). For the current CU, the candidate with the smallest cost is selected to be propagated.

[0052] In an example variation, affine merge candidates are also evaluated.

[0053] Embodiment 5 - Propagated motion information not used for spatial candidates In the fourth embodiment, additional latency is incurred because the propagation of motion parameters must be performed after the samples of the current CU are reconstructed. Then, the propagated motion parameters cannot be used to build motion candidates for the next CU without waiting for availability, which may result in additional latency. In this embodiment, the propagated motion information of the intra CU is not used for spatial candidates, but only for temporal candidates. Advantageously (in a variant), only in the fourth embodiment, the propagated motion information of the intra CU is not used for spatial candidates.

[0054] EMBODIMENT 6 - SUB-BLOCK DIVISION PROPAGATION MI propagation in the current intra block may be performed subblock by subblock. For example, if the neighboring CU consists of subpartitions, the MI parameters of each partition may be propagated in the current block, as shown in the example of Figure 17 for horizontal intra mode prediction direction (a) and diagonal intra mode prediction direction (b). In another variant, the neighboring MI may be picked up from the inter-coded subpartitions from the current CU, as shown in the example of Figure 18.

[0055] Example 7 - Backpropagation In general, the prediction signals of the top and left samples of a block coded in intra mode are well correlated with the adjacent CU reconstructed samples. Thus, these adjacent CU reconstructed samples are used to construct an intra prediction for the current CU. However, other samples (e.g., right and bottom samples) of the current CU may not be well correlated with the adjacent (left and / or top) CU reconstructed samples. In this embodiment, if the current CU is coded in intra mode and one left (or top) adjacent CU is coded in intra mode, the MI parameters of the current CU may be propagated to the MI buffer associated with the adjacent (left or top) CU coded in intra mode.

[0056] One embodiment of a method 1300 under general aspects described herein is shown in Figure 13. The method begins at start block 1301 and control passes to block 1310, which copies available motion information from a neighboring block to a buffer. Control passes from block 1310 to block 1320, which selects motion information from the buffer based on the intra mode of the video block. Control passes from block 1320 to block 1330, which encodes at least a portion of the video block using the selected motion information.

[0057] One embodiment of a method 1400 under general aspects described herein is shown in Figure 14. The method begins at start block 1401 and control passes to block 1410, which copies available motion information from a neighboring block to a buffer. Control passes from block 1410 to block 1420, which selects motion information from the buffer based on the intra mode of the video block. Control passes from block 1420 to block 1430, which decodes at least a portion of the video block using the selected motion information.

[0058] 15 illustrates one embodiment of an apparatus 1500 for encoding, decoding, compressing, or decompressing video data using any of the methods described above, or variations thereof. The apparatus includes a processor 1510 and may be interconnected through at least one port to a memory 1520. Additionally, both the processor 1510 and the memory 1520 may also have one or more additional interconnections to external connections.

[0059] The processor 1510 is also configured to either insert or receive information in a bitstream, and compress, encode, or decode using any of the aspects described.

[0060] The embodiments described herein include various aspects, including tools, features, embodiments, models, methods, and the like. Many of these aspects have been specifically described and described in a manner that may often sound limiting, at least to indicate their individual characteristics. However, this is for purposes of clarity of description and does not limit the applicability or scope of the aspects. In fact, all of the different aspects can be combined and substituted to provide further aspects. Moreover, these aspects can also be combined and substituted with aspects described in previous applications as well.

[0061] The aspects described and contemplated in this application can be implemented in many different forms. While Figures 1, 2, and 16 provide some embodiments, other embodiments are contemplated, and discussion of Figures 1, 2, and 16 is not intended to limit the scope of the implementations. At least one of the above aspects generally relates to video encoding and decoding, and at least one other aspect generally relates to transmitting a generated or encoded bitstream. These and other aspects can be implemented as a method, an apparatus, a computer-readable storage medium having stored therein instructions for encoding or decoding video data according to any of the described methods, and / or a computer-readable storage medium having stored therein a bitstream generated according to any of the described methods.

[0062] In this application, the terms "reconstructed" and "decoded" may be used interchangeably, the terms "pixel" and "sample" may be used interchangeably, and the terms "image", "picture", and "frame" may be used interchangeably. Typically, but not necessarily, the term "reconstructed" is used on the encoder side, while the term "decoded" is used on the decoder side.

[0063] Various methods are described herein, each of which includes one or more steps or acts for achieving the described method. Unless a specific order of steps or acts is required for the proper operation of the method, the order and / or use of specific steps and / or acts may be modified or combined.

[0064] Various methods and other aspects described herein may be used to modify modules of the video encoder 100 and video decoder 200, such as intra-prediction modules, entropy coding modules, and / or decoding modules (160, 360, 145, 330) as shown in Figures 1 and 2. Furthermore, aspects of the present disclosure are not limited to VVC or HEVC, but may be applied to other standards and recommendations, such as existing or future developments, and extensions of any such standards and recommendations, including VVC and HEVC. Unless otherwise indicated or technically precluded, aspects described herein may be used individually or in combination.

[0065] Various numerical values ​​are used in this application. The specific values ​​are for illustrative purposes and the described aspects are not limited to these specific values.

[0066] 1 shows an encoder 100. Variations of this encoder 100 are contemplated, but for clarity, the following describes the encoder 100 without describing all possible variations.

[0067] Before encoding, the video sequence may undergo pre-encoding processing (101), such as applying a color transformation to the input color picture (e.g., converting from RGB 4:4:4 to YCbCr 4:2:0) or performing a remapping of the input picture components to obtain a signal distribution that is more resilient to compression (e.g., using histogram equalization of one of the color components). Metadata can be associated with the pre-processing and added to the bitstream.

[0068] In the encoder 100, a picture is coded by the encoder elements, as described below. The picture to be coded is divided (102) into units, e.g., CUs, and processed. Each unit is coded, e.g., using either intra-mode or inter-mode. When a unit is coded in intra-mode, it performs intra-prediction (160). In inter-mode, motion estimation (175) and motion compensation (170) are performed. The encoder decides (105) whether intra-mode or inter-mode should be used to code the unit, and indicates the intra- or inter-decision, e.g., by a prediction mode flag. A prediction residual is calculated (110), e.g., by subtracting the prediction block from the original image block.

[0069] The prediction residual is then transformed (125) and quantized (130). The quantized transform coefficients, as well as the motion vectors and other syntax elements, are entropy coded (145) to output a bitstream. The encoder can skip the transform and apply quantization directly to the untransformed residual signal. The encoder can bypass both the transform and quantization, i.e., the residual is coded directly without applying a transform or quantization process.

[0070] The encoder decodes the coded block to provide a reference for further prediction. To decode the prediction residual, the quantized transform coefficients are dequantized (140) and inverse transformed (150). The decoded prediction residual and the predicted block are combined (155) to reconstruct an image block. An in-loop filter (165) is applied to the reconstructed picture to perform, for example, deblocking / sample adaptive offset (SAO) / adaptive loop filtering (ALF) filtering to reduce coding artifacts. The filtered image is stored in a reference picture buffer (180).

[0071] Figure 2 shows a block diagram of a video decoder 200. In the decoder 200, the bitstream is decoded by decoder elements, as described below. In general, the video decoder 200 performs a decoding pass that is inverse to the encoding pass described in Figure 1. Furthermore, the encoder 100 generally performs video decoding as part of the video data encoding.

[0072] In particular, the decoder's input includes a video bitstream, which may be generated by video encoder 100. The bitstream is first entropy decoded (230) to obtain transform coefficients, motion vectors, and other coded information. Picture partition information indicates how the picture is partitioned. Thus, the decoder may partition the picture according to the decoded picture partition information (235). To decode the prediction residual, the transform coefficients are dequantized (240) and inverse transformed (250). The decoded prediction residual and the predicted block are combined (255) to reconstruct an image block. A prediction block may be obtained (270) from intra prediction (260) or motion compensated prediction (i.e., inter prediction) (275). An in-loop filter (265) is applied to the reconstructed image. The filtered image is stored in a reference picture buffer (280).

[0073] The decoded picture may further undergo post-decoding processing (285), such as an inverse color conversion (e.g., from YcbCr 4:2:0 to RGB 4:4:4) or an inverse remapping that performs the inverse of the remapping process performed in the pre-encoding processing (101). The post-decoding processing can use metadata derived in the pre-encoding processing and signaled in the bitstream.

[0074] FIG. 16 illustrates a block diagram of an example of a system in which various aspects and embodiments are implemented. System 1000 may be embodied as a device including various components described below and configured to perform one or more of the aspects described herein. Examples of such devices include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected appliances, and servers. The elements of system 1000, alone or in combination, may be embodied in a single integrated circuit (IC), multiple ICs, and / or separate components. For example, in at least one embodiment, the processing elements and encoder / decoder elements of system 1000 are distributed across multiple ICs and / or separate components. In various embodiments, system 1000 communicatively couples to one or more other systems or other electronic devices, for example, via a communication bus or through dedicated input and / or output ports. In various embodiments, system 1000 is configured to implement one or more of the aspects described herein.

[0075] The system 1000 is configured to execute instructions loaded therein, for example, to implement various aspects described herein. The system 1000 includes at least one processor 1010. The processor 1010 may include embedded memory, input / output interfaces, and various other circuits known in the art. The system 1000 includes at least one memory 1020 (e.g., a volatile memory device and / or a non-volatile memory device). The system 1000 includes a storage device 1040, which may include non-volatile memory and / or volatile memory, including, but not limited to, an electrically erasable programmable read-only memory (EEPROM), a read-only memory (ROM), a programmable read-only memory (PROM), a random access memory (RAM), a dynamic random access memory (DRAM), a static random access memory (SRAM), a flash, a magnetic disk drive, and / or an optical disk drive. The storage devices 1040 may include, by way of non-limiting examples, internal storage devices, attached storage devices (including removable and non-removable storage devices), and / or network-accessible storage devices.

[0076] The system 1000 includes an encoder / decoder module 1030 configured to process data to provide, for example, encoded or decoded video, and the encoder / decoder module 1030 may include its own processor and memory. The encoder / decoder module 1030 represents a module that may be included in a device for performing encoding and / or decoding functions. As is known, a device may include one or both of an encoding module and a decoding module. Additionally, the encoder / decoder module 1030 may be implemented as a separate element of the system 1000, or may be incorporated within the processor 1010 as a combination of hardware and software, as is known to those skilled in the art.

[0077] Program code to be loaded into the processor 1010 or the encoder / decoder 1030 to perform various aspects described herein may be stored in the storage device 1040 and subsequently loaded into the memory 1020 for execution by the processor 1010. According to various embodiments, one or more of the processor 1010, the memory 1020, the storage device 1040, and the encoder / decoder module 1030 may store one or more of various items during execution of the processes described herein. Such stored items may include, but are not limited to, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from the processing of equations, expressions, operations, and computational logic.

[0078] In some embodiments, memory internal to the processor 1010 and / or the encoder / decoder module 1030 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, memory external to the processing device (e.g., the processing device can be either the processor 1010 or the encoder / decoder module 1030) is used for one or more of these functions. The external memory can be the memory 1020 and / or the storage device 1040, e.g., dynamic volatile memory and / or non-volatile flash memory. In some embodiments, the external non-volatile flash memory is used to store, for example, the television's operating system. In at least one embodiment, a fast external dynamic volatile memory such as RAM is used as working memory for video coding and decoding operations such as MPEG-2 (MPEG is called Moving Picture Experts Group, MPEG-2 is also called ISO / IEC 13818, 13818-1 is also known as H.222, and 13818-2 is also known as H.262), HEVC (HEVC is called High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2), or VVC (Versatile Video Coding, a new standard being developed by JVET).

[0079] Inputs to the elements of system 1000 may be provided through a variety of input devices as indicated in block 1130. Such input devices may include, but are not limited to, (i) a radio frequency (RF) section, for example, receiving an RF signal transmitted over the air by a broadcast station, (ii) a component (COMP) input terminal (or set of COMP input terminals), (iii) a universal serial bus (USB) input terminal, and / or (iv) a high definition multimedia interface (HDMI) input terminal. Although not shown in FIG. 16, other embodiments include composite video.

[0080] In various embodiments, the input devices of block 1130 have associated respective input processing elements known in the art. For example, the RF section may be associated with elements suitable for (i) selecting a desired frequency (also referred to as selecting a signal or band-limiting a signal to a frequency band), (ii) down-converting the selected signal, (iii) band-limiting again to a narrower frequency band to select a signal frequency band that may be referred to as a channel in a particular embodiment (for example), (iv) demodulating the down-converted band-limited signal, (v) performing error correction, and (vi) demultiplexing to select a desired stream of data packets. The RF section of various embodiments includes one or more elements that perform these functions, such as a frequency selector, a signal selector, a band limiter, a channel selector, a filter, a down-converter, a demodulator, an error corrector, and a demultiplexer. The RF section may include a tuner that performs various of these functions, including, for example, down-converting a received signal to a lower frequency (e.g., an intermediate frequency or a near-baseband frequency) or to baseband. In one embodiment of a set-top box, the RF section and its associated input processing elements receive RF signals transmitted over a wired (e.g., cable) medium and perform frequency selection by filtering, downconverting, and refiltering to a desired frequency band. Various embodiments rearrange the order of the above-mentioned (and other) elements, omit some of these elements, and / or add other elements that perform similar or different functions. Adding elements can include inserting elements between existing elements, such as, for example, inserting amplifiers and analog-to-digital converters. In various embodiments, the RF section includes an antenna.

[0081] Additionally, the USB and / or HDMI terminals may include respective interface processors for connecting the system 1000 to other electronic devices via USB and / or HDMI connections. It should be understood that various aspects of the input processing, e.g., Reed-Solomon error correction, may be implemented, for example, in a separate input processing IC or within the processor 1010, as desired. Similarly, aspects of the USB or HDMI interface processing may be implemented in a separate interface IC or within the processor 1010, as desired. The demodulated, error corrected, and demultiplexed streams are provided to various processing elements, including, for example, the processor 1010 and an encoder / decoder 1030 operating in combination with memory and storage elements, to process the data streams necessary for presentation to an output device.

[0082] The various elements of the system 1000 may be provided within an integrated housing in which the various elements may be interconnected and transmit data therebetween using suitable connection arrangements, e.g., internal buses known in the art, including the Inter-IC (I2C) bus, wiring, and printed circuit boards.

[0083] System 1000 includes a communication interface 1050 that enables communication with other devices over a communication channel 1060. Communication interface 1050 may include, but is not limited to, a transceiver configured to transmit and receive data over communication channel 1060. Communication interface 1050 may include, but is not limited to, a modem or a network card, although communication channel 1060 may be implemented in a wired medium and / or a wireless medium, for example.

[0084] In various embodiments, data is streamed or otherwise provided to the system 1000 using a wireless network such as a Wi-Fi network, e.g., IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers). The Wi-Fi signal in such embodiments is received via a communication channel 1060 adapted for Wi-Fi communication and a communication interface 1050. Typically, the communication channel 1060 in such embodiments is connected to an access point or router that provides access to external networks, including the Internet, to enable streaming applications and other over-the-top communications. In other embodiments, a set-top box that delivers data via an HDMI connection in the input block 1130 is used to provide the streamed data to the system 1000. In yet other embodiments, an RF connection in the input block 1130 is used to provide the streamed data to the system 1000. As indicated above, various embodiments provide the data in a non-streaming manner. Additionally, various embodiments use wireless networks other than Wi-Fi, e.g., a cellular network or a Bluetooth network.

[0085] The system 1000 can provide output signals to various output devices, including a display 1100, speakers 1110, and other peripheral devices 1120. The display 1100 of various embodiments includes, for example, one or more of a touch screen display, an organic light emitting diode (OLED) display, a curved display, and / or a foldable display. The display 1100 can be for a television, a tablet, a laptop, a mobile phone, or another device. Furthermore, the display 1100 can be integrated with other components (e.g., as in a smartphone) or can be separate (e.g., an external monitor for a laptop). In various examples of embodiments, the other peripheral devices 1120 include one or more of a standalone digital video disc (or digital versatile disc) (both terms are DVR), a disc player, a stereo system, and / or a lighting system. Various embodiments use one or more peripheral devices 1120 that provide functionality based on the output of the system 1000. For example, a disc player performs the functionality of playing the output of the system 1000.

[0086] In various embodiments, control signals are communicated between the system 1000 and the display 1100, speaker 1110, or other peripheral device 1120 using signaling such as AV.Link, Consumer Electronics Control (CEC), or other communication protocols that allow control between devices with or without user intervention. The output devices can be communicatively coupled to the system 1000 via dedicated connections through respective interfaces 1070, 1080, and 1090. Alternatively, the output devices can be connected to the system 1000 via the communication interface 1050 using the communication channel 1060. The display 1100 and speaker 1110 may be integrated into a single unit with other components of the system 1000 in an electronic device such as, for example, a television. In various embodiments, the display interface 1070 includes a display driver, such as, for example, a timing controller (T Con) chip.

[0087] For example, if the RF portion of input 1130 is part of a separate set-top box, display 1100 and speakers 1110 may alternatively be separate from one or more of the other components. In various embodiments where display 1100 and speakers 1110 are external components, the output signal may be provided via a dedicated output connection including, for example, an HDMI port, a USB port, or a COMP output.

[0088] The embodiments may be performed by computer software implemented by the processor 1010, or by hardware, or by a combination of hardware and software. As a non-limiting example, the embodiments may be implemented by one or more integrated circuits. The memory 1020 may be of any type suitable for the technology environment, and may be implemented using any suitable data storage technology, such as, as non-limiting examples, optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory devices. The processor 1010 may be of any type suitable for the technology environment, and may include, as non-limiting examples, one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a multi-core architecture-based processor.

[0089] Various implementations involve decoding. As used herein, "decoding" can encompass all or part of the processing performed on a received encoded sequence to generate a final output suitable for display, for example. In various embodiments, such processes include one or more of the processes typically performed by a decoder, such as, for example, entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such processes also or alternatively include processes performed by the decoders of the various implementations described herein.

[0090] As a further example, in one embodiment, "decoding" refers only to entropy decoding, in another embodiment, "decoding" refers only to differential decoding, and in another embodiment, "decoding" refers to a combination of entropy decoding and differential decoding. Whether the phrase "decoding process" is intended to refer specifically to a subset of operations or to the broader decoding process as a whole will be clear based on the context of the specific description and will be well understood by one of ordinary skill in the art.

[0091] Various implementations involve encoding. As can be seen from the above discussion of "decoding," "encoding" as used herein can encompass all or part of the processing performed on an input video sequence to generate, for example, an encoded bitstream. In various embodiments, such processes include one or more of the processes typically performed by an encoder, such as, for example, partitioning, differential encoding, transforming, quantization, and entropy encoding. In various embodiments, such processes also or alternatively include processes performed by the encoders of the various implementations described herein.

[0092] As a further example, in one embodiment, "encoding" refers to entropy encoding only, in another embodiment, "encoding" refers to differential encoding only, and in another embodiment, "encoding" refers to a combination of differential and entropy encoding. Whether the phrase "encoding process" is intended to refer specifically to a subset of operations or to the broader encoding process as a whole will be clear based on the context of the specific description and will be well understood by one of ordinary skill in the art.

[0093] It should be noted that the syntax elements used herein are descriptive terms, and therefore do not exclude the use of other syntax element names.

[0094] Where a figure is presented as a flow diagram, it should be understood that the figure also provides a block diagram of the corresponding apparatus. Similarly, where a figure is presented as a block diagram, it should be understood that the figure also provides a flow diagram of the corresponding method / process.

[0095] Various embodiments may relate to parametric models or rate-distortion optimization. In particular, during the encoding process, often due to computational complexity constraints, a balance or trade-off between rate and distortion is usually considered. This can be measured through a rate-distortion optimization (RDO) metric, or through least mean squares (LMS), mean absolute error (MAE), or other such measures. Rate-distortion optimization is usually formulated to minimize a rate-distortion function that is a weighted sum of rate and distortion. There are various approaches to solving the rate-distortion optimization problem. For example, these techniques may be based on extensive testing of all encoding options, including all considered modes or coding parameter values, but with a full evaluation of their coding costs and associated distortion of the reconstructed signal after encoding and decoding. To keep the coding complexity down, more rapid techniques may also be used, especially with the calculation of approximate distortion based on the prediction or prediction residual signal rather than the reconstructed signal. A mixture of these two approaches may also be used, such as by using approximate distortion for only some of the possible encoding options and full distortion for others. Other approaches evaluate only a subset of the considered encoding options. More generally, many approaches employ any of a variety of techniques to perform the optimization, but the optimization is not necessarily a complete assessment of both the coding cost and the associated distortion.

[0096] The implementations and aspects described herein may be implemented as, for example, a method or process, an apparatus, a software program, a data stream, or a signal. Even if discussed in the context of only a single implementation form (e.g., discussed only as a method), the implementation of the discussed features may also be implemented in other forms (e.g., an apparatus or a program). For example, an apparatus may be implemented in appropriate hardware, software, and firmware. The method may be implemented in a processor, which generally refers to a processing device, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Additionally, a processor may include, for example, a communication device, such as a computer, a mobile phone, a portable / personal digital assistant ("PDA"), and other devices that facilitate communication of information between end users.

[0097] References to "one embodiment" or "an embodiment" or "one implementation" or "an implementation", as well as other variations thereof, mean that a particular feature, structure, characteristic, etc. described in connection with that embodiment is included in at least one embodiment. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment", or "in one implementation" or "in an implementation" in various places throughout this application, as well as other variations, are not necessarily all referring to the same embodiment.

[0098] Additionally, the application may refer to "determining" various information. Determining information may include, for example, one or more of estimating information, calculating information, predicting information, or retrieving information from memory.

[0099] Additionally, the application may refer to "accessing" various information. Accessing information may include, for example, one or more of receiving information, retrieving information (e.g., from a memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0100] Additionally, the application may refer to "receiving" various information. Receiving, like "accessing," is intended to be a broad term. Receiving information may include, for example, one or more of accessing information or retrieving information (e.g., from a memory). Furthermore, "receiving" generally involves in some manner an operation, such as, for example, storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0101] For example, in the case of "A / B," "A and / or B," and "at least one of A and B," it should be understood that the use of any of the following " / ," "and / or," and "at least one of" is intended to encompass the selection of only the first enumerated alternative (A), or the selection of only the second enumerated alternative (B), or the selection of both alternatives (A and B). As a further example, in the case of "A, B, and / or C" and "at least one of A, B, and C," such language is intended to encompass the selection of only the first enumerated alternative (A), or the selection of only the second enumerated alternative (B), or the selection of only the third enumerated alternative (C), or the selection of only the first and second enumerated alternatives (A and B), or the selection of only the first and third enumerated alternatives (A and C), or the selection of only the second and third enumerated alternatives (B and C), or the selection of all three alternatives (A and B and C). This can be expanded as many times as the items listed, as would be apparent to one of ordinary skill in this and related arts.

[0102] Also, as used herein, the term "signaling" specifically means to indicate something to a corresponding decoder. For example, in certain embodiments, an encoder signals a particular one of multiple transforms, coding modes, or flags. Thus, in some embodiments, the same transform, parameter, or mode is used at both the encoder and decoder sides. Thus, for example, an encoder can transmit a particular parameter to a decoder so that the decoder can use the same particular parameter (explicit signaling). In contrast, if the decoder already has the particular parameter as well as other parameters, a non-transmitting signaling (implicit signaling) can be used to simply allow the decoder to know and select the particular parameter. By avoiding transmitting any actual functionality, bit savings are realized in various embodiments. It will be appreciated that signaling can be achieved in various ways. For example, one or more syntax elements, flags, etc. are used in various embodiments to signal information to a corresponding decoder. Although the above relates to the verb form of the term "signal", the term "signal" may also be used herein as a noun.

[0103] As will be apparent to one of ordinary skill in the art, implementations can result in a variety of signals formatted to carry information that can be, for example, stored or transmitted. Information can include, for example, instructions for performing a method or data generated by one of the described implementations. For example, a signal can be formatted to carry a bit stream of the described embodiments. For example, such a signal can be formatted as an electromagnetic wave (e.g., using a radio frequency portion of the spectrum) or as a baseband signal. Formatting can include, for example, encoding a data stream and modulating a carrier wave with the encoded data stream. The information that the signal carries can be, for example, analog or digital information. As is known, the signal can be transmitted over a variety of different wired or wireless links. The signal can be stored in a processor-readable medium.

[0104] The preceding sections describe several embodiments across various claim categories and types. The features of these embodiments may be provided alone or in any combination. Furthermore, the embodiments may include one or more of the following features, devices, or aspects across various claim categories and types, alone or in any combination.

[0105] One embodiment includes copying motion information from neighboring blocks to a buffer to be used to intra-encode / decode the current video block.

[0106] One embodiment includes using the above buffer information to encode / decode a video block.

[0107] One embodiment involves averaging motion vectors from two or more motion information when they have the same reference.

[0108] Other embodiments include any of the above methods in which the affine model includes the motion information.

[0109] Other embodiments include any of the above methods in which reconstructed samples of a current coding unit that is intra-coded are used to estimate candidate motion vectors that propagate to a buffer.

[0110] One embodiment includes a bitstream or signal that includes one or more syntax elements for performing the above functions, or variations thereof.

[0111] One embodiment includes a bitstream or signal that includes syntax conveying information generated according to any of the described embodiments.

[0112] An embodiment includes creating and / or transmitting and / or receiving and / or decoding according to any of the described embodiments.

[0113] An embodiment includes a method, process, apparatus, instruction storage medium, data storage medium, or signal according to any of the described embodiments.

[0114] One embodiment involves inserting syntax elements into the signaling that allow the decoder to determine the decoded information in a manner corresponding to that used by the encoder.

[0115] One embodiment includes creating, transmitting, receiving, and / or decoding a bitstream or signal that includes one or more of the described syntax elements, or variations thereof.

[0116] An embodiment includes a television, set-top box, mobile phone, tablet, or other electronic device that performs the conversion method according to any of the described embodiments.

[0117] One embodiment includes a television, set-top box, mobile phone, tablet, or other electronic device that performs the conversion method determination according to any of the described embodiments and displays the final image (e.g., using a monitor, screen, or other type of display).

[0118] One embodiment includes a television, set-top box, mobile phone, tablet, or other electronic device that selects, band-limits, or tunes (e.g., using a tuner) a channel to receive a signal containing the encoded image and performs a conversion method according to any of the described embodiments.

[0119] One embodiment includes a TV, set-top box, mobile phone, tablet, or other electronic device that receives a signal containing the encoded image wirelessly (eg, using an antenna) and performs the conversion method.

Claims

1. A method for encoding a video block, Copying available motion information from the block to the buffer, Select motion information from the buffer based on the intra mode of the video block, Using the selected motion information, encode at least a portion of the video block, A method that includes this.

2. It is a device, Memory and Processor and The processor is equipped with, Copying available motion information from the block to the buffer, Select motion information from the buffer based on the intra-mode of the video block, Using the selected motion information, encode at least a portion of the video block, A device configured to perform the following actions.

3. It is a method, Copying available motion information from the block to the buffer, Select motion information from the buffer based on the intra-mode of the video block, Using the selected motion information, decode at least a portion of the video block, A method that includes this.

4. It is a device, Memory and Processor and The processor is equipped with, Copying available motion information from the block to the buffer, Select motion information from the buffer based on the intra-mode of the video block, Using the selected motion information, decode at least a portion of the video block, A device configured to perform the following actions.

5. The method according to claim 1 or 3, wherein the motion information of the adjacent coding unit having the largest number of consecutive samples to the current coding unit including the video block is placed in the buffer.

6. The method according to claim 1 or 3, wherein the motion vector value is the average of the two highest available motion information when the motion information includes the same reference.

7. The method according to claim 6, wherein the direction of the current intra mode is used to select the position of the motion information to be added to the buffer.

8. The method according to claim 1 or 3, wherein the motion vector value is an average of several available motion pieces if the motion pieces include the same reference.

9. The method according to claim 1 or 3, wherein the motion information of a reference is rescaled using the picture sequence count of another reference.

10. The method according to claim 1 or 3, wherein an affine parameter model is used to fill the buffer of the current coding unit with spatially varying values.

11. The method according to claim 1 or 3, wherein a reconstructed sample of the intracoded current coding unit is used to estimate motion vector candidates to add to the buffer.

12. It is a device, The apparatus according to claim 4, (i) an antenna configured to receive a signal, wherein the signal includes the video block; (ii) a band limiter configured to limit the received signal to a frequency band including the video block; and (iii) a display configured to display an output representing the video block, A device equipped with the following features.

13. A non-temporary computer-readable medium comprising data content for playback using a processor, generated according to the method of claim 1.

Images