Method and apparatus for motion vector prediction based on sub-block motion vectors

Abstract

[0001] This disclosure generally relates to video encoding / decoding, and more particularly to motion vector prediction based on sub-block motion vectors. A method for processing a current block of a current frame in a coded video bitstream is disclosed. The method includes: receiving a coded video bitstream by a device; determining, based on the coded video bitstream, a current block of the current frame to be inter-predicted by at least one reference block of at least one reference frame; determining, based on the coded video bitstream, motion vectors (MVs) of the current block corresponding to each of the at least one reference block, where the MV of the current block will be predicted by the reference MV; and identifying, by the device, a sub-block MV of the MV reference block as a reference MV for predicting the MV of the current block.

Description

[Technical Field]

[0001] Incorporation by Reference This application is based on and claims the benefit of priority to U.S. Provisional Application No. 63 / 408,569, filed September 21, 2022, which is incorporated herein by reference in its entirety. This application is further based on and claims the benefit of priority to U.S. Nonprovisional Patent Application No. 18 / 122,823, filed March 17, 2023, which is incorporated herein by reference in its entirety.

[0002] This disclosure describes a set of advanced video coding techniques. More particularly, the disclosed techniques relate to implementing and enhancing motion vector prediction based on sub-block motion vectors. [Background technology]

[0003]

[0003] This background discussion provided herein is intended to generally present the context for the present disclosure. Work of the inventors currently identified to the extent that their work is described in this background section, and aspects of the discussion that may not otherwise be considered prior art at the time of filing this application, are not expressly or implicitly admitted to be prior art to the present disclosure.

[0004] Video encoding and decoding can be performed using inter-picture prediction with motion compensation. Uncompressed digital video can include a series of pictures, each having spatial dimensions of, for example, 1920 x 1080 luma samples and associated full or subsampled chroma samples. The series of pictures can have a fixed or variable picture rate (alternatively called a frame rate), for example, 60 pictures per second, or 60 frames per second. Uncompressed video has inherent bitrate requirements for streaming or data processing. For example, a video with a pixel resolution of 1920 x 1080, a frame rate of 60 frames / second, and chroma subsampling of 4:2:0 with 8 bits per pixel per color channel requires a bandwidth of nearly 1.5 Gbit / s. One hour of such video requires more than 600 GByte of storage space.

[0005]

[0005] One goal of video encoding and decoding can be the reduction of redundancy in an uncompressed input video signal through compression. Compression can, in some cases, help reduce the aforementioned bandwidth and / or storage space requirements by more than two orders of magnitude. Both lossless and lossy compression, as well as combinations thereof, can be employed. Lossless compression refers to a technique in which an exact copy of the original signal can be reconstructed from the compressed original signal through a decoding process. Lossy compression refers to an encoding / decoding process in which the original video information is not fully preserved during encoding and therefore cannot be fully restored during decoding. When using lossy compression, the reconstructed signal may not be identical to the original signal, but even with some information loss, the distortion between the original and reconstructed signal is small enough to make the reconstructed signal useful for the intended application. In the case of video, lossy compression is widely adopted in many applications. The amount of tolerable distortion depends on the application. For example, users of certain consumer video streaming applications can tolerate greater distortion than users of movie or television broadcasting applications. The compression ratio achievable by a particular encoding algorithm can be selected or adjusted to reflect different distortion tolerances; generally, the more distortion that can be tolerated, the more lossy and the greater the compression ratio that is possible with an encoding algorithm.

[0006] Video encoders and decoders can utilize techniques from several broad categories and processes, including, for example, motion compensation, Fourier transform, quantization, and entropy coding.

[0007]

[0007] Video codec technology can include a technique known as intra-coding. In intra-coding, sample values ​​are represented without reference to samples or other data from previously reconstructed reference pictures. In some video codecs, a picture is spatially subdivided into blocks of samples. When all blocks of samples are coded in intra mode, the picture can be called an intra-picture. Intra-pictures and their derivatives, such as independent decoder refresh pictures, can be used to reset the decoder state and can therefore be used as the first picture in a coded video bitstream and video session, or as a still image. The samples of the block after intra-prediction can then be transformed into the frequency domain, and the transform coefficients thus generated can be quantized before entropy coding. Intra-prediction refers to a technique that minimizes sample values ​​in the pre-transform domain. In some cases, the smaller the DC value and the smaller the AC coefficients after the transformation, the fewer bits are required to represent the block after entropy coding at a given quantization step size.

[0008]

[0008] Traditional intra-coding, such as that known from the MPEG-2 generation of encoding techniques, does not use intra-prediction. However, some newer video compression techniques include techniques that attempt to encode / decode blocks based on surrounding sample data and / or metadata obtained during the encoding and / or decoding of spatially neighboring blocks and that precede the intra-coded or decoded data block in decoding order. Such techniques are hereinafter referred to as "intra-prediction" techniques. It should be noted that, at least in some cases, intra-prediction uses only reference data from the current picture being reconstructed, and does not use reference data from other reference pictures.

[0009]

[0009] There can be many different forms of intra prediction. When two or more such techniques are available within a given video coding technique, the technique in use can be referred to as an intra prediction mode. In a particular codec, one or more intra prediction modes may be provided. In certain cases, a mode can have sub-modes and / or can be associated with various parameters, and the mode / sub-mode information and the intra coding parameters of a video block can be coded separately or collectively included in a mode codeword. Which codeword to use for a given mode, sub-mode, and / or parameter combination can also affect coding efficiency gains through intra prediction, and therefore, entropy coding techniques can be used to turn the codeword into a bitstream.

[0010]

[0010] Certain intra-prediction modes were introduced in H.264, improved in H.265, and further refined in newer coding techniques, such as the joint exploration model (JEM), versatile video coding (VVC), and benchmark set (BMS). In general, for intra-prediction, a predictor block can be formed using available neighboring sample values. For example, available values ​​of a specific set of neighboring samples along a specific direction and / or line may be copied into the predictor block. A reference to the direction in use can be coded in the bitstream or may itself be predicted.

[0011] 1A, a subset of nine predictor directions specified by the 33 possible intra predictor directions of H.265 (corresponding to the 33 angle modes of the 35 intra modes specified in H.265) is depicted in the lower right. The point (101) where the arrows converge represents the sample to be predicted. The arrows represent the direction in which neighboring samples are used to predict the sample at 101. For example, arrow (102) indicates that sample (101) is predicted from one or more neighboring samples located to the upper right and at an angle of 45 degrees from the horizontal. Similarly, arrow (103) indicates that sample (101) is predicted from one or more neighboring samples located to the lower left of sample (101) and at an angle of 22.5 degrees from the horizontal.

[0012]

[0012] With further reference to FIG. 1A, a square block (104) of 4x4 samples (indicated by a thick dotted line) is depicted at the upper left. The square block (104) includes 16 samples, each labeled with an "S," the block's position in the Y dimension (e.g., row index), and the block's position in the X dimension (e.g., column index). For example, sample S21 is the second sample (from the top) in the Y dimension and the first sample (from the left) in the X dimension. Similarly, sample S44 is the fourth sample in block (104) in both the Y and X dimensions. Because the block is 4x4 samples in size, S44 is at the bottom right. An example reference sample is also shown following a similar numbering scheme. The reference sample is labeled R, the sample's Y position (e.g., row index) and X position (column index) relative to block (104). In both H.264 and H.265, prediction samples that are adjacent to the block being reconstructed are used.

[0013]

[0013] Intra-picture prediction of block 104 may begin by copying reference sample values ​​from neighboring samples according to the signaled prediction direction. For example, assume that the coded video bitstream includes signaling indicating the prediction direction of the arrow (102) for this block 104—that is, the sample is predicted from one or more prediction samples located to the upper right and at a 45-degree angle from the horizontal. In such a case, samples S41, S32, S23, and S14 are predicted from the same reference sample R05. Thus, sample S44 is predicted from reference sample R08.

[0014] In certain cases, especially when the direction is not evenly divisible by 45 degrees, the values ​​of multiple reference samples may be combined, for example through interpolation, to calculate the reference sample.

[0015] The number of possible directions has increased as video coding technology continues to develop. In H.264 (2003), for example, nine different directions are available for intra prediction. This increased to 33 in H.265 (2013), and JEM / VVC / BMS can support up to 65 directions as of the time of this disclosure. Empirical studies have been conducted to help identify optimal intra prediction directions, and specific techniques in entropy coding may be used to encode these optimal directions with a small number of bits, accepting a specific bit penalty for the direction. Furthermore, the direction itself is sometimes predictable from neighboring directions used in intra prediction of decoded neighboring blocks.

[0016]

[0016] Figure 1B shows a schematic diagram (180) depicting 65 intra prediction directions according to JEM to illustrate the increasing number of prediction directions in various encoding techniques developed over time.

[0017]

[0017] The manner in which bits representing intra-prediction directions are mapped to prediction directions in the encoded video bitstream may vary from one video coding technique to another, and may range, for example, from a simple direct mapping of prediction directions to codewords, to intra-prediction modes, to complex adaptive schemes involving most probable modes, and similar techniques. However, in all cases, there may be certain directions for intra-prediction that are statistically less likely to occur in the video content than certain other directions. Because the goal of video compression is to reduce redundancy, in a well-designed video coding technique, these less likely directions may be represented with a greater number of bits than more likely directions.

[0018]

[0018] Inter-picture prediction or inter-prediction may be based on motion compensation. In motion compensation, sample data (reference picture) from a previously reconstructed picture or part thereof may be used for predicting a newly reconstructed picture or part of a picture (e.g., a block) after being spatially shifted in a direction indicated by a motion vector (hereinafter, MV). In some cases, the reference picture may be the same as the picture currently being reconstructed. The MV may have two dimensions, X and Y, or three dimensions, with the third dimension indicating the reference picture in use (similar to a temporal dimension).

[0019]

[0019] In some video compression techniques, the current MV applicable to a particular area of ​​sample data can be predicted from other MVs, such as other MVs related to other areas of sample data that are spatially adjacent to the area being reconstructed and precede the current MV in decoding order. Doing so can substantially reduce the total amount of data required to encode the MV by relying on the elimination of redundancy in correlated MVs, thereby improving compression efficiency. For example, when encoding an input video signal derived from a camera (known as raw video), there is a statistical possibility that an area larger than the area to which a single MV is applicable moves in a similar direction in the video sequence. Therefore, in some cases, MV prediction can work effectively because it can be predicted using similar motion vectors derived from MVs of neighboring areas. This makes the actual MV of a given area similar or identical to the MV predicted from surrounding MVs. Then, after entropy encoding, such an MV may be represented with fewer bits than would be used if the MV were encoded directly rather than predicted from neighboring MVs. In some cases, the MV prediction may be an example of lossless compression of a signal (i.e., the MV) derived from the original signal (i.e., the sample stream). In other cases, the MV prediction itself may be lossy, for example, due to rounding errors when computing the predictor from several surrounding MVs.

[0020]

[0020] H.265 / HEVC (ITU-T Recommendation H.265, "High Efficiency Video Coding", December 2016) describes various MV prediction mechanisms. Among the many MV prediction mechanisms specified by H.265, a technique hereinafter referred to as "spatial merging" will be described below.

[0021]

[0021] Specifically, referring to Figure 2, a current block (201) contains samples that the encoder found during the motion search process to be predictable from a spatially shifted previous block of the same size. Instead of directly encoding this MV, the MV can be derived from metadata associated with one or more reference pictures, such as the most recent reference picture (in decoding order), using the MV associated with any one of five surrounding samples denoted A0, A1, and B0, B1, B2 (202 to 206, respectively). In H.265, MV prediction can use predictors from the same reference picture used by neighboring blocks. Summary of the Invention [Means for solving the problem]

[0022] This disclosure describes various embodiments of methods, apparatus, and computer-readable storage media for motion vector prediction based on sub-block motion vectors.

[0023] According to one aspect, an embodiment of the present disclosure provides a method for decoding a current block of a current frame in a coded video bitstream. The method includes receiving, by a device, a coded video bitstream. The device includes a memory storing instructions and a processor in communication with the memory. The method further includes: determining, by the device based on the coded video bitstream, a current block of the current frame to be inter-predicted by at least one reference block of at least one reference frame; determining, by the device based on the coded video bitstream, motion vectors (MVs) of the current block corresponding to each of the at least one reference block, where the MV of the current block will be predicted by the reference MV; and identifying, by the device, a sub-block MV of the MV reference block as a reference MV for predicting the MV of the current block. The sub-block MV of the MV reference block is stored for use as a motion vector predictor (MVP).

[0024] According to another aspect, embodiments of the present disclosure provide an apparatus for processing a current block of a current frame in an encoded video bitstream. The apparatus includes a memory storing instructions and a processor in communication with the memory. When the processor executes the instructions, the processor is configured to cause the apparatus to perform the above-described method for video decoding and / or encoding.

[0025]

[0025] In another aspect, an embodiment of the present disclosure provides a non-transitory computer-readable medium storing instructions that, when executed by a computer for video decoding and / or encoding, cause the computer to implement the above-mentioned method for video decoding and / or encoding.

[0026]

[0026] These and other aspects and implementations thereof are described in more detail in the drawings, description, and claims.

[0027]

[0027] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings. [Brief explanation of the drawings]

[0028] [Figure 1A] FIG. 10 is a schematic diagram of an exemplary subset of intra-prediction direction modes. [Figure 1B]

[0029] FIG. 1 is a diagram of exemplary intra-prediction directions. [Figure 2]

[0030] FIG. 1 is a schematic diagram of a current block and its surrounding spatial merge candidates for motion vector prediction in one example. [Figure 3]

[0031] FIG. 3 is a simplified block diagram schematic of a communication system (300) according to an example embodiment. [Figure 4]

[0032] FIG. 4 is a simplified block diagram schematic of a communication system (400) according to an example embodiment. [Figure 5]

[0033] FIG. 2 is a schematic diagram of a simplified block diagram of a video decoder in accordance with an example embodiment; [Figure 6]

[0034] FIG. 2 is a schematic diagram of a simplified block diagram of a video encoder in accordance with an example embodiment; [Figure 7]

[0035] FIG. 10 is a block diagram of a video encoder according to another example embodiment. [Figure 8]

[0036] FIG. 10 is a block diagram of a video decoder according to another example embodiment. [Figure 9]

[0037] FIG. 1 is a diagram of a coding block partitioning scheme according to an example embodiment of the present disclosure. [Figure 10]

[0038] FIG. 10 is a diagram of another coding block partitioning scheme according to an example embodiment of the present disclosure. [Figure 11]

[0039] FIG. 10 is a diagram of another coding block partitioning scheme according to an example embodiment of the present disclosure. [Figure 12]

[0040] FIG. 10 is a diagram of an example partitioning of a base block into coding blocks according to an example partitioning scheme. [Figure 13]

[0041] FIG. 1 is a diagram of an example ternary partitioning scheme. [Figure 14]

[0042] FIG. 1 is a diagram of an example quadtree / binary tree coding block partitioning scheme. [Figure 15]

[0043] 1 is a diagram of a scheme for partitioning a coding block into multiple transform blocks and a coding order of the transform blocks according to an example embodiment of this disclosure. [Figure 16]

[0044] FIG. 10 is a diagram of another scheme for partitioning a coding block into multiple transform blocks and the coding order of the transform blocks according to an example embodiment of this disclosure. [Figure 17]

[0045] FIG. 10 is a diagram of another scheme for partitioning a coding block into multiple transform blocks according to an example embodiment of this disclosure. [Figure 18]

[0046] FIG. 1 is a diagram of the concept of hybrid inter prediction according to an example embodiment of this disclosure. [Figure 19]

[0047] FIG. 1 is a diagram of the concept of temporal interpolation prediction according to an example embodiment of the present disclosure. [Figure 20]

[0048] FIG. 2 is a diagram of an example implementation of motion samples used to derive model parameters for local warped motion prediction according to an embodiment of the present disclosure. [Figure 21]

[0049] FIG. 10 is a diagram of the concept of warped motion mode according to an example embodiment of the present disclosure. [Figure 22]

[0050] 1 is a flowchart of a method according to an example embodiment of the present disclosure. [Figure 23]

[0051] FIG. 1 is a schematic diagram of a computer system according to an example embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0029]

[0052] The present invention will now be described in detail below with reference to the accompanying drawings, which form a part hereof, and in which are shown by way of illustration specific embodiments. It should be noted, however, that the present invention may be embodied in many different forms, and thus, the subject matter included or claimed is not intended to be construed as limited to any of the embodiments that will be described below. It should also be noted that the present invention may be embodied as a method, device, component, or system. Thus, embodiments of the present invention may be in the form of, for example, hardware, software, firmware, or any combination thereof.

[0030]

[0053] Throughout this specification and claims, terms may have nuances in meaning that are suggested or implied in context beyond their explicitly stated meaning. The phrases "in one embodiment" or "in some embodiments" as used herein do not necessarily refer to the same embodiment, and the phrases "in another embodiment" or "in other embodiments" as used herein do not necessarily refer to different embodiments. Similarly, the phrases "in one implementation" or "in some implementations" as used herein do not necessarily refer to the same implementation, and the phrases "in another implementation" or "in other implementations" as used herein do not necessarily refer to different implementations. For example, the claimed subject matter is intended to include, in whole or in part, combinations of example embodiments / implementations.

[0031]

[0054] Generally, terminology may be understood, at least in part, from the usage in context. For example, as used herein, terms such as "and," "or," or "and / or" may include various meanings that may depend, at least in part, on the context in which such terms are used. Typically, "or" when used to relate a list such as A, B, or C is intended to refer to A, B, and C, which is used here in an inclusive sense, as well as A, B, or C, which is used here in an exclusive sense. Furthermore, as used herein, the terms "one or more" or "at least one" may be used to describe any feature, structure, or characteristic in a singular sense, or may be used to describe a combination of features, structures, or characteristics in a plural sense, depending, at least in part, on the context. Similarly, terms such as "a," "an," or "the" may be understood to convey a singular or plural sense, again depending, at least in part, on the context. Additionally, the terms "based on" or "determined by" may be understood as not necessarily intended to convey an exclusive set of elements, but instead allow for the presence of additional elements not necessarily expressly recited, again depending at least in part on the context.

[0032]

[0055] FIG. 3 illustrates a simplified block diagram of a communication system (300) according to an embodiment of the present disclosure. The communication system (300) includes multiple terminal devices capable of communicating with each other, e.g., via a network (350). For example, the communication system (300) includes a first pair of terminal devices (310) and (320) interconnected via the network (350). In the example of FIG. 3, the first pair of terminal devices (310) and (320) may perform unidirectional transmission of data. For example, the terminal device (310) may encode video data (e.g., a stream of video pictures captured by the terminal device (310)) for transmission to another terminal device (320) via the network (350). The encoded video data may be transmitted in the form of one or more coded video bitstreams. The terminal device (320) may receive the coded video data from the network (350), decode the coded video data to reconstruct the video pictures, and display the video pictures according to the reconstructed video data. One-way data transmission may be implemented in media serving applications, etc.

[0033]

[0056] In another example, the communication system (300) includes a second pair of terminal devices (330) and (340) that implement bidirectional transmission of encoded video data, such as may be implemented during a video conferencing application. For bidirectional transmission of data, in one example, each of the terminal devices (330) and (340) may encode video data (e.g., of a stream of video pictures captured by the terminal device) for transmission to the other of the terminal devices (330) and (340) over the network (350). Each of the terminal devices (330) and (340) may also receive encoded video data transmitted by the other of the terminal devices (330) and (340), decode the encoded video data to reconstruct the video pictures, and display the video pictures on an accessible display device in response to the reconstructed video data.

[0034]

[0057] In the example of FIG. 3 , the terminal devices (310), (320), (330), and (340) may be implemented as a server, a personal computer, and a smartphone, although the applicability of the underlying principles of the present disclosure may not be so limited. Embodiments of the present disclosure may also be implemented in desktop computers, laptop computers, tablet computers, media players, wearable computers, dedicated videoconferencing equipment, and the like. The network (350) represents any number or type of network that conveys coded video data between the terminal devices (310), (320), (330), and (340), including, for example, wireline and / or wireless communication networks. The communication network (350) may exchange data over circuit-switched, packet-switched, and / or other types of channels. Exemplary networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For the purposes of this discussion, the architecture and topology of the network (350) may not be important to the operation of the present disclosure unless explicitly described herein.

[0035]

[0058] 4 illustrates the arrangement of a video encoder and a video decoder in a video streaming environment as an example of application of the disclosed subject matter. The disclosed subject matter may be equally applicable to other video applications including, for example, video conferencing, digital TV broadcasting, gaming, virtual reality, storage of compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0036]

[0059] The video streaming system may include a video capture subsystem (413), which may include a video source (401), such as a digital camera, for producing a stream of uncompressed video pictures or images (402). In one example, the stream of video pictures (402) includes samples recorded by the digital camera of the video source 401. The stream of video pictures (402), depicted as a bold line to emphasize its larger data volume when compared to the encoded video data (404) (or coded video bitstream), can be processed by an electronic device (420) that includes a video encoder (403) coupled to the video source (401). The video encoder (403) may include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter, as described in more detail below. The encoded video data (404) (or encoded video bitstream (404)), depicted as a thin line to emphasize its smaller data volume when compared to the stream of uncompressed video pictures (402), can be stored on the streaming server (405) for future use or directly on a downstream video device (not shown). One or more streaming client subsystems, such as the client subsystems (406) and (408) of FIG. 4, can access the streaming server (405) to retrieve copies (407) and (409) of the encoded video data (404). The client subsystem (406) can include a video decoder (410), for example, within the electronic device (430). The video decoder (410) decodes the incoming copy of the encoded video data (407) to produce an outgoing stream of video pictures (411) that is uncompressed and ready to be rendered on a display (412) (e.g., a display screen) or other rendering device (not shown). Video decoder 410 may be configured to perform some or all of the various functions described in this disclosure.In some streaming systems, the encoded video data (404), (407), and (409) (e.g., video bitstreams) can be encoded according to a particular video encoding / compression standard. Examples of these standards include ITU-T Recommendation H.265. In one example, a developing video encoding standard is informally known as Versatile Video Coding (VVC). The disclosed subject matter may also be used in the context of VVC, as well as other video encoding standards.

[0037]

[0060] It should be noted that the electronic devices (420) and (430) may include other components (not shown). For example, the electronic device (420) may include a video decoder (not shown), and the electronic device (430) may also include a video encoder (not shown).

[0038]

[0061] 5 shows a block diagram of a video decoder (510) according to any of the embodiments of the present disclosure described below. The video decoder (510) can be included in an electronic device (530). The electronic device (530) can include a receiver (531) (e.g., receiving circuitry). The video decoder (510) can be used in place of the video decoder (410) in the example of FIG. 4.

[0039]

[0062] The receiver (531) may receive one or more coded video sequences to be decoded by the video decoder (510). In the same or another embodiment, one coded video sequence may be decoded at a time, with the decoding of each coded video sequence being independent of other coded video sequences. Each video sequence may be associated with multiple video frames or images. The coded video sequences may be received from a channel (501), which may be a hardware or software link to a storage device that stored the coded video data or a streaming source that transmitted the coded video data. The receiver (531) may receive the coded video data with other data, such as coded audio data and / or auxiliary data streams, and the coded video data may be forwarded to its respective processing circuitry (not shown). The receiver (531) may separate the coded video sequences from other data. A buffer memory (515) may be located between the receiver (531) and the entropy decoder / parser (520) (hereinafter "parser (520)") to prevent network jitter. In certain applications, the buffer memory (515) may be implemented as part of the video decoder (510). In other applications, the buffer memory (515) may be external to and separate from the video decoder (510) (not shown). In still other applications, there may be a buffer memory (not shown) external to the video decoder (510), for example, to prevent network jitter, or another additional buffer memory (515) internal to the video decoder (510), for example, to handle playback timing. When the receiver (531) is receiving data from a store / forward device with sufficient bandwidth and controllability or from an isosynchronous network, the buffer memory (515) may not be needed or may be small.For use with best-effort packet networks such as the Internet, a sufficiently sized buffer memory (515) may be required, and may be significantly larger. Such a buffer memory may be provided in an adaptable size and may be implemented at least in part in an operating system or similar element (not shown) external to the video decoder (510).

[0040]

[0063] The video decoder (510) may include a parser (520) for reconstructing symbols (521) from the encoded video sequence. These symbol categories include information used to manage the operation of the video decoder (510) and possibly information for controlling a rendering device, such as a display (512) (e.g., a display screen), which may or may not be an integral part of the electronic device (530), but may be coupled to the electronic device (530) as shown in FIG. 5. The control information for the rendering device may be in the form of a Supplemental Enhancement Information (SEI) message or a Video Usability Information (VUI) parameter set fragment (not shown). The parser (520) may parse / entropy decode the encoded video sequence received by the parser (520). The entropy coding of the coded video sequence may be in accordance with a video coding technique or standard and may follow various principles, including variable length coding, Huffman coding, context-independent or non-context-dependent arithmetic coding, etc. The parser (520) may extract a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder from the coded video sequence based on at least one parameter corresponding to the subgroup. The subgroup may include a group of pictures (GOP), a picture, a tile, a slice, a macroblock, a coding unit (CU), a block, a transform unit (TU), a prediction unit (PU), etc. Additionally, the parser (520) may extract coded video sequence information, such as transform coefficients (e.g., Fourier transform coefficients), quantization parameter values, motion vectors, etc.

[0041]

[0064] The parser (520) may perform entropy decoding / parsing operations on the video sequence received from the buffer memory (515) to produce symbols (521).

[0042]

[0065] The reconstruction of the symbols (521) may require several different processing or functional units, depending on the type of coded video picture or portion thereof (inter and intra pictures, inter and intra blocks, etc.), as well as other factors. The units required, and how they are required, may be controlled by subgroup control information parsed from the coded video sequence by the parser (520). The flow of such subgroup control information between the parser (520) and the following processing or functional units is not depicted for the sake of simplicity.

[0043]

[0066] In addition to the functional blocks already mentioned, the video decoder (510) can be conceptually subdivided into several functional units, as described below. In a practical implementation operating under commercial constraints, many of these functional units will interact closely with each other and may be at least partially integrated with each other. However, to clearly explain the various functions of the disclosed subject matter, a conceptual subdivision into functional units will be adopted in the following disclosure.

[0044]

[0067] The first unit may include a scalar / inverse transform unit (551). The scalar / inverse transform unit (551) may receive quantized transform coefficients and control information from the parser (520), including information indicating which type of inverse transform to use, block size, quantization factors / parameters, quantization scaling matrices, and lies as symbols (521). The scalar / inverse transform unit (551) may output blocks having sample values ​​that can be input to an aggregator (555).

[0045]

[0068] In some cases, the output samples of the scaler / inverse transform (551) may be for intra-coded blocks, i.e., blocks that can use prediction information from a previously reconstructed portion of the current picture rather than using prediction information from a previously reconstructed picture. Such prediction information can be provided by an intra-picture prediction unit (552). In some cases, the intra-picture prediction unit (552) may generate blocks of the same size and shape as the block being reconstructed using surrounding block information already reconstructed and stored in the current picture buffer (558). The current picture buffer (558), for example, buffers a partially reconstructed and / or fully reconstructed current picture. In some implementations, the aggregator (555) may add, on a sample-by-sample basis, the prediction information generated by the intra-prediction unit (552) to the output sample information as provided by the scaler / inverse transform unit (551).

[0046]

[0069] In other cases, the output samples of the scalar / inverse transform unit (551) may be for an inter-coded and potentially motion-compensated block. In such cases, the motion-compensated prediction unit (553) may access the reference picture memory (557) to fetch samples used for inter-picture prediction. After motion-compensating the fetched samples according to the symbols (521) for the block, these samples may be added to the output of the scalar / inverse transform unit (551) by the aggregator (555) to generate the output sample information (the output of unit 551 may be referred to as residual samples or residual signals). The addresses in the reference picture memory (557) from which the motion-compensated prediction unit (553) fetches the prediction samples may be controlled by a motion vector, available to the motion-compensated prediction unit (553) in the form of a symbol (521), which may have, for example, X and Y components (shift) and a reference picture component (time). Motion compensation may further include interpolation of sample values ​​as fetched from the reference picture memory (557) when sub-sample accurate motion vectors are in use, may be further associated with a motion vector prediction mechanism, etc.

[0047]

[0070] The output samples of the aggregator (555) may be subjected to various loop filtering techniques in the loop filter unit (556). Video compression techniques can include in-loop filter techniques controlled by parameters contained in the coded video sequence (also called the coded video bitstream) and made available to the loop filter unit (556) as symbols (521) from the parser (520), but can also include those that respond to meta-information obtained during decoding of previous portions (in decoding order) of the coded picture or coded video sequence, and those that respond to previously reconstructed, loop-filtered sample values. As described in more detail below, several types of loop filters may be included as part of the loop filter unit 556, in various orders.

[0048]

[0071] The output of the loop filter unit (556) may be a sample stream that can be output to a rendering device (512) or stored in a reference picture memory (557) for use during future inter-picture prediction.

[0049]

[0072] A particular coded picture can be used as a reference picture for future inter-picture prediction as soon as it is fully reconstructed. For example, once the coded picture corresponding to the current picture is fully reconstructed and the coded picture is identified as a reference picture (e.g., by the parser (520)), the current picture buffer (558) can become part of the reference picture memory (557), and an unused current picture buffer can be reallocated before beginning reconstruction of the next coded picture.

[0050]

[0073] The video decoder (510) may perform decoding operations according to a predetermined video compression technique adopted in a standard such as ITU-T Recommendation H.265. An encoded video sequence may conform to the syntax specified by the video compression technique or standard used, in the sense that the encoded video sequence conforms to both the syntax of the video compression technique or standard and the profile as documented in the video compression technique or standard. Specifically, a profile may select a specific tool from all tools available in the video compression technique or standard as the only tool available for use under this profile. To be standard-compliant, the complexity of the encoded video sequence may be within a range as defined by the level of the video compression technique or standard. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstructed sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. In some cases, the limits set by the levels can be further restricted through the Hypothetical Reference Decoder (HRD) specification and metadata for HRD buffer management signaled in the coded video sequence.

[0051]

[0074] In some example embodiments, the receiver (531) may receive additional (redundant) data along with the encoded video. The additional data may be included as part of the encoded video sequence. The additional data may be used by the video decoder (510) to properly decode the data and / or to more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0052]

[0075] 6 shows a block diagram of a video encoder (603) according to an example embodiment of the present disclosure. The video encoder (603) may be included in an electronic device (620). The electronic device (620) may further include a transmitter (640) (e.g., transmission circuitry). The video encoder (603) can be used in place of the video encoder (403) in the example of FIG. 4.

[0053]

[0076] The video encoder (603) may receive video samples from a video source (601) (not part of the electronic device (620) in the example of FIG. 6) that may capture video images to be encoded by the video encoder (603). In another example, the video source (601) may be provided as part of the electronic device (620).

[0054]

[0077] The video source (601) may provide a source video sequence to be encoded by the video encoder (603) in the form of a digital video sample stream, which can be of any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, etc.), any color space (e.g., BT.601 YCrCb, RGB, XYZ, etc.), and any suitable sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media serving system, the video source (601) may be a storage device capable of storing previously prepared video. In a video conferencing system, the video source (601) may be a camera that captures local image information as a video sequence. Video data may be provided as multiple individual pictures or images that convey motion when viewed in sequence. The picture itself may be organized as a spatial array of pixels, each of which can have one or more samples, depending on the sampling structure, color space, etc., used. Those skilled in the art can readily understand the relationship between pixels and samples. The following description focuses on samples.

[0055]

[0078] According to some example embodiments, the video encoder (603) may encode and compress pictures of a source video sequence into an encoded video sequence (643) in real time or under any other time constraints, as needed by the application. Enforcing an appropriate encoding speed constitutes one function of the controller (650). In some embodiments, the controller (650) may be functionally coupled to and control other functional units, such as those described below. The coupling is not depicted for simplicity. Parameters set by the controller (650) may include rate control-related parameters (picture skip, quantizer, lambda value for rate-distortion optimization techniques, etc.), picture size, group-of-picture (GOP) layout, maximum motion vector search range, etc. The controller (650) can be configured to have other appropriate functions for the video encoder (603) optimized for a particular system design.

[0056]

[0079] In some example embodiments, the video encoder (603) may be configured to operate within an encoding loop. As an overly simplified explanation, in one example, the encoding loop may include a source coder (630) (e.g., responsible for producing symbols, such as a symbol stream, based on an input picture to be encoded and a reference picture) and a (local) decoder (633) embedded in the video encoder (603). The decoder (633) reconstructs the symbols to produce sample data in a similar manner as a (remote) decoder would have produced it even if the embedded decoder 633 had processed the encoded video stream by the source coder 630 without entropy encoding (since any compression between the symbols and the encoded video bitstream in entropy encoding may be lossless in the video compression techniques contemplated by the disclosed subject matter). The reconstructed sample stream (sample data) is input to a reference picture memory (634). Because decoding the symbol stream results in bit-exact results independent of the decoder location (local or remote), the contents of the reference picture memory (634) are also bit-exact between the local and remote encoders. In other words, the predictive portion of the encoder "sees" the reference picture sampling exactly the same sample values ​​that the decoder "sees" when using prediction during decoding. This basic principle of reference picture synchrony (and the resulting drift when synchrony cannot be maintained, e.g., due to channel errors) is used to improve coding quality.

[0057]

[0080] The operation of the "local" decoder (633) can be the same as a "remote" decoder, such as the video decoder (510) already described in detail above in conjunction with Figure 5. Furthermore, and briefly referring to Figure 5, however, the entropy decoding portion of the video decoder (510), including the buffer memory (515) and parser (520), need not be entirely implemented in the local decoder (633) within the encoder, because symbols are available and the encoding / decoding of the symbols into an encoded video sequence by the entropy coder (645) and parser (520) can be lossless.

[0058]

[0081] An observation that can be made at this point is that any decoder technology other than parsing / entropy decoding, which may only be present in the decoder, may not necessarily be present in substantially identical functional form in the corresponding encoder. For this reason, the disclosed subject matter may sometimes focus on decoder operations in cooperation with the decoding portion of the encoder. Therefore, a description of the encoder technology may be omitted, as it is the reverse of the decoder technology, which is described generically. Only in certain areas or aspects is a more detailed description of the encoder provided below.

[0059]

[0082] During operation in some example implementations, the source coder (630) may perform motion-compensated predictive coding, which predictively codes an input picture with reference to one or more previously coded pictures from a video sequence designated as "reference pictures." In this manner, the coding engine (632) codes color channel differences (or residuals) between pixel blocks of the input picture and pixel blocks of reference pictures that may be selected as prediction references for the input picture. The terms "residual" and its adjective form "residual" are sometimes used interchangeably.

[0060]

[0083] The local video decoder (633) may decode the coded video data of pictures that may be designated as reference pictures based on symbols produced by the source coder (630). The operation of the coding engine (632) may advantageously be a lossy process. When the coded video data may be decoded in a video decoder (not shown in FIG. 6), the reconstructed video sequence may be a copy of the source video sequence, typically containing some errors. The local video decoder (633) may replicate the decoding process that may be performed on reference pictures by the video decoder and store the reconstructed reference pictures in a reference picture cache (634). In this way, the video encoder (603) may locally store copies of reconstructed reference pictures that have content in common with reconstructed reference pictures that will be retrieved by a far-end (remote) video decoder (without transmission errors).

[0061]

[0084] The predictor (635) may perform prediction searches for the coding engine (632). That is, to encode a new picture, the predictor (635) may search the reference picture memory (634) for sample data (as candidate reference pixel blocks) or for specific metadata, such as reference picture motion vectors, block shapes, etc., that may serve as appropriate prediction references for the new picture. The predictor (635) may operate on a sample block-by-pixel block basis to find appropriate prediction references. In some cases, as determined by search results obtained by the predictor (635), the input picture may have prediction references created from multiple reference pictures stored in the reference picture memory (634).

[0062]

[0085] The controller (650) may manage the coding operations of the source coder (630), including, for example, setting the parameters and subgroup parameters used to encode the video data.

[0063]

[0086] The output of all the aforementioned functional units may undergo entropy coding in an entropy coder (645), which transforms the symbols as produced by the various functional units into an encoded video sequence by lossless compression of the symbols through techniques such as Huffman coding, variable length coding, or arithmetic coding.

[0064]

[0087] The transmitter (640) may buffer the encoded video sequence as produced by the entropy coder (645) in preparation for transmission over a communication channel (660), which may be a hardware / software link to a storage device that will store the encoded video data. The transmitter (640) may merge the encoded video data from the video coder (603) with other data to be transmitted, such as encoded audio data and / or auxiliary data streams (sources not shown).

[0065]

[0088] The controller (650) may manage the operation of the video encoder (603). During encoding, the controller (650) may assign a particular coded picture type to each coded picture, which may affect the coding technique that may be applied to the respective picture. For example, pictures may often be assigned as one of the following picture types:

[0066]

[0089] An intra-picture (I-picture) may be coded and decoded without using any other pictures in a sequence as a source of prediction. Some video codecs allow different types of intra-pictures, including, for example, Independent Decoder Refresh ("IDR") pictures. Those skilled in the art are aware of these variations of I-pictures and their respective uses and characteristics.

[0067]

[0090] A predicted picture (P picture) may be coded and decoded using intra- or inter-prediction, using at most one motion vector and reference index to predict the sample values ​​of each block.

[0068]

[0091] Bidirectionally predicted pictures (B-pictures) may be coded and decoded using intra- or inter-prediction, using at most two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, multiple predicted pictures can use more than two reference pictures and associated metadata for the reconstruction of a single block.

[0069]

[0092] A source picture may generally be spatially subdivided into multiple sample coding blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples each) and coded block by block. Blocks may be predictively coded with reference to other (already coded) blocks as determined by the coding assignment applied to the block's respective picture. For example, blocks of an I-picture may be non-predictively coded or predictively coded with reference to previously coded blocks of the same picture (spatial prediction or intra-prediction). Pixel blocks of a P-picture may be predictively coded via spatial prediction or temporal prediction with reference to one previously coded reference picture. Blocks of a B-picture may be predictively coded via spatial prediction or temporal prediction with reference to one or two previously coded reference pictures. Source pictures or intermediate processed pictures may also be subdivided into other types of blocks for other purposes. The division of coding blocks and other types of blocks may or may not follow the same manner as described in more detail below.

[0070]

[0093] The video encoder (603) may perform encoding operations according to a given video encoding technique or standard, such as ITU-T Recommendation H.265. During its operation, the video encoder (603) may perform various compression operations, including predictive encoding operations that exploit temporal and spatial redundancies in the input video sequence. Thus, the encoded video data may conform to a syntax specified by the video encoding technique or standard used.

[0071]

[0094] In some example embodiments, the transmitter (640) may transmit additional data along with the encoded video. The source coder (630) may include data such as portions of the encoded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

[0072]

[0095] Video may be captured as a time sequence of multiple source pictures (video pictures). Intra-picture prediction (often abbreviated to intra-prediction) exploits spatial correlations within a given picture, while inter-picture prediction exploits temporal or other correlations between pictures. For example, a particular picture being encoded / decoded, called the current picture, may be partitioned into blocks. When a block in the current picture resembles a reference block in a previously coded and still buffered reference picture in the video, it may be coded by a vector, called a motion vector. A motion vector points to a reference block within a reference picture and may have a third dimension that identifies the reference picture in cases where multiple reference pictures are in use.

[0073]

[0096] In some example embodiments, bi-prediction techniques can be used for inter-picture prediction. Such bi-prediction techniques use two reference pictures, such as a first reference picture and a second reference picture, that both advance the current picture in the video in decoding order (but may be in the past or future, respectively, in display order). A block in the current picture can be coded by a first motion vector that points to a first reference block in the first reference picture and a second motion vector that points to a second reference block in the second reference picture. A block can be jointly predicted by combining the first reference block and the second reference block.

[0074]

[0097] Furthermore, to improve coding efficiency, merge mode techniques may be used during inter-picture prediction.

[0075]

[0098] According to some example embodiments of the present disclosure, prediction, such as inter-picture prediction and intra-picture prediction, is performed in units of blocks. For example, pictures in a sequence of video pictures are partitioned into coding tree units (CTUs) for compression, and the CTUs within a picture may have the same size, such as 128x128 pixels, 64x64 pixels, 32x32 pixels, or 16x16 pixels. In general, a CTU may include three parallel coding tree blocks (CTBs): one luma CTB and two chroma CTBs. Each CTU may be recursively quadtree split into one or more coding units (CUs). For example, a 64x64 pixel CTU may be split into one CU of 64x64 pixels or four CUs of 32x32 pixels. One or more of the 32x32 blocks may each be further split into four CUs of 16x16 pixels. In some example embodiments, each CU may be analyzed during encoding to determine a prediction type for the CU from various prediction types, such as an inter-prediction type or an intra-prediction type. The CU may be divided into one or more prediction units (PUs) according to temporal and / or spatial predictability. Generally, each PU includes one luma prediction block (PB) and two chroma PBs. In one embodiment, prediction operations during coding (encoding / decoding) are performed in units of prediction blocks. The division of a CU into PUs (or PBs of different color channels) may be performed in various spatial patterns. For example, a luma or chroma PB may include a matrix of sample values ​​(e.g., luma values), such as 8x8 pixels, 16x16 pixels, 8x16 pixels, 16x8 samples, etc.

[0076]

[0099] 7 shows a diagram of a video encoder (703) according to another example embodiment of this disclosure. The video encoder (703) is configured to receive a processed block (e.g., a predictive block) of sample values ​​in a current video picture in a sequence of video pictures and encode the processed block into an encoded picture that is part of an encoded video sequence. The example video encoder (703) may be used in place of the example video encoder (403) of FIG. 4.

[0077]

[0100] For example, the video encoder (703) receives a matrix of sample values ​​for a processing block, such as a predicted block of 8x8 samples. The video encoder (703) then determines, for example, using rate-distortion optimization (RDO), whether the processing block is best coded using intra mode, inter mode, or bi-predictive mode. When it is determined that the processing block is coded in intra mode, the video encoder (703) may encode the processing block into a coded picture using intra prediction techniques, and when it is determined that the processing block is coded in inter mode or bi-predictive mode, the video encoder (703) may encode the processing block into a coded picture using inter prediction or bi-predictive techniques, respectively. In some example embodiments, merge mode may be used as a sub-mode of inter-picture prediction, in which a motion vector is derived from one or more motion vector predictors without the benefit of coded motion vector components outside the predictors. In some other example embodiments, there may be motion vector components applicable to the subject block. Thus, the video encoder (703) may include components not explicitly shown in FIG. 7, such as a mode decision module, to determine the prediction mode of a processing block.

[0078]

[0101] In the example of Figure 7, the video encoder (703) includes an inter-encoder (730), an intra-encoder (722), a residual calculator (723), a switch (726), a residual encoder (724), a general controller (721), and an entropy encoder (725), linked together as shown in the example arrangement of Figure 7.

[0079]

[0102] The inter-encoder (730) is configured to receive a sample of a current block (e.g., a processing block), compare the block with one or more reference blocks in a reference picture (e.g., blocks in previous and subsequent pictures in display order), generate inter-prediction information (e.g., a description of redundant information due to inter-coding techniques, motion vectors, merge mode information), and calculate an inter-prediction result (e.g., a predicted block) based on the inter-prediction information using any suitable technique. In some examples, the reference picture is a decoded reference picture decoded based on the encoded video information using a decoding unit 633 incorporated in the example encoder 620 of FIG. 6 (such as shown as residual decoder 728 of FIG. 7, described in further detail below).

[0080]

[0103] The intra encoder (722) is configured to receive samples of a current block (e.g., a processing block), compare the block with previously encoded blocks in the same picture, generate transformed quantized coefficients, and in some cases, further generate intra prediction information (e.g., intra prediction direction information using one or more intra encoding techniques). The intra encoder (722) may calculate intra prediction results (e.g., predicted blocks) based on the intra prediction information and reference blocks in the same picture.

[0081]

[0104] The general controller (721) may be configured to determine general control data and control other components of the video encoder (703) based on the general control data. In one example, the general controller (721) determines a prediction mode for a block and sends a control signal to the switch (726) based on the prediction mode. For example, when the prediction mode is intra-mode, the general controller (721) controls the switch (726) to select an intra-mode result for use by the residual calculator (723) and controls the entropy encoder (725) to select intra-prediction information and include the intra-prediction information in the bitstream. When the prediction mode for the block is inter-mode, the general controller (721) controls the switch (726) to select an inter-prediction result for use by the residual calculator (723) and controls the entropy encoder (725) to select inter-prediction information and include the inter-prediction information in the bitstream.

[0082]

[0105] The residual calculator (723) may be configured to calculate the difference (residual data) between the received block and a prediction result of the block selected from the intra-encoder (722) or the inter-encoder (730). The residual encoder (724) may be configured to encode the residual data to generate transform coefficients. For example, the residual encoder (724) may be configured to convert the residual data from the spatial domain to the frequency domain to generate the transform coefficients. The transform coefficients then undergo a quantization process to obtain quantized transform coefficients. In various illustrative embodiments, the video encoder (703) further includes a residual decoder (728). The residual decoder (728) is configured to perform an inverse transform and generate decoded residual data. The decoded residual data can be used by the intra-encoder (722) and the inter-encoder (730) as appropriate. For example, the inter-encoder (730) can generate decoded blocks based on the decoded residual data and inter-prediction information, and the intra-encoder (722) can generate decoded blocks based on the decoded residual data and intra-prediction information. The decoded blocks are appropriately processed to generate decoded pictures, which can be buffered in a memory circuit (not shown) and used as reference pictures.

[0083]

[0106] The entropy encoder (725) may be configured to format a bitstream to include the encoded block and perform entropy encoding. The entropy encoder (725) may be configured to include various information in the bitstream. For example, the entropy encoder (725) may be configured to include general control data, selected prediction information (e.g., intra-prediction information or inter-prediction information), residual information, and other appropriate information in the bitstream. When encoding a block in an inter mode or a merged sub-mode of a bi-prediction mode, the residual information may be absent.

[0084]

[0107] 8 shows a diagram of an example video decoder (810) according to another embodiment of the present disclosure. The video decoder (810) is configured to receive coded pictures that are part of a coded video sequence and decode the coded pictures to generate reconstructed pictures. In one example, the video decoder (810) may be used in place of the video decoder (410) in the example of FIG. 4.

[0085]

[0108] In the example of Figure 8, the video decoder (810) includes an entropy decoder (871), an inter decoder (880), a residual decoder (873), a reconstruction module (874), and an intra decoder (872) linked together as shown in the example arrangement of Figure 8.

[0086]

[0109] The entropy decoder (871) can be configured to reconstruct, from the coded picture, certain symbols that represent syntax elements that make up the coded picture. Such symbols can include, for example, prediction information (e.g., intra- or inter-prediction information) that can identify the mode in which the block is coded (e.g., intra-mode, inter-mode, bi-prediction mode, merged sub-mode, or another sub-mode), certain samples or metadata used for prediction by the intra decoder (872) or inter decoder (880), residual information in the form of, for example, quantized transform coefficients, etc. In one example, when the prediction mode is an inter- or bi-prediction mode, the inter-prediction information is provided to the inter decoder (880), and when the prediction type is an intra-prediction type, the intra-prediction information is provided to the intra decoder (872). The residual information can undergo inverse quantization and is provided to the residual decoder (873).

[0087]

[0110] The inter decoder (880) may be configured to receive inter prediction information and generate inter prediction results based on the inter prediction information.

[0088]

[0111] The intra decoder (872) may be configured to receive intra prediction information and generate prediction results based on the intra prediction information.

[0089]

[0112] The residual decoder (873) may be configured to perform inverse quantization to extract dequantized transform coefficients, process the dequantized transform coefficients, and convert the residual from the frequency domain to the spatial domain. Additionally, the residual decoder (873) may utilize certain control information (to include quantization parameters (QPs)) that may be provided by the entropy decoder (871) (this may be only small data volume control information, so the data path is not depicted).

[0090]

[0113] The reconstruction module (874) may be configured to combine, in the spatial domain, the residual, as output by the residual decoder (873), and the prediction result (as output by the inter- or intra-prediction module, as the case may be), to form a reconstructed block that forms part of the reconstructed picture as part of the reconstructed video. Note that other appropriate operations, such as deblocking operations, may also be performed to improve visual quality.

[0091]

[0114] It should be noted that the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) may be implemented using any suitable technique. In some example embodiments, the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) may be implemented using one or more integrated circuits. In another embodiment, the video encoders (403), (603), and (603) and the video decoders (410), (510), and (810) may be implemented using one or more processors executing software instructions.

[0092]

[0115] Turning to block partitioning for encoding and decoding, general partitioning may start with a base block and may follow a predefined set of rules, a specific pattern, a partition tree, or any partition structure or scheme. The partitioning may be hierarchical and recursive. After dividing or partitioning the base block according to any or a combination of the example partitioning procedures described below or other procedures, a final set of partitions or coding blocks may be obtained. Each of these partitions may be at one of various partitioning levels in the partitioning hierarchy and may be of various shapes. Each of the partitions may be referred to as a coding block (CB). For various example partitioning implementations described further below, each resulting CB may be of any allowed size and partitioning level. Such partitions are referred to as coding blocks because they may form the unit at which some basic encoding / decoding decisions may be made, and at which encoding / decoding parameters may be optimized, determined, and signaled in the encoded video bitstream. The highest or deepest level in the final partition represents the depth of the coding block partitioning structure of the tree. The coding blocks may be luma coding blocks or chroma coding blocks. The CB tree structure for each color is sometimes called the coding block tree (CBT).

[0093]

[0116] The coding blocks of all color channels may be collectively referred to as a coding unit (CU). The hierarchical structure of all color channels may be collectively referred to as a coding tree unit (CTU). The partitioning pattern or structure of the various color channels in a CTU may or may not be the same.

[0094]

[0117] In some implementations, the partition tree schemes or structures used for the luma and chroma channels may not need to be the same. In other words, the luma and chroma channels may have separate coding tree structures or patterns. Furthermore, whether the luma and chroma channels use the same or different coding partition tree structures, and whether the actual coding partition tree structure should be used, may depend on whether the coded slice is a P slice, a B slice, or an I slice. For example, for an I slice, the chroma and luma channels may have separate coding partition tree structures or coding partition tree structure modes, while for a P or B slice, the luma and chroma channels may share the same coding partition tree scheme. When separate coding partition tree structures or modes are applied, the luma channel may be partitioned into CBs by one coding partition tree structure, and the chroma channel may be partitioned into chroma CBs by another coding partition tree structure.

[0095]

[0118] In some example implementations, a predetermined partitioning pattern may be applied to the base block. As shown in FIG. 9 , an example four-way partition tree may start at a first predefined level (e.g., a 64×64 block level or other size as the base block size), and the base block may be hierarchically partitioned down to a predefined lowest level (e.g., a 4×4 level). For example, the base block may follow four predefined partitioning options or patterns indicated by 902, 904, 906, and 908, and the partition designated as R may have the same partitioning option indicated in FIG. 9 repeated at a lower scale down to the lowest level (e.g., a 4×4 level), thereby allowing recursive partitioning. In some implementations, additional restrictions may be applied to the partitioning scheme of FIG. 9 . In the implementation of FIG. 9 , rectangular partitions (e.g., 1:2 / 2:1 rectangular partitions) may be allowed but are not allowed to be recursive, while square partitions are allowed to be recursive. Partitioning according to FIG. 9 with recursion generates a final set of coding blocks, if necessary. A coding tree depth may be further defined to indicate the division depth from the root node or root block. For example, the coding tree depth of the root node or root block, e.g., a 64×64 block, may be set to 0, and after the root block is further divided once according to FIG. 9, the coding tree depth increases by 1. The maximum or deepest level from the 64×64 base block to the smallest 4×4 partition should be 4 (starting from level 0) for the above scheme. Such a partitioning scheme may be applied to one or more of the color channels. Each color channel may be independently partitioned according to the scheme of FIG. 9 (e.g., a partitioning pattern or option from among predefined patterns may be independently determined for each color channel at each hierarchical level).Alternatively, two or more of the color channels may share the same hierarchical pattern tree of FIG. 9 (e.g., the same partitioning pattern or option from among predefined patterns may be chosen for two or more color channels per hierarchical level).

[0096]

[0119] FIG. 10 illustrates another example predefined partitioning pattern that allows recursive partitioning to form a partitioning tree. As shown in FIG. 10, an example 10-way partitioning structure or pattern may be predefined. The root block may begin at a predefined level (e.g., from a base block at a 128x128 level or a 64x64 level). The example partitioning structure of FIG. 10 includes various 2:1 / 1:2 and 4:1 / 1:4 rectangular partitions. The partition type with three sub-partitions, designated 1002, 1004, 1006, and 1008 in the second row of FIG. 10, may be referred to as a "T-type" partition. The "T-type" partitions 1002, 1004, 1006, and 1008 may be referred to as a left T-type, upper T-type, right T-type, and lower T-type. In some example implementations, none of the rectangular partitions of FIG. 10 are allowed to be further subdivided. A coding tree depth may be further defined to indicate the division depth from the root node or root block. For example, the coding tree depth of a root node or root block, such as a 128x128 block, may be set to 0, and after the root block is further divided once according to FIG. 10, the coding tree depth increases by 1. In some implementations, only the all-square partitions of 1010 may be allowed to recursively partition to the next level of the partitioning tree according to the pattern of FIG. 10. In other words, recursive partitioning may not be allowed for the square partitions in the T-type patterns 1002, 1004, 1006, and 1008. The partitioning procedure according to FIG. 10 with recursion generates a final set of coding blocks, if necessary. Such a scheme may be applied to one or more of the color channels. In some implementations, more flexibility may be added to the use of partitions below the 8x8 level. For example, 2x2 chroma inter prediction may be used in certain cases.

[0097]

[0120] In some other example implementations of coding block partitioning, a quadtree structure may be used to split a base block or an intermediate block into quadtree partitions. Such quadtree splitting may be applied hierarchically and recursively to any square-shaped partition. Whether a base block or an intermediate block or partition is further quadtree split may be adapted to various local characteristics of the base block or intermediate block / partition. The quadtree partitioning at picture boundaries may be further adapted. For example, implicit quadtree splitting may be implemented at picture boundaries so that a block continues to be quadtree split until its size fits the picture boundary.

[0098]

[0121] In some other example implementations, hierarchical binary partitioning from a base block may be used. In such a scheme, a base block or a mid-level block may be partitioned into two partitions. The partitioning may be horizontal or vertical. For example, horizontal partitioning may split a base block or a mid-level block into equal right and left partitions. Similarly, vertical partitioning may split a base block or a mid-level block into equal top and bottom partitions. Such partitioning may be hierarchical and recursive. For each base block or mid-level block, a determination may be made as to whether the binary partitioning scheme should continue and, if the scheme continues further, whether horizontal or vertical partitioning should be used. In some implementations, further partitioning may stop at a predefined minimum partition size (in one or both dimensions). Alternatively, further partitioning may stop as soon as a predefined partitioning level or depth from the base block is reached. In some implementations, the aspect ratio of the partitions may be limited. For example, the aspect ratio of a section may be no less than 1:4 (or no more than 4:1). Thus, a vertically elongated section with a vertical to horizontal aspect ratio of 4:1 may simply be further divided vertically into an upper section and a lower section, each with a vertical to horizontal aspect ratio of 2:1.

[0099]

[0122] In some other examples, a three-partitioning scheme may be used to partition the base block or any intermediate blocks, as shown in FIG. 13 . A three-partitioning scheme may be implemented vertically, as shown at 1302 in FIG. 13 , or horizontally, as shown at 1304 in FIG. 13 . The split ratio in the example in FIG. 13 is shown as 1:2:1 vertically or horizontally, but other ratios may be predefined. In some implementations, two or more different ratios may be predefined. Such a three-partitioning scheme may be used to complement a quadtree or two-partitioning structure, since such a ternary tree partitioning has the ability to capture an object at a block center within one contiguous partition, while quadtrees and two-partitions always split along the block center and therefore should split the object into separate partitions. In some implementations, the width and height of the partitions in the example ternary tree are always powers of two to avoid further transformations.

[0100]

[0123] The above partitioning schemes may be combined in any manner at different partitioning levels. As one example, the above-mentioned quadtree and binary partitioning schemes may be combined to partition a base block into a quadtree-binary-tree (QTBT) structure. In such a scheme, the base block or intermediate blocks / partitions may be quadtree-split or binary-split, if specified, subject to a set of predefined conditions. A specific example is illustrated in FIG. 14. In the example of FIG. 14, the base block is first quadtree-split into four partitions, as shown by 1402, 1404, 1406, and 1408. Each of the resulting partitions is then quadtree-partitioned into four further partitions (such as 1408), or binary-split into two further partitions at the next level (e.g., both symmetric, such as 1402 or 1406), or not split (such as 1404). Binary or quadtree splits may be recursively allowed for square-shaped partitions, as shown by the partition pattern of the overall example in 1410 and the corresponding tree structure / representation in 1420, where solid lines represent quadtree splits and dashed lines represent binary splits. A flag may be used at each binary split node (a non-leaf partition consisting of two) to indicate whether the binary split is horizontal or vertical. For example, as shown in 1420 in agreement with the partitioning structure in 1410, a flag "0" may represent a horizontal binary split and a flag "1" may represent a vertical binary split. In the case of quadtree split partitions, there is no need to indicate the split type, since a quadtree split always splits a block or partition both horizontally and vertically to produce four equal-sized sub-blocks / partitions. In some implementations, a flag "1" may represent a horizontal binary split and a flag "0" may represent a vertical binary split.

[0101]

[0124] In some example implementations of QTBT, the quadtree and binary splitting rule sets may be represented by the following predefined parameters and corresponding functions associated therewith: - CTU size: Root node size of the quadtree (size of the base block) - MinQTSize: Minimum allowable quadtree leaf node size - MaxBTSize: Maximum allowed binary tree root node size - MaxBTDepth: Maximum allowed binary tree depth - MinBTSize: The minimum allowable binary tree leaf node size In some example implementations of the QTBT partitioning structure, the CTU size may be set as 128x128 luma samples with two corresponding 64x64 blocks of chroma samples (when example chroma subsampling is considered and used), MinQTSize may be set as 16x16, MaxBTSize may be set as 64x64, MinBTSize (both width and height) may be set as 4x4, and MaxBTDepth may be set as 4. Quadtree partitioning may first be applied to the CTU to generate quadtree leaf nodes. The quadtree leaf nodes may have sizes from its minimum allowed size of 16x16 (i.e., MinQTSize) to 128x128 (i.e., CTU size). If the node is 128x128, it will not be initially split by the binary tree because its size exceeds MaxBTSize (i.e., 64x64). Otherwise, nodes that do not exceed MaxBTSize may be partitioned by a binary tree. In the example of FIG. 14, the base block is 128x128. According to a predefined rule set, the basic block can only be quadtree split. The base block has a partitioning depth of 0. Each of the resulting four partitions is 64x64, does not exceed MaxBTSize, and may undergo further quadtree or binary tree splits at level 1. The process continues. When the binary tree depth reaches MaxBTDepth (i.e., 4), further splits may not be considered. When a binary tree node has a width equal to MinBTSize (i.e., 4), further horizontal splits may not be considered. Similarly, when a binary tree node has a height equal to MinBTSize, further vertical splits are not considered.

[0102]

[0125] In some example implementations, the above QTBT scheme may be configured to support flexibility for luma and chroma to have the same or separate QTBT structures. For example, for P and B slices, the luma and chroma CTBs in one CTU may share the same QTBT structure. However, for an I slice, the luma CTB may be partitioned into CBs by a QTBT structure, and the chroma CTB may be partitioned into chroma CBs by a different QTBT structure. This means that CUs may be used to refer to different color channels in an I slice; for example, an I slice may consist of a coded block of a luma component or a coded block of two chroma components, while a CU in a P or B slice may consist of coded blocks of all three color components.

[0103]

[0126] In some other implementations, the QTBT scheme may be supplemented with the ternary scheme described above. Such implementations are sometimes referred to as multi-type tree (MTT) structures. For example, in addition to the binary division of nodes, one of the ternary partitioning patterns of FIG. 13 may be chosen. In some implementations, only square nodes may undergo ternary division. An additional flag may be used to indicate whether the ternary partitioning is horizontal or vertical.

[0104]

[0127] Two-level or multi-level tree designs, such as QTBT implementations, and QTBT implementations supplemented by three-partitions, may be primarily motivated by complexity reduction. Logically, the complexity of traversing a tree is T D where T represents the number of split types and D is the depth of the tree. Trade-offs may be made by using multiple types (T) while reducing the depth (D).

[0105]

[0128] In some implementations, the CB may be further partitioned. For example, the CB may be further partitioned into multiple prediction blocks (PBs) for intra- or inter-frame prediction during the encoding and decoding process. In other words, the CB may be further divided into different sub-partitions, and individual prediction decisions / configurations may be made. At the same time, the CB may be further partitioned into multiple transform blocks (TBs) to accurately represent the level at which a transform or inverse transform of video data is performed. The partitioning scheme of the CB into PBs and TBs may be the same or different. For example, each partitioning scheme may be implemented using a unique procedure based on, for example, various characteristics of the video data. The PB and TB partitioning schemes may be unrelated in some implementations. The PB and TB partitioning schemes and boundaries may be correlated in other implementations. In some implementations, for example, the TBs may be partitioned after the PB partitioning, and in particular, each PB may be determined following the partitioning of the coding block and then further partitioned into one or more TBs. For example, in some implementations, a PB may be split into one, two, four, or some other number of TBs.

[0106]

[0129] In some implementations, for the partitioning of base blocks into coding blocks and further into prediction blocks and / or transform blocks, the luma channel and the chroma channel may be treated separately. For example, in some implementations, partitioning of coding blocks into prediction blocks and / or transform blocks may be allowed for the luma channel, while such partitioning of coding blocks into prediction blocks and / or transform blocks may not be allowed for the chroma channel. In such implementations, transform and / or prediction of luma blocks may therefore only be performed at the coding block level. As another example, the minimum transform block size of the luma channel and the chroma channel may be different, e.g., coding blocks of the luma channel may be allowed to be partitioned into smaller transform and / or predictive blocks than the chroma channel. As yet another example, the maximum depth of partitioning of coding blocks into transform and / or predictive blocks may be different between the luma channel and the chroma channel, e.g., coding blocks of the luma channel may be allowed to be partitioned into deeper transform and / or predictive blocks than the chroma channel. As a specific example, a luma coding block may be partitioned into transform blocks of multiple sizes, which can be represented by recursive partitions down by up to two levels, and transform block shapes such as square, 2:1 / 1:2, and 4:1 / 1:4, as well as transform block sizes from 4x4 to 64x64 may be allowed. However, for chroma blocks, only the largest possible transform block specified for the luma block may be allowed.

[0107]

[0130] In some example implementations of the partitioning of coding blocks into PBs, the depth, shape, and / or other characteristics of the PB partitioning may depend on whether the PB is intra-coded or inter-coded.

[0108]

[0131] The partitioning of coding blocks (or prediction blocks) into transform blocks may be performed recursively or non-recursively in various illustrative schemes, including, but not limited to, quadtree division and predefined pattern division, and with further consideration of transform blocks at boundaries of coding blocks or prediction blocks. In general, the resulting transform blocks may be of different division levels, may not be the same size, and may not need to be square in shape (e.g., the resulting transform blocks can be rectangular with several allowed sizes and aspect ratios). Further examples are described in more detail below in connection with Figures 15, 16, and 17.

[0109]

[0132] However, in some other implementations, a CB obtained through any of the above partitioning schemes may be used as a basic or minimum coding block for prediction and / or transformation. In other words, no further division is performed for performing inter-prediction / intra-prediction and / or transformation. For example, a CB obtained from the above QTBT scheme may be directly used as a unit for performing prediction. Specifically, such a QTBT structure eliminates the concept of multiple partition types, i.e., such a QTBT structure eliminates the distinction between CUs, PUs, and TUs and supports more flexibility for the CU / CB partition shapes described above. In such a QTBT block structure, CUs / CBs can have square or rectangular shapes. Leaf nodes of such a QTBT are used as units for prediction and transformation processing without any further partitioning. This means that in such an example QTBT coding block structure, CUs, PUs, and TUs have the same block size.

[0110]

[0133] The various CB compartmentalization schemes described above, as well as further compartmentalization of the CB into PB and / or TB (not including PB / TB compartmentalization), may be combined in any manner. The following specific implementations are provided as non-limiting examples.

[0111]

[0134] Specific example implementations of coding block and transform block partitioning are described below. In such example implementations, a base block may be divided into coding blocks using recursive quadtree division or the predefined division patterns described above (such as those in FIGS. 9 and 10). For each level, local video data characteristics may determine whether further quadtree division of a particular partition should continue. The resulting CBs may be at various quadtree division levels and of various sizes. A decision as to whether a picture area should be coded using inter-picture (temporal) prediction or intra-picture (spatial) prediction may be made at the CB level (or at the CU level, for a total of three color channels). Each CB may be further divided into one, two, four, or other number of PBs according to a predefined PB division type. Within one PB, the same prediction process may be applied, and related information may be transmitted to the decoder for each PB. After obtaining the residual block by applying a prediction process based on the PB division type, the CB can be partitioned into TBs according to another quadtree structure similar to the coding tree of the CB. In this particular implementation, the CB or TB may be, but need not be, limited to a square shape. Furthermore, in this particular example, the PB may be square or rectangular for inter prediction, or only square for intra prediction. The coding block may be divided, for example, into four square-shaped TBs. Each TB may be further recursively divided (using quadtree division) into smaller TBs called residual quadtrees (RQTs).

[0112]

[0135] Another example implementation for partitioning a base block into CBs, PBs, and / or TBs is further described below. For example, rather than using multiple partition unit types such as those shown in FIG. 9 or FIG. 10, a quadtree with nested multitype trees using two- and three-split segmentation structures (e.g., QTBT or QTBT with three-split as described above) may be used. The separation of CBs, PBs, and TBs (i.e., partitioning CBs into PBs and / or TBs, and partitioning PBs into TBs) may be abandoned except when necessary for CBs with sizes too large for the maximum transform length; such CBs may require further partitioning. This example partitioning scheme may be designed to support more flexibility for CB partition shapes, so that both prediction and transformation can be performed on the CB level without further partitioning. In such a coding tree structure, the CBs may have a square or rectangular shape. Specifically, the coding tree block (CTB) may first be partitioned by a quadtree structure. The quadtree leaf nodes may then be further partitioned by a nested multitype tree structure. An example of a nested multitype tree structure using two or three splits is shown in FIG. 11. Specifically, the example multitype tree structure of FIG. 11 includes four split types, called vertical two-split (SPLIT_BT_VER) (1102), horizontal two-split (SPLIT_BT_HOR) (1104), vertical three-split (SPLIT_TT_VER) (1106), and horizontal three-split (SPLIT_TT_HOR) (1108). Therefore, CB corresponds to the leaf of the multitype tree. In this example implementation, as long as CB is not too large as the maximum transform length, this segmentation is used for both prediction and transform processing without any further partitioning. This means that in most cases, in a quadtree with a nested multitype tree coding block structure, CB, PB, and TB have the same block size. An exception occurs when the maximum supported transform length is smaller than the width or height of the color components of CB. In some implementations, in addition to the 2 or 3 splits, the nested pattern of FIG. 11 may further include a quadtree split.

[0113]

[0136] One specific example of a quadtree with a nested multitype tree coding block structure of block partitions (including quadtree, two-, and three-split options) for one base block is shown in FIG. 12. More specifically, FIG. 12 shows a base block 1200 quadtree split into four square partitions 1202, 1204, 1206, and 1208. The multitype tree structure of FIG. 11 and the decision to further use a quadtree for further splitting are made for each quadtree split partition. In the example of FIG. 12, partition 1204 is not further split. Partitions 1202 and 1208 each adopt another quadtree split. For partition 1202, the second-level quadtree-split top-left, top-right, bottom-left, and bottom-right partitions adopt a third-level split of the quadtree: horizontal two-split 1104 of FIG. 11, no split, and horizontal three-split 1108 of FIG. 11, respectively. Partition 1208 employs another quadtree splitting, with the second-level quadtree-split top-left, top-right, bottom-left, and bottom-right partitions employing a third-level split of vertical three-way split 1106 in FIG. 11 , no split, no split, and horizontal two-way split 1104 in FIG. 11 , respectively. Two of the subpartitions of the third-level top-left partition of 1208 are further split according to horizontal two-way split 1104 and horizontal three-way split 1108 in FIG. 11 , respectively. Partition 1206 employs a second-level splitting pattern according to vertical two-way split 1102 in FIG. 11 into two partitions that are further split according to horizontal three-way split 1108 and vertical two-way split 1102 in FIG. 11 at the third level. A fourth-level split is further applied to one of these according to horizontal two-way split 1104 in FIG. 11 .

[0114]

[0137] As a specific example of the above, the maximum luma transform size may be 64 × 64, and the maximum supported chroma transform size may be different from the luma, for example, at 32 × 32. Even if the CB in the above example of Figure 12 is not entirely further split into smaller PBs and / or TBs, when the width or height of the luma coding block or chroma coding block is greater than the maximum transform width or height, the luma coding block or chroma coding block may be automatically split in the horizontal and / or vertical directions to meet the transform size constraint in that direction.

[0115]

[0138] In the specific example of partitioning base blocks into CBs described above, the coding tree scheme may support the ability for luma and chroma to have separate block tree structures. For example, in the case of P and B slices, the luma and chroma CTBs in one CTU may share the same coding tree structure. In the case of an I slice, for example, the luma and chroma CTBs may have separate coding block tree structures. When separate block tree structures are applied, the luma CTB may be partitioned into luma CBs by one coding tree structure, and the chroma CTB is partitioned into chroma CBs by another coding tree structure. This means that a CU in an I slice may consist of a coding block of the luma component or a coding block of two chroma components, and that a CU in a P or B slice always consists of coding blocks of all three color components, unless the video is monochrome.

[0116]

[0139] When a coding block is further partitioned into multiple transform blocks, the transform blocks may be arranged in the bitstream according to various orders or scan patterns. Example implementations of partitioning coding or prediction blocks into transform blocks and the coding order of the transform blocks are described in further detail below. In some example implementations, as described above, the transform partitioning may support multiple shapes of transform blocks, such as 1:1 (square), 1:2 / 2:1, and 1:4 / 4:1, with transform block sizes ranging from 4×4 to 64×64, for example. In some implementations, when the coding block is smaller than or equal to 64×64, the transform block partitioning may be applied only to the luma component, and therefore, for chroma blocks, the transform block size is the same as the coding block size. Otherwise, if the coding block width or height is greater than 64, both the luma and chroma coding blocks may be implicitly split into multiples of min(W,64)×min(H,64) and min(W,32)×min(H,32), respectively.

[0117]

[0140] In some example implementations of transform block partitioning, for both intra- and inter-coded blocks, the coding blocks may be further partitioned into multiple transform blocks with a partition depth up to a predefined number of levels (e.g., two levels). The transform block partition depth and size may be related. For some example implementations, the mapping from the transform size of the current depth to the transform size of the next depth is shown below in Table 1.

[0118] [Table 1]

[0119]

[0141] Based on the example mapping in Table 1, for a 1:1 square block, the next level transform division may produce four 1:1 square sub-transform blocks. The transform partition may stop at, for example, 4x4. Thus, a transform size of 4x4 at the current depth corresponds to the same size of 4x4 at the next depth. In the example of Table 1, for a 1:2 / 2:1 non-square block, the next level transform division may produce two 1:1 square sub-transform blocks, while for a 1:4 / 4:1 non-square block, the next level transform division may produce two 1:2 / 2:1 sub-transform blocks.

[0120]

[0142] In some example implementations, additional restrictions on transform block partitioning may be applied to the luma component of an intra-coded block. For example, all sub-transform blocks at each level of transform partitioning may be restricted to be of equal size. For example, for a 32x16 coded block, level 1 transform partitioning creates two 16x16 sub-transform blocks, and level 2 transform partitioning creates eight 8x8 sub-transform blocks. In other words, second-level partitioning must be applied to all first-level sub-blocks to keep the transform units of equal size. An example of transform block partitioning for an intra-coded square block according to Table 1 is shown in Figure 15, with the coding order indicated by the arrows. Specifically, 1502 shows a square coded block. The first-level partitioning into four equal-sized transform blocks according to Table 1 is shown in 1504, with the coding order indicated by the arrows. All second-level partitioning of the first-level equal-sized block into 16 equal-sized transform blocks according to Table 1 is shown in 1506, with the coding order indicated by the arrows.

[0121]

[0143] In some example implementations, the above restrictions on intra-coding may not apply for the luma component of an inter-coded block. For example, after the first level of transform splitting, any one of the sub-transform blocks may be further split independently at another level. Thus, the resulting transform blocks may or may not be the same size. An example splitting of an inter-coded block into transform blocks in its coding order is shown in Figure 16. In the example of Figure 16, an inter-coded block 1602 is split into transform blocks at two levels according to Table 1. At the first level, the inter-coded block is split into four transform blocks of equal size. Then, only one of the four transform blocks (but not all of the transform blocks) is further split into four sub-transform blocks, resulting in a total of seven transform blocks with two different sizes, as indicated by 1604. The coding order of these seven transform blocks is indicated by the arrows at 1604 in Figure 16.

[0122]

[0144] In some example implementations, for chroma components, some additional restrictions on the transform blocks may apply: for example, for chroma components, the transform block size can be as large as the coding block size, but not smaller than a predefined size, such as 8x8.

[0123]

[0145] In some other example implementations, for coding blocks with width (W) or height (H) greater than 64, both luma and chroma coding blocks may be implicitly split into multiples of min(W,64) x min(H,64) and min(W,32) x min(H,32) transform units, respectively, where in this disclosure, "min(a,b)" may return the smaller value of a and b.

[0124]

[0146] Figure 17 further illustrates another alternative example scheme for partitioning a coding block or a prediction block into transform blocks. As shown in Figure 17, instead of using recursive transform partitioning, a predefined set of partitioning types may be applied to a coding block depending on the transform type of the coding block. In the particular example shown in Figure 17, one of six example partitioning types may be applied to split the coding block into various numbers of transform blocks. Such a scheme for generating transform block partitioning may be applied to a coding block or a prediction block.

[0125]

[0147] More specifically, the partitioning scheme of FIG. 17 provides up to six example partition types for any given transform type (transform type refers to the type of primary transform, such as ADST, for example). In this scheme, a transform partition type may be assigned for each coding block or predictive block, for example, based on rate-distortion cost. In an example, the transform partition type assigned to a coding block or predictive block may be determined based on the transform type of the coding block or predictive block. A particular transform partition type may correspond to a transform block division size and pattern, as shown by the six transform partition types illustrated in FIG. 17. The correspondence between various transform types and various transform partition types may be predefined. An example is shown below, where capitalized labels indicate transform partition types that may be assigned to a coding block or predictive block based on rate-distortion cost.

[0126]

[0148] ● PARTITION_NONE: Allocate a transformation size equal to the block size.

[0127]

[0149] ● PARTITION_SPLIT: Allocate a transformation size that is 1 / 2 the width of the block size and 1 / 2 the height of the block size.

[0128]

[0150] ● PARTITION_HORZ: Allocate a transform size that has the same width as the block size and 1 / 2 the height of the block size.

[0129]

[0151] ● PARTITION_VERT: Allocate a transformation size that has a width of 1 / 2 the block size and a height equal to the block size.

[0130]

[0152] ● PARTITION_HORZ4: Allocate a transformation size that has the same width as the block size and 1 / 4 of the height of the block size.

[0131]

[0153] ● PARTITION_VERT4: Allocates a transformation size that has a width of 1 / 4 of the block size and a height equal to the block size.

[0132]

[0154] In the above example, all transform partition types as shown in Figure 17 include uniform transform sizes for the partitioned transform blocks. This is by way of example only and not limitation. In some other implementations, mixed transform block sizes for the partitioned transform blocks of a particular partition type (or pattern) may be used.

[0133]

[0155] A video block (also referred to as a PB when not further partitioned into multiple predictive blocks, PB or CB) may be predicted in various manners rather than being directly encoded, thereby exploiting various correlations and redundancies in the video data to improve compression efficiency. Correspondingly, such prediction may be performed in various modes. For example, a video block may be predicted via intra prediction or inter prediction. In particular, in an inter prediction mode, a video block may be predicted by one or more other reference blocks or inter predictor blocks from one or more other frames via single-reference or mixed-reference inter prediction. For an inter prediction implementation, a reference block may be specified by its frame identifier (the temporal location of the reference block) and a motion vector that indicates the spatial offset between the current block being encoded or decoded and the reference block (the spatial location of the reference block). The reference frame identification and motion vector may be signaled in the bitstream. A motion vector, such as a spatial block offset, may be signaled directly or may itself be predicted by another reference motion vector or predictor motion vector. For example, the current motion vector may be predicted directly by a reference motion vector (e.g., of a candidate neighboring block) or by a combination of the reference motion vector and the motion vector difference (MVD) between the current and reference motion vectors. The latter is sometimes called merge mode with motion vector difference (MMVD). The reference motion vector may be identified in the bitstream, for example, as a pointer to a spatial neighboring block of the current block or a temporally neighboring but spatially aligned block.

[0134]

[0156] Inter prediction may be performed, for example, in single reference mode or combined reference mode. In some implementations, a skip flag may be initially included in the bitstream for the current block (or at a higher level) to indicate whether the current block is inter-coded and should not be skipped. If the current block is inter-coded, another flag may be further included in the bitstream to signal whether single reference mode or combined reference mode is used for predicting the current block. In single reference mode, one reference block may be used to generate a prediction block for the current block. In combined reference mode, two or more reference blocks may be used to generate a prediction block, for example, by weighted averaging. Combined reference mode is sometimes also referred to as two or more reference mode, two reference mode, or multiple reference mode. One or more reference blocks may be identified using one or more reference frame indices and, additionally, using one or more corresponding motion vectors, which indicate a shift between the reference block and the current block in location relative to the frame, e.g., horizontal and vertical pixels. For example, an inter-predicted block for a current block may be generated from a single reference block identified by one motion vector in a reference frame as the predictive block in single reference mode, while in mixed reference mode, the predictive block may be generated by a weighted average of two reference blocks in two reference frames indicated by two reference frame indices and two corresponding motion vectors. Motion vectors may be coded and included in the bitstream in various manners.

[0135]

[0157] In some implementations, an encoding or decoding system may maintain a decoded picture buffer (DPB). Some images / pictures may be kept in the DPB waiting to be displayed (in the decoding system), and some images / pictures in the DPB may be used as reference frames to enable inter-prediction (in the decoding or encoding system). In some implementations, reference frames in the DPB may be tagged as short-term or long-term references for the current image being encoded or decoded. For example, short-term reference frames may include frames used for inter-prediction of blocks in the current frame or in a predefined number (e.g., two) of subsequent video frames closest to the current frame in decoding order. Long-term reference frames may include frames in the DPB that can be used to predict image blocks in frames that are more than a predefined number of frames away from the current frame in decoding order. Information about such tags for short-term and long-term reference frames may be called a Reference Picture Set (RPS) and may be added to the header of each frame in the encoded bitstream. Each frame in the encoded video stream may be identified by a Picture Order Counter (POC), which may be absolutely numbered according to the playback sequence or may relate to a group of pictures starting from, for example, an I-frame.

[0136]

[0158] In some example implementations, one or more reference picture lists including identification of short-term and long-term reference frames for inter prediction may be formed based on information in the RPS. For example, a single picture reference list may be formed for unidirectional inter prediction and denoted as L0 reference (or reference list 0), while two picture reference lists may be formed for bidirectional inter prediction and denoted as L0 (or reference list 0) and L1 (or reference list 1), for each of the two prediction directions. The reference frames included in the L0 and L1 lists may be ordered in various predetermined manners. The lengths of the L0 and L1 lists may be signaled in the video bitstream. Unidirectional inter prediction may be a single reference mode, or may be a combined reference mode when multiple references for generating a prediction block by weighted averaging in a combined prediction mode are on the same side of a frame. Bidirectional inter prediction may be a combined mode only, since bidirectional inter prediction involves at least two reference blocks.

[0137]

[0159] In some implementations, a merge mode (MM) for inter prediction may be implemented. Generally, in the merge mode, one or more of the motion vectors in single-reference prediction or the motion vectors in mixed-reference prediction for the current PB may be derived from other motion vectors rather than being calculated and signaled independently. For example, in an encoding system, the current motion vector of the current PB may be represented by the difference between the current motion vector and one or more other already-encoded motion vectors (referred to as reference motion vectors). Such a motion vector difference, rather than the entire current motion vector, may be encoded and included in the bitstream and linked to the reference motion vector. Correspondingly, in a decoding system, the motion vector corresponding to the current PB may be derived based on the decoded motion vector difference and the decoded reference motion vector linked thereto. As a specific form of merge mode (MM) inter prediction in general, such inter prediction based on the motion vector difference may be referred to as merge mode with motion vector difference (MMVD). Therefore, MM in general, or MMVD in particular, may be implemented to exploit the correlation between motion vectors associated with different PBs to improve coding efficiency. For example, neighboring PBs may have similar motion vectors, and therefore the MVD may be small and efficiently codeable. As another example, motion vectors may be temporally correlated (between frames) for blocks that are similarly located / placed in space.

[0138]

[0160] In some example implementations, an MM flag may be included in the bitstream during the encoding process to indicate whether the current PB is in merge mode. Additionally or alternatively, an MMVD flag may be included during the encoding process and signaled in the bitstream to indicate whether the current PB is in MMVD mode. MM and / or MMVD flags or indicators may be provided at the PB level, CB level, CU level, CTB level, CTU level, slice level, frame level, picture level, sequence level, etc. As a specific example, both an MM flag and an MMVD flag may be included for the current CU, and the MMVD flag may be signaled immediately after the skip flag and MM flag to specify whether MMVD mode is used for the current CU.

[0139]

[0161] In some example implementations of MMVD, a list of reference motion vectors (RMVs), or MV predictor candidates for motion vector prediction, may be formed for the block being predicted. The list of RMV candidates may include a predetermined number (e.g., two) of MV predictor candidate blocks, and the motion vectors of the MV predictor candidate blocks may be used to predict the current motion vector. The RMV candidate blocks may include blocks selected from neighboring blocks in the same frame and / or from temporal blocks (e.g., co-located blocks in a following or subsequent frame of the current frame). These options represent blocks in spatial or temporal locations relative to the current block that are likely to have similar or identical motion vectors to the current block. The size of the list of MV predictor candidates may be predetermined. For example, the list may include two or more candidates. To be on the list of RMV candidates, a candidate block may be required, for example, to have the same reference frame (or multiple reference frames) as the current block, must exist (e.g., a boundary check needs to be performed when the current block is near the edge of a frame), must have already been encoded during the encoding process, and / or must have already been decoded during the decoding process. In some implementations, the list of merge candidates may be populated with spatially neighboring blocks (scanned in a specific predefined order) first, if available and if they satisfy the above conditions, and then with temporal blocks if space is still available in the list. Neighboring RMV candidate blocks may be selected, for example, from the blocks to the left and top of the current block. The list of RMV predictor candidates may be dynamically formed at various levels (sequence, picture, frame, slice, superblock, etc.) as a Dynamic Reference List (DRL). The DRL may be signaled in the bitstream.

[0140]

[0162] In some implementations, the actual MV predictor candidate used as the reference motion vector for predicting the motion vector of the current block may be signaled. In cases where the RMV candidate list contains two candidates, a one-bit flag called a merge candidate flag may be used to indicate the selection of the reference merge candidate. If the current block is predicted in mixed mode, each of the multiple motion vectors predicted using the MV predictor may be associated with a reference motion vector from the merge candidate list. The encoder may determine which of the RMV candidates more closely predicts the MV of the current coding block and signal the selection as an index to the DRL.

[0141]

[0163] In some example implementations of MMVD, after an RMV candidate is selected and used as a base motion vector predictor for predicting a motion vector, a motion vector difference (MVD or delta MV representing the difference between the motion vector to be predicted and a reference candidate motion vector) may be calculated in the encoding system. Such MVD may include information representing the magnitude and direction of the MV difference, and both the magnitude and direction of the MV difference may be signaled in the bitstream. The magnitude and direction of the motion difference may be signaled in various ways.

[0142]

[0164] In some example implementations of MMVD, a distance index may be used to specify magnitude information of the motion vector difference and to indicate one of a set of predefined offsets representing predefined motion vector differences from a starting point (reference motion vector). The MV offset according to the signaled index may then be added to the horizontal or vertical component of the start (reference) motion vector. Whether the horizontal or vertical component of the reference motion vector should be offset may be determined by the direction information of the MVD. Example predefined relationships between the distance index and the predefined offsets are specified in Table 2.

[0143] [Table 2]

[0144]

[0165] In some example implementations of MMVD, a direction index may be further signaled and used to indicate the direction of the MVD relative to the reference motion vector. In some implementations, the direction may be limited to one of horizontal and vertical. An example 2-bit direction index is shown in Table 3. In the example of Table 3, the interpretation of the MVD may differ depending on the information of the start / reference MV. For example, when the start / reference MV corresponds to a uni-prediction block or a bi-prediction block where both reference frame lists point to the same side of the current picture (i.e., when the POCs of the two reference pictures are both greater than or both less than the POC of the current picture), the code in Table 3 may specify the sign (direction) of the MV offset added to the start / reference MV. When the start / reference MV corresponds to a bi-predictive block with two reference pictures on different sides of the current picture (i.e., the POC of one reference picture is greater than the POC of the current picture and the POC of the other reference picture is less than the POC of the current picture), and when the difference between the reference POC in picture reference list 0 and the current frame is greater than the difference between the reference POC in picture reference list 1 and the current frame, the code in Table 3 may specify the sign of the MV offset added to the reference MV corresponding to the reference picture in picture reference list 0, and the sign of the offset of the MV corresponding to the reference picture in picture reference list 1 may have the opposite value (the opposite sign of the offset). Otherwise, if the difference between the reference POC in picture reference list 1 and the current frame is greater than the difference between the reference POC in picture reference list 0 and the current frame, the code in Table 3 may specify the sign of the MV offset added to the reference MV associated with picture reference list 1, and the sign of the offset to the reference MV associated with picture reference list 0 has the opposite value.

[0145] [Table 3]

[0146]

[0166] In some example implementations, the MVD may be scaled according to the POC difference per direction. If the POC difference in both lists is the same, no scaling is necessary. Otherwise, if the POC difference in reference list 0 is greater than the one in reference list 1, the MVD for reference list 1 is scaled. If the POC difference in reference list 1 is greater than list 0, the MVD for list 0 may be scaled in the same manner. If the start MV is uni-predicted, the MVD is added to the available MV or reference MV.

[0147]

[0167] In some example implementations of MVD coding and signaling for bidirectional mixed prediction, in addition to or as an alternative to separately coding and signaling two MVDs, symmetric MVD coding may be implemented such that only one MVD needs to be signaled and the other MVDs may be derived from the signaled MVD. In such implementations, motion information including reference picture indices for both list 0 and list 1 is signaled. However, for example, only the MVD associated with reference list 0 is signaled, and the MVD associated with reference list 1 is derived without being signaled. Specifically, at the slice level, a flag called "mvd_l1_zero_flag" may be included in the bitstream to indicate whether reference list 1 is not signaled in the bitstream. If this flag is 1, indicating that reference list 1 is equal to zero (and therefore not signaled), a bidirectional prediction flag called "BiDirPredFlag" may be set to 0, meaning there is no bidirectional prediction. Otherwise, if mvd_l1_zero_flag is zero, then if the closest reference picture in list 0 and the closest reference picture in list 1 form a forward-backward pair of reference pictures or a backward-forward pair of reference pictures, then BiDirPredFlag may be set to 1, and the reference pictures in both list 0 and list 1 are short-term reference pictures. Otherwise, BiDirPredFlag is set to 0. A BiDirPredFlag of 1 may indicate that a symmetric mode flag is additionally signaled in the bitstream. A decoder may extract the symmetric mode flag from the bitstream when BiDirPredFlag is 1. The symmetric mode flag may, for example, be signaled at the CU level (if necessary) and may indicate whether symmetric MVD coding mode is used for the corresponding CU.When the symmetric mode flag is 1, it indicates the use of symmetric MVD coding mode and that only the reference picture indices for both list 0 and list 1 (called "mvp_l0_flag" and "mvp_l1_flag") are signaled, along with the MVD associated with list 0 (called "MVD0"), and that the other motion vector difference, "MVD1," is to be derived rather than signaled. For example, MVD1 may be derived as -MVD0. Thus, in the example symmetric MVD mode, only one MVD is signaled.

[0148]

[0168] In some other example implementations of MV prediction, a harmonized scheme may be used to implement general merge-mode MMVD, as well as some other types of MV prediction, for both single-reference mode and combined-reference mode MV prediction. Various syntax elements may be used to signal the manner in which the MV of the current block is predicted.

[0149]

[0169] For example, for single criterion mode, the following MV prediction modes may be signaled:

[0150]

[0170] NEARMV - Use one of the motion vector predictors (MVP) in the list directly pointed to by the DRL (Dynamic Reference List) index with no MVD.

[0151]

[0171] NEWMV - Use one of the motion vector predictors (MVP) in the list signaled by the DRL index as a reference and apply a delta to the MVP (e.g., using MVD).

[0152]

[0172] GLOBALMV - Use motion vectors based on frame-level global motion parameters.

[0153]

[0173] Similarly, if a combined reference inter prediction mode uses two reference frames corresponding to two MVs to be predicted, the following MV prediction modes may be signaled:

[0154]

[0174] NEAR_NEARMV - Use one of the motion vector predictors (MVPs) in the list signaled by the DRL index without MVD for each of two of the MVs to be predicted.

[0155]

[0175] NEAR_NEWMV - Use one of the motion vector predictors (MVP) in the list signaled by the DRL index as a reference MV without MVD to predict the first of two motion vectors, and use one of the motion vector predictors (MVP) in the list signaled by the DRL index as a reference MV with an additionally signaled delta MV (MVD) to predict the second of two motion vectors.

[0156]

[0176] NEW_NEARMV - Use one of the motion vector predictors (MVP) in the list signaled by the DRL index as a reference MV without MVD to predict the second of two motion vectors, and use one of the motion vector predictors (MVP) in the list signaled by the DRL index as a reference MV with the additionally signaled delta MV (MVD) to predict the first of two motion vectors.

[0157]

[0177] NEW_NEWMV - Use one of the motion vector predictors (MVPs) in the list signaled by the DRL index as the reference MV and use this together with the additionally signaled delta MV to predict for each of the two MVs.

[0158]

[0178] GLOBAL_GLOBALMV - Use the MV from each reference based on its frame-level global motion parameters.

[0159]

[0179] Thus, the term "NEAR" above refers to MV prediction using a reference MV without any MVD as a general merge mode, while the term "NEW" refers to MV prediction using a reference MV and offsetting the reference MV with a signaled or induced MVD, as in the MMVD mode. In the case of hybrid inter prediction, for example, the two MVDs may be correlated, and both the reference base motion vector and the motion vector delta may generally be different or unrelated between the two references or two MVDs, even though such correlation may be exploited to reduce the amount of information required to signal the two motion vector deltas. To exploit such correlation, joint signaling of the two MVDs may be implemented and indicated in the bitstream, as described in more detail below.

[0160]

[0180] In some example implementations of the MVD, a predefined pixel resolution for the MVD may be allowed. For example, 1 / 8 pixel motion vector accuracy (or precision) may be allowed. The above-mentioned MVDs for various MV prediction modes may be constructed and signaled in various manners. In some implementations, various syntax elements may be used to signal the above-mentioned motion vector differences in reference frame list 0 or list 1.

[0161]

[0181] For example, a syntax element called "mv_joint" may specify which components of the associated motion vector difference are non-zero. In the case of MVD, this is signaled for all non-zero components together. For example, mv_joint has the following values: 0 may indicate no non-zero MVD along the horizontal or vertical direction. A 1 may indicate that there is non-zero MVD along the horizontal direction only. 2 may indicate that there is non-zero MVD along the vertical direction only. 3 may indicate that there is non-zero MVD along both the horizontal and vertical directions.

[0162]

[0182] When the "mv_joint" syntax element for MVD signals that there are no non-zero MVD components, no further MVD information may be signaled. However, if the "mv_joint" syntax signals that there are one or two non-zero components, additional syntax elements may be further signaled for each non-zero MVD component, as described below.

[0163]

[0183] For example, a syntax element called "mv_sign" may be used to additionally specify whether the corresponding motion vector difference component is positive or negative.

[0164]

[0184] As another example, a syntax element called "mv_class" may be used to specify a class of a motion vector difference from among a predefined set of classes for the corresponding non-zero MVD component. The predefined classes for motion vector differences may be used, for example, to divide a continuous magnitude space of motion vector differences into non-overlapping ranges, with each range corresponding to an MVD class. Thus, the signaled MVD class indicates the magnitude range of the corresponding MVD component. In the example implementation shown in Table 4 below, higher classes correspond to motion vector differences with larger magnitude ranges. In Table 4, the symbol (n,m] is used to represent a range of motion vector differences greater than n pixels and less than or equal to m pixels.

[0165] [Table 4]

[0166]

[0185] In some other examples, a syntax element called "mv_bit" may be further used to specify the integer portion of the offset between a non-zero motion vector difference component and the starting magnitude of the correspondingly signaled MV class magnitude range. Thus, mv_bit may indicate the magnitude or size of the MVD. The number of bits required in "my_bit" to signal the full range per MVD class may vary depending on the MV class. As an example, MV_CLASS0 and MV_CLASS1 in the implementation of Table 4 may require only a single bit to indicate a 1 or 2 integer pixel offset from a starting MVD of 0, and each higher MV_CLASS in the example implementation of Table 4 may require progressively one more bit for "mv_bit" than the previous MV_CLASS.

[0167]

[0186] In some other examples, a syntax element called “mv_fr” may be further used to specify the first two fractional bits of the motion vector difference for the corresponding non-zero MVD component, while a syntax element called “mv_hp” may be used to specify the third fractional bit (high-resolution bit) of the motion vector difference for the corresponding non-zero MVD component. The two “mv_fr” bits essentially provide ¼-pixel MVD resolution, while the “mv_hp” bits may further provide ⅛-pixel resolution. In some other implementations, two or more “mv_hp” bits may be used to provide MVD pixel resolution finer than ⅛-pixel. In some example implementations, additional flags may be signaled at one or more of various levels to indicate whether ⅛-pixel or higher MVD resolutions are supported. If an MVD resolution does not apply to a particular coding unit, the above syntax element for the corresponding unsupported MVD resolution may not be signaled.

[0168]

[0187] In some example implementations above, fractional resolution may be independent of different MVD classes. In other words, similar options for motion vector resolution may be provided using a predefined number of "mv_fr" and "mv_hp" bits to signal fractional MVD of non-zero MVD components, regardless of the magnitude of the motion vector difference.

[0169]

[0188] However, in some other example implementations, the resolution of motion vector differences in various MVD scale classes may be distinguished or adaptive. Specifically, a high-resolution MVD with a large MVD scale for a higher MVD class may not result in a statistically significant improvement in compression efficiency or coding gain. Therefore, the MVD may be coded with a reduced resolution (integer pixel resolution or fractional pixel resolution) for a larger MVD scale range corresponding to a higher MVD scale class. Similarly, the MVD may generally be coded with a reduced resolution (integer pixel resolution or fractional pixel resolution) for a larger MVD value. Such MVD-class-dependent or MVD-scale-dependent MVD resolution may generally be referred to as an adaptive MVD resolution, a scale-dependent adaptive MVD resolution, or a scale-dependent MVD resolution. The term "resolution" may also be referred to as "pixel resolution." The adaptive MVD resolution may be implemented in various manners, as described by the following example implementations, to achieve better overall compression efficiency. In particular, due to the statistical observation that treating the MVD resolution of a large-scale or high-class MVD at a level similar to that of a small-scale or low-class MVD in a non-adaptive manner may not significantly improve the inter-prediction residual coding efficiency of blocks with a large-scale or high-class MVD, the reduction in the number of signaling bits by aiming for a less precise MVD may be greater than the additional bits required to code the inter-prediction residual as a result of such a less precise MVD. In other words, using a higher MVD resolution for a large-scale or high-class MVD may not produce more coding gain than using a lower MVD resolution.

[0170]

[0189] In some typical example implementations, the pixel resolution or precision of the MVD may decrease or may stop increasing as the MVD class increases. Decreasing the pixel resolution of the MVD corresponds to a coarser MVD (or a larger step from one MVD level to the next MVD level). In some implementations, the correspondence between MVD pixel resolution and MVD class may be specified, pre-defined, or pre-configured, and thus may not need to be signaled in the encoded bitstream.

[0171]

[0190] In some example implementations, the MV classes in Table 3 may each be associated with a different MVD pixel resolution.

[0172]

[0191] In some example implementations, each MVD class may be associated with a single allowed resolution. In some other implementations, one or more MVD classes may each be associated with two or more optional MVD pixel resolutions. A signal in the bitstream for a current MVD component having such an MVD class may therefore be followed by additional signaling to indicate the optional pixel resolution selected for the current MVD component. In some example implementations, adaptively allowed MVD pixel resolutions may include, but are not limited to, (in descending order of resolution) 1 / 64-pel (pixel), 1 / 32-pel, 1 / 16-pel, 1 / 8-pel, 1 / 4-pel, 1 / 2-pel, 1-pel, 2-pel, 4-pel, etc. Thus, each ascending MVD class may be associated with one of these MVD pixel resolutions in non-ascending order. In some implementations, an MVD class may be associated with two or more of the above resolutions, with a higher resolution being lower than or equal to the lower resolution of the preceding MVD class. For example, if MV_CLASS_3 in Table 4 is associated with optional 1-pel and 2-pel resolutions, then the highest resolution that MV_CLASS_4 in Table 4 may be associated with would be 2-pel. In some other implementations, the highest allowable resolution of an MV class may be higher than the lowest allowable resolution of the preceding (lower) MV class. However, the average allowable resolutions of ascending MV classes may simply be in non-ascending order.

[0173]

[0192] In some implementations, when fractional pixel resolution higher than 1 / 8 pel is allowed, the "mv_fr" and "mv_hp" signaling may be correspondingly expanded to a total of more than 3 fractional bits.

[0174]

[0193] In some example implementations, fractional pixel resolution may be allowed for MVD classes below or equal to the threshold MVD class. For example, fractional pixel resolution may be allowed only for MVD_class_0 and not for all other MV classes in Table 4. Similarly, fractional pixel resolution may be allowed only for MVD classes below or equal to any one of the other MV classes in Table 4. For other MVD classes above the threshold MVD class, only integer pixel resolution of the MVD is allowed. In this manner, fractional resolution signaling, such as one or more of the "mv-fr" and / or "mv-hp" bits, may not need to be signaled for MVDs signaled with MVD classes above or equal to the threshold MVD class. For MVD classes having resolutions lower than 1 pixel, the number of bits of the "mv-bit" signaling may be further reduced. For example, for MV_CLASS_5 in Table 4, the range of the MVD pixel offset is (32,64], and therefore 5 bits are required to signal the entire range with 1-pel resolution. However, if MV_CLASS_5 is associated with 2-pel MVD resolution (a resolution lower than 1-pel resolution), 4 bits instead of 5 bits may be required for "mv-bit", and none of "mv-fr" and "mv-hp" need to be signaled following the signaling of "mv_class" as MV-CLASS_5.

[0175]

[0194] In some example implementations, fractional pixel resolution may only be allowed for MVDs with integer values ​​below a threshold integer pixel value. For example, fractional pixel resolution may only be allowed for MVDs smaller than 5 pixels. Corresponding to this example, fractional resolution may be allowed for MV_CLASS_0 and MV_CLASS_1 in Table 4, but not for all other MV classes. As another example, fractional pixel resolution may only be allowed for MVDs smaller than 7 pixels. Corresponding to this example, fractional resolution may be allowed for MV_CLASS_0 and MV_CLASS_1 in Table 4 (which have a range below 5 pixels), but not for MV_CLASS_3 and above (which have a range above 5 pixels). For MVDs belonging to MV_CLASS_2, whose pixel range encompasses 5 pixels, fractional pixel resolution of the MVD may or may not be allowed depending on the "mv-bit" value. Fractional pixel resolution may be allowed if the 'm-bit' value is signaled as 1 or 2 (so the integer part of the signaled MVD is 5 or 6 calculated as the start of the pixel range of MV_CLASS_2 with an offset of 1 or 2 as indicated by 'm-bit'). Otherwise, if the 'mv-bit' value is signaled as 3 or 4 (so that the integer part of the signaled MVD is 7 or 8), fractional pixel resolution may not be allowed.

[0176]

[0195] In some other implementations, for MV classes equal to or greater than the threshold MV class, only a single MVD value may be allowed. For example, such a threshold MV class may be MV_CLASS_2. Thus, MV_CLASS_2 and above may only be allowed to have a single MVD value, and no fractional pixel resolution. The single allowed MVD value for these MV classes may be predefined. In some examples, the single allowed value may be the high end value of the respective range for these MV classes in Table 4. For example, MV_CLASS_2 through MV_CLASS_10 may be greater than or equal to the threshold class of MV_CLASS_2, and the single allowed MVD values ​​for these classes may be predefined as 8, 16, 32, 64, 128, 256, 512, 1024, and 2048, respectively, as shown in Table 5. In other examples, the single allowed value may be the center value of the respective range for these MV classes in Table 4. For example, MV_CLASS_2 through MV_CLASS_10 may exceed the class threshold, and the single allowed MVD values ​​for these classes may be predefined as 3, 6, 12, 24, 48, 96, 192, 384, 768, and 1536, respectively. Any other value within the range may also be defined as the single allowed resolution for the respective MVD class.

[0177] [Table 5]

[0178]

[0196] In the above implementation, when the signaled "mv_class" is equal to or exceeds a predefined MVD class threshold, the "mv_class" signaling alone is sufficient to determine the MVD value. The magnitude and direction of the MVD should then be determined using "mv_class" and "mv_sign."

[0179]

[0197] Thus, when MVD is signaled for only one reference frame (from reference frame list 0 or list 1, but not from both), or when MVD is signaled for two reference frames together, the accuracy (or resolution) of the MVD may depend on the class of the associated motion vector difference and / or the magnitude of the MVD in Table 3. Various other adaptable MVD resolution schemes depending on the MVD magnitude or class are envisioned.

[0180]

[0198] Moving to various composite inter-prediction modes in which each MV is predicted by a reference motion vector and may be coded by an MVD, the two MVDs may be signaled separately or jointly in the bitstream, as described above. Thus, in some example implementations, in addition to the above-described NEAR_NEARMV, NEAR_NEWMV, NEW_NEARMV, NEW_NEWMV, and GLOBAL_GLOBALMV modes, another inter-prediction mode called JOINT_NEWMV may be introduced for a mode in which the MVDs of reference list 0 and reference list 1 are signaled together. Specifically, when the inter-prediction mode is indicated as NEW_NEWMV, the MVDs of reference list 0 and reference list 1 are signaled separately, while when the inter-prediction mode is indicated as JOINT_NEWMV mode, the MVDs of reference list 0 and reference list 1 are signaled together. In particular, for joint MVD, only one MVD called joint_delta_mv may need to be signaled and transmitted in the bitstream, and the MVDs of reference list 0 and reference list 1 may be derived from joint_delta_mv. The derived MVD may then be combined with the reference motion vectors in reference list 0 or reference list 1 to generate two motion vectors for locating reference blocks for hybrid inter prediction.

[0181]

[0199] In some implementations of composite inter prediction, the JOINT_NEWMV mode may be signaled together with the NEAR_NEARMV, NEAR_NEWMV, NEW_NEARMV, NEW_NEWMV, and GLOBAL_GLOBALMV modes. In such implementations, syntax for indicating any one of these alternative composite inter prediction modes may be included in the bitstream at any one of various signaling levels (e.g., sequence level, picture level, frame level, slice level, tile level, superblock level, etc.). Alternatively, the JOINT_NEWMV mode may be implemented as a sub-mode of the NEW_NEWMV mode. In other words, during the NEW_NEWMV mode, the two MVDs of the two reference blocks are either signaled together (thus being a JOINT_NEWMV sub-mode) or not signaled (being another sub-mode of the NEW_NEWMV mode). In such an implementation, a first syntax element may be included in the bitstream to indicate any one of the NEAR_NEARMV, NEAR_NEWMV, NEW_NEARMV, NEW_NEWMV, and GLOBAL_GLOBALMV modes, and when the first syntax element indicates that the NEW_NEWMV mode has been selected for a coding block, a second syntax element may be further included in the bitstream and extractable by a decoder to indicate whether the MVDs of the coding block are signaled separately or together.

[0182]

[0200] For implementations of joint MVD in composite inter prediction, the MVD associated with the reference MV may be derived from a signaled joint MVD from the bitstream, such as the joint_delta_mv described above. Such derivation may involve, for example, scaling the signaled joint MVD to obtain one or both of the two MVDs. In other words, the signaled joint MVD may be scaled before being added to the motion vector predictor (MVP) or the reference MV. As a result of the scaling, the precision or pixel resolution of the scaled MVD may differ from the tolerance precision of the motion vector difference. In some example implementations, such an MVD scaled from the jointly signaled MVD may first be quantized to the tolerance precision of the MVD of the current picture or slice or tile or superblock or coded block before being added to the reference MVD for generating a motion vector.

[0183]

[0201] In some example implementations, the frame index of a reference frame in a composite inter-prediction mode may be signaled in a bitstream. The frame index may correspond to a picture order counter (POC) associated with the reference frame. The distance between the reference frame and the current frame may be defined and expressed as the difference between the corresponding POCs. The direction of the reference frame (before or after the current frame) may be expressed by a code. Thus, a signed distance may be used to represent the position of the reference frame relative to the current frame. The reference frames for the composite inter-prediction mode may be referred to as a first reference frame and a second reference frame.

[0184]

[0202] In some example implementations, when JOINT_NEWMV mode is signaled and the POC distances between two reference frames and the current frame are different, the MVD may be scaled for reference list 0 or reference list 1 based on the POC distances. Specifically, the distance between reference frame list 0 and the current frame may be represented as td0, and the distance between reference frame list 1 and the current frame may be represented as td1. When td0 is equal to or greater than td1, joint_mvd may be used directly for reference list 0, and the MVD of reference list 1 may be derived from joint_mvd based on equation (1).

[0185]

number

[0186]

[0203] Otherwise, if td1 is equal to or greater than td0, then joint_mvd is used directly for reference list 1, and the MVD for reference list 0 is derived from joint_mvd based on equation (2).

[0187]

number

[0188]

[0204] Returning to the motion vector predictors (MVPs) and MVP lists discussed above, in some example implementations, such a motion vector predictor list may be established for a group of coding blocks (e.g., a superblock) that includes motion vector candidates for predicting a motion vector for each predictive block of the superblock. The maximum size of the MVP list may be predefined or configured. Candidates in the MVP list may be established using a predefined set of rules. For example, these candidates may be selected from already reconstructed motion vectors belonging to blocks spatially close to the current coding block or superblock (referred to as spatial motion vector predictors, or SMVPs). Alternatively or additionally, these candidates may belong to blocks in a reference frame of the current coding block or superblock (referred to as temporal motion vector predictors, or TMVPs). Spatial motion vector predictors may be adjacent or non-adjacent SMVPs. Adjacent SMVPs may refer to motion vector predictors belonging to predictive blocks adjacent to the current coding block or superblock. Non-adjacent SMVPs may refer to motion vector predictors belonging to predictive blocks that are not immediately next to the current coding block or superblock. Other types of MVP candidates may be further derived from the reconstructed motion vectors. As another example, as described in more detail below, one or more additional MVP banks may be maintained as one of the sources for establishing the MVP list.

[0189]

[0205] The MVP list may be constructed to hold a predetermined number of reconstructed MVP candidates (SMVP, TMVP, or other derived MVPs, or other types of MVP candidates) on both the encoder and decoder sides of the current coding block or superblock. When encoding the current prediction block in inter-prediction mode, the encoder should select an MVP from the candidates in the MVP candidate list that provides optimal coding efficiency as a predictor of the motion vector for the current prediction block. The index of the selected MVP in the MVP list may be signaled in the bitstream. Correspondingly, the decoder should update the MVP list for the current coding block or superblock at the same time the bitstream is reconstructed, extract the MVP index of the current inter-predicted prediction block, obtain the MVP from the MVP candidate list according to the extracted MVP index in the MVP list, and use the MVP as a predictor of the motion vector of the current prediction block to reconstruct the motion vector of the current prediction block (e.g., by combining the motion vector predictor extracted from the MVP list with the corresponding MVD). For example, the MVP list may represent a stack having a predetermined fixed size.

[0190]

[0206] For example, the SMVP may be derived from spatial neighbor prediction blocks, including adjacent spatial neighbor prediction blocks and non-adjacent spatial neighbor prediction blocks, where adjacent spatial neighbor prediction blocks are directly adjacent to the current block or superblock to the top or left of the block or superblock (assuming they are previous blocks that have already been reconstructed), and non-adjacent spatial neighbor prediction blocks are close to but not directly adjacent to the current block or superblock.

[0191]

[0207] In some other example implementations, an inter-coding mode using adaptive MVD resolution (AMVD), sometimes named AMVDMV, is added to the single reference case. When the AMVDMV mode is selected, this indicates that AMVD is applied to the signal MVD.

[0192]

[0208] In some implementations, a flag, sometimes named amvd_flag, is added during JOINT_NEWMV mode to indicate whether AMVD is applied to the joint MVD coding mode. When adaptive MVD resolution is applied to the joint MVD coding mode, named joint AMVD coding, the MVDs of two reference frames are signaled together, and the precision of the MVD is implicitly determined by the MVD magnitude. Otherwise, the MVDs of two (or more) reference frames are signaled together, and conventional MVD coding is applied.

[0193]

[0209] In some other example implementations, several composite inter modes, such as those shown in Figure 18, may be used as inter prediction modes. Essentially, these modes create a prediction for block 1810 of current frame 1802 by combining two hypotheses from two different reference frames: reference frame 0 (1804) with motion vector 0 (MV0) and reference frame 1 (1806) with motion vector 1 (MV1). Thus, in this mode, two motion information components (e.g., MV0 and MV1) may be sent in the bitstream per block as overhead. In some implementations, the two reference frames may be two adjacent neighboring frames, e.g., reference frame 0 is the previous neighboring frame and reference frame 1 is the next neighboring frame.

[0194]

[0210] In some implementations, motion vectors may be predicted using predictors from spatial and temporal neighbors or historical motion vectors, but the bytes used for motion information may still be very significant for many content and applications.

[0195]

[0211] In some other example implementations, a temporal interpolated prediction (TIP) mode / process may be used for some inter-prediction modes, as shown in Figure 19. In this process, information in two reference frames is used to interpolate between reference frame 0 (F i-1 , 1904) and reference frame 1 (F i+1 , 1906) and uses a simple interpolation process to estimate the current frame (F i , 1901 and / or 1902) may be combined and projected to the same time instance. Multiple TIP modes are supported. In one TIP mode, a temporally interpolated frame (TIP frame, 1902) may be used as an additional reference frame. In some implementations, a motion vector (MV0) may be used to predict block 1910 in the current frame (1901) based on the TIP frame (1902).

[0196]

[0212] In some implementations, the current frame F i The coding blocks of F may directly reference the interpolated frame and utilize information derived from two different references with only the overhead cost of a single inter prediction mode. In another TIP mode, the temporally interpolated frame is used to encode the current frame F while skipping any other traditional encoding steps. i may be directly assigned as the output of the decoding process for . This mode may exhibit significant coding and complexity benefits, especially for low bitrate applications.

[0197]

[0213] In some implementations, there may be several complex methods for interpolating frames between two reference frames, such as frame rate up-conversion (FRUC) techniques, and achieving a good trade-off between complexity and compression quality may be a necessary constraint when designing new encoding tools. A simple method may include reusing already available motion information in reference frames without the need to perform any additional motion search. Simulation results show that this simple method may achieve good quality while maintaining a low-complexity implementation, which leads to improved coding efficiency.

[0198]

[0214] In some other example implementations, adaptive motion vector resolution (AMVR) may be used to support several MV precisions. For example, one AMVR scheme may support a total of seven MV precisions (8, 4, 2, 1, 1 / 2, 1 / 4, 1 / 8). For each prediction block, the AVM encoder searches all supported precision values ​​and signals the best precision to the decoder.

[0199]

[0215] In some implementations, two precision sets are supported to reduce the encoder's execution time. Each precision set contains four predefined precisions. The precision set is adaptively selected at the frame level based on the frame's maximum precision value. The maximum precision may be signaled in the frame header. Table 6 shows an example outlining the supported precision values ​​based on the frame-level maximum precision.

[0200] [Table 6]

[0201]

[0216] In some implementations, a frame-level flag may be used to indicate whether the MV of a frame includes sub-pixel (pel) precision. AMVR is enabled only when the value of the cur_frame_force_integer_mv flag is 0. In AMVR, when the block precision is less than maximum precision, the motion model and interpolation filter are not signaled. When the block precision is less than maximum precision, the motion mode is inferred to be translational motion, and the interpolation filter is inferred to be a normal interpolation filter. Similarly, when the block precision is 4-pel or 8-pel, the inter-intra mode is not signaled and is inferred to be 0.

[0202]

[0217] In various embodiments of the present disclosure, the precision of an MV is expressed in values ​​(typically in pixels (pels)). Precisions with larger values ​​may be referred to as coarser precisions (or lower precisions), and / or precisions with smaller values ​​may be referred to as finer precisions (or higher precisions). For example, a first precision with 4-pels is coarser (or lower) than a second precision with 1-pel, and a third precision with 1 / 2-pels is finer (or higher) than a fourth precision with 2-pels.

[0203]

[0218] In various embodiments of the present disclosure, the precision with the maximum allowable value may be referred to as the minimum precision, and the precision with the minimum allowable value may be referred to as the maximum precision.

[0204]

[0219] In various embodiments of the present disclosure, when the precision value increases from a smaller value to a larger value, this may be referred to as "downshifting" the precision, i.e., changing the precision from finer precision (higher precision) to coarser precision (lower precision), and when the precision value decreases from a larger value to a smaller value, this may be referred to as "upshifting" the precision, i.e., changing the precision from coarser precision (lower precision) to finer precision (higher precision).

[0205]

[0220] In some other example implementations, a warped motion mode may be used. Conventional motion compensation may assume a translational motion model between the reference block and the target block, and warped motion may utilize an affine model. For example, the affine motion model may be represented by equation (3).

[0206]

number

[0207]

[0221] where [x,y] are the coordinates of the original pixel, and [x',y'] are the warped coordinates of the reference block.,To specify the warped motion, up to six parameters are required,,as can be seen from equation (3): a3 and b3 specify the,conventional translation MV, a1 and b2 specify the scaling along the MV,,and a2 and b1 specify the rotation.

[0208]

[0222] In some implementations of global distortion motion compensation, global motion information is signaled for each inter reference frame, and the inter reference frame includes a global motion type and several motion parameters. Table 7 shows examples of global motion types and the number of associated parameters.

[0209] [Table 7]

[0210]

[0223] In some implementations, after signaling the reference frame index, when global motion is selected, the global motion type and parameters associated with the given reference frame are used for the current coding block.

[0211]

[0224] In some implementations of local distortion motion compensation, for inter-coded blocks, local distortion motion may be allowed when some or all of the following conditions are met: the current block uses a single reference prediction, the width or height of the coded block is 8 or greater, and / or at least one of the adjacent neighboring blocks uses the same reference frame as the current block.

[0212]

[0225] In some implementations, when local distortion motion is used for the current block, the affine model parameters are estimated by mean-square minimization of the difference between the reference and the modeled projection based on the MV of the current block and its adjacent neighboring blocks. To estimate the parameters of the local distortion motion, when the neighboring block uses the same reference frame as the current block, a projection sample pair is obtained, which is the center sample in the neighboring block and its corresponding sample in the reference frame. After this, three spare samples are generated by shifting the center position by four equal samples in one or both dimensions. These spare samples are also considered to be projection sample pairs to ensure the stability of the model parameter estimation process.

[0213]

[0226] In some implementations, the MVs of neighboring blocks are used to derive motion parameters, called motion samples. The motion samples are selected from neighboring blocks that use the same reference frame as the current block. Note that the warped motion prediction mode is only enabled for blocks that use a single reference frame.

[0214]

[0227] For example, referring to FIG. 20, relative to a current block (2080), the motion vectors of neighboring blocks B0 (2000), B1 (2010), and B2 (2020) are referred to as MV0, MV1, and MV2, respectively. The current block is predicted using uni-prediction with reference frame Ref0. The neighboring block B0 (2000) is predicted using mixed prediction with reference frames Ref0 and Ref1, the neighboring block B1 (2010) is predicted using uni-prediction with reference frame Ref0, and the neighboring block B2 (2020) is predicted using mixed prediction with reference frames Ref0 and Ref2. The motion vector MV0 of B0 Ref0 , B1 MV1 Ref0 , and MV2 for B2 Ref0 may be used as the motion sample to derive the affine motion parameters of the current block.

[0215]

[0228] In some other example implementations, a local warp extension mode and / or a local warp delta mode may be supported as two types of warp motion models. In addition to translational motion, AVM also supports warp motion compensation. A global warp model is associated with each reference frame, with each of the four non-translation parameters (e.g., a_1, a_2, b_1, and b_2 in equation (3)) having 12-bit precision, and the translational motion vectors (e.g., a_3 and b_3 in equation (3)) being coded with 15-bit precision. A coding block may provide a reference frame index and choose it for direct use. The global warp model captures frame-level scaling and rotation and thus primarily focuses on robust motion across the entire frame. Additionally, a local warp model at the coding block level may also be supported. In the local warp mode, also known as WARPED_CAUSAL, the warp parameters of the current block are derived by fitting a model to nearby motion vectors using least squares.

[0216]

[0229] In some implementations, the new warp motion mode, warp extension mode, may be called WARP_EXTEND. The idea of ​​this mode is to smoothly extend the motion of neighboring blocks into the current block, but with some ability to modify the warp parameters. This allows complex warp motion to be represented and spread across multiple blocks while minimizing block artifacts.

[0217]

[0230] In some implementations, to achieve this, the WARP_EXTEND mode applied to the NEWMV block constructs a new warp model based on two constraints: the first constraint is that the per-pixel motion vectors generated by the new warp model may follow the per-pixel motion vectors in the neighboring block, and the second constraint is that the pixel at the center of the current block may have a per-pixel motion vector that matches the signaled motion vector of the block as a whole. For example, referring to Figure 21, when the neighboring block (2120) to the left of the current block (2110) is warped, a model for fitting motion vectors may be used.

[0218]

[0231] In some implementations, these two constraints imply certain equations involving the distortion parameters of the neighboring block and the current block. These equations can then be solved to calculate the distortion model of the current block. Mathematically, when (A,...,F) represents the neighboring distortion model and (A',...,F') represents the new distortion model, the following equation applies for each point along the common edge according to the first constraint:

[0219]

number

[0220]

[0232] In some implementations, points along an edge have different y values, but they all have the same x value. This means that the y coefficient must be the same on both sides of equation (4) (so B'=B and D'=D), while the x coefficient satisfies the two equations with respect to the other coefficients. B'=B D'=D A'x+E'=Ax+E C'x+F'=Cx+F (5)

[0221]

[0233] In the last two equations of equation (5), x is the horizontal position of a vertical column of pixels and is therefore effectively a constant.

[0222]

[0234] In some implementations, the second constraint specifies that the motion vector at the center of the block must be equal to the one signaled using the NEWMV mechanism. This can result in two additional equations, resulting in six simultaneous equations in six variables with a unique solution. These equations can be solved very efficiently in both software and hardware, and the solutions boil down to basic addition / subtraction / multiplication and division by powers of two. Therefore, this mode is significantly simpler than existing least-squares-based local distortion tools.

[0223]

[0235] In some implementations, when there are multiple neighboring blocks that can be used to extend from, some method must be used to select which block to extend. This is a similar problem to that encountered in motion vector prediction: when there are several possible motion vectors from nearby blocks, one motion vector must be selected as the basis for NEWMV encoding. A similar solution may be used and extended to handle the need for WARP_EXTEND. This may be done by tracking the source of each motion vector prediction. Thus, WARP_EXTEND is only enabled when the selected motion vector prediction is taken directly from a neighboring block. Therefore, the block may be used as a single "neighboring block" in the remainder of the algorithm.

[0224]

[0236] In some implementations, to make this case inexpensive to code, WARP_EXTEND is allowed to be used for a NEAR MV block when the neighbor's distortion model is very good as is without requiring any further modifications. The neighbor selection is the same as for NEW MV, except that this time the neighbor may need to be warped rather than simply translational. Thus, when this is true and WARP_EXTEND is selected, the neighbor's distortion model parameters may be copied to the current block.

[0225]

[0237] In some implementations, one of the motion modes is a warp delta mode, called WARP_DELTA. In this mode, the distortion model of a block is coded as a delta from a predicted distortion model, similar to how motion vectors are coded as deltas from a predicted motion vector. The prediction may be obtained from the global motion model (if any) or from neighboring blocks.

[0226]

[0238] In some implementations, to avoid having multiple schemes that encode the same predicted distortion model, some or all of the following restrictions may apply: when the mode is NEARMV or NEWMV, the same neighbor selection logic as described may be used for WARP_EXTEND, and when this results in a warped neighboring block, the model of the block (without applying the rest of the WARP_EXTEND logic) may be used as the prediction. In some implementations, when any or all of the above restrictions are not met, the global distortion model may be used as the base.

[0227]

[0239] In some implementations, a delta may be coded for each non-translation parameter, and / or the translation portion of the model may be adjusted so that the motion vector for each pixel at the center of the block matches the overall motion vector for the block. Methods that involve explicitly coding a delta for each warp parameter may use more bits to encode than other warp modes. Thus, WARP_DELTA may be disabled for blocks that are relatively small in size, such as blocks smaller than 16x16. In some implementations, the decoding logic may be quite simple and can represent more complex motion that other warp modes cannot.

[0228]

[0240] In some other example implementations, merge with motion vector difference (MMVD) may be used for skip or merge modes with motion vector representation methods. In some implementations, MMVD reuses merge candidates, and the merge candidates may be used in VVC. A candidate may be selected from among the merge candidates and further extended by the proposed motion vector representation method. In some implementations, MMVD provides simplified signaling for new motion vector representations. The representation method includes a starting point, a motion magnitude, and a motion direction. In some implementations, the MMVD technique uses a merge candidate list, and the merge candidate list may be used in VVC. In some implementations, only candidates with the default merge type (MRG_TYPE_DEFAULT_N) are considered for MMVD extension.

[0229]

[0241] In some implementations, the base candidate index defines the starting point. Table 8 shows some example base candidate indices that indicate the best candidate from among the candidates in the list. In some implementations, when the number of base candidates is equal to 1, the base candidate IDX may not be signaled.

[0230] [Table 8]

[0231]

[0242] In some implementations, the distance index may indicate motion magnitude information. The distance index indicates a predefined distance from the starting point information. Table 2 may be used as an example of the predefined distance.

[0232]

[0243] In some implementations, the direction index represents the direction of the MVD relative to the start point. Table 3 may be used as an example of direction indexes representing four directions.

[0233]

[0244] In some implementations, the MMVD flag may be signaled immediately after sending the skip and merge flags. When the skip and merge flags are true, the MMVD flag is parsed. When the MMVD flag is equal to 1, the MMVD syntax is parsed. When the MMVD flag is not 1, the AFFINE flag is parsed. When the AFFINE flag is equal to 1, this is AFFINE mode, and / or when the AFFINE flag is not 1, the skip / merge index is parsed for the VTM's skip / merge mode.

[0234]

[0245] In some implementations, only coded block-level MVs are stored in memory and used for spatial and temporal motion vector prediction. Increasingly, coding tools such as local distortion motion (including distortion extension and distortion delta), temporal interpolation prediction, and bidirectional optical flow create sub-block (e.g., 8x8) MVs for finer motion compensation. When these sub-block MVs are discarded after the corresponding motion compensation, this can result in suboptimal performance for motion vector prediction. This disclosure describes various embodiments for sub-block motion vector storage and use, thus improving video encoding / decoding techniques and / or increasing encoding / decoding efficiency.

[0235]

[0246] The various embodiments and / or implementations described in this disclosure may be implemented separately or in combination in any order. Furthermore, each of the method (or embodiment), encoder, and decoder may be implemented by processing circuitry (e.g., one or more processors or one or more integrated circuits). The one or more processors execute programs stored in a non-transitory computer-readable medium. In this disclosure, the term block may be interpreted as a prediction block, a coding block, or a coding unit (CU).

[0236]

[0247] 22 shows a flowchart 2200 of an example method according to the principles underlying the above-described implementations for sub-block motion vector storage and use. The example decoding method flow begins at 2201 and may include some or all of the following steps: S2210 (receiving a coded video bitstream); S2220 (determining, based on the coded video bitstream, a current block of a current frame to be inter-predicted by at least one reference block of at least one reference frame); S2230 (determining, based on the coded video bitstream, motion vectors (MVs) for the current block corresponding to each of the at least one reference block, where the MV of the current block will be predicted by the reference MV); and / or S2240 (identifying, by the device, a sub-block MV of the MV reference block as a reference MV for predicting the MV of the current block). The example method stops at S2299.

[0237]

[0248] The sub-block MVs of the MV reference block are stored for use as motion vector predictors (MVPs). In some implementations, one or more sub-block MVs after motion compensation may be stored for use as MVPs in predicting neighboring blocks.

[0238]

[0249] In any part or combination of the above implementations, before identifying a sub-block MV of the MV reference block as a reference MV for predicting the MV of the current block, method 2200 may further include predicting the MV reference block by sub-block motion compensation to calculate the sub-block MV, wherein the sub-block motion compensation includes at least one of local distortion motion prediction, bidirectional optical flow, or smooth inter prediction mode.

[0239]

[0250] In any part or combination of the above implementations, before identifying a sub-block MV of the MV reference block as a reference MV for predicting the MV of the current block, method 2200 may further include predicting the MV reference block by warped motion prediction to calculate the sub-block MV, where the warped motion prediction includes at least one of local warped motion prediction, global warped motion prediction, warped extension mode, or warped delta mode.

[0240]

[0251] In any part or combination of the above implementations, before identifying the sub-block MV of the MV reference block as the reference MV for predicting the MV of the current block, method 2200 may further include predicting the MV reference block by warped motion prediction to calculate the sub-block MV, and / or downshifting the precision of the sub-block MV to maximum precision for the MV of the current block.

[0241]

[0252] In any part or combination of the above implementations, before identifying the sub-block MV of the MV reference block as the reference MV for predicting the MV of the current block, method 2200 may further include predicting the MV reference block by warped motion prediction to calculate the sub-block MV, and / or downshifting the precision of the sub-block MV to maximum precision for the motion vector predictor (MVP).

[0242]

[0253] In any part or combination of the above implementations, before identifying a sub-block MV of the MV reference block as a reference MV for predicting the MV of the current block, method 2200 may further include predicting the MV reference block by a bidirectional optical flow mode to calculate the sub-block MV.

[0243]

[0254] In any part or combination of the above implementations, before identifying the sub-block MV of the MV reference block as the reference MV for predicting the MV of the current block, method 2200 may further include predicting the MV reference block by a bidirectional optical flow mode to calculate the sub-block MV, and / or downshifting the precision of the sub-block MV to maximum precision for motion compensation, or downshifting the precision of the sub-block MV to maximum precision for a motion vector predictor (MVP).

[0244]

[0255] In any part or combination of the above implementations, before identifying a sub-block MV of the MV reference block as a reference MV for predicting the MV of the current block, method 2200 may further include predicting the MV reference block by a sub-block motion compensation mode to calculate a sub-block MV and a block MV, and / or the step of identifying a sub-block MV of the MV reference block as a reference MV for predicting the MV of the current block may include selecting a sub-block MV from the sub-block MV and the block MV as the reference MV for predicting the MV of the current block based on at least one of code information, mode information, or a type of MVP.

[0245]

[0256] In any part or combination of the above implementations, before identifying a sub-block MV of the MV reference block as a reference MV for predicting the MV of the current block, method 2200 may further include predicting the MV reference block by at least one sub-block motion compensation mode to calculate multiple sub-block MVs, and / or obtaining a weighted average of multiple sub-block MVs as the sub-block MV of the MV reference block.

[0246]

[0257] In any part or combination of the above implementations, before identifying a sub-block MV of the MV reference block as a reference MV for predicting the MV of the current block, method 2200 may further include predicting the MV reference block by a warped motion prediction mode to calculate the sub-block MV, and / or the MV of the current block is predicted by a spatial motion vector prediction (SMVP) mode based on the sub-block MV, and / or the sub-block MV corresponds to the access position or center position of the MV reference block.

[0247]

[0258] In any part or combination of the above implementations, before identifying a sub-block MV of the MV reference block as a reference MV for predicting the MV of the current block, method 2200 may further include predicting the MV reference block by a warped motion prediction mode to calculate the sub-block MV, and / or the MV of the current block is predicted by a temporal-based vector prediction mode or warped motion prediction based on the sub-block MV.

[0248]

[0259] In any part or combination of the above implementations, before identifying a sub-block MV of the MV reference block as a reference MV for predicting the MV of the current block, method 2200 may further include predicting the MV reference block by a bidirectional optical flow mode to calculate the sub-block MV, and / or the MV of the current block is predicted by a temporal-based motion vector prediction mode based on the sub-block MV.

[0249]

[0260] In any part or combination of the above implementations, before identifying the sub-block MV of the MV reference block as the reference MV for predicting the MV of the current block, method 2200 may further include extracting, from the coded video bitstream, high-level syntax indicating whether the sub-block MV is used to predict the MV of the current block at one of the following levels: sequence level, frame level, picture level, or slice level.

[0250]

[0261] In any part or combination of the above implementations, in response to the current block being at the top of a coding tree unit (CTU) and the sub-block MV corresponding to the neighboring sub-block above the current block, the MV of the current block is predicted based on the sub-block MV by a motion vector prediction mode other than spatial motion vector prediction (SMVP).

[0251]

[0262] In various embodiments of this disclosure, the precision of an MV is expressed in values ​​(typically in pixels (pels)). The precision with the smallest allowed value may be referred to as the maximum precision. When the precision value increases from a smaller value to a larger value, this may be referred to as "downshifting" the precision, i.e., changing the precision from a finer (or higher) precision to a coarser (or lower) precision.

[0252]

[0263] In various embodiments in the present disclosure, the sub-block MVs of the current block may be stored in memory after corresponding sub-block motion compensation, such as (local distortion, or bidirectional optical flow, smooth inter prediction mode), so that these sub-block MVs are ready to be used as candidate MVs in the (temporal and / or spatial) MVP of the next block during the video encoding / decoding process.

[0253]

[0264] In one non-limiting example, when a current block uses warped motion prediction, such as, but not limited to, local warped motion, global warped motion, warped extension mode, warped delta mode, etc., sub-block MVs are calculated and stored in memory. These stored sub-block MVs are ready to be used as candidate MVs in the (temporal and / or spatial) MVP of the next block during the video encoding / decoding process.

[0254]

[0265] In another non-limiting example, when a current block uses warped motion prediction, sub-block MVs are calculated, downshifted to the highest allowed MV precision (e.g., 1 / 16 luma pel in some cases) for motion compensation of the next block to save internal memory, and stored in memory. These stored sub-block MVs are ready to be used as candidate MVs in the (temporal and / or spatial) MVP of the next block during the video encoding / decoding process.

[0255]

[0266] In another non-limiting example, when a current block uses warped motion prediction, sub-block MVs are calculated, downshifted to the highest MVP precision (e.g., 1 / 8 luma pel in some cases) and stored in memory to save internal memory. These stored sub-block MVs are ready to be used as candidate MVs in the (temporal and / or spatial) MVP of the next block during the video encoding / decoding process.

[0256]

[0267] In another non-limiting example, when the current block uses bidirectional optical flow (such as OPFL mode in AVM), the sub-block MVs for bidirectional optical flow motion compensation are stored in memory in the same manner as described in any of the above examples. These stored sub-block MVs are ready to be used as candidate MVs in the (temporal and / or spatial) MVP of the next block during the video encoding / decoding process.

[0257]

[0268] In another non-limiting example, both block-level MVs and sub-block-level MVs are stored in memory, and selection for block-level MVs and sub-block-level MVs can be conditionally performed based on code information or based on different modes or types of (temporal and / or spatial) MVPs. These stored sub-block MVs are ready to be used as candidate MVs in the (temporal and / or spatial) MVP of the next block during the video encoding / decoding process.

[0258]

[0269] In another non-limiting example, a sub-block level MV may result from a weighted average of multiple sub-blocks coded using separate motion compensation with different MVs, such as sub-blocks in motion warping. That is, the sub-block size for MV storage and the sub-block size for motion compensation can be different. These stored sub-block MVs are ready to be used as candidate MVs in the (temporal and / or spatial) MVP of the next block during the video encoding / decoding process.

[0259]

[0270] In various embodiments of the present disclosure, the sub-block MVs are stored and the stored sub-block MVs may be used in the temporal and / or spatial MVPs of the next block.

[0260]

[0271] In one non-limiting example, sub-block MVs from warped motion prediction are used for spatial motion vector prediction, including but not limited to adjacent spatial motion vector predictors, non-adjacent spatial motion vector predictors, and reference MV banks. During spatial motion vector prediction (SMVP), sub-block MVs at the access position or sub-block MVs at the center position of neighboring warped motion prediction blocks can be used as spatial MV predictors.

[0261]

[0272] In another non-limiting example, the sub-block MVs from the distorted motion prediction are used for temporal-based motion vector prediction, such as, but not limited to, temporal motion vector prediction (TMVP) or temporal interpolated prediction (TIP).

[0262]

[0273] In another non-limiting example, sub-block MVs from warp motion prediction are used for distortion model calculation, i.e., during collection of neighboring block MVs, sub-block MVs are used instead of the entire block MV when the neighboring block uses warp motion.

[0263]

[0274] In another non-limiting example, sub-block MVs from bidirectional optical flow are only allowed to be used for temporal-based motion vector prediction, and SMVP blocking and distortion model calculation usage will reduce the complexity of the hardware implementation.

[0264]

[0275] In another non-limiting example, one high-level syntax is signaled at the sequence level / frame level / picture level / slice level to specify whether a sub-block motion vector is available for the next block. The syntax elements for the high-level syntax may be included in one of a slice header, a picture header, a sequence header, a video parameter set (VPS), a picture parameter set (PPS), a sequence parameter setting (SPS), etc.

[0265]

[0276] In various embodiments of the present disclosure, to reduce line memory requirements, when a block is at the top of a superblock (e.g., CTU), sub-block MVs from neighboring sub-blocks above the current block (including the top left and top right, or MV banks) are not allowed to be used as the SMVP of the current block.

[0266]

[0277] In the embodiments and implementations of the present disclosure, any steps and / or operations may be combined or arranged in any quantity or order as desired. Two or more of the steps and / or operations may be performed in parallel. The embodiments and implementations of the present disclosure may be used separately or in combination in any order. Furthermore, each of the method (or embodiment), encoder, and decoder may be implemented by processing circuitry (e.g., one or more processors or one or more integrated circuits). In one example, the one or more processors execute a program stored on a non-transitory computer-readable medium. The embodiments of the present disclosure may apply to luma blocks or chroma blocks. The term block may be interpreted as a prediction block, a coding block, or a coding unit (CU). The term block here may also be used to refer to a transform block. In the following sections, when referring to a block size, the block size may refer to the block width or height, the maximum value of the width and height, the minimum value of the width and height, the area size (width * height), or the aspect ratio of the block (width:height or height:width).

[0267]

[0278] The techniques described above can be implemented as computer software using computer-readable instructions and physically stored on one or more computer-readable media. For example, Figure 23 illustrates a computer system (2300) suitable for implementing certain embodiments of the disclosed subject matter.

[0268]

[0279] Computer software can be coded using any suitable machine or computer language, which may be subjected to mechanisms such as assembling, compiling, linking, etc. to produce code comprising instructions that can be executed by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., either directly or through translation, microcode execution, etc.

[0269]

[0280] The instructions may be executable by various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet of Things devices, and the like.

[0270]

[0281] 23 for computer system (2300) are exemplary in nature and are not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing embodiments of the present disclosure. The arrangement of components should not be interpreted as having any dependency or requirement regarding any one or combination of components illustrated in the exemplary embodiment of computer system (2300).

[0271]

[0282] The computer system (2300) may include certain human interface input devices that can respond to input by one or more human users through, for example, tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, clapping), visual input (e.g., gestures), and olfactory input (not shown). The human interface devices can also be used to capture certain media that are not necessarily directly related to conscious human input, such as audio (e.g., speech, music, ambient sounds), images (e.g., scanned images, photographic images obtained from a still image camera), and video (e.g., two-dimensional video, three-dimensional video, including stereoscopic video).

[0272]

[0283] The input human interface devices may include one or more (only one of each depicted) of a keyboard (2301), a mouse (2302), a trackpad (2303), a touchscreen (2310), a data glove (not shown), a joystick (2305), a microphone (2306), a scanner (2307), and a camera (2308).

[0273]

[0284] The computer system (2300) may further include certain human interface output devices. Such human interface output devices may stimulate one or more of the human user's senses, for example, through tactile output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touchscreen (2310), data gloves (not shown), or joystick (2305), although haptic feedback devices that do not function as input devices are also possible), audio output devices (such as speakers (2309), headphones (not shown)), visual output devices (such as screens (2310), to include CRT screens, LCD screens, plasma screens, and OLED screens, each with or without touchscreen input capability and each with or without haptic feedback capability—some of which may be capable of outputting two-dimensional visual output or three- or more-dimensional output through means such as stereoscopic output, virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).

[0274]

[0285] The computer system (2300) may also include human-accessible storage devices and their associated media, such as optical media including CD / DVD ROM / RW (2320), including media such as CD / DVD (2321), thumb drives (2322), removable hard drives or solid state drives (2323), older magnetic media such as tape and floppy disks (not shown), and specialized ROM / ASIC / PLD-based devices (not shown) such as security dongles.

[0275]

[0286] Furthermore, those skilled in the art will understand that the term "computer-readable medium" as used in connection with the subject matter of this disclosure does not encompass transmission media, carrier waves, or other transient signals.

[0276]

[0287] The computer system 2300 may also include an interface 2354 to one or more communications networks 2355. For example, the networks may be wireless, wireline, or optical. Furthermore, the networks may be local, wide-area, metropolitan, vehicular, and industrial, real-time, delay-tolerant, and the like. Examples of networks include local area networks such as Ethernet; wireless LANs; cellular networks, such as to include GSM, 3G, 4G, 5G, LTE, and the like; TV wireline or wireless wide-area digital networks, such as to include cable TV, satellite TV, and terrestrial television broadcast; and vehicular and industrial networks, such as to include CAN buses. Particular networks generally require an external network interface adapter attached to a particular general-purpose data port or peripheral bus 2349 (e.g., a USB port on the computer system 2300); others are generally integrated into the core of the computer system 2300 by attachment to a system bus, as described below (e.g., an Ethernet interface to a PC computer system, or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system (2300) can communicate with other entities. Such communication can be one-way receive-only (e.g., television broadcast), one-way transmit-only (e.g., CANbus to a specific CANbus device), or bidirectional, e.g., to other computer systems using local or wide-area digital networks. Specific protocols and protocol stacks are available for use with each of these networks and network interfaces, as described above.

[0277]

[0288] The aforementioned human interface devices, human-accessible storage devices, and network interfaces may be attached to the core (2340) of the computer system (2300).

[0278]

[0289] The core (2340) may include one or more central processing units (CPUs) (2341), graphics processing units (GPUs) (2342), specialized programmable processing units (2343) in the form of field programmable gate areas (FPGAs), hardware accelerators (2344) for specific tasks, graphics adapters (2350), etc. These devices may be connected through a system bus (2348), along with read-only memory (ROM) (2345), random access memory (2346), internal mass storage (2347) such as an internal non-user-accessible hard drive, SSD, etc. In some computer systems, the system bus (2348) may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may be attached directly to the core's system bus (2348) or through a peripheral bus (2349). In one example, the screen (2310) can be connected to a graphics adapter (2350). Peripheral bus architectures include PCI, USB, and the like.

[0279]

[0290] The CPU (2341), GPU (2342), FPGA (2343), and accelerator (2344) can execute specific instructions that can combine to make up the aforementioned computer code. This computer code can be stored in ROM (2345) or RAM (2346). Temporary data can also be stored in RAM (2346), while permanent data can be stored, for example, in internal mass storage (2347). Rapid storage and retrieval to and from any of the memory devices can be enabled through the use of cache memory, which can be closely associated with one or more of the CPU (2341), GPU (2342), mass storage (2347), ROM (2345), RAM (2346), etc.

[0280]

[0291] The computer-readable medium can bear computer code for performing various computer-executed operations. The medium and computer code can be those specially designed and constructed for the purposes of the present disclosure, or they can be of the kind well known and available to those skilled in the computer software arts.

[0281]

[0292] As a non-limiting example, a computer system (2300) having an architecture, and specifically a core (2340), can provide functionality as a result of a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media can be user-accessible mass storage, as introduced above, as well as media associated with specific storage of the core (2340) that is non-transitory in nature, such as core internal mass storage (2347) or ROM (2345). Software implementing various embodiments of the present disclosure can be stored in such devices and executed by the core (2340). The computer-readable media can include one or more memory devices or chips, depending on particular needs. The software can cause the core (2340), and specifically the core's processor (including a CPU, GPU, FPGA, etc.), to perform certain processes, or certain portions of certain processes, described herein, including defining data structures stored in RAM (2346) and modifying such data structures in accordance with the software-defined processes. Additionally or alternatively, a computer system may provide functionality as a result of logic hardwired or otherwise embodied in circuitry (e.g., accelerator (2344)) that can operate in place of or together with software to perform particular processes, or particular portions of particular processes, described herein. References to software may encompass logic, and vice versa, where appropriate. References to computer-readable media may encompass, where appropriate, circuitry (such as an integrated circuit (IC)) that stores software for execution, circuitry that embodies logic for execution, or both. The present disclosure encompasses any appropriate combination of hardware and software.

[0282]

[0293] While this disclosure has described several exemplary embodiments, there are alterations, permutations, and various substitute equivalents that are within the scope of this disclosure. It will thus be recognized that those skilled in the art will be capable of devising numerous systems and methods that, while not explicitly shown or described herein, embody the principles of the present disclosure and are therefore within the spirit and scope of the present disclosure.

[0283] Appendix A: Acronyms IBC: Intra-block copy IntraBC: Intrablock copy JEM: Joint Exploration Model VVC: Versatile Video Coding BMS: Benchmark Set MV: Motion Vector HEVC: High Efficiency Video Coding SEI:Supplementary Enhancement Information VUI: Video Usability Information GOP: Group of Pictures TU: Conversion unit PU: Prediction Unit CTU: Coding Tree Unit CTB: coding tree block PB: Predicted Block HRD: Hypothetical Reference Decoder SNR: Signal to Noise Ratio CPU: Central Processing Unit GPU: Graphics Processing Unit CRT: cathode ray tube LCD: Liquid crystal display OLED: Organic Light Emitting Diode CD: Compact Disc DVD: Digital Video Disc ROM: Read-only memory RAM: Random Access Memory ASIC: Application Specific Integrated Circuit PLD: Programmable Logic Device LAN: Local Area Network GSM: Global System for Mobile Communications LTE: Long Term Evolution CANBus: Controller Area Network Bus USB: Universal Serial Bus PCI: Peripheral Component Interconnect FPGA: Field Programmable Gate Area SSD: Solid State Drive IC: Integrated Circuit HDR: High Dynamic Range SDR: Standard Dynamic Range JVET: Joint Video Exploration Team MPM: Most Probable Mode WAIP: Wide-Angle Intra Prediction CU: Coding Unit PU: Prediction Unit TU: Conversion unit CTU: Coding Tree Unit PDPC: Position Dependent Prediction Combination ISP: Intra Sub-Partition SPS: Sequence Parameter Setting PPS: Picture Parameter Set APS: Calibration parameter set VPS: Video Parameter Set DPS: Decoding Parameter Set ALF: Adaptive Loop Filter SAO: Sample Adaptive Offset CC-ALF: Cross-Component Adaptive Loop Filter CDEF: Constrained Directional Enhancement Filter CCSO: Cross-Component Sample Offset LSO: Local Sample Offset LR: Loop Restoration Filter AV1:AOMedia Video 1 AV2:AOMedia Video 2 RPS: Reference Picture Set DPB: Decoded Picture Buffer MMVD: Merge mode with motion vector difference IntraBC or IBC: Intrablock copy BV: Block Vector BVD: Block Vector Difference RSM: Reference Sample Memory BM: bilateral matching AMVD: Adaptive Motion Vector Difference AMVR: Adaptive Motion Vector Resolution AMVDR: Adaptive Motion Vector Difference Resolution [Explanation of symbols]

[0284] 101 points, sample 102 Arrow 103 Arrow 104 Square Blocks 180 Schematic 201 Current Block 300 Communication Systems 350 Network 310 Terminal Devices 320 terminal devices 330 Terminal Devices 340 Terminal Devices 400 Communication Systems 401 Video Source 402 Video picture or image stream, video picture stream 403 Video Encoder 404 Encoded Video Data, Encoded Video Bitstream 405 Streaming Server 406 Client Subsystem 407 Copy of encoded video data, encoded video data 408 Client Subsystem 409 Copy of encoded video data, encoded video data 410 Video Decoder 411 Video Picture Stream 412 Display 413 Video Capture Subsystem 420 Electronic Devices 430 Electronic Devices 501 Channel 510 Video Decoder 512 displays, rendering devices 515 buffer memory 520 Entropy Decoder / Parser, Parser 521 Symbol 530 Electronic Devices 531 Receiver 551 Scaler / Inverse Transformation Unit, Scaler / Inverse Transformation, Unit 552 intra picture prediction unit, intra prediction, intra prediction unit 553 Motion Compensated Prediction Unit, Motion Compensated Prediction 555 Aggregator 556 Loop filter unit, loop filter 557 Reference Picture Memory 558 Current Picture Buffer 601 Video Sources 603 Video Encoder 620 Electronic Devices 630 Source Coder 632 encoding engine 633 decoder, local video decoder, (local) decoder, decoding unit 634 Reference Picture Memory, Reference Picture Cache 635 Predictor 640 Transmitter 643 coded video sequence 645 Entropy Coder 650 Controller 660 Communication Channels 703 Video Encoder 721 General Controller 722 Intra Encoder 723 Residual Calculator 724 Residual Encoder 725 Entropy Encoder 726 Switch 728 Residual Decoder 730 Interencoder 810 Video Decoder 871 Entropy Decoder 872 Intra Decoder 873 Residual Decoder 874 Reconstruction Module, Reconstruction 880 Interdecoder 1002 "T-type" section, T-type pattern 1004 "T-type" section, T-type pattern 1006 "T-type" section, T-type pattern 1008 "T-type" section, T-type pattern 1010 All square plots 1102 Vertical binary fission, SPLIT_BT_VER 1104 Horizontal binary split, SPLIT_BT_HOR 1106 Vertical split into three, SPLIT_TT_VER 1108 Horizontal split into three, SPLIT_TT_HOR 1202 Square Plot, Plot 1204 Square Plot, Plot 1206 Square Plot, Plot 1208 Square Plot, Plot 1602 Inter-coded blocks 1802 Current Frame 1804 Reference Frame 0 1806 Reference Frame 1 1810 Block 1901 Current Frame (F i ) 1902 Current Frame (Fi ), time interpolation frame, TIP frame 1904 Reference Frame 0 (F i-1 ) 1906 Reference Frame 1 (F i+1 ) 1910 coding block, block 2000 Next block B0 2010 Next block B1 2020 Next block B2 2080 current block 2110 Current Block 2120 Next block 2300 Computer Systems 2301 Keyboard 2302 Mouse 2303 Trackpad 2305 Joystick 2306 Microphone 2307 Scanner 2308 Camera 2309 Speaker 2310 Touchscreen, Screen 2320 CD / DVD ROM / RW 2321 CD / DVD and other media 2322 thumb drive 2323 Removable Hard Drive or Solid State Drive 2340 cores 2341 Central Processing Unit (CPU) 2342 Graphics Processing Unit (GPU) 2343 FPGA, Specialized Programmable Processing Unit 2344 ACCL, Hardware Accelerator, Accelerator 2345 Read-Only Memory (ROM) 2346 Random Access Memory, RAM 2347 Internal Mass Storage, Mass Storage, Core Internal Mass Storage 2349 Peripheral bus 2350 graphics adapter 2354 Interface, Network Interface 2355 Communication Network

Claims

1. 1. A method for decoding a current block of a current frame in a coded video bitstream, comprising: receiving an encoded video bitstream by a device comprising a memory storing instructions and a processor in communication with said memory; determining, by the device based on the encoded video bitstream, the current block of the current frame to be inter-predicted with at least one reference block of at least one reference frame; determining, by the device based on the encoded video bitstream, motion vectors (MVs) of a current block corresponding to each of the at least one reference block, wherein the MVs of the current block are predicted by reference MVs; identifying, by the device, sub-block MVs of a MV reference block as reference MVs for predicting the MV of the current block, the sub-block MVs of the MV reference block being stored for use as motion vector predictors (MVPs); A method comprising:

2. before the step of identifying the sub-block MV of the MV reference block as the reference MV for predicting the MV of the current block, predicting the MV reference block by sub-block motion compensation to calculate the sub-block MV, wherein the sub-block motion compensation includes at least one of local warped motion prediction, bidirectional optical flow, or smooth inter prediction mode; The method of claim 1 further comprising:

3. before the step of identifying the sub-block MV of the MV reference block as the reference MV for predicting the MV of the current block, predicting the MV reference block by warped motion prediction to calculate the sub-block MV, wherein the warped motion prediction includes at least one of local warped motion prediction, global warped motion prediction, warped extension mode, or warped delta mode; The method of claim 1 further comprising:

4. before the step of identifying the sub-block MV of the MV reference block as the reference MV for predicting the MV of the current block, predicting the MV reference block by warped motion prediction to calculate the sub-block MV; downshifting the precision of the sub-block MVs to full precision for the current block MVs; The method of claim 1 further comprising:

5. before the step of identifying the sub-block MV of the MV reference block as the reference MV for predicting the MV of the current block, predicting the MV reference block by warped motion prediction to calculate the sub-block MV; downshifting the precision of the sub-block MV to full precision for a motion vector predictor (MVP); The method of claim 1 further comprising:

6. before the step of identifying the sub-block MV of the MV reference block as the reference MV for predicting the MV of the current block, predicting the MV reference block by a bidirectional optical flow mode to calculate the sub-block MV; The method of claim 1 further comprising:

7. before the step of identifying the sub-block MV of the MV reference block as the reference MV for predicting the MV of the current block, predicting the MV reference block by a bidirectional optical flow mode to calculate the sub-block MV; downshifting the precision of said sub-block MV to full precision for motion compensation; or downshifting the precision of the sub-block MV to full precision for a motion vector predictor (MVP); The method of claim 1 further comprising:

8. Prior to the step of identifying the sub-block MV of the MV reference block as the reference MV for predicting the MV of the current block, the method further comprises: predicting the MV reference block according to a sub-block motion compensation mode to calculate the sub-block MV and block MV; further comprising the step of identifying the sub-block MV of the MV reference block as the reference MV for predicting the MV of the current block, selecting the sub-block MV from the sub-block MV and the block MV as the reference MV for predicting the MV of the current block based on at least one of code information, mode information, or type of MVP; The method of claim 1 , comprising:

9. before the step of identifying the sub-block MV of the MV reference block as the reference MV for predicting the MV of the current block, predicting the MV reference block according to at least one sub-block motion compensation mode to calculate a plurality of sub-block MVs; obtaining a weighted average of the plurality of sub-block MVs as a sub-block MV of the MV reference block; The method of claim 1 further comprising:

10. Prior to the step of identifying the sub-block MV of the MV reference block as the reference MV for predicting the MV of the current block, the method further comprises: predicting the MV reference block according to a warped motion prediction mode to calculate the sub-block MV; further comprising The motion vector of the current block is predicted by a spatial motion vector prediction (SMVP) mode based on the sub-block motion vectors; The sub-block MV corresponds to an access position or a center position of the MV reference block. The method of claim 1.

11. Prior to the step of identifying the sub-block MV of the MV reference block as the reference MV for predicting the MV of the current block, the method further comprises: predicting the MV reference block according to a warped motion prediction mode to calculate the sub-block MV; further comprising The motion vector of the current block is predicted by a temporal-based vector prediction mode or a warped motion prediction mode based on the sub-block motion vectors. The method of claim 1.

12. Prior to the step of identifying the sub-block MV of the MV reference block as the reference MV for predicting the MV of the current block, the method further comprises: predicting the MV reference block by a bidirectional optical flow mode to calculate the sub-block MV; further comprising the MV of the current block is predicted by a temporal-based motion vector prediction mode based on the sub-block MV; The method of claim 1.

13. before the step of identifying the sub-block MV of the MV reference block as the reference MV for predicting the MV of the current block, extracting from the coded video bitstream a high-level syntax indicating whether the sub-block MV is used to predict the MV of the current block at one of the following levels: sequence level, frame level, picture level, or slice level. The method of claim 1 further comprising:

14. In response to the current block being at the top of a coding tree unit (CTU) and the sub-block MV corresponding to an upper neighboring sub-block of the current block, the MV of the current block is predicted based on the sub-block MV by a motion vector prediction mode other than spatial motion vector prediction (SMVP). The method of claim 1.

15. 1. An apparatus for processing a current block of a current frame in a coded video bitstream, comprising: a memory storing instructions; a processor in communication with said memory, said processor being configured, when said processor executes said instructions, to cause said apparatus to perform a method according to any one of claims 1 to 14; An apparatus comprising:

16. 15. A non-transitory computer-readable storage medium having stored thereon instructions, the instructions being configured, when executed by a processor, to cause the processor to perform a method according to any one of claims 1 to 14.

Images