Interpolation filters for adaptive motion vector differential resolution.

Abstract

The present disclosure relates generally to video coding, and more particularly to a method and system for selecting an interpolation filter used to generate an inter-predicted block based on a motion vector magnitude and / or a motion vector pixel resolution. For example, a method for processing a video block of a video stream is disclosed. The method may include receiving a flag from a video stream, determining based on the received flag that the video block is to be inter-predicted by a reference frame according to a motion vector (MV), deriving an MVD (MV difference) from an MVD of the video block and a reference MV, obtaining a signal indicative of an interpolation filter, the signal being based on at least one of a resolution and a magnitude of the MVD, selecting an interpolation filter in response to the signal, and performing inter-prediction of the video block from the reference frame according to the MV and the selected interpolation filter.

Description

[Technical field]

[0001] Incorporation by Reference This application is based on and claims the benefit of priority to U.S. Non-Provisional Patent Application No. 17 / 956,559, filed September 29, 2022, which is based on and claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 286,327, filed December 6, 2021, both of which are entitled "Interpolation Filters for Adaptive Motion Vector Difference Resolution." The entirety of this prior patent application is incorporated herein by reference.

[0002] FIELD OF THE DISCLOSURE This disclosure relates generally to video coding, and more particularly to methods and systems for selecting interpolation filters used to generate inter-predicted blocks based on motion vector magnitude and / or motion vector pixel resolution. [Background technology]

[0003] This background discussion provided herein is intended to provide a general overview of the background of the present disclosure, and to the extent that it is described in this background section, the work of the currently named inventors and other aspects of the description that may not have attained the status of prior art at the time of filing of this application are not admitted, expressly or impliedly, as prior art to the present disclosure.

[0004] Video encoding and decoding can be performed using inter-picture prediction with motion compensation. An uncompressed digital video may contain a sequence of pictures, each of which has a certain spatial dimension, for example 1920x1080 luma samples, and associated full or subsampled chroma samples. The sequence of pictures may have a picture rate (also called frame rate) that may be constant or variable, for example 60 pictures per second or frames per second. Uncompressed video has specific bitrate requirements for streaming and data processing. For example, a video with a pixel resolution of 1920x1080, a frame rate of 60 per second, and chroma subsampling of 4:2:0 with 8 bits per pixel per color channel requires a bandwidth of about 1.5 Gbit / s. One hour of such video requires more than 600 GByte of storage space.

[0005] One of the goals of video encoding and decoding is to reduce redundancy in the compressed input video signal by compression. Compression can be useful in reducing the bandwidth and / or storage requirements mentioned above, in some cases by more than one order of magnitude. Both lossless and lossy compression, as well as combinations of these, can be used. Lossless compression refers to techniques where an exact replica of the original signal can be reconstructed from the compressed original signal through a decoding process. Lossy compression refers to an encoding / decoding process where the original video information is not fully preserved during encoding and cannot be fully restored during decoding. With lossy compression, the reconstructed signal may not be identical to the original signal, but the distortion between the original and reconstructed signal is small enough that the reconstructed signal can be used for the intended application, although some information is lost. In the case of video, lossy compression is widely used for many applications. The amount of acceptable distortion depends on the application. For example, users of certain consumer streaming applications may tolerate more distortion than users of cinema or television broadcast applications. The compression ratio achievable by a particular encoding algorithm can be selected or adjusted to reflect different distortion tolerances: higher tolerable distortion typically allows for encoding algorithms that result in larger losses and higher compression ratios.

[0006] Techniques from several broad categories and steps may be used in video encoders and decoders, including, for example, motion compensation, Fourier transform, quantization, and entropy coding.

[0007] Video codec techniques can include a technique known as intra-coding, which represents sample values ​​without reference to samples or other data from a previously reconstructed reference picture. Some video codecs spatially subdivide a picture into blocks of samples. If all blocks of samples are coded in intra mode, the picture can be called an intra-picture. Intra-pictures and their derivatives, such as independent decoder refresh pictures, can be used to reset the decoder state, and can be used as the first image of a coded video bitstream or video session, or as a still image. The samples of the block after intra prediction can then undergo a transform to the frequency domain, and the transform coefficients so generated can be quantized before entropy coding. Intra-prediction represents a technique that minimizes the sample values ​​in the pre-transform domain. In some cases, the smaller the DC value and the smaller the AC coefficients after the transform, the smaller the size of the bits required for a given quantization step to represent the block after entropy coding.

[0008] Traditional intra-coding, such as coding known from the MPEG-2 generation of coding techniques, does not use intra-prediction. However, some newer video compression techniques include techniques that attempt to code / decode blocks based on surrounding sample data and / or metadata obtained during the coding and / or decoding of, for example, spatial neighbors or that precede the block of intra-coded or decoded data in decoding order. Such techniques are hereinafter referred to as "intra-prediction" techniques. It is noted that, at least in some examples, intra-prediction only uses reference data of the current picture being reconstructed and not reference data of other reference pictures.

[0009] There may be many different forms of intra prediction. When more than one such technique is available for a given video coding technique, the technique used may be referred to as an intra prediction mode. One or more intra prediction modes may be implemented in a particular codec. In some examples, a mode may have sub-modes and / or may be associated with various parameters, and the mode / sub-mode information and intra coding parameters for a block of video may be coded separately or may be included together in a mode codeword. Which codeword is used for a given mode, sub-mode and / or parameter combination may affect the coding efficiency improvement provided by intra prediction, and therefore may affect the entropy coding technique used to convert the codeword into a bitstream.

[0010] Certain modes of intra prediction were introduced in H.264, improved in H.265, and further improved by new coding techniques such as the joint exploration model (JEM), versatile video coding (VVC), and benchmark set (BMS). Typically, intra prediction can use the available values ​​of neighboring samples to form a prediction block. For example, the available values ​​of a certain set of neighboring samples along a given direction and / or line can be replicated in the prediction block. Reference values ​​in the direction used can be coded in the bitstream, or the reference values ​​themselves can be predicted.

[0011] Referring to FIG. 1A, a subset of nine prediction directions specified by the 33 possible intra prediction directions of H.265 (corresponding to the 33 angular modes of the 35 intra modes described in H.265) is shown at the bottom right. The point where the arrows meet (101) represents the sample being predicted. The arrows represent the direction of predicting the sample 101 using neighboring samples. For example, the arrow (102) indicates predicting the sample (101) from one or more neighboring samples toward the upper right at an angle of 45 degrees from the horizontal. Similarly, the arrow (103) indicates predicting the sample (101) from one or more neighboring samples toward the lower left of the sample (101) at an angle of 22.5 degrees from the horizontal.

[0012] Continuing with FIG. 1A, a square block (104) of 4×4 samples is shown at the top left (indicated by a thick dashed line). The square block (104) contains 16 samples, each with an index including "S" and the sample's position in the Y dimension (e.g., row index) and the sample's position in the X dimension (e.g., column index). For example, sample S21 is the second sample in the Y dimension (from the top) and the first sample in the X dimension (from the left). Similarly, sample S44 is the fourth sample in the block (104) in both the Y and X dimensions. If the size of the block is 4×4 samples, S44 is at the bottom right. Further examples of reference samples are shown following a similar numbering scheme. The reference samples are indexed with R and the Y position (e.g., row index) and X position (column index) of the reference sample relative to the block (104). In both H.264 and H.265, prediction samples that are adjacent and near the block are used during reconstruction.

[0013] Intra prediction of block 104 may start by duplicating reference sample values ​​from neighboring samples according to a signaled prediction direction. For example, assume that the coded video bitstream includes signaling for block 104 indicating a prediction direction of arrow (102), i.e., predicting samples from one or more predicted samples toward the top right at an angle of 45 degrees from the horizontal. In such a case, predict samples S41, S32, S23, and S14 from the same reference sample R05. Then predict sample S44 from reference sample R08.

[0014] In some instances, particularly where the orientation is not evenly divisible by 45 degrees, values ​​of multiple reference samples may be combined, for example through interpolation, to calculate the reference sample.

[0015] As video coding techniques continue to develop, the number of possible directions has increased. In H.264 (2003), for example, nine different directions are available for intra prediction. This has increased to 33 in H.265 (2013), but JEM / VVC / BMS can support up to 65 directions at the time of this disclosure. Experimental studies have been performed to help identify the optimal intra prediction direction, but some techniques of entropy coding can be used to code such an optimal direction for a small number of bits, while accepting some bit penalty for the direction. Furthermore, in some cases, the direction itself can be predicted from nearby directions used in intra prediction of the neighboring blocks being decoded.

[0016] FIG. 1B shows a schematic diagram (180) illustrating the 65 intra prediction directions according to the JEM, with many prediction directions being shown for various ever-evolving coding techniques.

[0017] The manner in which bits representing intra-prediction directions are mapped to prediction directions in an encoded video bitstream may vary for different video coding techniques, and may range from simple direct mapping of prediction directions to techniques such as complex matching schemes that result in intra-prediction modes, codewords, and most likely modes. In all cases, however, there may be some directions of intra-prediction that are statistically less likely to occur in the video content than some other directions. Because the goal of video compression is to reduce redundancy, these less likely directions may be represented by more bits than more likely directions if the video coding technique is properly designed.

[0018] Inter-picture prediction (or inter-prediction) may be based on motion compensation, in which sample data from a previously reconstructed picture or part of it (reference picture) may be used to predict a newly reconstructed picture or part of a picture (e.g., a block) after being spatially shifted in a direction indicated by a motion vector (hereafter MV). In some cases, the reference picture may be the same as the picture at the time the reconstruction is being performed. The MV may have two dimensions X and Y, or three dimensions, with the third dimension being an index for the reference picture being used (similar to the temporal dimension).

[0019] In some video compression techniques, a current MV applicable to a region of sample data can be predicted from other MVs, e.g., from other MVs related to other regions of sample data that are spatially adjacent to the region being reconstructed and that precede the current MV in decoding order. In this way, the total amount of data required to encode the MV can be substantially reduced as the redundancy of correlated MVs is removed, thereby increasing compression efficiency. For example, when encoding an input video signal obtained from a camera (called raw video), MV prediction can work effectively because there is a statistical probability that a region larger than the region to which one MV is applicable moves in a similar direction in the video sequence, and therefore, in some cases, the region can be predicted using similar motion vectors derived from MVs of nearby regions. As a result, the actual MV of a given region is similar or identical to the MV predicted from the surrounding MVs. After entropy encoding, the MV can be represented with fewer bits than would be used if the MV were directly encoded without prediction from one or more nearby MVs. In some examples, MV prediction can be an example of a lossless compression of a signal (i.e., MV) derived from the original signal (i.e., sample stream). In other cases, the MV prediction itself may be lossy, for example due to rounding errors when computing the prediction from several surrounding MVs.

[0020] Various MV prediction mechanisms are described in H.265 / HEVC (ITU-T Rec. H.265, “High Efficiency Video Coding”, December 2016). Among the many MV prediction mechanisms described in H.265, a technique hereinafter referred to as “spatial merging” is described below.

[0021] Specifically, referring to Fig. 2, a current block (201) comprises samples that are detected by the encoder during the motion search process and are predictable from a spatially shifted previous block of the same size. Instead of directly encoding the MV, the MV can be derived from metadata associated with one or more reference pictures, e.g., the MV can be derived from the most recent (decoding order) reference picture using the MV associated with any one of the surrounding five samples, denoted A0, A1 and B0, B1, B2 (202-206, respectively). In H.265, MV prediction can use predicted values ​​obtained from the same reference picture used by neighboring blocks. Summary of the Invention [Means for solving the problem]

[0022] FIELD OF THE DISCLOSURE This disclosure relates generally to video coding, and more particularly to methods and systems for selecting interpolation filters used to generate inter-predicted blocks based on motion vector magnitude and / or motion vector pixel resolution.

[0023] In one implementation, a method for processing a video block of a video stream is disclosed. The method may include receiving a flag from the video stream, determining based on the received flag that the video block is to be inter-predicted by a reference frame according to a motion vector (MV), deriving an MV from an MVD (MV difference) of the video block and a reference MV, obtaining a signal indicative of an interpolation filter, the signal being based on at least one of a resolution and a size of the MVD, selecting an interpolation filter according to the signal, and performing inter-prediction of the video block from the reference frame according to the MV and the selected interpolation filter.

[0024] In the above implementation, obtaining a signal may include determining a magnitude of the MVD based on an MV class index associated with the video block, the MV class index being extracted from the video stream, and obtaining a signal to set to a default interpolation filter if the MV class index is not less than a threshold MV class index.

[0025] In any one of the above implementations, the default interpolation filter is an interpolation filter from a plurality of interpolation filters that form a pool of candidate interpolation filters to be used for video blocks having an MV class index less than a threshold MV class index.

[0026] In any one of the above implementations, the default interpolation filter includes a REGULAR, SMOOTH, SHARP, or BILINEAR interpolation filter.

[0027] In any one of the above implementations, obtaining a signal may include determining that an adaptive pixel resolution is to be applied to the MVD, determining a pixel resolution of the MVD from a plurality of adaptive MVD pixel resolutions, and obtaining a signal for setting a default interpolation filter in response to the pixel resolution of the MVD being at one or more predefined levels.

[0028] In any one of the above implementations, the one or more predefined levels include an MVD resolution level that does not exceed a predefined MVD pixel resolution threshold.

[0029] In any one of the above implementations, the pixel resolution of the MVD is determined based on the MV class index of the video block extracted from the video stream.

[0030] In any one of the above implementations, the predefined MVD pixel resolution threshold corresponds to the MV class index threshold.

[0031] In any one of the above implementations, the default interpolation filter is an interpolation filter from a plurality of interpolation filters that form a pool of candidate interpolation filters to be used for image blocks having one or more adaptive MVD pixel resolutions that are not the default level.

[0032] In any one of the above implementations, the default interpolation filter includes one of a REGULAR, SMOOTH, SHARP, or BILINEAR interpolation filter.

[0033] In any one of the above implementations, the step of obtaining a signal may include a step of determining an MV class index corresponding to a magnitude of the MVD from the video stream, a step of setting a default subset of predefined interpolation filters from the set of interpolation filters if the MV class index of the MV is not less than a predefined MV class index threshold, and a step of extracting a selection indicator from the video stream as a signal for selecting an interpolation filter from the subset of predefined interpolation filters.

[0034] In any one of the above implementations, the step of obtaining a signal may include a step of determining that an adaptive pixel resolution is to be applied to the MVD; a step of determining a pixel resolution of the MVD from a plurality of adaptive MVD pixel resolutions; a step of setting a default subset of predefined interpolation filters from a set of interpolation filters in response to the pixel resolution of the MVD not exceeding a predefined MVD pixel resolution threshold; and a step of extracting a selection indicator from the video stream as a signal for selecting an interpolation filter from the subset of predefined interpolation filters.

[0035] In any one of the above implementations, the step of obtaining a signal may include a step of determining that an adaptive pixel resolution is applied to the MVD of the video block; a step of determining a pixel resolution of the MVD from a plurality of adaptive MVD pixel resolutions or a step of determining an MV class index of the MV from a plurality of MV class indexes; a step of identifying a set of contexts for encoding the interpolation filter selection indicator based on the pixel resolution of the MVD or the MV class index; and a step of decoding the video stream according to the set of contexts to obtain the interpolation filter selection indicator as a signal.

[0036] In any one of the above implementations, the step of obtaining the signal may include determining that an adaptive pixel resolution is applied to the MVD; determining an MV class index of the video block from a plurality of MV class indexes; decoding the video stream using a first set of contexts to obtain an interpolation filter selection indicator as a signal if the MV class index is not less than a threshold MV class index; and decoding the video stream using a second set of contexts different from the first set of contexts to obtain the interpolation filter selection indicator as a signal if the MV class index is less than the threshold MV class index.

[0037] In any one of the above implementations, obtaining the signal may include setting a default interpolation filter in response to determining that the MVD pixel resolution of the MVD is a non-fractional pixel resolution.

[0038] In any one of the above implementations, the step of obtaining a signal may include a step of setting a default subset of predefined interpolation filters from a set of interpolation filters in response to determining that the MVD pixel resolution of the MVD is a non-fractional pixel resolution, and a step of extracting a selection indicator from the video stream as a signal for selecting an interpolation filter from the subset of predefined interpolation filters.

[0039] Aspects of the present disclosure also provide a video encoding or decoding device or apparatus that includes circuitry configured to perform any of the implementations of the above methods.

[0040] Aspects of the present disclosure also provide a non-transitory computer-readable medium storing instructions that, when executed by a computer, cause the computer to perform a method for video decoding and / or encoding.

[0041] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings. [Brief description of the drawings]

[0042] [Figure 1A] 1 shows a schematic diagram of an exemplary subset of intra-prediction directional modes. [Figure 1B] 1 shows a diagram of a typical intra prediction direction. [Diagram 2] 1 illustrates a schematic diagram of a current block and its surrounding spatial merging candidates used for motion vector prediction in one example. [Diagram 3] 1 shows a simplified block diagram schematic of a communication system (300) according to one embodiment. [Figure 4] 1 shows a simplified block diagram of a communication system (400) according to one embodiment. [Diagram 5] FIG. 2 shows a schematic diagram of a simplified block diagram of a video decoder according to one embodiment. [Figure 6] 1 shows a schematic diagram of a simplified block diagram of a video encoder according to an embodiment; [Figure 7] 4 shows a block diagram of a video encoder according to another embodiment; [Figure 8] 4 shows a block diagram of a video decoder according to another embodiment; [Figure 9] 1 illustrates a coding block division method according to an embodiment of the present disclosure. [Figure 10] 1 illustrates another coding block partitioning scheme according to an embodiment of the present disclosure. [Figure 11] 1 illustrates another coding block partitioning scheme according to an embodiment of the present disclosure. [Figure 12] 1 shows an example of dividing a base block into coding blocks according to a typical division method. [Figure 13] An example of a three-way split method is shown below. [Figure 14] An example of a quadtree / binary tree coding block division method is shown below. [Figure 15] 4 illustrates a scheme for dividing a coding block into multiple transform blocks and a coding order for the transform blocks according to an embodiment of the present disclosure. [Figure 16] 4 illustrates another scheme for splitting a coding block into multiple transform blocks and the coding order of the transform blocks according to an embodiment of the present disclosure. [Figure 17] 4 illustrates another scheme for splitting a coding block into multiple transform blocks according to an embodiment of the present disclosure. [Figure 18] 1 shows a flowchart of a method according to an embodiment of the present disclosure. [Figure 19] FIG. 1 shows a schematic diagram of a computer system according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0043] Throughout the specification and claims, terms may have special meanings suggested or implied by the context beyond the scope of their stated meaning. The phrases "in one embodiment" or "in some embodiments" used in this disclosure do not necessarily refer to the same embodiment, and the phrases "in another embodiment" or "in other embodiments" used in this disclosure do not necessarily refer to different embodiments. Similarly, the phrases "in one implementation" or "in some implementations" used in this disclosure do not necessarily refer to the same implementation, and the phrases "in another implementation" or "in other implementations" used in this disclosure do not necessarily refer to different implementations. The intention, for example, is that the claimed subject matter includes a combination of exemplary embodiments / implementations in whole or in part.

[0044] Overall, the usage of a term may be at least partially determined by how it is used in context. For example, terms such as "and," "or," or "and / or" as used in this disclosure may include various meanings that may depend, at least in part, on the context in which the term is used. Typically, when "or" is used to link an enumeration such as A, B, or C, "or" is intended to mean A, B, or C, in this case used in an inclusive sense, and "or" is intended to mean A, B, or C, in this case used in an exclusive sense. In addition, the terms "one or more" or "at least one" as used in this disclosure may be used to describe any feature, structure, or characteristic in the singular sense or may be used to describe a combination of features, structures, or characteristics in the plural sense, at least in part depending on the context. Similarly, terms such as "a," "an," or "the" may be used to recognize the use in the singular or to recognize the use in the plural, at least in part depending on the context. In addition to the above, the terms "based on" or "determined" may be understood, again at least in part depending on the context, as not necessarily intended to imply a closed enumerated set of factors, and even to allow for the existence of other factors not necessarily expressly described. FIG. 3 illustrates a simplified block diagram of a communication system (300) according to an embodiment of the present disclosure. The communication system (300) includes a plurality of terminal devices that can communicate with each other, e.g., via a network (350). For example, the communication system (300) includes a first pair of terminal devices (310) and (320) that are interconnected via the network (350). In the example of FIG. 3, the first pair of terminal devices (310) and (320) may perform unidirectional transmission of data. For example, the terminal device (310) may encode video data (e.g., video data of a stream of video pictures captured by the terminal device (310)) for transmission to the other terminal device (320) via the network (350).The encoded video data may be transmitted as one or more encoded video bitstreams. A terminal device (320) may receive the encoded video data from a network (350), decode the encoded video data to recover video pictures, and display video pictures according to the recovered video data. Unidirectional data transmission may also be implemented in media provisioning applications, etc.

[0045] In another example, the communication system (300) includes a second pair of terminal devices (330) and (340) for bidirectional transmission of encoded video data, such as may be implemented in videoconferencing applications. For bidirectional transmission of data, by way of example, each of the terminal devices (330) and (340) may encode video data (e.g., video data of a stream of video pictures captured by the terminal device) for transmission over the network (350) to the other of the terminal devices (330) and (340). Each of the terminal devices (330) and (340) may also receive the encoded video data transmitted by the other of the terminal devices (330) and (340), decode the encoded video data to recover the video pictures, and display the video pictures corresponding to the recovered video data on an accessible display device.

[0046] In the example of FIG. 3, the terminal devices (310), (320), (330), and (340) may be implemented as a server, a personal computer, and a smartphone, although the scope of the underlying principles of the present disclosure is not limited in this manner. The embodiments of the present disclosure may be implemented in desktop computers, laptop computers, tablet computers, media players, wearable computers, dedicated videoconferencing equipment, and / or the like. The network (350) 9 represents any number and type of network that conveys encoded video data between the terminal devices (310), (320), (330), and (340), including, for example, wired and / or wireless communication networks. The communication network (350) 9 may exchange data over circuit-switched channels, packet-switched channels, and / or other types of channels. Representative networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For purposes of illustration of the present example, unless expressly described in the present application, the architecture and topology of the network (350) may not be important to the operation of the present disclosure.

[0047] Figure 4 shows an arrangement of a video encoder and a video decoder in a video streaming environment as an example application of the disclosed subject matter. The disclosed subject matter may be similarly applicable to other applications of video including, for example, video conferencing, digital television broadcasting, gaming, virtual reality, storage of compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0048] A video streaming system may include a video source (401), e.g., a video capture subsystem (413) that may include a digital camera, that generates a stream of uncompressed video pictures or images (402). In one example, the stream of video pictures (402) includes samples recorded by the digital camera of the video source 401. The stream of video pictures (402), shown in bold to emphasize its large amount of data when compared to the encoded video data (404) (i.e., the encoded video bitstream), may be processed by an electronic device (420) that includes a video encoder (403) that is connected to the video source (401). The video encoder (403) may include hardware, software, or a combination thereof that enables or implements aspects of the disclosed subject matter, as described in more detail below. The encoded video data (404) (i.e., encoded video bitstream (404)) is shown with thin lines to emphasize its small amount of data when compared to the uncompressed video picture stream (402), which may be stored on the streaming server (405) for later use or directly on a downstream video device (not shown). One or more streaming client subsystems, such as the client subsystems (406) and (408) of FIG. 4, may access the streaming server (405) to obtain copies (407) and (409) of the encoded video data (404). The client subsystem (406) may include a video decoder (410), for example, in an electronic device (430). The video decoder (410) receives and decodes the copy of the encoded video data (407) and generates an output stream of video pictures (411) that is uncompressed and that can be rendered on a display (412) (e.g., a display screen) or other rendering device (not shown). Video decoder 410 may be configured to perform some or all of the various functions described in this disclosure.In some streaming systems, the encoded video data 404, 407, and 409 (e.g., a video bitstream) may be encoded according to a particular video encoding / compression standard. Examples of such standards include ITU-T Recommendation H.265. In one example, a video encoding standard under development is informally known as Versatile Video Coding (VVC). The disclosed subject matter may be used with respect to VVC and other video encoding standards.

[0049] It is noted that electronic devices 420 and 430 can include other components (not shown). For example, electronic device 420 can include a video decoder (not shown), and similarly, electronic device 430 can include a video encoder (not shown).

[0050] 5 shows a block diagram of a video decoder (510) according to any of the embodiments of the present disclosure described below. The video decoder (510) can be included in an electronic device (530). The electronic device (530) can include a receiver (531) (e.g., a receiving circuit). The video decoder (510) can be used in place of the video decoder (410) in the example of FIG. 4.

[0051] The receiver (531) may receive one or more encoded video sequences to be decoded by the video decoder (510). One encoded video sequence at a time may be decoded in the same embodiment or in another embodiment, and the decoding of each encoded video sequence is independent of the other encoded video sequences. Each video sequence may relate to multiple video frames or images. The encoded video sequences may be received from a channel (501), which may be a hardware / software coupled to a storage device that stores the encoded video data or a streaming source that transmits the encoded video data. The receiver (531) may receive the encoded video data along with other data, such as encoded audio data and / or ancillary data streams, and may forward the other data to corresponding processing circuitry (not shown). The receiver (531) may separate the encoded video sequences from the other data. To address network jitter, a buffer memory (515) may be disposed between the receiver (531) and the entropy decoder / parser (520) (hereinafter "parser (520)"). In some applications, the buffer memory (515) may be implemented as part of the video decoder (510). In other applications, the buffer memory (515) may be external to the video decoder (510) and may be separate from the video decoder (510) (not shown). In still other applications, a buffer memory (not shown) may be external to the video decoder (510), for example to handle network jitter, and another additional buffer memory (515) may be present within the video decoder (510), for example to handle playback timing. If the receiver (531) receives data from a store and forward device or an isochronous network with sufficient bandwidth and controllability, the buffer memory (515) may not be required, or may be small. For use with best effort packet networks such as the Internet, a sufficiently sized buffer memory (515) may be required, and the size of the buffer memory (515) may be relatively large.Such a buffer memory may be implemented with any suitable size and may be implemented, at least in part, in an operating system or similar element (not shown) external to the video decoder (510).

[0052] The video decoder (510) may include a parser (520) that reconstructs symbols (521) from the encoded video sequence. Such categories of symbols include information used to manage the operation of the video decoder (510), and may include information to control a rendering device, such as a display (512) (e.g., a display screen) that may or may not be an integral part of the electronic device (530) and may be connected to the electronic device (530), as shown in FIG. 5. The control information for one or more rendering devices may take the form of a Supplemental Enhancement Information (SEI) message or a Video Usability Information (VUI) parameter set fragment (not shown). The parser (520) may parse / entropy decode the encoded video sequence received by the parser (520). The entropy coding of the encoded video sequence may conform to video coding techniques and standards and may follow a variety of principles, including variable length coding, Huffman coding, arithmetic coding with or without context-sensitive coding, etc. The parser (520) may extract from the encoded video sequence a set of subgroup parameters for at least one subgroup of pixels in the video decoder based on at least one parameter corresponding to the subgroup. A subgroup may include a Group of Pictures (GOP), a picture, a tile, a slice, a macroblock, a Coding Unit (CU), a block, a Transform Unit (TU), a Prediction Unit (PU), etc. The parser (520) may also extract from encoded video sequence information such as transform coefficients (e.g., Fourier transform coefficients), quantizer parameter values, motion vectors, etc.

[0053] The parser (520) may generate symbols (521) by performing an entropy decoding / parser operation on the video sequence received from the buffer memory (515).

[0054] The reconstruction of the symbols (521) may involve a number of different processing or functional entities depending on the type of encoded video picture or portion thereof (e.g., inter-picture and intra-picture, inter-block and intra-block) and other factors. Which entities are involved and how they are involved may be controlled by subgroup control information parsed by the parser (520) from the encoded video sequence. For simplicity, the flow of such subgroup control information between the parser (520) and the multiple processing or functional entities described below is not shown.

[0055] Further to the functional blocks already described, the video decoder (510) may be conceptually subdivided into a number of functional units, as described below. In an actual implementation operating within commercial constraints, many of these functional units may interact closely with each other and may be at least partially integrated with each other. However, the following disclosure employs a conceptual subdivision into functional units in order to clearly describe and explain the various features of the disclosed subject matter.

[0056] The first section may include a scalar / inverse transform unit (551), which may receive quantized transform coefficients as symbols (521) from the parser (520) and control information including information indicating which type of inverse transform to use, block size, quantization factors / quantization parameters, quantization scaling matrices, and states. The scalar / inverse transform unit (551) may output blocks comprising sample values ​​that may be input to an aggregater (555).

[0057] In some examples, the output samples of the scaler / inverse transform (551) may be suitable for intra-coded blocks, i.e., blocks that do not use predictive information from previously reconstructed pictures, but may use predictive information from previously reconstructed portions of the current picture. Such predictive information may be provided by an intra-predictor (552). In some examples, the intra-predictor (552) may generate blocks of the same size and shape as the block being reconstructed using information from surrounding blocks that have already been reconstructed and stored in a current picture buffer (558). The current picture buffer (558) may, for example, buffer a partially reconstructed current picture and / or a fully reconstructed current picture. In some implementations, the aggregator (555) may add, on a sample-by-sample basis, the predictive information generated by the intra-predictor (552) to the output sample information provided by the scaler / inverse transform (551).

[0058] In other examples, the output samples of the scaler / inverse transform unit (551) may be suitable for inter-coded blocks, and possibly motion-compensated blocks. In such examples, the motion compensation prediction unit (553) may access the reference picture memory (557) to obtain samples used for inter-picture prediction. The obtained samples may be motion-compensated according to the symbols (521) related to the block, and then added to the output of the scaler / inverse transform unit (551) (the output of the unit 551 may be referred to as a residual sample or a residual signal) by the aggregator (555) to generate output sample information. The addresses in the reference picture memory (557) from which the motion compensation prediction unit (553) obtains prediction samples may be controlled by a motion vector, which is available to the motion compensation prediction unit (553) in the form of a symbol (521) that may have, for example, X, Y components (shift) and a reference picture component (time). Motion compensation may also include interpolation of sample values ​​obtained from a reference picture memory (557) when sub-sample accurate motion vectors are used, or may involve motion vector prediction mechanisms, etc.

[0059] The output samples of the aggregater (555) may be subjected to various loop filtering techniques in the loop filter unit (556). While the video compression techniques may include in-loop filter techniques controlled by parameters contained in the coded video sequence (also referred to as the coded video bitstream) and made available to the loop filter unit (556) as symbols (521) from the parser (520), the video compression techniques may also be sensitive to meta-information obtained when decoding previous (decode order) portions of the coded picture or coded video sequence, and may also be sensitive to previously reconstructed and loop filtered sample values. Several types of loop filters may be included as part of the loop filter unit 556, in various orders, as will be described in more detail below.

[0060] The output of the loop filter unit (556) can be a sample stream that can be output to a rendering device (512) as well as stored in a reference picture memory (557) for later use in inter-picture prediction.

[0061] Once some coded pictures are fully reconstructed, they can be used as reference pictures for subsequent inter-picture prediction. For example, once a coded picture corresponding to a current picture is fully reconstructed and the coded picture is identified (e.g., by the parser (520)) as a reference picture, the current picture buffer (558) can become part of the reference picture memory (557), and a new current picture buffer can be reallocated before starting the reconstruction of the next coded picture.

[0062] The video decoder (510) may perform decoding operations according to a predefined video compression technique employed in a given standard (e.g., ITU-T Recommendation H.265). An encoded video sequence is said to be compliant with the syntax defined by the video compression technique or standard used in the sense that the encoded video sequence adheres to both the syntax of the video compression technique or standard and the profile described in the video compression technique or standard document. In particular, a profile may select a predefined tool from all tools available in the video compression technique or standard as the only tools that can be used according to the profile. To be compliant with a standard, the complexity of the encoded video sequence may be within limits defined by the level of the video compression technique or standard. In some examples, the level may limit the maximum picture size, the maximum frame rate, the maximum reconstruction sample rate (e.g., measured in million samples per second), the maximum reference picture size, etc. The limits imposed by the levels may in some instances be further restricted by the specifications of a Hypothetical Reference Decoder (HRD) and metadata for HRD buffer management that is signaled in the encoded video sequence.

[0063] In some embodiments, the receiver (531) may receive ancillary (redundant) data along with the encoded video. The ancillary data may be included as part of the encoded video sequence. The ancillary data may be used by the video decoder (510) to properly decode the data and / or to more accurately reconstruct the original video data. For example, the ancillary data may take the form of temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0064] 6 shows a block diagram of a video encoder (603) according to one embodiment of the present disclosure. The video encoder (603) may be included in an electronic device (620). The electronic device (620) may further include a transmitter (640) (e.g., a transmission circuit). The video encoder (603) may be used in place of the video encoder (403) in the example of FIG. 4.

[0065] The video encoder (603) may receive video samples from a video source (601) (which in the example of FIG. 6 is not part of the electronic device (620)) capable of capturing one or more video images to be encoded by the video encoder (603). In another example, the video source (601) may be implemented as part of the electronic device (620).

[0066] The video source (601) may provide a source video sequence to be encoded by the video encoder (603) as a stream of digital video samples that may be of any suitable bit depth (e.g.: 8-bit, 10-bit, 12-bit, ...), any color space (e.g. BT.601 YCrCb, RGB, XYZ...), and any suitable sampling structure (e.g. YCrCb 4:2:0, YCrCb 4:4:4). In a media presentation system, the video source (601) may be a storage device capable of storing pre-prepared video. In a video conferencing system, the video source (601) may be a camera capturing on-site image information as a video sequence. The video data may be provided as a number of individual pictures or images that achieve motion when viewed in sequence. A picture may simply be organized as a spatial array of pixels, where each pixel may comprise one or more samples depending on the sampling structure, color space, etc. being used. The relationship between pixels and samples is easily understood by those skilled in the art. The following description focuses on samples.

[0067] According to some embodiments, the video encoder (603) may encode and compress pictures of a source video sequence into an encoded video sequence (643) in real-time or subject to any other time constraints required by the application. Enforcing the maintenance of an appropriate encoding rate constitutes one of the functions of the controller (650). In some embodiments, the controller (650) may be operatively connected to other functionalities, described below, which are controlled by the controller (650). For simplicity, the connections are not shown. Parameters set by the controller (650) may include rate control related parameters (picture skip, quantizer, lambda value for rate-distortion optimization techniques, ...), picture size, group of pictures (GOP) layout, search range for longest motion vector, etc. The controller (650) may be configured to have other appropriate functions for the video encoder (603) optimized for a particular system design.

[0068] In some embodiments, the video encoder (603) may be configured to operate in an encoding loop. In a simplified explanation, in one example, the encoding loop may include a source coder (630) (e.g., responsible for generating symbols, such as a symbol stream, based on an input picture to be encoded and one or more reference pictures) and a decoder (633) built into the video encoder (603) (located here, not elsewhere). The decoder (633) reconstructs the symbols and generates sample data as if it were generated by a decoder (located elsewhere), even though the built-in decoder 633 processed the video stream encoded by the source coder 630 without entropy encoding (any compression between the symbols and the encoded video bitstream in entropy encoding may be lossless compression included in the video compression techniques contemplated in the disclosed subject matter). The reconstructed sample stream (sample data) is input to a reference picture memory (634). Since decoding the symbol stream will yield bit-accurate results independent of the decoder's location (here or elsewhere), the contents of the reference picture memory (634) will also be bit-accurate with the encoder at this location compared to the encoder at another location. In other words, the predictive part of the encoder "sees" exactly the same sample values ​​as the reference picture samples as if the decoder "sees" when using prediction during decoding. This fundamental principle of reference picture synchrony (and the resulting drift when synchrony cannot be maintained, e.g. due to channel errors) is used to improve coding quality.

[0069] The operation of the "in-place" decoder (633) may be the same as that of a decoder such as the "elsewhere" video decoder (510) already described in detail above in conjunction with Figure 5. However, and with some reference to Figure 5, because symbols are available and the encoding of symbols by the entropy coder (645) and parser (520) and decoding into an encoded video sequence may be lossless, the entropy decoding portion of the video decoder (510), including the buffer memory (515) and parser (520), may not be fully implemented in the in-place decoder (633) of the encoder.

[0070] At this point, it can be said that any decoder technology other than parsing / entropy decoding, which may only exist in the decoder, must also necessarily exist in the corresponding encoder in substantially the same functional form. For this reason, the disclosed subject matter may occasionally focus on the operation of the decoder in relation to the decoding portion of the encoder. Thus, a description of the encoder technology may be omitted, since it is the reverse of the decoder technology described in its entirety. Only certain parts or aspects of the encoder are described in more detail below.

[0071] In operation of some implementations, the source coder (630) may perform motion-compensated predictive coding, which predictively codes an input picture with reference to one or more previously coded pictures from a video sequence designated as "reference pictures." In this manner, the coding engine (632) codes color channel differences (i.e., residual values) between pixel blocks of the input picture and pixel blocks of one or more reference pictures that can be selected as one or more prediction references for the input picture. The terms "residue" and its adjective form "residual" may be used interchangeably.

[0072] The video decoder (633) at this location may decode the encoded video data of the pictures that can be designated as reference pictures based on the symbols generated by the source coder (630). It may be useful for the operation of the coding engine (632) to be a lossy process. If the encoded video data may be decoded by some video decoder (not shown in FIG. 6), the reconstructed video sequence may generally be a copy of the source video sequence with some errors. The video decoder (633) at this location may repeat the decoding process that may be performed by the video decoder on the reference pictures, such that the reconstructed reference pictures are stored in the reference picture cache (634). In this way, the video encoder (603) may store copies of reconstructed reference pictures at this location that have common content with the reconstructed reference pictures that will be obtained by the far-end (elsewhere) video decoder (without transmission errors).

[0073] The predictor (635) may perform a prediction search for the coding engine (632). That is, for a new picture to be coded, the predictor (635) may search the reference picture memory (634) to obtain sample data (sample data as candidate reference pixel blocks) and to obtain reference picture motion vectors, block shapes, and other specific metadata that can serve as suitable prediction references for the new picture. The predictor (635) may operate using one sample block per pixel block to detect suitable prediction references. In some examples, the input picture may include prediction references obtained from multiple reference pictures stored in the reference picture memory (634), as determined by the search results obtained by the predictor (635).

[0074] For example, the controller (650) may manage the encoding operations of the source coder (630), including setting the parameters and subgroup parameters used to encode the video data.

[0075] The output of all the above mentioned functionalities may be subjected to entropy coding in an entropy coder (645), which converts the symbols produced by the various functionalities into an encoded video sequence by losslessly compressing the symbols according to techniques such as Huffman coding, variable length coding, arithmetic coding, etc.

[0076] The transmitter (640) may buffer one or more encoded video sequences generated by the entropy coder (645) in preparation for transmission over a communication channel (660), and may be hardware / software coupled to a storage device that stores the encoded video data. The transmitter (640) may also integrate other data to be transmitted, such as encoded audio data and / or ancillary data streams (sources not shown), with the encoded video data obtained from the video coder (603).

[0077] A controller (650) may manage the operation of the video encoder (603). During encoding, the controller (650) may assign to each encoded picture a particular encoded picture type, which can affect the encoding technique that can be applied to the respective picture. For example, pictures can often be assigned as one of the following picture types:

[0078] An Intra Picture (I-picture) can be described as a picture that can be coded and decoded without using any other picture in the sequence as a basis for prediction. In some video codecs, various types of Intra pictures are possible, including, for example, Independent Decoder Refresh (IDR) pictures. Such variations of I-pictures and their corresponding applications and features will occur to those skilled in the art.

[0079] A predictive picture (P-picture) can be defined as a picture that can be coded and decoded using intra- or inter-prediction, using at most one motion vector and reference index to predict the sample values ​​of each block.

[0080] A bi-directionally predictive picture (B-picture) can be said to be a picture that can be coded and decoded using intra- or inter-prediction and up to two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, a multiple-predictive picture can use more than two reference pictures and associated metadata for the reconstruction of a block.

[0081] In general, a source picture may be spatially subdivided into multiple sample coding blocks (e.g., blocks of 4x4, 8x8, 4x8 or 16x16 samples, respectively) and the blocks may be coded one by one. Blocks may be coded predictively with reference to other (already coded) blocks, as determined by the coding assignment applied to the picture corresponding to each block. For example, blocks of an I picture may be coded without prediction or with prediction with reference to already coded blocks of the same picture (spatial prediction or intra prediction). Pixel blocks of a P picture may be coded predictively with spatial prediction or with temporal prediction with reference to one previously coded reference picture. Blocks of a B picture may be coded predictively with spatial prediction or with temporal prediction with reference to one or two previously coded reference pictures. Source pictures or intermediate processing pictures may be subdivided into other types of blocks for other purposes. The division of coding blocks and other types of blocks may or may not follow the same manner, which is described in more detail below.

[0082] The video encoder (603) may perform encoding operations according to a predefined video encoding technique or standard (e.g., ITU-T Recommendation H.265), in which the video encoder (603) may perform various compression operations, including predictive encoding operations that exploit temporal and spatial redundancy in the input video sequence, and thus the encoded video data may conform to a syntax defined by the video encoding technique or standard being used.

[0083] In some embodiments, the transmitter (640) may transmit ancillary data along with the encoded video. The source coder (630) may include such data as part of the encoded video sequence. The ancillary data may comprise temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures or slices, SEI messages, VUI parameter set fragments, etc.

[0084] A video may be captured in time order as multiple source pictures (video pictures). Intra-picture prediction (often abbreviated as intra-prediction) exploits spatial correlation within a given picture, while inter-picture prediction exploits correlation (temporal or other correlation) between pictures. For example, a particular picture being coded / decoded, called the current picture, may be divided into blocks. If a block in the current picture is similar to a reference block in a previously coded reference picture in the video that is still buffered, the block in the current picture may be coded by a vector, called a motion vector. The motion vector points towards the reference block in the reference picture, and may have a third dimension to identify the reference picture when multiple reference pictures are used.

[0085] In some embodiments, a bi-prediction technique can be used for inter-picture prediction. This bi-prediction technique uses two reference pictures, such as a first reference picture and a second reference picture, both of which precede the current picture in decoding order (or alternatively may be past or future in display order, respectively) in the video. A block in the current picture can be coded with a first motion vector pointing towards a first reference block in the first reference picture and a second motion vector pointing towards a second reference block in the second reference picture. A combination of the first and second reference blocks can be used together to predict a block.

[0086] Furthermore, the merge mode method can be used for inter-picture prediction to improve coding efficiency.

[0087] According to some embodiments of the present disclosure, prediction such as inter-picture prediction or intra-picture prediction is performed on a block-by-block basis. For example, a picture in a sequence of video pictures may be divided into coding tree units (CTUs) for compression, and the CTUs in a picture may have the same size (e.g., 64×64 pixels, 32×32 pixels, or 16×16 pixels). In general, a CTU may include three parallel coding tree blocks (CTBs), i.e., one luma CTB and two chroma CTBs. Each CTU may be recursively quadtree-divided into one or more coding units (CUs). For example, a 64×64 pixel CTU may be divided into one CU of 64×64 pixels or four CUs of 32×32 pixels. Each of one or more of the 32×32 blocks may be further divided into four CUs of 16×16 pixels. In some embodiments, during encoding, each CU may be analyzed to determine the prediction type of the CU from various prediction types, such as an inter prediction type or an intra prediction type. A CU may be divided into one or more Prediction Units (PUs) according to temporal and / or spatial predictability. In general, each PU includes one luma prediction block (PB) and two chroma PBs. In one embodiment, the prediction operation of coding (encoding / decoding) is performed in units of prediction blocks. The division of a CU into PUs (or PBs of different color channels) may be performed in various spatial patterns. A luma PB or a chroma PB may include a matrix of sample values ​​(e.g., luma values), such as, for example, 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 samples, etc.

[0088] 7 shows a diagram of a video encoder (703) according to another embodiment of the present disclosure. The video encoder (703) is configured to receive a processed block of sample values ​​(e.g., a predictive block) in a current video picture in a sequence of video pictures and to encode the processed block into an encoded picture that is part of an encoded video sequence. The example video encoder (703) may be used in place of the example video encoder (403) of FIG. 4.

[0089] For example, the video encoder (703) receives a matrix of sample values ​​for a processing block, such as a prediction block of 8 by 8 samples. The video encoder (703) then determines, for example using rate-distortion optimization (RDO), whether the processing block is best coded using intra mode, inter mode, or bi-prediction mode. If it is determined that the processing block is coded in intra mode, the video encoder (703) may use intra prediction methods to code the processing block into a coded picture, and if it is determined that the processing block is coded in inter mode or bi-prediction mode, the video encoder (703) may use inter prediction or bi-prediction methods, respectively, to code the processing block into a coded picture. In some embodiments, merge mode may be used as a sub-mode of inter-picture prediction that derives motion vectors from one or more motion vector predictors without the aid of coded motion vector components outside the predictor. In some other embodiments, there may be motion vector components applicable to the block of interest. Accordingly, the video encoder (703) may include components not explicitly shown in FIG. 7, such as a mode decision module that determines the prediction mode of a processing block.

[0090] In the example of FIG. 7, the video encoder (703) includes an inter-encoder (730), an intra-encoder (722), a residual value calculator (723), a switch (726), a residual value encoder (724), an overall controller (721), and an entropy encoder (725), connected together as shown in the example arrangement of FIG.

[0091] The inter encoder (730) is configured to receive a sample of a current block (e.g., a processing block), compare the block to one or more reference blocks in a reference picture (e.g., blocks in previous and subsequent pictures in display order), generate inter prediction information (e.g., a description of redundant information according to an inter coding scheme, motion vectors, merge mode information), and calculate an inter prediction result (e.g., a predicted block) based on the inter prediction information using any suitable technique. In some examples, the reference picture is a decoded reference picture that has been decoded based on the coded video information using a decoder 633 (shown as residual decoder 728 in FIG. 7, described in more detail below) incorporated in the example encoder 620 of FIG. 6.

[0092] The intra encoder (722) is configured to receive samples of a current block (e.g., a processing block), compare the block to previously coded blocks in the same picture, generate transformed quantized coefficients, and possibly also generate intra prediction information (e.g., intra prediction direction information according to one or more intra coding techniques). The intra encoder (722) may calculate an intra prediction result (e.g., a predicted block) based on the intra prediction information and a reference block in the same picture.

[0093] The overall controller (721) may be configured to determine overall control data and control other components of the video encoder (703) based on the overall control data. In one example, the overall controller (721) determines a prediction mode for the block and provides a control signal to the switch (726) based on the prediction mode. For example, if the prediction mode is an intra-mode, the overall controller (721) controls the switch (726) to select an intra-mode result for use by the residual value calculator (723) and controls the entropy encoder (725) to select intra-prediction information and include the intra-prediction information in the bitstream, and if the prediction mode for the block is an inter-mode, the overall controller (721) controls the switch (726) to select an inter-prediction result for use by the residual value calculator (723) and controls the entropy encoder (725) to select inter-prediction information and include the inter-prediction information in the bitstream.

[0094] The residual value calculator (723) may be configured to calculate a difference (residual value data) between the received block and a prediction result for the block selected from the intra-encoder (722) or the inter-encoder (730). The residual value encoder (724) may be configured to encode the residual value data to generate transform coefficients. For example, the residual value encoder (724) may be configured to transform the residual value data from the spatial domain to the frequency domain to generate transform coefficients. The transform coefficients are then subjected to a quantization process to obtain quantized transform coefficients. In various embodiments, the video encoder (703) also includes a residual decoder (728). The residual decoder (728) is configured to perform an inverse transform to generate decoded residual value data. The decoded residual value data can be used appropriately by the intra-encoder (722) and the inter-encoder (730). For example, the inter encoder (730) can generate a decoded block based on the decoded residual value data and the inter prediction information, and the intra encoder (722) can generate a decoded block based on the decoded residual value data and the intra prediction information. The decoded block is appropriately processed to generate a decoded picture, which can be buffered in a memory circuit (not shown) and used as a reference picture.

[0095] The entropy encoder (725) may be configured to format a bitstream to include the encoded block to perform entropy encoding. The entropy encoder (725) may be configured to include various information in the bitstream. For example, the entropy encoder (725) may be configured to include overall control data, selected prediction information (e.g., intra-prediction information or inter-prediction information), residual value information, and other suitable information in the bitstream. When encoding a block in a merged sub-mode of either an inter mode or a bi-prediction mode, the residual value information may not be present.

[0096] 8 shows a diagram of an example video decoder (810) according to another embodiment of the present disclosure. The video decoder (810) is configured to receive coded pictures that are part of an encoded video sequence and to decode the coded pictures to generate reconstructed pictures. In one example, the video decoder (810) may be used in place of the example video decoder (410) of FIG. 4.

[0097] In the example of Figure 8, the video decoder (810) includes an entropy decoder (871), an inter decoder (880), a residual decoder (873), a reconstruction module (874), and an intra decoder (872), connected together as shown in the example arrangement of Figure 8.

[0098] The entropy decoder (871) may be configured to reconstruct from the coded picture certain symbols that represent syntax elements that make up the coded picture. Such symbols may include, for example, prediction information (e.g., intra-mode, inter-mode, bi-predicted mode, merge submode, or another submode) that may identify the mode used to code the block, certain samples or metadata used for prediction by the intra-decoder (872) or inter-decoder (880), and residual information, for example in the form of quantized transform coefficients. In one example, if the prediction mode is an inter-mode or bi-predicted mode, the inter-prediction information is provided to the inter-decoder (880), and if the prediction type is an intra-prediction type, the intra-prediction information is provided to the intra-decoder (872). The residual information may be inverse quantized and provided to the residual decoder (873).

[0099] An inter decoder (880) may be configured to receive the inter prediction information and to generate an inter prediction result based on the inter prediction information.

[0100] The intra decoder (872) may be configured to receive the intra prediction information and to generate a prediction result based on the intra prediction information.

[0101] The residual decoder (873) may be configured to perform inverse quantization to extract dequantized transform coefficients, and process the dequantized transform coefficients to transform the residual from the frequency domain to the spatial domain. The residual decoder (873) may also utilize certain control information (to include quantizer parameters (QP)), which may be provided by the entropy decoder (871) (this may be control information with only a small amount of data, so the data path is not shown).

[0102] The reconstruction module (874) may be configured to combine, in the spatial domain, the residual output by the residual decoder (873) with the prediction result (output by the inter-prediction module or the intra-prediction module, as the case may be) to form a reconstructed block that forms part of the reconstructed picture as part of the reconstructed video. It is noted that other suitable operations, such as a deblocking operation, may also be performed to improve video quality.

[0103] It is noted that the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) may be implemented using any suitable technique. In some embodiments, the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) may be implemented using one or more integrated circuits. In other embodiments, the video encoders (403), (603), and (603) and the video decoders (410), (510), and (810) may be implemented using one or more processors executing software instructions.

[0104] Focusing on block partitioning used for encoding and decoding, a typical partitioning may start from a base block and follow a predefined set of rules, a specific pattern, a partitioning tree, or any partitioning structure or scheme. The partitioning may be hierarchical or recursive. After dividing or partitioning the base block according to the example partitioning procedures described below or other procedures or any combination thereof, a final set of partitions or coding blocks may be obtained. Each such partition may be at one of various partitioning levels of the partitioning hierarchy and may be of various shapes. Each partition may be referred to as a coding block (CB). In various partitioning implementations described further below, each resulting CB may be of any allowed size and any allowed partitioning level. Such partitions are referred to as coding blocks because they may form units on which some basic encoding / decoding decisions can be made and on which optimization, determination, and signaling of encoding / decoding parameters can be performed in the coded video bitstream. The highest or deepest level of the final division represents the depth of the coding block division structure of the tree. The coding block may be a luma coding block or a chroma coding block. The CB tree structure of each color may be called a coding block tree (CBT).

[0105] The coding blocks for all color channels may be collectively referred to as a coding unit (CU). The hierarchical structure for all color channels may be collectively referred to as a coding tree unit (CTU). The division pattern or structure of various color channels in a CTU may or may not be the same.

[0106] In some implementations, the partition tree scheme or structure used for the luma and chroma channels may not need to be the same. In other words, the luma and chroma channels may have different coding tree structures or patterns. Furthermore, whether the same or different coding partition tree structures are used for the luma and chroma channels, and the actual coding partition tree structure used may depend on whether the slice being coded is a P slice, a B slice, or an I slice. For example, in an I slice, the chroma and luma channels may have different coding partition tree structures or coding partition tree structure modes, while in a P slice or a B slice, the luma and chroma channels may share the same coding partition tree scheme. When different coding partition tree structures or modes are applied, the luma channel can be partitioned into CBs by a certain coding partition tree structure, and the chroma channel can be partitioned into chroma CBs by another coding partition tree structure.

[0107] In some implementations, a predefined partitioning pattern may be applied to the base block. As shown in FIG. 9, an example of a four-way partitioning tree may start from an initial predefined level (for base block size, e.g., 64×64 block level or other size) and partition the base block down to a predefined lowest level (e.g., 4×4 level). For example, the base block may be subjected to four predefined partitioning options or partitioning patterns shown by 902, 904, 906 and 908, and the partitioning shown by R may be used for recursive partitioning in that the same partitioning options shown in FIG. 9 may be repeated on a smaller scale down to the lowest level (e.g., 4×4 level). In some implementations, other limitations may be applied to the partitioning scheme of FIG. 9. In the implementation of FIG. 9, a rectangular partitioning part (e.g., 1:2 / 2:1 rectangular partitioning) may be used, but the rectangular partitioning part may not be recursive, while the square partitioning part may be recursive. The final set of coding blocks is generated by partitioning according to FIG. 9 with recursion when necessary. The coding tree depth may further be defined to indicate the division depth starting from the root node or block. For example, the coding tree depth of the root node or block (e.g., a 64×64 block) may be set to 0, and after the root block is divided once according to FIG. 9, the coding tree depth is increased by 1. The maximum or deepest level from the 64×64 base block to the 4×4 minimum division is 4 (starting from level 0) in the above scheme. Such a division scheme may be applied to one or more of the color channels. Each color channel may be divided individually according to the scheme of FIG. 9 (e.g., the division pattern or option may be determined from the predefined pattern for each color channel at each hierarchical level separately). Alternatively, two or more of the color channels may share the same hierarchical pattern tree of FIG. 9 (e.g., the same division pattern or option may be selected from the predefined pattern for two or more color channels at each hierarchical level).

[0108] FIG. 10 shows another example of a predefined partitioning pattern that allows a partitioning tree to be formed by recursive partitioning. As shown in FIG. 10, ten example partitioning structures or patterns can be predefined. A root block can originate from a predefined level (e.g., a base block at the 128×128 level or the 64×64 level). The example partitioning structures in FIG. 10 include various 2:1 / 1:2 rectangular partitions and 4:1 / 1:4 rectangular partitions. The partitioning type with three subdivision indicators 1002, 1004, 1006, and 1008 in the second row of FIG. 10 may be applied to a "T-type" partition. The "T-type" partitions 1002, 1004, 1006, and 1008 may be referred to as left T-type, upper T-type, right T-type, and lower T-type. In some implementations, none of the rectangular partitions in FIG. 10 may be further subdivided. A coding tree depth may be further defined to indicate the partitioning depth originating from the root node or block. For example, the coding tree depth of the root node or block (e.g., a 128x128 block) can be set to 0, and the coding tree depth increases by 1 after the root block is split once according to FIG. 10. In some implementations, the recursive split to the next level of the split tree according to the pattern of FIG. 10 can only use the all-square splits in 1010. In other words, the square splits in the T-type patterns 1002, 1004, 1006, and 1008 cannot be used for recursive splits. The final set of coding blocks is generated using a splitting procedure according to FIG. 10 with recursion when necessary. Such a scheme can be applied to one or more of the color channels. In some implementations, more flexibility can be added to the use of splits below the 8x8 level. For example, 2x2 chroma inter prediction can be used in some examples.

[0109] In another implementation of coding block partitioning, a quadtree structure can be used to partition a base block or an intermediate block into quadtree partitions. Such quadtree partitioning can be applied hierarchically and recursively to any square-shaped partition. Depending on various local characteristics of the base block or intermediate block / partition, the possibility of further quadtree partitioning of the base block or intermediate block or partition can be optimized. Furthermore, the quadtree partitioning at picture boundaries can be optimized. For example, an implicit quadtree partitioning can be performed at picture boundaries such that a block continues to be quadtree partitioned until its size fits the picture boundary.

[0110] Other implementations may use hierarchical bisections starting from the base block. In such a scheme, the base block or mid-level block may be divided into two divisions. The bisections may be either horizontal or vertical. For example, a horizontal bisection may divide the base block or mid-level block into equal left and right divisions. Similarly, a vertical bisection may divide the base block or mid-level block into equal top and bottom divisions. Such bisections may be hierarchical and recursive. For each base block or mid-level block, a decision may be made as to whether to continue the bisection scheme, and if so, whether to use a horizontal or vertical bisection. In some implementations, further divisions may be stopped at a predefined minimum division size (either a minimum division size in one dimension or a minimum division size in both dimensions). Alternatively, further divisions may be stopped once a predefined division level or depth starting from the base block is reached. In some implementations, the aspect ratio of the divisions may be limited. For example, the aspect ratio of the divisions may not be less than 1:4 (or more than 4:1). Thus, for a vertical strip-shaped section with a vertical to horizontal aspect ratio of 4:1, it can only be further divided vertically into upper and lower sections, each of which has a vertical to horizontal aspect ratio of 2:1.

[0111] In yet some other examples, a third division scheme as shown in FIG. 13 can be used to divide the base block or any intermediate block. The third division pattern can be implemented vertically as shown in FIG. 13 at 1302 or horizontally as shown in FIG. 13 at 1304. Although the example division ratio in FIG. 13 is shown as 1:2:1 both vertically and horizontally, other ratios may be predefined. In some implementations, two or more different ratios may be predefined. Such a third division scheme can be used to complement a quadtree or binary division structure in that such triple tree division can capture objects in the middle of a block in one continuous division, whereas quadtrees and binary trees always divide along the block middle, causing objects to be divided into separate divisions. In some implementations, the width and height of a division in a typical triple tree is always a power of two to avoid further transformations.

[0112] The above partitioning schemes can be combined in any manner at various partitioning levels. As an example, the above-mentioned quadtree partitioning scheme and binary partitioning scheme may be combined to partition the base block into a quadtree-binary-tree (QTBT) structure. In such a scheme, the base block or intermediate blocks / partitions may be either quadtree partitioned or binary partitioned according to a predefined set of conditions (if conditions are specified). A particular example is shown in FIG. 14. In the example of FIG. 14, the base block is first quadtree partitioned into four partitions shown at 1402, 1404, 1406 and 1408. Each of the resulting partitions is then either quadtree partitioned into four further partitions at the next level (e.g. 1408), binary partitioned into two further partitions (e.g. either horizontally or vertically as in 1402 or 1406, both of which are symmetrical), or not partitioned (e.g. 1404). As shown in the overall exemplary division pattern in 1410 and the corresponding tree structure / representation in 1420, binary division or quadtree division can be used recursively for square shaped divisions. In 1410 and 1420, solid lines represent quadtree division and dashed lines represent binary division. A flag can be used at each binary division node (non-leaf binary division) to indicate whether the binary division is horizontal or vertical. For example, as shown in 1420, which is consistent with the division structure in 1410, a flag "0" can be said to represent a horizontal binary division and a flag "1" can be said to represent a vertical binary division. For the divisions of quadtree division, there is no need to indicate the division type because in quadtree division, a block or division is always divided both horizontally and vertically to generate four sub-blocks / divisions of equal size. In some implementations, a flag "1" may represent a horizontal binary division and a flag "0" may represent a vertical binary division.

[0113] In some implementations of QTBT, the quadtree and binary partitioning rule sets can be represented by the following predefined parameters and associated corresponding functions: -CTU size: quadtree root node size (base block size) -MinQTSize: The minimum allowable quadtree leaf node size -MaxBTSize: Maximum allowable binary tree root node size -MaxBTDepth: Maximum allowed binary tree depth -MinBTSize: The minimum allowable binary tree leaf node size In some implementations of the QTBT partitioning structure, the CTU size can be set as 128x128 luma samples with corresponding 64x64 blocks of chroma samples (assuming typical chroma subsampling), MinQTSize can be set as 16x16, MaxBTSize can be set as 64x64, MinBTSize (both width and height) may be set as 4x4, and MaxBTDepth can be set as 4. A quadtree partition can be applied to the CTU first to generate quadtree leaf nodes. The quadtree leaf nodes can range in size from 16x16 (i.e. MinQTSize), which is the minimum size a quadtree leaf node can have, to 128x128 (i.e. CTU size). If the node is 128x128, it will not be split by the binary tree first since its size exceeds MaxBTSize (i.e. 64x64). Otherwise, the node that does not exceed MaxBTSize can be split by the binary tree. In the example of FIG. 14, the base block is 128x128. According to the default rule set, only quadtree decomposition can be performed on the basic block. The decomposition depth of the base block is zero. Each of the four resulting decompositions is 64x64, which does not exceed MaxBTSize, and each decomposition can be further quadtree or bintree decomposed at level 1. The process proceeds. No further decompositions can be assumed when the bintree depth reaches MaxBTDepth (i.e. 4). No further horizontal decompositions can be assumed when the width of the bintree node is equal to MinBTSize (i.e. 4). Similarly, no further vertical decompositions can be assumed when the height of the bintree node is equal to MinBTSize.

[0114] In some implementations, the above QTBT scheme can be configured to accommodate luma and chroma flexibility with the same QTBT structure, or luma and chroma flexibility with separate QTBT structures. For example, in P and B slices, the luma CTB and chroma CTB in one CTU may share the same QTBT structure. Meanwhile, in I slices, the luma CTB may be split into CBs with a QTBT structure, and the chroma CTB may be split into chroma CBs with another QTBT structure. This means that CUs can be used to apply to different color channels in an I slice, for example, an I slice may consist of a coding block of a luma component or a coding block of two chroma components, and a CU in a P or B slice may consist of coding blocks of all three color components.

[0115] In other implementations, the QTBT method can be supplemented with the above-mentioned third division method. Such implementations are sometimes referred to as multi-type-tree (MTT) structures. For example, in addition to the bisection of the nodes, one of the third division patterns in Figure 13 can be selected. In some implementations, only square nodes can be subjected to third division. An additional flag can be used to indicate whether the third division is horizontal or vertical.

[0116] The primary motivation for designing two-level and multi-level trees, such as the QTBT implementation and the QTBT implementation complemented by a trichotomous partition, is to reduce complexity. In theory, the complexity over the tree is T D where T is the number of split types and D is the depth of the tree. A tradeoff can be made by using multiple types (T) while reducing the depth (D).

[0117] In some implementations, the CB can be further divided. For example, the CB can be further divided into multiple prediction blocks (PBs) for intraframe or interframe prediction during the encoding and decoding processes. In other words, the CB can be further divided into different subdivisions, and separate prediction decisions / configurations can be made. In parallel, the CB can be further divided into multiple transform blocks (TBs) to define the level at which the transformation or inverse transformation of the video data is performed. The division of the CB into PBs and TBs may be the same or different. For example, each division scheme may be performed using its own procedure, for example, based on various characteristics of the video data. In some implementations, the PB division scheme and the TB division scheme may be unrelated. In other implementations, the PB and TB division schemes and the boundaries may be correlated. In some implementations, for example, the TB may be divided after the PB division, and in particular, each PB may be further divided into one or more TBs after being determined according to the division of the coding blocks. For example, in some implementations, the PB may be divided into one TB, two TBs, four TBs, or other number of TBs.

[0118] In some implementations, the luma and chroma channels may be treated differently when splitting the base blocks into coding blocks and further into prediction and / or transform blocks. For example, in some implementations, the division of the coding blocks into prediction and / or transform blocks may be used for the luma channel, whereas the division of the coding blocks into prediction and / or transform blocks may not be used for one or more chroma channels. Thus, such implementations may only perform the transformation and / or prediction of the luma blocks at the coding block level. In another example, the minimum transform block size of the luma channel and one or more chroma channels may be different, e.g., allowing the coding blocks of the luma channel to be divided into smaller transform and / or predictive blocks than the chroma channels. In yet another example, the maximum depth of the division of the coding blocks and / or predictive blocks into transform blocks may be different for the luma channel and the chroma channels, e.g., allowing the coding blocks of the luma channel to be divided into transform and / or predictive blocks that are deeper than the one or more chroma channels. In a particular example, a luma coding block may be partitioned into transform blocks of multiple sizes, which may be expressed as a recursive partition going down up to two levels, allowing transform block shapes such as square, 2:1 / 1:2, and 4:1 / 1:4, and transform block sizes from 4x4 to 64x64. In contrast, a chroma block may only allow the largest possible transform block specified for the luma block.

[0119] In some implementations of partitioning a coding block into PBs, the depth, shape, and / or other characteristics of the PB partition may depend on whether the PB is intra-coded or inter-coded.

[0120] The division of the coding block (or prediction block) into transform blocks may be performed recursively or non-recursively in various typical manners, including but not limited to quadtree division and predefined pattern division, and may be performed by further considering the transform blocks at the boundaries of the coding block or the prediction block. Generally speaking, the resulting transform blocks may be transform blocks of different division levels, may not be transform blocks of the same size, and may not be required to be square in shape (e.g., they may be rectangular with some allowable sizes and aspect ratios). Further examples are described in more detail below with respect to Figures 15, 16, and 17.

[0121] In contrast to the above, in another implementation, the CB obtained through any of the above partitioning methods may be used as a basic coding block, i.e., a minimum coding block, for prediction and / or transformation. In other words, no further partitioning is performed for inter / intra prediction and / or transformation purposes. For example, the CB obtained from the above QTBT method may be directly used as a unit for performing prediction. Specifically, the concept of multiple partition types is eliminated in such a QTBT structure, i.e., the separation of CU, PU, ​​and TU is eliminated, to accommodate the above-mentioned higher flexibility of CU / CB partition shapes. In such a QTBT block structure, CU / CB can have both square and rectangular shapes. The leaf nodes of such a QTBT are used as units for prediction and transformation processes without any further partitioning. This means that the block sizes of CU, PU, ​​and TU are the same in such an example QTBT coding block structure.

[0122] The various CB division schemes described above and further division of the CB into PB and / or TB (not including PB / TB division) may be combined in any manner. The following specific implementations are provided as non-limiting examples.

[0123] In the following, a specific implementation of the coding block division and the transform block division is described. In such an implementation, the base block can be divided into coding blocks using recursive quadtree division or the above-mentioned predefined division patterns (such as the division patterns in Figs. 9 and 10). For each level, local video data characteristics can determine whether to continue further quadtree division of a particular division. The obtained CBs can be CBs of different quadtree division levels and CBs of different sizes. The decision of whether the picture area should be coded using inter-picture (temporal) prediction or intra-picture (spatial) prediction can be made at the CB level (it can also be made at the CU level for all three color channels). Each CB can be further divided into one PB, two PBs, four PBs, or other number of PBs according to the predefined PB division type. Within one PB, the same prediction process can be applied, and related information can be sent to the decoder on a PB-by-PB basis. After obtaining the residual block by applying the prediction process based on the PB division type, the CB can be divided into TBs according to another quadtree structure similar to the coding tree of the CB. In this particular implementation, the CB or TB may be limited to a square shape, but is not required to be limited to this. Furthermore, in this particular implementation, the PB may be square or rectangular in shape in the case of inter prediction, and the PB may be limited to a square in the case of intra prediction. The coding block may be divided into, for example, four square-shaped TBs. Each TB may be further recursively divided (using quadtree division) into smaller TBs, which are called Residual Quadtrees (RQTs).

[0124] In the following, another implementation of splitting a base block into CB, PB, and / or TB will be further described. For example, instead of using multiple division unit types such as those shown in FIG. 9 or FIG. 10, a quadtree with nested multi-type tree (e.g., QTBT or QTBT with ternary division described above) using a bisection division structure and a trisection division structure can be used. Separation of CB, PB, and TB (i.e., splitting CB into PB and / or TB, and splitting PB into TB) can be discontinued except when necessary for CBs whose size is too large for the maximum transform length (such CBs may need to be further split). This exemplary division scheme can be designed to accommodate higher flexibility of CB division shape, so that both prediction and transformation can be performed on the CB level without further division. In such a coding tree structure, CB can have either a square shape or a rectangular shape. In particular, a coding tree block (CTB) can be first divided by a quadtree structure. Then, the quadtree leaf nodes can be further divided by a nested multi-type tree structure. An example of a nested multi-type tree structure using a bisection or trisection is shown in FIG. 11. In particular, the exemplary multi-type tree structure in FIG. 11 includes four split types called vertical bisection (SPLIT_BT_VER) (1102), horizontal bisection (SPLIT_BT_HOR) (1104), vertical trisection (SPLIT_TT_VER) (1106), and horizontal trisection (SPLIT_TT_HOR) (1108). Then, CB corresponds to the leaf of the multi-type tree. In this implementation, this segmentation is used for both prediction and transformation processes without any further splitting, except when CB is too large for the maximum transform length. This means that in most cases, the block sizes of CB, PB, and TB are the same in the quadtree with nested multi-type tree coding block structure. An exception occurs when the maximum transform length that can be accommodated is less than the width or height of the color components of CB. In some implementations, in addition to a binary or trichotomous division, the nested pattern of FIG. 11 can further include a quadtree division.

[0125] One particular example of quadtrees (including quadtree options, binary partition options and tri-partition options) with nested multi-type tree coding block structures of block partitions for one base block is shown in Figure 12. In more detail, Figure 12 shows a base block 1200 being quadtree partitioned into four square partitions 1202, 1204, 1206 and 1208. For each partition of the quadtree partition, it is determined whether the multi-type tree structure and quadtrees of Figure 11 should be further used for further partitioning. In the example of Figure 12, partition 1204 is not further partitioned. A different quadtree partition is used for each of partitions 1202 and 1208. In the splitter 1202, the upper left, upper right, lower left and lower right subdivisions are respectively subjected to third level splitting, namely quadtree, horizontal bisection 1104 in FIG. 11, no split and horizontal bisection 1108 in FIG. 11. Another quadtree split is used in the splitter 1208, and the upper left, upper right, lower left and lower right subdivisions are respectively subjected to third level splitting, namely vertical bisection 1106 in FIG. 11, no split, no split and horizontal bisection 1104 in FIG. 11. Two of the subdivisions of the third level upper left subdivision in 1208 are further split according to horizontal bisection 1104 in FIG. 11 and horizontal bisection 1108 in FIG. 11, respectively. The division 1206 is divided into two divisions using a second level division pattern according to the vertical bisection division 1102 of Figure 11, and the two divisions are further divided at a third level according to the horizontal third division 1108 and the vertical bisection division 1102 of Figure 11. A fourth level division is then applied to one of these according to the horizontal bisection division 1104 of Figure 11.

[0126] In the particular example above, the maximum luma transform size may be 64x64, and the maximum supported chroma transform size can be different from the luma (e.g., 32x32). Although the exemplary CB in Figure 12 above is not typically further divided into smaller PBs and / or TBs, if the width or height of a luma or chroma coded block exceeds the maximum transform width or height, the luma or chroma coded block can still be automatically divided in the horizontal and / or vertical directions to satisfy the transform size limitation in that direction.

[0127] In the specific example of dividing the base block into CBs described above, the coding tree scheme can accommodate that luma and chroma can have separate block tree structures, as described above. For example, in P slices and B slices, the luma CTB and chroma CTB in one CTU may share the same coding tree structure. In an I slice, for example, luma and chroma may have separate coding block tree structures. When separate block tree structures are applied, the luma CTB can be divided into luma CBs by a certain coding tree structure, and the chroma CTB is divided into chroma CBs by another coding tree structure. This means that a CU in an I slice can consist of a coding block of a luma component or a coding block of two chroma components, and a CU in a P slice or B slice always consists of coding blocks of all three color components, except when the image is a monochrome image.

[0128] If a coding block is further divided into multiple transform blocks, the transform blocks in the coding block may be ordered in the bitstream according to various orders or may be ordered in the bitstream according to a scanning manner. Implementations of dividing a coding block or a prediction block into transform blocks and the coding order of the transform blocks are described in more detail below. In some implementations, as described above, the transform partitioning can accommodate transform blocks of multiple shapes (e.g., 1:1 (square), 1:2 / 2:1, and 1:4 / 4:1, when the transform block size ranges from, e.g., 4×4 to 64×64). In some implementations, when the coding block is 64×64 or smaller, the transform block partitioning can be applied only to the luma component such that the transform block size is the same for chroma blocks as the coding block size. In cases other than those mentioned above, when the width or height of the coding block exceeds 64, both the luma coding block and the chroma coding block can be implicitly divided into transform blocks that are multiples of min(W,64)×min(H,64) and transform blocks that are multiples of min(W,32)×min(H,32), respectively.

[0129] In some implementations of transform block partitioning, for intra-coded blocks and inter-coded blocks, the coding block can be further partitioned into multiple transform blocks using a partition depth up to a predefined number of levels (e.g., 2 levels). The partition depth and partition size of the transform block can be related. For some implementations, the mapping from the transform size of the current depth to the transform size of the next depth is shown in Table 1 as follows: [Table 1]

[0130] Based on the example of the correspondence in Table 1, for a 1:1 square block, the next level transform splitter can produce four 1:1 square sub-transform blocks. The transform splitting can be stopped at, for example, 4×4. Thus, the transform size of the current depth, which is 4×4, corresponds to the same size of 4×4 at the next depth. In the example of Table 1, for a non-square 1:2 / 2:1 block, the next level transform splitter can produce two 1:1 square sub-transform blocks, whereas for a non-square 1:4 / 4:1 block, the next level transform splitter can produce two 1:2 / 2:1 sub-transform blocks.

[0131] In some implementations, further restrictions on transform block partitioning can be applied for luma components of intra-coded blocks. For example, all sub-transform blocks at each level of transform partitioning can be restricted to be of equal size. For example, for a 32×16 coding block, the level 1 transform partitioning results in two 16×16 sub-transform blocks, and the level 2 transform partitioning results in eight 8×8 sub-transform blocks. In other words, it is ensured that the second level partitioning is applied to all first level sub-blocks to keep the transform units of the same size. An example of transform block partitioning for an intra-coded square block according to Table 1 is shown in FIG. 15 with the coding order indicated by the arrows. Specifically, 1502 denotes a square coding block. The first level partitioning according to Table 1 into four transform blocks of equal size is shown in 1504 with the coding order indicated by the arrows. The second level partitioning of all first level equal-sized blocks according to Table 1 into 16 transform blocks of equal size is shown in 1506 with the coding order indicated by the arrows.

[0132] In some implementations, the luma components of an inter-coded block may not be subject to the above restrictions for intra-coding. For example, after the first level of transform partitioning, any one of the sub-transform blocks may be further partitioned at an additional level independently of the others. Thus, the resulting transform blocks may or may not be transform blocks of the same size. An example of partitioning an inter-coded block into transform blocks is shown in FIG. 16 along with their coding order. In the example of FIG. 16, an inter-coded block 1602 is partitioned into two levels of transform blocks according to Table 1. At the first level, the inter-coded block is partitioned into four transform blocks of the same size. Then, only one of the four transform blocks (but not all four) is further partitioned into four sub-transform blocks, resulting in a total of seven transform blocks of two different sizes as shown in 1604. An example of the coding order of these seven transform blocks is shown by the arrows at 1604 in FIG. 16.

[0133] In some implementations, some further restrictions on the transform blocks for one or more chroma components may be applied, for example, the transform block size for one or more chroma components may be as large as the coding block size, but not less than a predefined size (e.g., 8x8).

[0134] In some other implementations, for coding blocks with either width (W) or height (H) greater than 64, both luma and chroma coding blocks can be implicitly divided into multiples of min(W,64)×min(H,64) and multiples of min(W,32)×min(H,32) transform units, respectively, where in this disclosure min(a,b) can return the smaller of a and b.

[0135] Figure 17 further illustrates an example of another alternative scheme for splitting a coding block or a prediction block into transform blocks. As shown in Figure 17, instead of using recursive transform partitioning, a predefined set of partition types can be applied to the coding block depending on the transform type of the coding block. In the particular example shown in Figure 17, one of six typical partition types can be applied to split the coding block into various numbers of transform blocks. Such a scheme that causes transform block partitioning can be applied to both coding blocks and prediction blocks.

[0136] More specifically, the partitioning scheme of FIG. 17 provides up to six typical partitioning types for any given transform type (the transform type referred to here refers to the transform type of the main transform, such as ADST). In this scheme, any coding block or prediction block can be assigned a transform partitioning type based on, for example, a rate-distortion cost. In one example, the transform partitioning type assigned to a coding block or prediction block can be determined based on the transform type of the coding block or prediction block. As shown by the six transform partitioning types shown in FIG. 17, a particular transform partitioning type can correspond to a transform block partitioning size and pattern. The correspondence between different transform types and different transform partitioning types can be defined. Examples are shown below. The capitalized descriptions indicate the transform partitioning types that can be assigned to a coding block or prediction block based on the rate-distortion cost.

[0137] ·PARTITION_NONE: Allocate transformation size equal to block size.

[0138] ·PARTITION_SPLIT: Allocates a transformation size that is 1 / 2 the width of the block size and 1 / 2 the height of the block size.

[0139] ·PARTITION_HORZ: Allocates a transformation size that has the same width as the block size and 1 / 2 the height of the block size.

[0140] ·PARTITION_VERT: Allocates a transformation size that has half the width of the block size and the same height as the block size.

[0141] ·PARTITION_HORZ4: Allocates a transformation size that has the same width as the block size and 1 / 4 of the height of the block size.

[0142] ·PARTITION_VERT4: Allocates a transformation size that has 1 / 4 the width of the block size and the same height as the block size.

[0143] In the above example, all of the transform partition types shown in Figure 17 include the same transform size for the partitioned transform blocks. This is by way of example only and not limitation. In other implementations, a mix of different transform block sizes may be used for the partitioned transform blocks of a particular partition type (i.e., pattern).

[0144] The PBs resulting from any of the above partitioning schemes (i.e., the CB, which can also be called the PB when not further partitioned into predictive blocks) can then become individual blocks used for coding by either intra-prediction or inter-prediction. In the inter-prediction of the current PB, a residual between the current block and the predictive block can be generated, coded and included in the coded bitstream.

[0145] Inter prediction can be performed, for example, in single reference mode or synthetic reference mode. In some implementations, a skip flag can be included in the bitstream used for the current block (or at a higher level) first to indicate whether the current block is to be inter-coded and not skipped. If the current block is inter-coded, another flag can be further included in the bitstream to signal whether a single reference mode or a synthetic reference mode is used to predict the current block. In single reference mode, one reference block can be used to generate a prediction block for the current block. In synthetic reference mode, two or more reference blocks can be used to generate a prediction block, for example, by weighted averaging. The synthetic reference mode may be referred to as one or more reference modes, two or more reference modes, or multiple reference modes. The one or more reference blocks can be identified using one or more reference frame indices and further using one or more corresponding motion vectors indicating a shift in position, for example, horizontal and vertical pixels, between the one or more reference blocks and the current block. For example, in a single reference mode, an inter-predicted block of a current block may be generated from a single reference block identified as a predictive block by one motion vector in a reference frame, whereas in a combined reference mode, a predictive block may be generated by a weighted average of two reference blocks in two reference frames indicated by two reference frame indexes and two corresponding motion vectors. The one or more motion vectors may be coded in various ways and included in the bitstream.

[0146] In some implementations, a decoded picture buffer (DPB) may be managed in the encoding system or the decoding system. Some images / pictures may be managed in the DPB in a display pending state (in the decoding system), and some images / pictures in the DPB may be used as reference frames to enable inter-prediction (in the decoding system or the encoding system). In some implementations, the reference frames in the DPB may be tagged as either short-term or long-term references for the current image being encoded or decoded. For example, short-term reference frames may include frames used for inter-prediction of blocks in the current frame or blocks in a predetermined number (e.g., two) subsequent video frames closest to the current frame in decoding order. Long-term reference frames may include frames in the DPB that may be used to predict image blocks in frames that are more than the predetermined number of frames away from the current frame in decoding order. Information regarding such tags of short-term and long-term reference frames may be referred to as a Reference Picture Set (RPS), and this information may be added to the header of each frame in the encoded bitstream. Each frame in a coded video stream can be identified by a Picture Order Counter (POC), which may be counted according to absolute playback order or may relate to a set of pictures, starting with, for example, an I-frame.

[0147] In some implementations, one or more reference picture lists including identities of short-term and long-term reference frames used for inter-prediction may be formed based on information in the RPS. For example, one picture reference list may be formed for unidirectional inter-prediction, denoted as L0 reference (also referred to as reference list 0). In contrast, two picture reference lists may be formed for bidirectional inter-prediction, denoted as L0 (also referred to as reference list 0) and L1 (also referred to as reference list 1) for each of the two prediction directions. The order of the reference frames included in the L0 and L1 lists may be determined in various predefined manners. The lengths of the L0 and L1 lists may be signaled in the video bitstream. The unidirectional inter-prediction may be in either a single reference mode or a synthetic reference mode, where the references used to generate the prediction block by weighted averaging in the synthetic prediction mode are on the same side of the block to be predicted. Bidirectional inter-prediction is limited to the synthetic mode in that at least two reference blocks are required for bidirectional inter-prediction.

[0148] In some implementations, a merge mode (MM) for inter prediction may be implemented. In general, in the merge mode, one or more of the motion vectors of the single reference prediction or the motion vectors of the synthetic reference prediction of the current PB may be derived from one or more other motion vectors, rather than being calculated and signaled independently. For example, in the encoding system, one or more current motion vectors of the current PB may be represented by one or more differences between one or more current motion vectors and one or more other motion vectors (referred to as reference motion vectors) that have already been encoded. Such one or more differences of one or more motion vectors, rather than the one or more current motion vectors themselves, may be encoded and included in the bitstream, and may be linked to one or more reference motion vectors. Correspondingly, in the decoding system, one or more motion vectors corresponding to the current PB may be derived based on one or more decoded motion vector differences and one or more decoded reference motion vectors linked thereto. As a specific form of the general merge mode (MM) inter prediction, such inter prediction based on one or more motion vector differences may be referred to as a merge mode with motion vector difference (MMVD). Thus, MM can be implemented in general and MMVD can be implemented in specific cases to exploit correlations between motion vectors associated with different PBs to improve coding efficiency. In one example, motion vectors of nearby PBs may be similar, so the MVD may be small and the motion vectors of nearby PBs can be coded efficiently. In another example, motion vectors may be temporally (between frames) correlated for blocks that are spatially similarly located / positioned.

[0149] In some implementations, an MM flag may be included in the bitstream during the encoding process to indicate whether the current PB is of merge mode. Additionally or alternatively, an MMVD flag may be included in the bitstream during the encoding process to signal whether the current PB is in MMVD mode. MM and / or MMVD flags or indicators may be provided at the PB level, CB level, CU level, CTB level, CTU level, slice level, picture level, etc. In certain examples, both an MM flag and an MMVD flag may be included and used for the current CU, and an MMVD flag may be signaled immediately after the skip flag and the MM flag to specify whether the MMVD mode is used for the current CU.

[0150] In some implementations of MMVD, a list of reference motion vector (RMV) candidates for motion vector prediction, i.e., MV predictor candidates, can be formed and used for the block being predicted. The list of RMV candidates can include a predefined number (e.g., two) of MV predictor candidate blocks whose motion vectors can be used to predict the current motion vector. The RMV candidate blocks can include blocks selected from neighboring blocks in the same frame and / or from temporal blocks (e.g., blocks at the same location in a frame before or after the current frame). These selections represent blocks that are at a spatial or temporal location relative to the current block and that are likely to have a similar or identical motion vector to the current block. The size of the list of MV predictor candidates can be predefined. For example, the list may include two or more candidates. To be listed as an RMV candidate, for example, a candidate block may be required to have the same reference frame (or reference frames) as the current block, and the candidate block must exist (e.g., a boundary check needs to be performed if the current block is close to a frame edge), and must have already been coded during the coding process and / or must have already been decoded during the decoding process. In some implementations, the list of merge candidates may first include spatially neighboring blocks (scanned in a certain predefined order) if they are available and satisfy the above conditions, followed by the temporal block if there is still room available in the list. Neighboring RMV candidate blocks may be selected, for example, from the blocks to the left and above the current block. The list of RMV predictor candidates can be dynamically formed as Dynamic Reference Lists (DRLs) at different levels (sequence, picture, frame, slice, superblock, etc.). The DRLs can be signaled in the bitstream.

[0151] In some implementations, the current MV predictor candidate used as the reference motion vector for predicting the motion vector of the current block can be signaled. When the RMV candidate list includes two candidates, a one-bit flag called a merge candidate flag can be used to indicate the selection of the reference merge candidate. When the current block is predicted in a compositing mode, each of the multiple motion vectors predicted using the MV predictor can be associated with a reference motion vector obtained from the merge candidate list. The RMV candidate used for closer prediction of the MV of the current coding block can be determined by the encoder, and the selection can be signaled as an index into the DRL.

[0152] In some implementations of MMVD, after an RMV candidate is selected and used as a base motion vector predictor for a predicted motion vector, a motion vector differential (also called MVD delta MV, which represents the difference between the predicted motion vector and the reference candidate motion vector) can be calculated by the coding system. Such MVD can include information representing the magnitude of the MV difference and the direction of the MV difference, both of which can be signaled in the bitstream. The motion difference magnitude and motion difference direction can be signaled in various ways.

[0153] In some implementations of MMVD, a distance index can be used to specify the magnitude information of the motion vector differential and to indicate one of a set of predefined offsets representing a predefined motion vector differential from the origin (reference motion vector). Then, the MV offset according to the signaled index can be added to either the horizontal or vertical component of the origin (reference) motion vector. The directionality information of the MVD can determine whether to offset the horizontal or vertical component of the reference motion vector. An example of a predefined relationship between the distance index and the predefined offsets is listed in Table 2. [Table 2]

[0154] In some implementations of MMVD, a direction index can be further signaled and used to indicate the direction of the MVD with respect to the reference motion vector. In some implementations, the direction can be restricted to either horizontal or vertical. An example of a 2-bit direction index is shown in Table 3. In the example of Table 3, the interpretation of the MVD can change depending on the information of the origin / reference MV. For example, if the origin / reference MV corresponding to a uni-predictive block or a bi-predictive block with both reference frame lists faces the same side of the current picture (i.e., the POCs of the two reference pictures are both greater than or both less than the POC of the current picture), the code in Table 3 can be said to refer to the code (direction) of the MV offset added to the origin / reference MV. If the origin / reference MV corresponds to a bi-predictive block with two reference pictures on different sides of the current picture (i.e., the POC of one reference picture is greater than the POC of the current picture and the POC of the other reference picture is less than the POC of the current picture) and the difference between the reference POC in picture reference list 0 and the current frame exceeds the difference between the reference POC in picture reference list 1 and the current frame, the code in Table 3 refers to the code of the MV offset added to the reference MV corresponding to the reference picture in picture reference list 0, and the code of the offset of the MV corresponding to the reference picture in picture reference list 1 has an inverse value (the inverse sign of the offset). In the other case, if the difference between the reference POC in picture reference list 1 and the current frame exceeds the difference between the reference POC in picture reference list 0 and the current frame, the code in Table 3 refers to the code of the MV offset added to the reference MV associated with picture reference list 1, and the code of the offset of the reference MV associated with picture reference list 0 has an inverse value. [Table 3]

[0155] In some implementations, the MVD can be scaled according to the difference in POC in each direction. If the difference in POC in both lists is the same, no scaling is required. Otherwise, if the difference in POC in reference list 0 exceeds that of reference list 1, scale the MVD of reference list 1. If the POC difference of reference list 1 is larger than that of list 0, the MVD of list 0 can be scaled in the same manner. If the origin MV is uni-predicted, add the MVD to the available MV, i.e., the reference MV.

[0156] In some implementations of MVD coding and signaling for bidirectional synthetic prediction, in addition to or instead of coding and signaling two MVDs separately, symmetric MVD coding may be performed such that only one MVD needs to be signaled and the other MVD can be derived from the signaled MVD. In such implementations, motion information including reference picture indexes of both list-0 and list-1 is signaled. However, for example, only the MVD related to reference list-0 is signaled, and the MVD related to reference list-1 is derived without being signaled. In particular, at the slice level, a flag called "mvd_l1_zero_flag" may be included in the bitstream to indicate whether reference list-1 is not signaled in the bitstream. If this flag is 1, indicating that reference list-1 is equal to zero (and thus not signaled), a bidirectional prediction flag called "BiDirPredFlag" may be set to 0, meaning that no bidirectional prediction is performed. Differently from the above, when mvd_l1_zero_flag is zero, BiDirPredFlag can be set to 1 when the closest reference picture in list-0 and the closest reference picture in list-1 form a forward-backward pair of reference pictures or a backward-forward pair of reference pictures, and both the reference pictures in list-0 and list-1 are short-term reference pictures. Otherwise, BiDirPredFlag is set to zero. BiDirPredFlag being 1 indicates that the symmetric mode flag is further signaled in the bitstream. When BiDirPredFlag is 1, the symmetric mode flag can be extracted by the decoder from the bitstream. For example, the symmetric mode flag can be signaled at the CU level (if necessary), and the symmetric mode flag can indicate whether the symmetric MVD coding mode is in use for the corresponding CU.If the symmetric mode flag is 1, it indicates that a symmetric MVD coding mode is used, and only the reference picture indexes of both list-0 and list-1 (referred to as "mvp_l0_flag", "mvp_l1_flag") are signaled together with the MVD related to list-0 (referred to as "MVD0"), and the other motion vector differential ("MVD1") is to be derived without being signaled. For example, MVD1 can be derived as -MVD0. Thus, only one MVD is signaled in a typical symmetric MVD mode. In other implementations of MV prediction, for single reference mode and synthetic reference mode MV prediction, a cooperative scheme can be used to implement general merge mode, MMVD, or some other type of MV prediction. Various syntax elements can be used to signal how to predict the MV of the current block.

[0157] For example, in single reference mode, the following MV prediction modes can be signaled:

[0158] NEARMV: does not use any MVD, but directly uses one of the motion vector predictors (MVPs) in a list indicated by a Dynamic Reference List (DRL) index.

[0159] NEWMV: Using one of the motion vector predictors (MVPs) in the list signaled by the DRL index as a reference, apply the delta to the MVP (eg, apply it using the MVD).

[0160] GLOBALMV: Uses motion vectors based on frame-level global motion parameters.

[0161] Similarly, for a synthesized reference inter-prediction mode with two reference frames corresponding to the two MVs to be predicted, the following MV prediction modes can be signaled:

[0162] NEAR_NEARMV: For each of two of the predicted MVs, use one of the motion vector predictors (MVPs) in the list signaled by the DRL index without using the MVD.

[0163] NEAR_NEWMV: To predict the first of two motion vectors, one of the motion vector predictors (MVP) in the list signaled by the DRL index is used as a reference MV without using the MVD, and to predict the second of two motion vectors, one of the motion vector predictors (MVP) in the list signaled by the DRL index is used as a reference MV together with the further signaled delta MV (MVD).

[0164] NEW_NEARMV: To predict the second of two motion vectors, one of the motion vector predictors (MVP) in the list signaled by the DRL index is used as a reference MV without using the MVD, and to predict the first of two motion vectors, one of the motion vector predictors (MVP) in the list signaled by the DRL index is used as a reference MV together with the further signaled delta MV (MVD).

[0165] NEW_NEWMV: Use one of the motion vector predictors (MVPs) in the list signaled by the DRL index as a reference MV, together with the further signaled delta MV that predicts for each of the two MVs.

[0166] GLOBAL_GLOBALMV: The MV obtained from each reference is used based on the frame-level global motion parameters of this MV.

[0167] Thus, the term "NEAR" above refers to MV prediction using a reference MV without MVD as in the general merge mode, whereas the term "NEW" refers to MV prediction using a reference MV and offsetting this MV with the signaled MVD as in the MMVD mode. In synthetic inter prediction, both the reference-based motion vector and the motion vector delta may be typically different or unrelated between both references. This may be the case even though they may be correlated and such correlation may be exploited to reduce the amount of information required to signal the two motion vector deltas. In such a scenario, signaling the two MVDs together may be implemented and signaled in the bitstream.

[0168] The Dynamic Reference List (DRL) described above can be used to dynamically manage and populate a set of indexed motion vectors that are considered as candidate motion vector predictors.

[0169] In some implementations, a default pixel resolution of the MVD can be used. For example, a motion vector precision (or accuracy) of 1 / 8 pixel can be used. The MVDs of the various MV prediction modes described above can be configured and signaled in different ways. In some implementations, different syntax elements can be used to signal one or more of the motion vector differentials in the reference frame list 0 or list 1.

[0170] For example, a syntax element called "mv_joint" can indicate which components of its associated motion vector differential are non-zero. For MVD, all non-zero components are combined to signal this syntax element. For example, mv_joint can be expressed as: 0, which can indicate the absence of non-zero MVD in either the horizontal or vertical direction, 1, which can be shown to have non-zero MVD only in the horizontal direction, It can be shown that there is a non-zero MVD only in the vertical direction, It can be shown that there is a non-zero MVD in the horizontal and vertical directions. It has the value:

[0171] If the "mv_joint" syntax element of the MVD signals that there are no non-zero MVD components, then no further MVD information needs to be signaled, however, if the "mv_joint" syntax signals that there are one or two non-zero components, then additional syntax elements can be further signaled for each non-zero MVD component as described below.

[0172] In one example, a syntax element called "mv_sign" may be used to further indicate whether the corresponding motion vector differential component is positive or negative.

[0173] In another example, a syntax element called "mv_class" may be used to specify a class of motion vector differentials from a predefined set of classes of corresponding non-zero MVD components. For example, the predefined classes of motion vector differentials may be used to divide a contiguous magnitude space of motion vector differentials into non-overlapping ranges, each range corresponding to an MVD class. Thus, the signaled MVD class indicates a magnitude range of the corresponding MVD component. In the implementation shown in Table 4 below, higher classes correspond to motion vector differentials with a larger magnitude range. In Table 4, the symbol (n,m] is used to represent a range of motion vector differentials that is greater than n pixels and less than or equal to m pixels. [Table 4]

[0174] In another example, a syntax element called "mv_bit" may be further used to indicate the integer portion of the offset between a non-zero motion vector differential component and the origin magnitude of the corresponding signaled MV class magnitude range. Thus, mv_bit may be said to indicate the magnitude or amplitude of the MVD. The number of bits required in "my_bit" to signal the full range of each MVD class may vary depending on the MV class function. In this example, MV_CLASS 0 and MV_CLASS 1 of the implementation of Table 4 may require only one bit to indicate an integer pixel offset of 1 or 2 from the origin MVD of 0, with each higher MV_CLASS in the implementation of Table 4 requiring one more "mv_bit" than the MV_CLASS immediately below it.

[0175] In another example, a syntax element called "mv_fr" can be further used to indicate the first two fractional bits of the motion vector differential of the corresponding non-zero MVD component, while a syntax element called "mv_hp" can be further used to indicate the third fractional bit (high resolution bit) of the motion vector differential of the corresponding non-zero MVD component. The two bits "mv_fr" necessarily achieve a quarter pixel MVD resolution, while the "mv_hp" bit can further achieve a one-eighth pixel resolution. In another implementation, more than one "mv_hp" bit can be used to achieve a MVD pixel resolution finer than one-eighth pixel. In some implementations, further flags can be signaled at one or more of various levels to indicate whether an MVD resolution of one-eighth pixel or more is supported. If an MVD resolution is not applied to a particular coding unit, the above syntax elements of the corresponding non-supported MVD resolution may not be signaled.

[0176] In some implementations of the above, the fractional resolution may be independent of the different classes of MVDs, in other words, the same selection of motion vector resolution can be achieved with the default number of bits of "mv_fr" and "mv_hp" used to signal the fractional MVD of the non-zero MVD components, regardless of the magnitude of the motion vector differential.

[0177] On the other hand, in other implementations, there may be differences in the resolution of the motion vector difference for different MVD size classes. In particular, a high-resolution MVD with a large MVD size for a higher MVD class may not provide a statistically significant improvement in compression efficiency. Therefore, the MVD may be encoded with a reduced resolution (integer pixel resolution or fractional pixel resolution) for a range of large MVD sizes corresponding to a higher MVD size class. Similarly, the MVD may be encoded with a reduced resolution (integer pixel resolution or fractional pixel resolution) for large MVD values ​​in most cases. Such MVD class-dependent or MVD size-dependent MVD resolutions are often referred to as adaptive MVD resolutions, size-dependent adaptive MVD resolutions, or size-dependent MVD resolutions. Furthermore, the term "resolution" may be referred to as "pixel resolution". The adaptive MVD resolutions for different objects may be implemented as illustrated by the following implementations that provide a good overall compression efficiency. In particular, due to statistical considerations that non-adaptive processing of the MVD resolution of a large magnitude or high class MVD at the same level as the MVD resolution of a small magnitude or low class MVD may not significantly improve the inter-prediction residual coding efficiency of a block using a large magnitude or high class MVD, the reduction in the number of signaling bits that would result from reducing the MVD accuracy may exceed the increase in bits required to code the inter-prediction residual as a result of such a reduction in MVD accuracy. In other words, using a high MVD resolution of a large magnitude or high class MVD may not result in a larger coding gain than using a low MVD resolution.

[0178] In some typical implementations, increasing the MVD class may result in a decrease in the pixel resolution, i.e., precision, of the MVD, or may remain unchanged. A decrease in pixel resolution of the MVD corresponds to a coarser MVD (i.e., a larger step from a given MVD level to the next). In some implementations, the relationship between MVD pixel resolution and MVD class may be specified, defaulted, or preset, and may not need to be signaled in the encoded bitstream.

[0179] In some implementations, each of the MV classes in Table 3 can be associated with a different MVD pixel resolution.

[0180] In some implementations, each MVD class can be associated with one possible resolution. In other implementations, one or more MVD classes can be associated with two or more optionally selectable MVD pixel resolutions. Thus, the signaling of the bitstream of the current MVD component with such MVD class can be followed by further signaling to indicate the optionally selectable pixel resolution selected for the current MVD component.

[0181] In some implementations, the adaptively enabled MVD pixel resolutions may include, but are not limited to, 1 / 64pel (pixels), 1 / 32pel, 1 / 16pel, 1 / 8pel, 1-4pel, 1 / 2pel, 1pel, 2pel, 4pel... (in decreasing order of resolution). Thus, each of the ascending MVD classes may be associated with one of these MVD pixel resolutions in a non-ascending order. In some implementations, an MVD class may be associated with more than one of the above resolutions, where a higher resolution may be less than or equal to a lower resolution of the previous MVD class. For example, if MV_CLASS_3 in Table 4 is associated with selectable 1pel and 2pel resolutions, then the highest resolution that MV_CLASS_4 in Table 4 may be associated with is 2pel. In other implementations, the highest allowed resolution of a given MV class may exceed the lowest allowed resolution of the previous (lower) MV class, but only as long as it does not increase the average allowed resolution of the ascending MV classes.

[0182] In some implementations, if fractional pixel resolution greater than 1 / 8 pel is permitted, the "mv_fr" and "mv_hp" signaling may be correspondingly extended to a total of 3 or more fractional bits.

[0183] In some implementations, only fractional pixel resolutions of MVD classes less than or equal to the threshold MVD class may be allowed. For example, only fractional pixel resolutions of MVD-CLASS 0 may be allowed, and fractional pixel resolutions of all other MV classes in Table 4 may not be allowed. Similarly, only fractional pixel resolutions of MVD classes less than or equal to any of the other MV classes in Table 4 may be allowed. For other MVD classes above the threshold MVD class, only integer pixel resolutions of the MVD are allowed. In this case, it is possible to avoid signaling fractional resolution signaling such as one or more of the "mv-fr" and / or "mv-hp" bits of the MVD signaled with MVD classes greater than or equal to the threshold MVD class. For MVD classes with resolutions less than one pixel, the number of bits of "mv-bit" signaling may be further reduced. For example, for MV_CLASS_5 in Table 4, the MVD pixel offset range is (32,64], so 5 bits are required to signal the entire range when the resolution is 1 pel. In contrast, if MV_CLASS_5 is associated with an MVD resolution of 2 pel (a lower resolution than 1 pixel), then 4 bits are required for "mv-bits" instead of 5 bits, and neither "mv-fr" nor "mv-hp" need to be signaled after signaling an "mv_class" such as MV-CLASS_5.

[0184] In some implementations, only fractional pixel resolutions of MVDs with integer values ​​less than a threshold integer pixel value may be allowed. In one example, only fractional pixel resolutions of MVDs less than 5 pixels may be allowed. To accommodate this example, fractional resolutions of MV_CLASS_0 and MV_CLASS_1 in Table 4 may be allowed, and fractional resolutions of all other MV classes may not be allowed. In another example, only fractional pixel resolutions of MVDs less than 7 pixels may be allowed. To accommodate this example, fractional resolutions of MV_CLASS_0 and MV_CLASS_1 in Table 4 (with ranges less than 5 pixels) may be allowed, and fractional resolutions of MV_CLASS_3 and above (with ranges greater than 5 pixels) may not be allowed. For MVDs belonging to MV_CLASS_2, whose pixel range includes 5 pixels, fractional pixel resolutions of the MVD may or may not be allowed depending on the "mv-bit" value. If the "m-bit" value is signaled as 1 or 2 (signaled such that the integer part of the signaled MVD, calculated as the start of the pixel range of MV_CLASS_2, is 5 or 6 and the offset indicated by "m-bit" is 1 or 2), then fractional pixel resolution may be allowed. Otherwise, if the "mv-bit" value is signaled as 3 or 4 (signaled such that the integer part of the signaled MVD is 7 or 8), then fractional pixel resolution may not be allowed.

[0185] In other implementations, only one MVD value may be allowed for MV classes equal to or greater than the threshold MV class. For example, such a threshold MV class may be MV_CLASS 2. Thus, for MV_CLASS_2 and above, only one MVD value and no fractional pixel resolution may be allowed. A single allowed MVD value for these MV classes may be predefined. In some examples, the allowed single value may be the higher end value of the range for each of these MV classes in Table 4. For example, MV_CLASS_2 through MV_CLASS_10 may be equal to or greater than the threshold class that is MV_CLASS 2, and the single allowed MVD values ​​for these classes may be predefined as 8, 16, 32, 64, 128, 256, 512, 1024, and 2048, respectively. In other examples, the allowed single value may be the middle value of the range for each of such MV classes in Table 4. For example, MV_CLASS_2 through MV_CLASS_10 may be higher than the class thresholds, and the single allowed MVD values ​​for these classes may be defined as 3, 6, 12, 24, 48, 96, 192, 384, 768, and 1536, respectively. Any other value within the range may be defined as the single allowed resolution for the respective MVD class.

[0186] In the above implementation, it is sufficient to signal "mv_class" to determine the MVD value if the signaled "mv_class" is greater than or equal to the predefined MVD class threshold. After that, the magnitude and direction of the MVD are determined using "mv_class" and "mv_sign".

[0187] Therefore, when the MVD of only one reference frame (a reference frame from either reference frame list 0 or list 1, but not from both) is signaled, or when the MVDs of two reference frames are signaled together, the accuracy (i.e., resolution) of the MVD can depend on the associated class of motion vector differential in Table 3 and / or the magnitude of the MVD.

[0188] In some other implementations, the pixel resolution or precision of the MVD may decrease or remain unchanged as the MVD size is increased. For example, the pixel resolution may depend on the integer part of the MVD size. In some implementations, fractional pixel resolution may be allowed only if the MVD size is less than or equal to a width threshold. At the decoder, the integer part of the MVD size may be extracted from the bitstream first. The pixel resolution may then be determined, and then a determination may be made as to whether any fractional MVD is present in the bitstream and needs to be parsed (e.g., if fractional pixel resolution of a particular extracted MVD integer size is not allowed, the fractional MVD bits may not be included in the bitstream that requires extraction). The above implementations regarding MVD class-dependent adaptive MVD pixel resolution apply to MVD size-dependent adaptive MVD pixel resolution. In certain examples, it may be allowed to only have a default value of 1 for MVD classes that exceed or include a size threshold.

[0189] For a current coding block, when in either a single-frame inter prediction mode or a synthetic inter prediction mode, and regardless of whether one or more motion vectors of the inter prediction are based on one or more MVDs, after the motion vector of the current coding block relative to the reference frame is determined, a predictive element block can be generated from the reference frame (can be generated from either the reference frame of the encoder or the decoder). In some implementations, a predictive element block in a corresponding predicted frame can be generated or derived by interpolating samples in the predicted frame based on the corresponding motion vector. In one example, in some scenarios, a motion vector corresponding to a reference frame of the current coding block may be a fractional motion vector. In particular, a motion vector indicated as one of a plurality of reference motion vectors in one or more motion vector reference lists may be a fractional motion vector in either or both horizontal and vertical dimensions. In another example, the MVD may be a fractional MVD in either or both horizontal and vertical dimensions. In such scenarios, a predictive element block indicated by a motion vector in a corresponding reference frame may not directly match a particular sample block in the reference frame, but may need to be interpolated between sample blocks.

[0190] Such interpolation can be implemented, for example, by using filters called interpolation filters. The interpolation filter applied to a particular current coding block can be selected from a set of predefined or preconfigured interpolation filters. Interpolation filtering can be implemented separately in each of the two dimensions (horizontal and vertical). Thus, an interpolation filter can be implemented as a one-dimensional filter that can be applied to either the horizontal or vertical dimension. In some implementations, some of these interpolation filters can include predefined finite impulse response (FIR) filters. For example, the predefined or preconfigured interpolation filters may include, but are not limited to, a 6-tap FIR filter (REGULAR), a 6-tap FIR filter (SMOOTH), an 8-tap FIR filter (SHARP), and a 2-tap BILINEAR filter.

[0191] In some implementations, when the dual filter mode is enabled, the interpolation filters for the horizontal and vertical dimensions can be independent in implementation. Whether or not the dual filter mode is enabled can be signaled at various levels (e.g., sequence header, picture header, frame header, slice header, coding tree level, macroblock level, block level). In the dual filter mode, the interpolation filters to be used in each of the two dimensions can also be signaled separately at the various levels. Otherwise, when the dual filter mode is not enabled, the selection of only one interpolation filter to be used / shared in both the horizontal and vertical dimensions of the coding block can be signaled. The interpolation filter to be used can also be signaled at the various levels. Two sets of interpolation filters available for selection in the dual filter mode and for selection in the single filter mode can be specified separately. These sets can be the same set of filters or different sets of filters.

[0192] Statistical studies have shown that when a motion vector difference (MVD) and a reference motion vector (MV) are used to derive an actual MV of a coding block depending on the magnitude and / or pixel resolution of the MVD, a certain type of interpolation filter can facilitate achieving better coding gain than other types of interpolation filters when used to generate a predictor component block. Therefore, in order to improve signaling efficiency while keeping the degradation of coding gain negligible, a small set of interpolation filters can be defined when defining a set of interpolation filters as candidate interpolation filters for a certain MVD magnitude or pixel resolution value or value range.

[0193] Similarly, statistical studies have also shown that depending on the actual MV size and / or pixel resolution of a block, certain types of interpolation filters can facilitate achieving better coding gain than other types of interpolation filters when used to generate a predictor component block. Thus, to improve signaling efficiency while keeping the degradation of coding gain negligible, a small set of interpolation filters can be predefined when predefining a set of interpolation filters as candidate interpolation filters for a particular MV size or pixel resolution value or value range. The actual MV may be a MV directly signaled in the video stream, or may be signaled as a reference motion vector in the DRL described above, or may be a combination of a reference motion vector in the DRL and a reference motion vector in the MVD.

[0194] In some implementations, the MV or MVD magnitudes can be used as a basis for selecting one or more interpolation filters. For example, one or more values ​​or value ranges of the MV or MVD magnitudes can be predefined. Each MV or MVD magnitude value or value range can be predefined to a predefined subset of the interpolation filters in the full set of candidate interpolation filters. Thus, the correspondence between the MV or MVD magnitude values ​​or value ranges and the subset of the interpolation filters can be predefined and known to the encoder and the decoder. The encoder can then determine the subset of the interpolation filters corresponding to a particular MV or MVD magnitude value or value range during the encoding process, and the subset can also be derived by the decoder based on the corresponding MV or MVD magnitude value or value range without any further signaling in the bitstream.

[0195] Similarly, the MV pixel resolution or the MVD pixel resolution can be used to select one or more interpolation filters. For example, one or more values ​​or value ranges of the MV or MVD pixel resolution can be predefined. Each value or value range of the MV or MVD pixel resolution can be associated with a predefined subset of interpolation filters in the entire set of candidate interpolation filters. Thus, the correspondence between the values ​​or value ranges of the MV or MVD pixel resolution and the subset of interpolation filters can be predefined and known to the encoder and the decoder. The subset of interpolation filters corresponding to a particular MV or MVD pixel resolution value or value range can then be determined by the encoder during the encoding process, and the subset can also be derived by the decoder based on the corresponding MV or MVD pixel resolution value or value range without any further signaling in the bitstream.

[0196] The number of interpolation filters in each of the above subsets of interpolation filters for different MV or MVD magnitudes or pixel resolutions can be changed. In some implementations, the above specific subset of interpolation filters can include two or more candidate interpolation filters. In this scenario, for a corresponding MV or MVD magnitude or pixel resolution value or value range, the encoder can further select a specific filter from the subset of interpolation filters at various levels (e.g., block level, slice level, frame level, picture level, sequence level, etc.) (e.g., by selecting a filter that achieves a superior coding gain), and the encoder can include corresponding signaling in the bitstream (referred to as an interpolation filter selection indicator) indicating a specific filter selected from the subset of interpolation filters defined by the MV or MVD magnitude or pixel resolution value or value range. Since the subset of interpolation filters may include a small number of filters compared to the entire set of candidate interpolation filters, the number of bits required to signal a specific filter selected from the subset is small, which improves signaling efficiency.

[0197] In other implementations, a particular subset of the above interpretation filters may include only one default interpolation filter. In this scenario, the MVD magnitude value or value range that corresponds to the default predefined interpolation filter is directly mapped. Therefore, there is no need to include further signaling in the bitstream about the particular predefined interpolation filter selected. The predefined default interpolation filter can be specified directly from the MVD miniguide value or value range.

[0198] In some implementations, a threshold level of the magnitude or pixel resolution of the MV or MVD can be predefined to define the various magnitude or pixel resolution value ranges. In some specific implementations, a threshold level can be predefined to define the magnitude or pixel resolution of the MV or MVD falling into two ranges, one above the threshold level and the other below the threshold level, each corresponding to a predefined subset of the interpolation filters.

[0199] In one example, a threshold level of MV or MVD magnitude may be predefined. A first predefined subset of interpolation filters may be used as candidate interpolation filters for blocks with MV or MVD magnitudes above the threshold level, while a second predefined subset of interpolation filters may be used as candidate interpolation filters for blocks with MV or MVD magnitudes not above the threshold level. A particular interpolation filter selected from the subset of filters may be further signaled in the bitstream. In a particular example, the first subset of interpolation filters corresponding to MV or MVD magnitudes above the threshold level may include one default interpolation filter, while the second subset of interpolation filters corresponding to MV or MVD magnitudes not above the threshold level may include the entire set (or a smaller set) of candidate interpolation filters. In such an implementation, no further signaling of the selected interpolation filter is required when the MV or MVD magnitude is above the threshold level.

[0200] In another example, a pixel resolution threshold level of the MV or MVD may be predefined. A first predefined subset of interpolation filters may be used as candidate interpolation filters for blocks with a pixel resolution of the MV or MVD below the threshold level, while a second predefined subset of interpolation filters may be used as candidate interpolation filters for blocks with a pixel resolution of the MV or MVD not below the threshold level. The particular interpolation filter selected using the subset of filters may be further signaled in the bitstream. In a particular example, the first subset of interpolation filters corresponding to a pixel resolution of the MV or MVD below the threshold level may include one default interpolation filter, while the second subset of interpolation filters corresponding to a pixel resolution of the MV or MVD not below the threshold level may include the entire set (or a smaller set) of candidate interpolation filters. In such an implementation, no further signaling of the selected interpolation filter is required when the pixel resolution of the MV or MVD is below the threshold level.

[0201] In some implementations, the MVD magnitude level can be represented by the MV class index described above in connection with Table 4. Thus, the various MV classes exemplarily shown in Table 4 can be defined with a set of MV threshold classes and further put into two class ranges. Each MV class range can correspond to a default pre-defined subset of the interpolation filters. In a particular example, one MV class threshold may be pre-defined to provide two MV class ranges. In particular, the threshold MV class may be set as MV_CLASS_0. Alternatively, the threshold MV class may be set as MV_CLASS_1 or other MV class. Taking the MV class threshold MV_CLASS_1 as an example, the class range of MV classes above MV_CLASS_1 (MV_CLASS_2 to MV_CLASS_10 inclusive) may be associated with a first pre-defined default subset of the interpolation filters, while the MV classes MV_CLASS_0 and MV_CLASS_1 may be associated with a second pre-defined default subset of the interpolation filters. For example, a first subset of interpolation filters corresponding to MV_CLASS_2 through MV_CLASS_10 may include one default interpolation filter, and a second subset of interpolation filters corresponding to MV_CLASS_0 and MV_CLASS_1 may include the entire set (or a smaller set) of candidate interpolation filters.

[0202] In some implementations, the MVD pixel resolution levels can be divided into various predefined ranges, each corresponding to a predefined subset of interpolation filters. In the adaptive MVD implementation described above, the MVD pixel resolution of a particular block may be one of a whole set of candidate pixel resolutions, including (but not limited to) 1 / 64, 1 / 32, 1 / 16, 1 / 8, 1 / 4, 1 / 2, 1, 2, 4, .... Although any type of division of pixel resolution ranges can be predefined (n some specific implementations), the available MVD pixel resolutions can be divided into two ranges, a first range being MVD pixel resolutions less than 1 pixel (lower pixel resolutions correspond to larger pixel values, and thus the first range includes resolutions of 1 pixel, 2 pixels, 4 pixels, ...) and a second range being MVD pixel resolutions greater than 1 pixel (including all the fractional pixel resolutions listed above). The first range can be associated with a first default subset of interpolation filters, while the second range can be associated with a second default subset of interpolation filters. The first subset may, for example, include one default interpolation filter, while the second subset may, for example, include the entire set (or a smaller set) of candidate interpolation filters.

[0203] In some specific implementations, when fractional MVD of a coding block is not allowed, none of the subset of interpolation filters may be the entire set of candidate interpolation filters. In other words, such implementations allow only a subset of the interpolation filters, not the entire set of candidate interpolation filters. In particular, only one subset of the interpolation filters may be used for selection when fractional MVD is not allowed. The one subset of the interpolation filters may be part of the entire set of allowed candidate interpolation filters. For example, the subset of the interpolation filters may include a REGULAR filter and a SMOOTH filter. In other implementations, the one subset of the interpolation filters may include one default interpolation filter. In other words, in an MVD implementation where fractional MVD is not allowed, one default interpolation filter may be predefined. In such implementations, when no fractional MVD is allowed in an adaptive MVD implementation, it may not be necessary to signal the selected interpolation filter in the bitstream. The one default interpolation filter may be, for example, one of a REGULAR filter, a SMOOTH filter, a SHARP filter, or a BILINEAR filter.

[0204] In other implementations, a correspondence or mapping between MVD widths and adaptive MVD pixel resolutions (represented by the MV class index in Table 4) can be predefined, as described above. In such a scenario, the predefined division of MV classes into different ranges for mapping to different subsets of interpolation filters corresponds to or is consistently related to the division of the MVD pixel resolution into different pixel resolution ranges.

[0205] The above implementation recognizes statistical considerations that show that for large MVD magnitudes, i.e., low MVD pixel resolutions, using the entire set of available candidate interpolation filters for selection, while requiring high signaling overhead, does not provide significant coding gain compared to using a smaller set of candidate interpolation filters, and does not provide significant coding gain even compared to using a single default interpolation filter.

[0206] In the various implementations above, the entire set of candidate interpolation filters may include, but are not limited to, one or more of a 6-tap FIR filter (REGULAR), a 6-tap FIR filter (SMOOTH), an 8-tap FIR filter (SHARP), and a 2-tap BILINEAR filter. The subset of interpolation filters may be any combination of filters in the entire set of interpolation filters. A specific example of the subset of interpolation filters may include a REGULAR filter and a SMOOTH filter. If the subset of filters includes one default interpolation filter, such default interpolation filter may be any filter in the entire set of candidate interpolation filters. In particular, one default filter may be defined as one of a REGULAR filter and a SMOOTH filter.

[0207] In some of the above implementations, the signaling of the selected interpolation filter (e.g., selected from a subset of the interpolation filters or the entire set of the interpolation filters) may be entropy coded according to a context that depends on the magnitude or pixel resolution of the MV or MVD. In a particular example, the entropy coding context may be determined based on whether an adaptive MVD resolution is applied and / or the MVD pixel resolution or MV class of the MVD. For example, if an adaptive MVD resolution is applied, the MVD pixel resolution or MV class may be divided into multiple ranges associated with multiple subsets of the interpolation filters. The signaling with each subset of the selected interpolation filter may be coded with a particular set of contexts. Thus, each MVD pixel resolution range or MV class range or subset of the interpolation filters will correspond to a set of coding contexts. Such sets of coding contexts may be determined individually to maximize coding efficiency.

[0208] In some implementations, such MVD pixel resolution ranges or MV class ranges can be defined by multiple thresholds. In one implementation, two ranges defined by one threshold can be defined. This threshold (either the MVD pixel resolution threshold or the MV class index threshold) is used to determine the two ranges and the corresponding predefined subsets of interpolation filters. Signaling in a first subset of interpolation filters can be coded according to a first set of contexts, while signaling with a second subset of interpolation filters can be coded according to a second set of contexts. Such implementations can achieve even larger coding gains, since the number of filters in each subset may be different, and the statistics of the selected filters in each subset may also be different for many coding blocks.

[0209] FIG. 18 shows a flowchart 1800 of an example method according to the principles underlying the above implementations of selecting and signaling an interpolation filter. The flow of the example method starts at 1801. At S1810, a video stream is received. At S1820, it is determined that a video block is inter-predicted by a reference frame according to a motion vector (MV). At S1830, it is determined from the video stream that an MVD (MV difference) of the video block and a reference MV is derived. At S1840, an interpolation filter is selected according to a resolution or size of the MVD. At S1850, an inter-predicted block of the video block is generated from the reference frame according to the MV and the interpolation filter. The example method stops at S1899.

[0210] In the embodiments and implementations of the present disclosure, any steps and / or operations may be combined or arranged in any total number or order as desired. Two or more of the steps and / or operations may be performed in parallel. The embodiments and implementations of the present disclosure may be used separately or combined in any order. Furthermore, each of the method (or embodiment), the encoder, and the decoder may be implemented by a processing circuit (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored on a non-transitory computer-readable medium. The embodiments of the present disclosure may be applied to a luma block or a chroma block. The term "block" may be interpreted as a prediction block, a coding block, or a coding unit (i.e., CU). The term "block" in this disclosure may also be used to refer to a transform block. In the following, when describing a block size, it may refer to the block width or height, or the maximum value of the width and height, or the minimum value of the width and height, or the area dimension (width x height), or the aspect ratio of the block (width:height or height:width).

[0211] The techniques described above can be implemented using computer readable instructions and as computer software physically stored on one or more computer readable media. For example, Figure 19 illustrates a computer system (1900) suitable for implementing certain embodiments of the disclosed subject matter.

[0212] Computer software may be encoded using any suitable machine or computer language, which may be processed by assembly, compilation, linking, or similar mechanisms to generate code comprising instructions which may be executed directly, interpreted, executed by microcode, or the like, by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc.

[0213] The instructions may be executed in various types of computers or components thereof including, for example, a personal computer, a tablet computer, a server, a smart phone, a gaming console, an Internet of Things device, and the like.

[0214] 19 for computer system (1900) are, of course, exemplary and are not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing the embodiments of the present disclosure, nor should the configuration of components be construed as imposing any dependency or requirement on any one or combination of components illustrated in the exemplary embodiment of computer system (1900).

[0215] The computer system (1900) may include certain human interface input devices that may be responsive to input by one or more human users through, for example, tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, clapping), visual input (e.g., gestures), and olfactory input (not shown). The human interface devices may also be used to capture media that is not necessarily directly associated with conscious human input, such as audio (e.g., speech, music, ambient sounds), images (e.g., scanned images, photographic images captured from a still image camera), and video (e.g., two-dimensional video, three-dimensional video, including stereoscopic video).

[0216] The input human interface devices may include one or more (only one of each shown) of a keyboard (1901), a mouse (1902), a trackpad (1903), a touch screen (1910), a data glove (not shown), a joystick (1905), a microphone (1906), a scanner (1907), and a camera (1908).

[0217] The computer system (1900) may also include some human interface output devices. Such human interface output devices may stimulate one or more of the human user's senses, for example, through haptic output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., touch screen (1910), haptic feedback via data gloves (not shown) or joystick (1905), although haptic feedback devices that are not input devices may also be present), audio output devices (speakers (1909), headphones (not shown), etc.), visual output devices (screens (1910) including CRT screens, LCD screens, plasma screens, OLED screens, etc., with or without touch screen input capabilities, with or without haptic feedback capabilities, some of which may be capable of outputting two-dimensional visual output or three or more dimensional output by means of stereoscopic video output devices, virtual reality glasses (not shown), holographic displays, smoke tanks (not shown), etc.), and printers (not shown).

[0218] The computer system (1900) may also include human-handleable storage devices and associated media, such as optical media, including CD / DVD ROM / RW (1920) with CD / DVD or similar media (1921), thumb drives (1922), removable hard drives or solid state drives (1923), legacy magnetic media, such as tapes or floppy disks (not shown), and dedicated ROM / ASIC / PLD based devices, such as security dongles (not shown).

[0219] Those skilled in the art will also appreciate that the term "computer-readable medium" as used in connection with the subject matter disclosed herein does not include transmission media, carrier waves, or other transitory signals.

[0220] The computer system (1900) may also include an interface (1954) to one or more communications networks (1955). The networks may be, for example, wireless, wired, optical, etc. Furthermore, the networks may be local, wide area, metropolitan, vehicular, industrial, real-time, delay tolerant, etc. Examples of networks include local area networks such as Ethernet, cellular networks including WLAN, GSM, 3G, 4G, 5G, LTE, etc., television wired or wireless wide area digital networks including cable, satellite, and terrestrial television, vehicular and industrial networks including CAN bus, etc. Some networks typically require an external network interface adapter that is attached to some general-purpose data port or peripheral bus (1949) (e.g., a USB port on the computer system (1900)), while others are typically built into the core of the computer system (1900) by being attached to a system bus as described below (e.g., an Ethernet interface built into a PC computer system, or a cellular network interface built into a smartphone computer system). Using any of these networks, the computer system (1900) can communicate with others. Such communication can be one-way, receive-only (e.g., television broadcast), one-way transmit-only (e.g., a CANbus to a specific CANbus device), or two-way, e.g., two-way communication to another computer system using a local or wide area digital network. Several protocols and protocol stacks can be used for each of these networks and network interfaces described above.

[0221] The above-mentioned human interface devices, human-accessible storage devices and network interfaces may be attached to the core (1940) of the computer system (1900).

[0222] The core (1940) may include one or more central processing units (CPUs) (1941), graphics processing units (GPUs) (1942), dedicated programmable processing units in the form of field programmable gate areas (FPGAs) (1943), hardware accelerators (1944) for specific tasks, graphics adapters (1950), etc. These devices may be connected through a system bus (1948) along with read only memory (ROM) (1945), random access memory (1946), and internal mass storage (1947) such as a hard disk or SSD that is not accessible to the user. In some computer systems, the system bus (1948) may be manually operable in the form of one or more physical plugs that allow expansion by adding additional CPUs, GPUs, etc. Peripheral devices may be attached directly to the core's system bus (1948) or through a peripheral bus (1949). In one example, a screen (1910) may be connected to the graphics adapter (1950). Peripheral bus architectures include PCI, USB, etc.

[0223] The CPU (1941), GPU (1942), FPGA (1943) and accelerator (1944) can execute a number of instructions that can combine to form the above computer code. The computer code can be stored in a ROM (1945) or in a RAM (1946). Data that does not change can be stored, for example, in an internal mass storage (1947), while data that changes can also be stored in the RAM (1946). A cache memory, which can be closely associated with one or more of the CPU (1941), GPU (1942), mass storage (1947), ROM (1945), RAM (1946), etc., can be used to enable fast storage and retrieval from any of the memory devices.

[0224] The computer-readable medium may bear computer code for performing various computer-implemented operations, and the medium and computer code may be those specially designed and constructed for the purposes of the present disclosure, or they may be of the kind well known and available to those having skill in the computer software arts.

[0225] As a non-limiting example, a computer system having the architecture (1900), and in particular the core (1940), may perform functions as a result of a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media may be media related to mass storage that can be handled by a user as introduced above, but may also be specific storage of the core (1940) that has a non-transitory nature, such as the core's internal mass storage (1947) or ROM (1945). Software implementing various embodiments of the present disclosure may be stored in such devices and executed by the core (1940). The computer-readable media may include one or more memory devices or chips depending on the particular requirements. The software may cause the core (1940), and in particular the processor (including a CPU, GPU, FPGA, etc.) in the core (1940), to perform a particular process or a particular part of a particular process described in the present application (including determining data structures to be stored in RAM (1946) and modifying such data structures according to the process defined by the software). Additionally or alternatively, the computer system may perform functions as a result of hard-wired logic or may be otherwise implemented in circuitry (e.g., accelerators (1944)) that may act in place of or cooperate with software to perform particular processes or portions of particular processes described in this application. Where appropriate, references to "software" may encompass logic, and vice versa. Where appropriate, references to "computer-readable medium" may encompass circuitry (such as integrated circuits (ICs)) that stores the software to be executed, circuitry that implements the logic to be executed, or both. This disclosure includes any suitable combination of hardware and software.

[0226] While this disclosure has described several exemplary embodiments, there are alterations, permutations, and various substitute equivalents, which are within the scope of this disclosure. Thus, it will be appreciated that those skilled in the art will be able to devise numerous systems and methods that embody the principles of this disclosure and are therefore within its spirit and scope, even if not explicitly shown or described in this application. Appendix A: Acronyms JEM: joint exploration model VVC: versatile video coding BMS:benchmark set MV: Motion Vector HEVC:High Efficiency Video Coding SEI:Supplementary Enhancement Information VUI:Video Usability Information GOP: Groups of Picture TU: Transform Unit PU: Prediction Unit CTU: Coding Tree Unit CTB: Coding Tree Block PB: Prediction Block HRD:Hypothetical Reference Decoder SNR: Signal Noise Ratio CPU: Central Processing Unit GPU: Graphics Processing Unit CRT: Cathode Ray Tube LCD: Liquid Crystal Display OLED:Organic Light-Emitting Diode CD:Compact Disc DVD: Digital Video Disc ROM:Read-Only Memory RAM:Random Access Memory ASIC:Application-Specific Integrated Circuit PLD:Programmable Logic Device LAN:Local Area Network GSM:Global System for Mobile communication LTE:Long-Term Evolution CANBus:Controller Area Network Bus USB:Universal Serial Bus PCI:Peripheral Component Interconnect FPGA:Field Programmable Gate Area SSD:solid-state drive IC:Integrated Circuit HDR:high dynamic range SDR:standard dynamic range JVET:Joint Video Exploration Team MPM:most probable mode WAIP:Wide-Angle Intra Prediction CU:Coding Unit PU:Prediction Unit TU:Transform Unit CTU:Coding Tree Unit PDPC:Position Dependent Prediction Combination ISP:Intra Sub-Partition SPS:Sequence Parameter Setting PPS:Picture Parameter Set APS:Adaptation Parameter Set VPS:Video Parameter Set DPS:Decoding Parameter Set ALF:Adaptive Loop Filter SAO:Sample Adaptive Offset CC-ALF:Cross-Component Adaptive Loop Filter CDEF:Constrained Directional Enhancement Filter CCSO:Cross-Component Sample Offset LSO:Local Sample Offset LR:Loop Restoration Filter AV1:AOMedia Video 1 AV2:AOMedia Video 2 MVD:Motion Vector difference CfL:Chroma from Luma SDT:Semi Decoupled Tree SDP:Semi Decoupled Partitioning SST:Semi Separate Tree SB:Super Block IBC(すなわちIntraBC):Intra Block Copy CDF:Cumulative Density Function SCC:Screen Content Coding GBI:Generalized Bi-prediction BCW:Bi-prediction with CU-level Weight CIIP: Combined intra-inter prediction POC: Picture Order Count RPS: Reference Picture Set DPB: Decoded Picture Buffer MMVD:Merge Mode with Motion Vector Difference [Explanation of symbols]

[0227] R01~R70 Reference Sample S11~S44 Sample 101 points 102 Arrow 103 Arrow 104 Square Block 201 Current Block 202~206 Sample 300 Communication Systems 310 Terminal Equipment 320 Terminal Equipment 330 Terminal Equipment 340 Terminal Equipment 350 Communication Network 400 Communication Systems 401 Video Source 402 Stream 403 Video Encoder 404 Video Bitstream 405 Streaming Server 406 Client Subsystem 407 Duplication 408 Client Subsystem 409 Duplication 410 Video Decoder 411 Output Stream 412 Display 413 Video Imaging Subsystem 420 Electronic equipment 430 Electronic equipment 501 Channel 510 Video Decoder 512 Display 515 Buffer Memory 520 Parser 521 Symbols 530 Electronic equipment 531 Receiver 551 Scaler / Inverse Converter 552 Intra-picture prediction unit (intra prediction unit) 553 Compensation Prediction Department 555 Collector 556 Loop filter section 557 Reference Picture Memory 558 Current Picture Buffer 601 Video Source 603 Video Encoder 620 Electronic equipment 630 Source Coder 632 encoding engine 633 Video Decoder 634 Reference Picture Memory 635 Predictor 640 Transmitter 643 Video Sequence 645 Entropy Coder 650 Controller 660 Communication Channel 703 Video Encoder 721 General Controller 722 Intra Encoder 723 Residual value calculator 724 Residual Value Encoder 725 Entropy Encoder 726 Switch 728 Residual Decoder 730 Interencoder 810 Video Decoder 871 Entropy Decoder 872 Intra Decoder 873 Residual Decoder 874 Reconstruction Module 880 Interdecoder 902~908 Division pattern 1002~1008 T-type pattern 1010 Pattern 1102 Vertical Bisection 1104 horizontal bisection 1106 Vertical thirds 1108 Horizontal third division 1200 Base Block 1202~1208 Split part 1302 and 1304 Tripartite Pattern 1410 Division Pattern 1420 Tree Structure 1402~1408 Split part 1502 Square Encoding Block 1504 and 1506 division methods 1602 Inter-coding Block 1604 Division method 1800 Flowchart 1900 Computer Systems 1901 Keyboard 1902 Mouse 1903 Trackpad 1905 Joystick 1906 Mike 1907 Scanner 1908 Camera 1909 Audio Output Device 1910 Touch Screen 1920 CD / DVD ROM / RW 1921 Medium 1922 Thumb Drive 1923 Solid State Drive 1940 Core 1941 CPU 1942 GPU 1943 FPGA 1944 Hardware Accelerator 1945 ROM 1946 RAM 1947 Mass Storage 1948 System Bus 1949 Peripheral Bus 1950 Graphics Adapter 1954 Interface 1955 Communication Network

Claims

1. A method executed by one or more processors for processing video blocks of a video stream, comprising: receiving a flag from the video stream; determining, based on the received flag, that the video block is to be inter-predicted by a reference frame according to a motion vector (MV); deriving the MV from an MVD (MV difference) of the video block and a reference MV; obtaining a signal indicative of an interpolation filter, said signal being based on at least one of a resolution and a magnitude of said MVD; selecting the interpolation filter in response to the signal; performing inter-prediction of the video block from the reference frame according to the MV and the selected interpolation filter; A method comprising:

2. The step of obtaining a signal comprises: determining the magnitude of the MVD based on a MV class index associated with the video block, the MV class index being extracted from the video stream; obtaining the signal for setting a default interpolation filter if the MV class index is not less than a threshold MV class index; The method of claim 1 , comprising:

3. 3. The method of claim 2, wherein the default interpolation filter is an interpolation filter from a plurality of interpolation filters that form a pool of candidate interpolation filters to be used for video blocks having MV class indices less than the threshold MV class index.

4. The method of claim 2 , wherein the predefined interpolation filter comprises a regular, smooth, sharp, or bilinear interpolation filter.

5. The step of obtaining a signal comprises: determining that an adaptive pixel resolution is applied to the MVD; determining a pixel resolution of the MVD from a plurality of adaptive MVD pixel resolutions; obtaining the signal for setting a default interpolation filter in response to the pixel resolution of the MVD being at one or more predetermined levels; The method of claim 1 , comprising:

6. The method of claim 5 , wherein the one or more predefined levels comprise an MVD resolution level that does not exceed a predefined MVD pixel resolution threshold.

7. The method of claim 6 , wherein the pixel resolution of the MVD is determined based on an MV class index of the video block extracted from the video stream.

8. The method of claim 7 , wherein the predetermined MVD pixel resolution threshold corresponds to an MV class index threshold.

9. 6. The method of claim 5, wherein the default interpolation filter is one of a plurality of interpolation filters that form a pool of candidate interpolation filters to be used for video blocks having an adaptive MVD pixel resolution other than the one or more default levels.

10. The method of claim 5 , wherein the predefined interpolation filter comprises one of a regular, smooth, sharp, or bilinear interpolation filter.

11. The step of obtaining a signal comprises: determining an MV class index corresponding to the magnitude of the MVD from the video stream; setting a default subset of pre-defined interpolation filters from a set of interpolation filters if the MV class index of the MV is not less than a pre-defined MV class index threshold; extracting a selection indicator from the video stream as the signal for selecting the interpolation filter from the subset of predefined interpolation filters; The method of claim 1 , comprising:

12. The method of claim 11 , wherein the subset of predefined interpolation filters includes a regular interpolation filter and a smooth interpolation filter.

13. The step of obtaining a signal comprises: determining that an adaptive pixel resolution is applied to the MVD; determining a pixel resolution of the MVD from a plurality of adaptive MVD pixel resolutions; setting a default subset of pre-defined interpolation filters from a set of interpolation filters in response to the pixel resolution of the MVD not exceeding a pre-defined MVD pixel resolution threshold; extracting a selection indicator from the video stream as the signal for selecting the interpolation filter from the subset of predefined interpolation filters; The method of claim 1 , comprising:

14. The method of claim 13 , wherein the subset of predefined interpolation filters includes a regular interpolation filter and a smooth interpolation filter.

15. The step of obtaining a signal comprises: determining that an adaptive pixel resolution is applied to the MVD of the video block; determining a pixel resolution of the MVD from a plurality of adaptive MVD pixel resolutions or determining an MV class index of the MV from a plurality of MV class indexes; identifying a set of contexts for encoding an interpolation filter selection indicator based on the pixel resolution of the MVD or the MV class index; decoding the video stream according to the set of contexts to obtain the interpolation filter selection indicator as the signal; The method of claim 1 , comprising:

16. The step of obtaining a signal comprises: determining that an adaptive pixel resolution is applied to the MVD; determining an MV class index for the video block from a plurality of MV class indexes; decoding the video stream using a first set of contexts to obtain an interpolation filter selection indicator as the signal if the MV class index is not less than a threshold MV class index; decoding the video stream using a second set of contexts different from the first set of contexts if the MV class index is less than the threshold MV class index to obtain the interpolation filter selection indicator as the signal; The method of claim 1 , comprising:

17. 2. The method of claim 1, wherein obtaining the signal comprises obtaining the signal for setting the interpolation filter to a default in response to determining that the MVD pixel resolution of the MVD is a non-fractional pixel resolution.

18. The step of obtaining a signal comprises: setting a default subset of pre-defined interpolation filters from a set of interpolation filters in response to determining that the MVD pixel resolution of the MVD is a non-fractional pixel resolution; extracting a selection indicator from the video stream as the signal for selecting the interpolation filter from the subset of predefined interpolation filters; The method of claim 1 , comprising:

19. The method of claim 18 , wherein the subset of predefined interpolation filters includes a regular interpolation filter and a smooth interpolation filter.

20. A device configured to perform a method according to any one of claims 1 to 19.

21. A computer program product for causing one or more processors to carry out a method according to any one of claims 1 to 19.

Images