Temporal motion vector predictor candidate search

Abstract

The present disclosure relates generally to video coding, and more particularly to methods and systems for determining temporal motion vector predictor (TMVP) candidates for inter prediction in video coding. For example, the disclosed methods include limiting the number of TMVP candidates in a motion vector predictor (MVP) list and providing various search mechanisms to promote MVP candidate diversity among TMVP and other types of MVP candidates and improve coding efficiency.

Description

[Technical field]

[0001] Incorporation by Reference

[0001] This application claims the benefit of priority to U.S. Non-Provisional Application No. 18 / 051,397, filed on October 31, 2022, and entitled "Temporal Motion Vector Predictor Candidates Search," and U.S. Provisional Patent Application No. 63 / 402,386, filed on August 30, 2022, and entitled "Temporal Motion Vector Predictor Candidates Improvement," each of which is incorporated by reference in its entirety herein.

[0002] FIELD OF THE DISCLOSURE

[0002] This disclosure relates generally to video coding, and more particularly to methods and systems for determining temporal motion vector predictor candidates for inter prediction in video coding. [Background technology]

[0003]

[0003] This background description provided herein is intended to generally present the context of the present disclosure. The work of the currently identified inventors is not expressly or implicitly admitted to be prior art to the present disclosure, to the extent that it is described in this background section, as well as aspects of the description that may not be considered prior art at the time of filing of this application.

[0004]

[0004] Video encoding and decoding can be performed using inter-picture prediction with motion compensation. Uncompressed digital video can include a sequence of pictures, each having spatial dimensions of, for example, 1920x1080 luma samples and associated full or subsampled chroma samples. The sequence of pictures can have a fixed or variable picture rate (alternatively called frame rate), for example, 60 pictures per second, i.e., 60 frames per second. Uncompressed video has inherent bit rate requirements for streaming or data processing. For example, a video with pixel resolution of 1920x1080, frame rate of 60 frames / second, and chroma subsampling 4:2:0 with 8 bits per pixel per color channel requires a bandwidth of nearly 1.5 Gbit / s. One hour of such video requires more than 600 GByte of storage space.

[0005]

[0005] One objective of video encoding and decoding may be the reduction of redundancy in an uncompressed input video signal through compression. Compression may, in some cases, help reduce the aforementioned bandwidth and / or storage space requirements by more than two orders of magnitude. Both lossless and lossy compression, as well as combinations thereof, may be employed. Lossless compression refers to techniques where an exact copy of the original signal can be reconstructed from the compressed original signal through a decoding process. Lossy compression refers to an encoding / decoding process where the original video information is not fully preserved during encoding and therefore cannot be fully restored during decoding. When using lossy compression, the reconstructed signal may not be identical to the original signal, but even with some information loss, the distortion between the original and reconstructed signals is small enough to make the reconstructed signal useful for the intended application. In the case of video, lossy compression is widely adopted in many applications. The amount of tolerable distortion depends on the application. For example, users of a particular consumer video streaming application can tolerate more distortion than users of a movie or television broadcasting application. The compression ratio achievable by a particular encoding algorithm can be selected or adjusted to reflect different distortion tolerances; generally, the more distortion that can be tolerated, the more encoding algorithms that result in greater loss and greater compression ratios are possible.

[0006]

[0006] Video encoders and decoders can utilize techniques from a number of broad categories and processes, including, for example, motion compensation, Fourier transform, quantization, and entropy coding.

[0007]

[0007] Video codec technology can include a technique known as intra-coding. In intra-coding, sample values ​​are represented without reference to samples or other data from a previously reconstructed reference picture. In some video codecs, a picture is spatially subdivided into blocks of samples. When all blocks of samples are coded in intra mode, the picture can be called an intra-picture. Intra-pictures and their derivatives, such as independent decoder refresh pictures, can be used to reset the state of the decoder and can therefore be used as the first picture in a coded video bitstream and video session or as a still image. The samples of the block after intra-prediction can then be transformed into the frequency domain, and the transform coefficients so generated can be quantized before entropy coding. Intra-prediction represents a technique that minimizes sample values ​​in the domain before transformation. In some cases, the smaller the DC value after transformation and the smaller the AC coefficients, the fewer bits are required at a given quantization step size to represent the block after entropy coding.

[0008]

[0008] Traditional intra-coding, such as that known from MPEG-2 generation coding techniques, does not use intra-prediction. However, some newer video compression techniques include techniques that attempt to code / decode a block based on surrounding sample data and / or metadata obtained during the encoding and / or decoding of spatially adjacent blocks and that precede the intra-coded or decoded data block in decoding order. Such techniques are hereafter referred to as "intra-prediction" techniques. It should be noted that in at least some cases, intra-prediction uses only reference data from the current picture being reconstructed, and not from other reference pictures.

[0009]

[0009] There may be many different forms of intra prediction. When more than one of such techniques are available within a given video coding technique, the technique in use may be referred to as an intra prediction mode. In a particular codec, one or more intra prediction modes may be provided. In certain cases, a mode may have sub-modes and / or may be associated with various parameters, and the mode / sub-mode information and the intra coding parameters of a video block may be coded separately or collectively included in a mode codeword. Which codeword should be used for a given mode, sub-mode, and / or parameter combination may also affect the coding efficiency improvement through intra prediction, and therefore entropy coding techniques may be used to turn the codeword into a bitstream.

[0010]

[0010] Certain intra prediction modes were introduced in H.264, improved in H.265, and further improved in newer coding techniques such as the joint exploration model (JEM), versatile video coding (VVC), and benchmark set (BMS). In general, for intra prediction, it is possible to form a predictor block using nearby sample values ​​that become available. For example, available values ​​of a particular set of nearby samples along a particular direction and / or line may be copied to the predictor block. The reference of the direction in use may be coded in the bitstream or may itself be predicted.

[0011]

[0011] Referring to FIG. 1A, a subset of 9 predictor directions specified among 33 possible intra predictor directions of H.265 (corresponding to 33 angle modes among 35 intra modes specified in H.265) is depicted at the bottom right. The point (101) where the arrows converge represents the sample to be predicted. The arrows represent the direction in which adjacent samples are used to predict the sample at 101. For example, arrow (102) indicates that sample (101) is predicted from one or more adjacent samples located to the upper right at an angle of 45 degrees from the horizontal. Similarly, arrow (103) indicates that sample (101) is predicted from one or more adjacent samples located to the lower left of sample (101) at an angle of 22.5 degrees from the horizontal.

[0012]

[0012] With further reference to FIG. 1A, at the top left is depicted a square block (104) of 4x4 samples (indicated by a thick dotted line). The square block (104) contains 16 samples, each labeled with an "S", the block's position in the Y dimension (e.g., row index), and the block's position in the X dimension (e.g., column index). For example, sample S21 is the second sample (from the top) in the Y dimension and the first sample (from the left) in the X dimension. Similarly, sample S44 is the fourth sample of the block (104) in both the Y and X dimensions. Since the block is 4x4 samples in size, S44 is at the bottom right. Also shown are examples of reference samples following a similar numbering scheme. The reference samples are labeled with R, the sample's Y position (e.g., row index), and X position (column index) relative to the block (104). In both H.264 and H.265, prediction samples that are close and neighboring to the block being reconstructed are used.

[0013]

[0013] The intra-picture prediction of a block 104 may start by copying reference sample values ​​from adjacent samples according to the signaled prediction direction. For example, assume that the coded video bitstream includes signaling indicating the prediction direction of the arrow (102) for this block 104 - that is, the sample is predicted from one or more prediction samples located at an angle of 45 degrees from the horizontal and to the upper right. In such a case, samples S41, S32, S23, and S14 are predicted from the same reference sample R05. Thus, sample S44 is predicted from reference sample R08.

[0014] In certain cases, particularly when the orientation is not evenly divisible by 45 degrees, the values ​​of multiple reference samples may be combined, for example through interpolation, to calculate the reference sample.

[0015]

[0015] The number of possible directions has increased as video coding technology continues to develop. In H.264 (2003), for example, nine different directions are available for intra prediction. This has increased to 33 in H.265 (2013), and JEM / VVC / BMS can support up to 65 directions at the time of this disclosure. Empirical studies have been conducted to help identify optimal intra prediction directions, and certain techniques in entropy coding may be used to encode these optimal directions with a small number of bits at the expense of a certain bit penalty for the direction. Furthermore, the direction itself is sometimes predictable from similar directions used in intra prediction of nearby decoded blocks.

[0016]

[0016] FIG. 1B shows a schematic diagram (180) depicting 65 intra prediction directions according to JEM to illustrate the increasing number of prediction directions in various encoding techniques developed over time.

[0017]

[0017] The manner in which bits representing intra-prediction directions are mapped to prediction directions in the encoded video bitstream may vary from one video coding technique to another, and may range, for example, from a simple direct mapping of prediction directions to codewords, to intra-prediction modes, to complex adaptive schemes involving maximum likelihood modes, and similar techniques. However, in all cases, there may be certain directions for intra-prediction that are statistically less likely to occur in the video content than certain other directions. Since the goal of video compression is to reduce redundancy, in a well-designed video coding technique, these less likely directions may be represented with a larger number of bits than the more likely directions.

[0018]

[0018] Inter-picture prediction or inter-prediction may be based on motion compensation. In motion compensation, sample data (reference picture) from a previously reconstructed picture or part thereof may be used for prediction of a newly reconstructed picture or part of a picture (e.g., a block) after being spatially shifted in a direction indicated by a motion vector (hereinafter, MV: motion vector). In some cases, the reference picture may be the same as the picture currently being reconstructed. The MV may have two dimensions X and Y, or three dimensions, with the third dimension indicating the reference picture in use (similar to the temporal dimension).

[0019]

[0019] In some video compression techniques, the current MV applicable to a particular area of ​​sample data is predictable from other MVs, e.g., other MVs related to other areas of sample data that are spatially adjacent to the area being reconstructed and that precede the current MV in decoding order. In this way, the total amount of data required to encode the MV can be substantially reduced by relying on the removal of redundancy in correlated MVs, thereby improving compression efficiency. For example, when encoding an input video signal derived from a camera (known as raw video), MV prediction can work effectively because there is a statistical possibility that areas larger than the area to which a single MV is applicable move in a similar direction in the video sequence, and therefore, in some cases, can be predicted using similar motion vectors derived from the MVs of nearby areas. This makes the actual MV of a given area similar or identical to the MV predicted from the surrounding MVs. Such an MV may then be represented, after entropy encoding, with fewer bits than would be used if the MV were encoded directly instead of predicted from nearby MVs. In some cases, the MV prediction may be an example of a lossless compression of a signal (i.e., the MV) derived from the original signal (i.e., the sample stream). In other cases, the MV prediction may itself be lossy, for example due to rounding errors when computing the predictor from some surrounding MVs.

[0020]

[0020] Various MV prediction mechanisms are described in H.265 / HEVC (ITU-T Recommendation H.265, "High Efficiency Video Coding", December 2016). Among the many MV prediction mechanisms specified by H.265, a technique hereafter called "spatial merging" is described below.

[0021]

[0021] Specifically, referring to Fig. 2, a current block (201) contains samples that the encoder finds during the motion search process to be predictable from a previous block of the same size but spatially shifted. Instead of directly encoding this MV, the MV can be derived from metadata associated with one or more reference pictures, such as the most recent reference picture (in decoding order), using the MV associated with any one of the five surrounding samples denoted as A0, A1, and B0, B1, B2 (202 to 206, respectively). In H.265, MV prediction can use predictors from the same reference picture used by adjacent blocks. Summary of the Invention

[0022]

[0022] The present disclosure relates generally to video coding, and more particularly to a method and system for determining temporal motion vector predictor (TMVP) candidates for inter-prediction in video coding. For example, the disclosed method includes limiting the number of TMVP candidates in a motion vector predictor (MVP) list and provides various search mechanisms to promote MVP candidate diversity between TMVP and other types of MVP candidates and improve coding efficiency.

[0023]

[0023] In an example implementation, a method for processing a current predicted block of a current frame in a video stream is disclosed. The method may include determining that a current prediction block should be inter-predicted by a reference block in a reference frame and that a motion vector of the current prediction block should be predicted by a reference motion vector; identifying a set of candidate prediction blocks in the current frame as a search pool for temporal motion vector predictor (TMVP) candidates for the current prediction block; searching the set of candidate prediction blocks according to a search order to identify up to N TMVP candidates for the current prediction block, where the searching ends in response to N TMVP candidates being identified, where N is a positive integer; constructing a motion vector predictor (MVP) list indicating a set of MVP candidates, where the set of MVP candidates includes one or more of a set of spatial MVP (SMVP) candidates and the up to N TMVP candidates; extracting an MVP index of the current prediction block from the video stream; and identifying a reference motion vector for inter-predicting the current prediction block based on the extracted MVP index and the MVP list.

[0024]

[0024] In the above implementation, the current prediction block belongs to a current superblock, the current superblock includes multiple prediction blocks, and the set of candidate prediction blocks includes at least a subset of the multiple prediction blocks of the current superblock.

[0025]

[0025] In any one of the above implementations, the multiple predictive blocks of the current superblock form a predictive block array having a column dimension and a row dimension, the reconstructed adjacent predictive blocks of the current superblock are above the current superblock in the row dimension and / or to the left of the current superblock in the column dimension, and the search order for searching the set of candidate predictive blocks for up to N TMVP candidates includes starting away from and moving closer to the reconstructed adjacent predictive blocks of the current superblock along at least one of the column and row dimensions of the predictive block array.

[0026]

[0026] In any one of the above implementations, the search order for searching a set of candidate prediction blocks for up to N TMVP candidates may include starting from the bottom row of the prediction block array and proceeding to the top row, and proceeding row-by-row from the rightmost prediction block to the leftmost prediction block, or starting from the rightmost column of the prediction block array and proceeding to the leftmost column, and proceeding column-by-column from the bottommost prediction block to the topmost prediction block, or searching the prediction block array along a diagonal direction based on the column and row dimensions, and proceeding from the rightmost and bottommost prediction block of the prediction block array toward the leftmost and topmost prediction block.

[0027]

[0027] In any one of the above implementations, the set of candidate prediction blocks includes a subset of the multiple prediction blocks having M prediction blocks skipped from the multiple prediction blocks, where M is a non-negative integer.

[0028]

[0028] In any one of the above implementations, the skipped M prediction blocks include at least the leftmost column of the prediction block array.

[0029]

[0029] In any one of the above implementations, the skipped M prediction blocks include at least the top row of the prediction block array.

[0030]

[0030] In any one of the above implementations, the skipped M prediction blocks include at least the top row and the leftmost column of the prediction block array.

[0031]

[0031] In any one of the above implementations, the set of candidate prediction blocks excludes any additional prediction blocks outside the current superblock.

[0032]

[0032] In any one of the above implementations, the set of candidate prediction blocks further includes at least one additional prediction block outside the current superblock.

[0033]

[0033] In any one of the above implementation forms, the method may further include searching a plurality of predictive blocks of the current superblock to identify up to N1 TMVP candidates, where N1 is a positive integer less than or equal to N, and ceasing searching the plurality of predictive blocks of the current superblock in response to N1 TMVP candidates being identified from the plurality of predictive blocks.

[0034]

[0034] In any one of the above implementation forms, up to one TMVP candidate is identified from at least one additional predictive block outside the current superblock, and the method further includes ceasing to search the at least one additional predictive block outside the current superblock in response to one TMVP candidate being identified from the at least one additional predictive block outside the current superblock.

[0035]

[0035] In any one of the above implementation forms, the set of candidate prediction blocks includes up to L prediction blocks from a plurality of prediction blocks, where L is a positive integer less than the total number of prediction blocks in the current superblock.

[0036]

[0036] In the above implementation, L=1, and the set of candidate prediction blocks includes only the rightmost and bottommost prediction blocks of the prediction block array. In the above implementation, the maximum L prediction blocks are uniformly distributed within the multiple prediction blocks of the current superblock.

[0037]

[0037] In any one of the above implementations, the position of the set of candidate prediction blocks is determined by encoded information including motion vectors of spatially adjacent blocks of the current superblock or the number of SMVP candidates already included in the MVP list.

[0038]

[0038] In any one of the above implementations, the method further includes, in response to there being N1 SMVP candidates in the MVP list, further limiting the number of TMVP candidates in the MVP list to N2, where N1 and N2 are positive integers and N2 is less than or equal to N.

[0039]

[0039] In any one of the above implementation forms, the context for signaling the index in the MVP list associated with the current prediction block and the inter prediction mode depends on whether at least N TMVPs are included in the MVP list.

[0040]

[0040] In any one of the above implementation forms, limiting the TMVP candidates in the MVP list to a maximum of N is in response to the current prediction block being coded under a bidirectional inter-prediction mode.

[0041]

[0041] Furthermore, aspects of the present disclosure provide an electronic device or apparatus including circuitry or a processor configured to perform any of the implementations of the above methods.

[0042]

[0042] Furthermore, an aspect of the present disclosure provides a non-transitory computer-readable medium storing instructions that, when executed by an electronic device, cause the electronic device to perform any one of the implementations of the above methods.

[0043] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings. [Brief description of the drawings]

[0044] [Figure 1A] FIG. 13 is a schematic diagram of an example subset of intra-prediction direction modes. [Figure 1B]

[0045] 1 is a diagram of an example intra-prediction direction. [Diagram 2]

[0046] FIG. 2 is a schematic diagram of a current block and its surrounding spatial merging candidates for motion vector prediction in one example. [Diagram 3]

[0047] FIG. 3 is a schematic diagram of a simplified block diagram of a communication system (300) according to an embodiment. [Figure 4]

[0048] FIG. 4 is a schematic diagram of a simplified block diagram of a communication system (400) according to an embodiment. [Diagram 5]

[0049] FIG. 2 is a schematic diagram of a simplified block diagram of a video decoder according to an embodiment; [Figure 6]

[0050] FIG. 2 is a schematic diagram of a simplified block diagram of a video encoder according to an embodiment; [Figure 7]

[0051] FIG. 4 is a block diagram of a video encoder according to another embodiment. [Figure 8]

[0052] FIG. 4 is a block diagram of a video decoder according to another embodiment. [Figure 9]

[0053] FIG. 2 is a diagram of a coding block partitioning scheme according to an embodiment of the present disclosure. [Figure 10]

[0054] FIG. 13 is a diagram of another coding block partitioning scheme according to an embodiment of the present disclosure. [Figure 11]

[0055] FIG. 13 is a diagram of another coding block partitioning scheme according to an embodiment of the present disclosure. [Figure 12]

[0056] FIG. 13 is a diagram of a partitioning of an example implementation of base blocks into coding blocks according to an example partitioning scheme. [Figure 13]

[0057] FIG. 2 is a diagram of an exemplary ternary partitioning scheme. [Figure 14]

[0058] FIG. 2 is a diagram of an example quadtree / binary tree coding block partitioning scheme. [Figure 15]

[0059] 1 is a diagram of a scheme for partitioning a coding block into multiple transform blocks and a coding order of the transform blocks according to an embodiment of the present disclosure. [Figure 16]

[0060] FIG. 13 is a diagram of another scheme for partitioning a coding block into multiple transform blocks and a coding order for the transform blocks according to an embodiment of the present disclosure. [Figure 17]

[0061] FIG. 13 is a diagram of another scheme for partitioning a coding block into multiple transform blocks according to an embodiment of the present disclosure. [Figure 18]

[0062] FIG. 1 is a diagram of a search process for spatial motion vector predictor candidates for a superblock. [Figure 19]

[0063] 4 is a diagram of an example process for determining a temporal motion vector predictor for a current predictive block by linear projection. [Figure 20]

[0064] FIG. 2 is a diagram of example intra- and outer-prediction blocks of a superblock. [Figure 21]

[0065] FIG. 13 is a diagram of an example new motion vector predictor candidate generation for a single inter-predicted block. [Figure 22]

[0066] FIG. 13 is a diagram of an example new motion vector predictor candidate generation for a composite inter-predicted block. [Diagram 23]

[0067] FIG. 13 is a diagram of an example reference motion vector candidate bank update process. [Figure 24]

[0068] 4 is a diagram of an example order for building a motion vector predictor list. [Diagram 25]

[0069] 4 is a diagram of an example search order for identifying temporal motion vector predictor candidates for a current super-block. [Figure 26] 4 is a diagram of an example search order for identifying temporal motion vector predictor candidates for a current super-block. [Figure 27] 4 is a diagram of an example search order for identifying temporal motion vector predictor candidates for a current super-block. [Figure 28] 4 is a diagram of an example search order for identifying temporal motion vector predictor candidates for a current super-block. [Figure 29] 4 is a diagram of an example search order for identifying temporal motion vector predictor candidates for a current super-block. [Diagram 30] 4 is a diagram of an example search order for identifying temporal motion vector predictor candidates for a current super-block. [Diagram 31]

[0070] 1 is a flowchart of a method according to an embodiment of the present disclosure. [Diagram 32]

[0071] FIG. 1 is a schematic diagram of a computer system according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0045]

[0072] Throughout this specification and the claims, terms may have subtle differences in meaning suggested or implied in context beyond the explicitly stated meaning. The phrases "in one embodiment" or "in some embodiments" as used herein do not necessarily refer to the same embodiment, and the phrases "in another embodiment" or "in other embodiments" as used herein do not necessarily refer to different embodiments. Similarly, the phrases "in one implementation" or "in some implementations" as used herein do not necessarily refer to the same implementation, and the phrases "in another implementation" or "in other implementations" as used herein do not necessarily refer to different implementations. For example, the claimed subject matter is intended to include a combination of the exemplary embodiments / implementations in whole or in part.

[0046]

[0073] Generally, terminology may be understood at least in part from the usage in context. For example, terms such as "and", "or", or "and / or" as used herein may include various meanings that may depend at least in part on the context in which such terms are used. Typically, "or" when used to relate a list such as A, B, or C is intended to mean A, B, and C, which is used herein in an inclusive sense, as well as A, B, or C, which is used herein in an exclusive sense. Furthermore, the terms "one or more" or "at least one" as used herein may be used to describe any feature, structure, or characteristic in a singular sense, or may be used to describe a combination of features, structures, or characteristics in a plural sense, depending at least in part on the context. Similarly, terms such as "a", "an", or "the" may be understood to convey a singular sense or a plural sense, depending at least in part on the context. Further, the terms "based on" or "determined by" may be understood as not necessarily intended to convey an exclusive set of elements, but instead allow for the presence of additional elements not necessarily expressly recited, again depending at least in part on the context. FIG. 3 illustrates a simplified block diagram of a communication system (300) according to an embodiment of the present disclosure. The communication system (300) includes multiple terminal devices capable of communicating with each other, for example, via a network (350). For example, the communication system (300) includes a first pair of terminal devices (310) and (320) interconnected via the network (350). In the example of FIG. 3, the first pair of terminal devices (310) and (320) may implement a unidirectional transmission of data. For example, the terminal device (310) may encode video data (e.g., of a stream of video pictures captured by the terminal device (310)) for transmission to the other terminal device (320) via the network (350).The encoded video data can be transmitted in the form of one or more coded video bitstreams. The terminal device (320) may receive the coded video data from the network (350), decode the coded video data to reconstruct the video picture, and display the video picture according to the reconstructed video data. The one-way data transmission may be implemented in a media serving application, etc.

[0047]

[0074] In another example, the communication system (300) includes a second pair of terminal devices (330) and (340) implementing bidirectional transmission of encoded video data, such as may be implemented during a video conferencing application. In the case of bidirectional transmission of data, in one example, each of the terminal devices (330) and (340) may encode video data (e.g., of a stream of video pictures captured by the terminal device) for transmission to the other of the terminal devices (330) and (340) over the network (350). Each of the terminal devices (330) and (340) may also receive encoded video data transmitted by the other of the terminal devices (330) and (340), decode the encoded video data to recover the video pictures, and display the video pictures on an accessible display device in response to the recovered video data.

[0048]

[0075] In the example of FIG. 3, the terminal devices (310), (320), (330), and (340) may be implemented as a server, a personal computer, and a smartphone, although the applicability of the basic principles of the present disclosure may not be so limited. The embodiments of the present disclosure may be implemented in desktop computers, laptop computers, tablet computers, media players, wearable computers, dedicated video conferencing equipment, and the like. The network (350) represents any number or type of network that conveys encoded video data between the terminal devices (310), (320), (330), and (340), including, for example, wireline and / or wireless communication networks. The communication network (350) 9 may exchange data over circuit-switched, packet-switched, and / or other types of channels. Representative networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For the purposes of this discussion, the architecture and topology of the network (350) may not be important to the operation of the present disclosure unless explicitly described herein.

[0049]

[0076] 4 illustrates an arrangement of a video encoder and a video decoder in a video streaming environment as an example of application of the disclosed subject matter. The disclosed subject matter may be equally applicable to other video applications including, for example, video conferencing, digital TV broadcasting, gaming, virtual reality, storage of compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0050]

[0077] The video streaming system may include a video capture subsystem (413), which may include a video source (401), such as a digital camera, for producing a stream of uncompressed video pictures or images (402). In one example, the stream of video pictures (402) includes samples recorded by the digital camera of the video source 401. The stream of video pictures (402), depicted as a bold line to emphasize its large data volume when compared to the encoded video data (404) (or encoded video bitstream), can be processed by an electronic device (420) that includes a video encoder (403) coupled to the video source (401). The video encoder (403) may include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter as described in more detail below. The encoded video data (404) (or encoded video bitstream (404)), depicted as thin lines to emphasize its smaller data volume when compared to the stream of uncompressed video pictures (402), can be stored on the streaming server (405) for future use or directly on a downstream video device (not shown). One or more streaming client subsystems, such as the client subsystems (406) and (408) of FIG. 4, can access the streaming server (405) to retrieve copies (407) and (409) of the encoded video data (404). The client subsystem (406) can include a video decoder (410), for example in the electronic device (430). The video decoder (410) decodes the incoming copy of the encoded video data (407) and produces an output stream of video pictures (411) that is uncompressed and can be rendered on a display (412) (e.g., a display screen) or other rendering device (not shown). The video decoder 410 may be configured to perform some or all of the various functions described in this disclosure.In some streaming systems, the encoded video data (404), (407), and (409) (e.g., video bitstreams) may be encoded according to a particular video encoding / compression standard. Examples of these standards include ITU-T Recommendation H.265. In one example, a video encoding standard under development is informally known as Versatile Video Coding (VVC). The disclosed subject matter may also be used in the context of VVC, as well as other video encoding standards.

[0051]

[0078] It should be noted that the electronic devices (420) and (430) may include other components (not shown). For example, the electronic device (420) may include a video decoder (not shown), and the electronic device (430) may also include a video encoder (not shown).

[0052]

[0079] 5 shows a block diagram of a video decoder (510) according to any of the embodiments of the present disclosure below. The video decoder (510) can be included in an electronic device (530). The electronic device (530) can include a receiver (531) (e.g., receiving circuitry). The video decoder (510) can be used in place of the video decoder (410) in the example of FIG. 4.

[0053]

[0080] The receiver (531) may receive one or more encoded video sequences to be decoded by the video decoder (510). In the same or another embodiment, one encoded video sequence may be decoded at a time, and the decoding of each encoded video sequence is independent of the other encoded video sequences. Each video sequence may be associated with multiple video frames or images. The encoded video sequences may be received from a channel (501), which may be a hardware / software link to a storage device that has stored the encoded video data or a streaming source that has transmitted the encoded video data. The receiver (531) may receive the encoded video data with other data, such as encoded audio data and / or auxiliary data streams, and the encoded video data may be forwarded to its respective processing circuitry (not shown). The receiver (531) may separate the encoded video sequences from the other data. A buffer memory (515) may be disposed between the receiver (531) and the entropy decoder / parser (520) (hereafter “parser (520)”) to prevent network jitter. In certain applications, the buffer memory (515) may be implemented as part of the video decoder (510). In other applications, the buffer memory (515) may be external to and separate from the video decoder (510) (not shown). In still other applications, there may be a buffer memory (not shown) external to the video decoder (510), e.g., to prevent network jitter, and another additional buffer memory (515) internal to the video decoder (510), e.g., to handle playback timing. When the receiver (531) is receiving data from a store / forward device with sufficient bandwidth and controllability, or from an isosynchronous network, the buffer memory (515) may not be necessary or may be small.For use with best-effort packet networks such as the Internet, a sufficiently sized buffer memory (515) may be required, which may be quite large. Such a buffer memory may be provided in an adaptable size and may be implemented at least in part in an operating system or similar element (not shown) external to the video decoder (510).

[0054]

[0081] The video decoder (510) may include a parser (520) for reconstructing symbols (521) from the encoded video sequence. These categories of symbols include information used to manage the operation of the video decoder (510) and possibly information for controlling a rendering device, such as a display (512) (e.g., a display screen), which may or may not be an integral part of the electronic device (530), but may be coupled to the electronic device (530) as shown in FIG. 5. The control information for the rendering device may be in the form of a Supplemental Enhancement Information (SEI) message or a Video Usability Information (VUI) parameter set fragment (not shown). The parser (520) may parse / entropy decode the encoded video sequence received by the parser (520). The entropy coding of the coded video sequence may conform to a video coding technique or standard and may follow various principles including variable length coding, Huffman coding, context-independent or non-context-dependent arithmetic coding, etc. The parser (520) may extract a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder from the coded video sequence based on at least one parameter corresponding to the subgroup. The subgroup may include a group of pictures (GOP), a picture, a tile, a slice, a macroblock, a coding unit (CU), a block, a transform unit (TU), a prediction unit (PU), etc. Additionally, the parser (520) may extract coded video sequence information such as transform coefficients (e.g., Fourier transform coefficients), quantization parameter values, motion vectors, etc.

[0055]

[0082] The parser (520) may perform an entropy decoding / parsing operation on the video sequence received from the buffer memory (515) to produce symbols (521).

[0056]

[0083] The reconstruction of the symbols (521) may require a number of different processing or functional units, depending on the type of coded video picture or part thereof (inter and intra pictures, inter and intra blocks, etc.), as well as other factors. The units required, and how they are required, may be controlled by subgroup control information parsed from the coded video sequence by the parser (520). The flow of such subgroup control information between the parser (520) and the following processing or functional units is not depicted for the sake of simplicity.

[0057]

[0084] In addition to the functional blocks already mentioned, the video decoder (510) can be conceptually subdivided into several functional units as described below. In a practical implementation operating under commercial constraints, many of these functional units may closely interact with each other and may be at least partially integrated with each other. However, in order to clearly explain the various functions of the disclosed subject matter, a conceptual subdivision into functional units is adopted in the following disclosure.

[0058]

[0085] The first unit may include a scalar / inverse transform unit (551), which may receive quantized transform coefficients and control information from the parser (520), including information indicating which type of inverse transform to use, block size, quantization factors / parameters, quantization scaling matrices, and lies as symbols (521). The scalar / inverse transform unit (551) may output blocks having sample values ​​that can be input to an aggregator (555).

[0059]

[0086] In some cases, the samples output from the scaler / inverse transform (551) may be for intra-coded blocks, i.e., blocks that can use prediction information from a previously reconstructed portion of the current picture, rather than using prediction information from a previously reconstructed picture. Such prediction information can be provided by an intra-picture prediction unit (552). In some cases, the intra-picture prediction unit (552) may generate blocks of the same size and shape of the block being reconstructed using surrounding block information already reconstructed and stored in a current picture buffer (558). The current picture buffer (558) may, for example, buffer a partially reconstructed current picture and / or a fully reconstructed current picture. In some implementations, the aggregator (555) may add the prediction information generated by the intra prediction unit (552) to the output sample information provided by the scaler / inverse transform unit (551) on a sample-by-sample basis.

[0060]

[0087] In other cases, the output samples of the scalar / inverse transform unit (551) may be for an inter-coded and potentially motion-compensated block. In such cases, the motion compensated prediction unit (553) may access the reference picture memory (557) to fetch samples used for inter-picture prediction. After motion compensating the fetched samples according to the symbol (521) for the block, these samples may be added to the output of the scalar / inverse transform unit (551) by the aggregator (555) to generate output sample information (the output of unit 551 may be referred to as residual samples or residual signals). The addresses in the reference picture memory (557) from which the motion compensated prediction unit (553) fetches prediction samples may be controlled by a motion vector and are available to the motion compensated prediction unit (553) in the form of a symbol (521) that may have, for example, X, Y components (shift) and a reference picture component (time). Motion compensation may further include interpolation for sample values ​​fetched from the reference picture memory (557) when sub-sample accurate motion vectors are used, may be further associated with a motion vector prediction mechanism, etc.

[0061]

[0088] The output samples of the aggregator (555) may be subjected to various loop filtering techniques in the loop filter unit (556). Video compression techniques may include in-loop filter techniques controlled by parameters contained in the coded video sequence (also called coded video bitstream) and made available to the loop filter unit (556) as symbols (521) from the parser (520), but may also include those that are responsive to meta-information obtained during decoding of a previous portion (in decoding order) of the coded picture or coded video sequence, and those that are responsive to previously reconstructed and loop filtered sample values. As described in more detail below, several types of loop filters may be included as part of the loop filter unit 556, in various orders.

[0062]

[0089] The output of the loop filter unit (556) may be a sample stream that can be output to a rendering device (512) and stored in a reference picture memory (557) for use during future inter-picture prediction.

[0063]

[0090] A particular coded picture can be used as a reference picture for future inter-picture prediction as soon as it is fully reconstructed. For example, once a coded picture corresponding to a current picture is fully reconstructed and the coded picture is identified as a reference picture (e.g., by the parser (520)), the current picture buffer (558) can become part of the reference picture memory (557), and an unused current picture buffer can be reallocated before beginning reconstruction of a subsequent coded picture.

[0064]

[0091] The video decoder (510) may perform decoding operations according to a given video compression technique adopted in a standard such as ITU-T Recommendation H.265. The encoded video sequence may conform to the syntax specified by the video compression technique or standard used in the sense that the encoded video sequence conforms to both the syntax of the video compression technique or standard and the profile as documented in the video compression technique or standard. In particular, a profile may select a particular tool from all the tools available in the video compression technique or standard as the only tool available for use under this profile. To be standard-compliant, the complexity of the encoded video sequence may be within a range as defined by the level of the video compression technique or standard. In some cases, the level limits the maximum picture size, the maximum frame rate, the maximum reconstructed sample rate (e.g., measured in megasamples per second), the maximum reference picture size, etc. In some cases, the limits set by the levels can be further restricted through a Hypothetical Reference Decoder (HRD) specification and metadata for HRD buffer management signaled in the encoded video sequence.

[0065]

[0092] In some implementations, the receiver (531) may receive additional (redundant) data along with the encoded video. The additional data may be included as part of the encoded video sequence. The additional data may be used by the video decoder (510) to properly decode the data and / or to more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0066]

[0093] 6 shows a block diagram of a video encoder (603) according to an embodiment of the present disclosure. The video encoder (603) may be included in an electronic device (620). The electronic device (620) may further include a transmitter (640) (e.g., a transmission circuit device). The video encoder (603) may be used in place of the video encoder (403) in the example of FIG. 4.

[0067]

[0094] The video encoder (603) may receive video samples from a video source (601) (not part of the electronic device (620) in the example of FIG. 6), which may capture video images to be encoded by the video encoder (603). In another example, the video source (601) may be provided as part of the electronic device (620).

[0068]

[0095] The video source (601) may provide a source video sequence to be encoded by the video encoder (603) in the form of a digital video sample stream, which may be of any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 YCrCb, RGB, XYZ...), and any suitable sampling structure (e.g., YCrCb4:2:0, YCrCb4:4:4). In a media serving system, the video source (601) may be a storage device capable of storing pre-prepared videos. In a video conferencing system, the video source (601) may be a camera that captures local image information as a video sequence. The video data may be provided as a number of individual pictures or images that convey motion when viewed in sequence. The pictures themselves may be organized as a spatial array of pixels, each of which may have one or more samples depending on the sampling structure, color space, etc., in use. Those skilled in the art can easily understand the relationship between pixels and samples. The following description focuses on examples.

[0069]

[0096] According to some implementations, the video encoder (603) may encode and compress pictures of a source video sequence into an encoded video sequence (643) in real-time or under any other time constraint as required by the application. Enforcing a proper encoding speed constitutes one function of the controller (650). In some embodiments, the controller (650) may be functionally coupled to and control other functional units as described below. The coupling is not depicted for the sake of brevity. Parameters set by the controller (650) may include rate control related parameters (picture skip, quantizer, lambda value for rate distortion optimization techniques, ...), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (650) can be configured to have other appropriate functions for the video encoder (603) optimized for a particular system design.

[0070]

[0097] In some implementations, the video encoder (603) may be configured to operate within an encoding loop. As an oversimplified explanation, in one example, the encoding loop may include a source coder (630) (e.g., responsible for producing symbols, such as a symbol stream, based on an input picture to be encoded and a reference picture) and a (local) decoder (633) built into the video encoder (603). The decoder (633) reconstructs symbols to produce sample data in a manner similar to that produced by a (remote) decoder, even if the built-in decoder 633 processes the video stream encoded by the source coder 630 without entropy encoding (as any compression between the symbols in the entropy encoding and the encoded video bitstream may be lossless in the video compression techniques considered in the disclosed subject matter). The reconstructed sample stream (sample data) is input to a reference picture memory (634). Since the decoding of the symbol stream results in bit-exact results independent of the decoder location (local or remote), the contents of the reference picture memory (634) are also bit-exact between the local and remote encoders. In other words, the prediction part of the encoder "sees" exactly the same sample values ​​as the reference picture samples that the decoder would "see" when using prediction during decoding. This basic principle of reference picture synchrony (and the resulting drift if synchrony cannot be maintained, e.g., due to channel errors) is used to improve coding quality.

[0071]

[0098] The operation of the "local" decoder (633) may be the same as a "remote" decoder, such as the video decoder (510) already described in detail above in conjunction with Figure 5. However, and also with brief reference to Figure 5, because symbols are available and the encoding / decoding of symbols into an encoded video sequence by the entropy coder (645) and parser (520) may be lossless, the entropy decoding portion of the video decoder (510), including the buffer memory (515) and parser (520), may not be implemented entirely in the local decoder (633) within the encoder.

[0072]

[0099] At this point, it can be said that decoder techniques other than parsing / entropy decoding that are present only in the decoder must necessarily be present in the corresponding encoder in substantially identical functional form. For this reason, the subject matter disclosed may sometimes focus on the decoder operation in cooperation with the decoding portion of the encoder. Thus, a description of the encoder techniques may be omitted, since they are the inverse of the decoder techniques described generically. Only in certain areas or aspects is a more detailed description of the encoder provided below.

[0073]

[0100] During operation in some implementations, the source coder (630) may perform motion-compensated predictive coding, which predictively codes an input picture with reference to one or more previously coded pictures from a video sequence designated as "reference pictures." In this manner, the coding engine (632) codes color channel differences (or residuals) between pixel blocks of the input picture and pixel blocks of reference pictures that may be selected as prediction references for the input picture. The terms "residual" and its adjective form "residual" are sometimes used interchangeably.

[0074]

[0101] The local video decoder (633) may decode the encoded video data of pictures that may be designated as reference pictures based on the symbols produced by the source coder (630). The operation of the encoding engine (632) may advantageously be a lossy process. When the encoded video data is decoded in a video decoder (not shown in FIG. 6), the reconstructed video sequence may be a copy of the source video sequence, which typically contains some errors. The local video decoder (633) may replicate the decoding process that may be performed by the video decoder on the reference pictures and store the reconstructed reference pictures in a reference picture cache (634). In this way, the video encoder (603) may locally store copies of reconstructed reference pictures that have common content with the reconstructed reference pictures that will be obtained by a far-end (remote) video decoder (without transmission errors).

[0075]

[0102] The predictor (635) may perform a prediction search for the coding engine (632). That is, to code a new picture, the predictor (635) may search the reference picture memory (634) for sample data (as candidate reference pixel blocks) or for specific metadata, such as reference picture motion vectors, block shapes, etc., that may serve as suitable prediction references for the new picture. The predictor (635) may operate on a sample block-by-pixel block basis to find a suitable prediction reference. In some cases, as determined by the search results obtained by the predictor (635), the input picture may have prediction references created from multiple reference pictures stored in the reference picture memory (634).

[0076]

[0103] The controller (650) may manage the encoding operations of the source coder (630), including, for example, setting the parameters and subgroup parameters used to encode the video data.

[0077]

[0104] The output of all the aforementioned functional units may be subjected to entropy coding in an entropy coder (645), which converts the symbols produced by the various functional units into an encoded video sequence by losslessly compressing the symbols according to techniques such as Huffman coding, variable length coding, or arithmetic coding.

[0078]

[0105] The transmitter (640) may buffer the encoded video sequence produced by the entropy coder (645) in preparation for transmission over a communication channel (660), which may be a hardware / software link to a storage device that is to store the encoded video data. The transmitter (640) may merge the encoded video data from the video coder (603) with other data to be transmitted, such as, for example, encoded audio data and / or auxiliary data streams (sources not shown).

[0079]

[0106] A controller (650) may manage the operation of the video encoder (603). During encoding, the controller (650) may assign each encoded picture a particular encoded picture type, which may affect the encoding technique that may be applied to the respective picture. For example, pictures may often be assigned one of the following picture types:

[0080]

[0107] An intra picture (I-picture) may be one that is coded and decoded without using any other picture in a sequence as a source of prediction. Some video codecs allow different types of intra pictures, including, for example, Independent Decoder Refresh ("IDR") pictures. Those skilled in the art are aware of these variations of I-pictures and their respective uses and characteristics.

[0081]

[0108] A predicted picture (P picture) may be encoded and decoded using intra- or inter-prediction, using at most one motion vector and reference index to predict the sample values ​​of each block.

[0082]

[0109] Bidirectionally predicted pictures (B-pictures) may be coded and decoded using intra- or inter-prediction using at most two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, multiple predicted pictures can use more than two reference pictures and associated metadata for the reconstruction of a single block.

[0083]

[0110] A source picture may generally be spatially subdivided into a number of sample coding blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples, respectively) and coded block by block. A block may be predictively coded with reference to other (already coded) blocks determined by the coding assignment applied to the respective picture of the block. For example, a block of an I picture may be non-predictively coded or may be predictively coded with reference to already coded blocks of the same picture (spatial or intra prediction). A pixel block of a P picture may be predictively coded via spatial prediction or via temporal prediction with reference to one previously coded reference picture. A block of a B picture may be predictively coded via spatial prediction or via temporal prediction with reference to one or two previously coded reference pictures. A source picture or an intermediate processed picture may be subdivided into other types of blocks for other purposes. The division of coding blocks and other types of blocks may or may not follow the same aspects as described in more detail below.

[0084]

[0111] The video encoder (603) may perform encoding operations that comply with a given video encoding technique or standard, such as ITU-T Recommendation H.265. During its operation, the video encoder (603) may perform various compression operations, including predictive encoding operations that exploit temporal and spatial redundancy in the input video sequence. Thus, the encoded video data may conform to a syntax specified by the video encoding technique or standard used.

[0085]

[0112] In some embodiments, the transmitter (640) may transmit additional data along with the encoded video. The source coder (630) may include such data as part of the encoded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

[0086]

[0113] Video may be captured as multiple source pictures (video pictures) in a time sequence. Intra-picture prediction (often abbreviated to intra prediction) exploits spatial correlations of a given picture, while inter-picture prediction exploits temporal or other correlations between pictures. For example, a particular picture being encoded / decoded, called the current picture, may be partitioned into blocks. A block in the current picture may be coded by a vector, called a motion vector, when it resembles a reference block in a previously coded and still buffered reference picture in the video. A motion vector points to a reference block in a reference picture, and may have a third dimension that identifies the reference picture in cases where multiple reference pictures are in use.

[0087]

[0114] In some embodiments, bi-prediction techniques can be used for inter-picture prediction. According to such bi-prediction techniques, two reference pictures, such as a first reference picture and a second reference picture, are used that precede the current picture in the video in decoding order (but may be in the past or future, respectively, in display order). A block in the current picture can be coded by a first motion vector that points to a first reference block in the first reference picture and a second motion vector that points to a second reference block in the second reference picture. A block can be jointly predicted by combining the first reference block and the second reference block.

[0088]

[0115] Furthermore, to improve coding efficiency, merge mode techniques may be used during inter-picture prediction.

[0089]

[0116] According to some embodiments of the present disclosure, predictions such as inter-picture prediction and intra-picture prediction are performed on a block-by-block basis. For example, pictures in a sequence of video pictures are partitioned into coding tree units (CTUs) for compression, and the CTUs in a picture may have the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. In general, a CTU may include three parallel coding tree blocks (CTBs), one luma CTB and two chroma CTBs. Each CTU may be recursively quad-tree partitioned into one or more coding units (CUs). For example, a CTU of 64×64 pixels may be partitioned into one CU of 64×64 pixels, or four CUs of 32×32 pixels. Each of one or more of the 32×32 blocks may be further partitioned into four CUs of 16×16 pixels. In some implementations, each CU may be analyzed during encoding to determine a prediction type for the CU from among various prediction types, such as an inter prediction type or an intra prediction type. The CU may be divided into one or more prediction units (PUs) according to temporal and / or spatial predictability. In general, each PU includes one luma prediction block (PB) and two chroma PBs. In one embodiment, the prediction operation during coding (encoding / decoding) is performed in units of prediction blocks. The division of the CU into PUs (or PBs of different color channels) may be performed in various spatial patterns. For example, a luma or chroma PB may include a matrix of sample values ​​(e.g., luma values), such as 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 samples, etc.

[0090]

[0117] 7 shows a diagram of a video encoder (703) according to another embodiment of the present disclosure. The video encoder (703) is configured to receive a processed block of sample values ​​(e.g., a predictive block) in a current video picture in a sequence of video pictures and to encode the processed block into an encoded picture that is part of an encoded video sequence. The example video encoder (703) may be used in place of the example video encoder (403) of FIG. 4.

[0091]

[0118] For example, the video encoder (703) receives a matrix of sample values ​​for a processing block, such as a predictive block of 8x8 samples. The video encoder (703) then determines, for example using rate distortion optimization (RDO), whether the processing block is best coded using an intra mode, an inter mode, or a bi-predictive mode. When it is determined that the processing block is coded in an intra mode, the video encoder (703) may use an intra prediction technique to encode the processing block into a coded picture, and when it is determined that the processing block is coded in an inter mode or a bi-predictive mode, the video encoder (703) may use an inter prediction or bi-predictive technique to encode the processing block into a coded picture, respectively. In some implementations, a merge mode may be used as a sub-mode of inter-picture prediction, in which case a motion vector is derived from one or more motion vector predictors without the benefit of any coded motion vector components outside the predictors. In some other implementations, there may be motion vector components applicable to the subject block. Thus, the video encoder (703) may include components not explicitly shown in FIG. 7, such as a mode decision module, to determine the prediction mode of a processing block.

[0092]

[0119] In the example of Figure 7, the video encoder (703) includes an inter-encoder (730), an intra-encoder (722), a residual calculator (723), a switch (726), a residual encoder (724), a general controller (721), and an entropy encoder (725), linked together as shown in the example arrangement of Figure 7.

[0093]

[0120] The inter encoder (730) is configured to receive a sample of a current block (e.g., a processing block), compare the block with one or more reference blocks in a reference picture (e.g., blocks in previous and subsequent pictures in display order), generate inter prediction information (e.g., a description of redundant information due to inter encoding techniques, motion vectors, merge mode information), and calculate an inter prediction result (e.g., a prediction block) based on the inter prediction information using any suitable technique. In some examples, the reference picture is a decoded reference picture that has been decoded based on the encoded video information using a decoding unit 633 incorporated in the example encoder 620 of FIG. 6 (shown as a residual decoder 728 of FIG. 7, as described in more detail below).

[0094]

[0121] The intra encoder (722) is configured to receive samples of a current block (e.g., a processing block), compare the block with previously encoded blocks in the same picture, generate transformed quantized coefficients, and in some cases, further generate intra prediction information (e.g., intra prediction direction information according to one or more intra encoding techniques). The intra encoder (722) may calculate intra prediction results (e.g., a prediction block) based on the intra prediction information and reference blocks in the same picture.

[0095]

[0122] The general controller (721) may be configured to determine general control data and control other components of the video encoder (703) based on the general control data. In one example, the general controller (721) determines a prediction mode of the block and sends a control signal to the switch (726) based on the prediction mode. For example, when the prediction mode is an intra mode, the general controller (721) controls the switch (726) to select an intra mode result for use by the residual calculator (723) and controls the entropy encoder (725) to select intra prediction information and include the intra prediction information in the bitstream, and when the prediction mode of the block is an inter mode, the general controller (721) controls the switch (726) to select an inter prediction result for use by the residual calculator (723) and controls the entropy encoder (725) to select inter prediction information and include the inter prediction information in the bitstream.

[0096]

[0123] The residual calculator (723) may be configured to calculate a difference (residual data) between a received block and a prediction result of a block selected from the intra-encoder (722) or the inter-encoder (730). The residual encoder (724) may be configured to encode the residual data to generate transform coefficients. For example, the residual encoder (724) may be configured to convert the residual data from a spatial domain to a frequency domain to generate transform coefficients. The transform coefficients then undergo a quantization process to obtain quantized transform coefficients. In various embodiments, the video encoder (703) further includes a residual decoder (728). The residual decoder (728) is configured to perform an inverse transform and generate decoded residual data. The decoded residual data can be suitably used by the intra-encoder (722) and the inter-encoder (730). For example, the inter-encoder (730) can generate decoded blocks based on the decoded residual data and the inter-prediction information, and the intra-encoder (722) can generate decoded blocks based on the decoded residual data and the intra-prediction information. The decoded blocks are appropriately processed to generate a decoded picture, which can be buffered in a memory circuit (not shown) and used as a reference picture.

[0097]

[0124] The entropy encoder (725) may be configured to format a bitstream to include the encoded block and perform entropy encoding. The entropy encoder (725) may be configured to include various information in the bitstream. For example, the entropy encoder (725) may be configured to include general control data, selected prediction information (e.g., intra-prediction information or inter-prediction information), residual information, and other suitable information in the bitstream. When encoding a block in an inter mode or a merged sub-mode of a bi-prediction mode, the residual information may be absent.

[0098]

[0125] 8 shows a diagram of an example video decoder (810) according to another embodiment of the present disclosure. The video decoder (810) is configured to receive encoded pictures that are part of an encoded video sequence and decode the encoded pictures to generate reconstructed pictures. In one example, the video decoder (810) may be used in place of the video decoder (410) in the example of FIG. 4.

[0099]

[0126] In the example of FIG. 8, the video decoder (810) includes an entropy decoder (871), an inter decoder (880), a residual decoder (873), a reconstruction module (874), and an intra decoder (872) linked together as shown in the example arrangement of FIG. 8.

[0100]

[0127] The entropy decoder (871) can be configured to reconstruct from the coded picture certain symbols that represent syntax elements that make up the coded picture. Such symbols can include, for example, prediction information (e.g., intra-mode, inter-mode, bi-predictive mode, merged sub-mode, or another sub-mode) that can identify the mode in which the block is coded, certain samples or metadata used for prediction by the intra-decoder (872) or inter-decoder (880), residual information in the form of, for example, quantized transform coefficients, etc. In one example, when the prediction mode is an inter- or bi-predictive mode, the inter-predictive information is provided to the inter-decoder (880), and when the prediction type is an intra-predictive type, the intra-predictive information is provided to the intra-decoder (872). The residual information can undergo inverse quantization and is provided to the residual decoder (873).

[0101]

[0128] The inter decoder (880) may be configured to receive the inter prediction information and to generate inter prediction results based on the inter prediction information.

[0102]

[0129] The intra decoder (872) may be configured to receive intra prediction information and to generate a prediction result based on the intra prediction information.

[0103]

[0130] The residual decoder (873) may be configured to perform inverse quantization to extract dequantized transform coefficients, and process the dequantized transform coefficients to convert the residual from the frequency domain to the spatial domain. Additionally, the residual decoder (873) may utilize certain control information (to include quantization parameters (QP)) that may be provided by the entropy decoder (871) (this may be only a small amount of control information, so the data path is not depicted).

[0104]

[0131] The reconstruction module (874) may be configured to combine in the spatial domain the residual output by the residual decoder (873) and the prediction result (possibly output by the inter- or intra-prediction module) to form a reconstructed block that forms part of the reconstructed picture as part of the reconstructed video. It should be noted that other suitable operations, such as deblocking operations, may also be performed to improve visual quality.

[0105]

[0132] It should be noted that the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) may be implemented using any suitable technique. In some implementations, the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) may be implemented using one or more integrated circuits. In other embodiments, the video encoders (403), (603), and (603) and the video decoders (410), (510), and (810) may be implemented using one or more processors executing software instructions.

[0106]

[0133] Returning to the description of block partitioning for encoding and decoding, a general partitioning may start from a base block and may follow a predefined set of rules, a specific pattern, a partition tree, or any partition structure or scheme. The partitioning may be hierarchical and recursive. After splitting or partitioning the base block according to any of the example partitioning procedures described below or other procedures or combinations thereof, a final set of partitions or coding blocks may be obtained. Each of these partitions may be at one of various partitioning levels in the partitioning hierarchy and may be of various shapes. Each of the partitions may be referred to as a coding block (CB). For various example partitioning implementations described further below, each resulting CB may be of any of the allowed sizes and partitioning levels. Such partitions are referred to as coding blocks because they may form the units at which some basic encoding / decoding decisions may be made and encoding / decoding parameters may be optimized, determined, and signaled in the encoded video bitstream. The highest or deepest level of the final partition represents the depth of the coding block partitioning structure of the tree. A coding block may be a luma coding block or a chroma coding block. The CB tree structure for each color may be called a coding block tree (CBT).

[0107]

[0134] The coding blocks of all color channels may be collectively referred to as a coding unit (CU). The hierarchical structure of all color channels may be collectively referred to as a coding tree unit (CTU). The partitioning pattern or structure of the various color channels in a CTU may or may not be the same.

[0108]

[0135] In some implementations, the partition tree scheme or structure used for the luma and chroma channels may not need to be the same. In other words, the luma and chroma channels may have separate coding tree structures or patterns. Furthermore, whether the luma and chroma channels use the same coding partition tree structure or different coding partition tree structures, and whether the actual coding partition tree structure should be used, may depend on whether the coded slice is a P slice, a B slice, or an I slice. For example, for an I slice, the chroma and luma channels may have separate coding partition tree structures or coding partition tree structure modes, while for a P or B slice, the luma and chroma channels may share the same coding partition tree scheme. When separate coding partition tree structures or modes are applied, the luma channel may be partitioned into CB by one coding partition tree structure, and the chroma channel may be partitioned into chroma CB by another coding partition tree structure.

[0109]

[0136] In some implementations, a pre-determined partitioning pattern may be applied to the base block. As shown in FIG. 9, an example 4-way partition tree may start from a first pre-defined level (e.g., 64×64 block level or other size as the base block size), and the base block may be partitioned hierarchically down to a pre-defined lowest level (e.g., 4×4 level). For example, the base block may follow four pre-defined partitioning options or patterns indicated by 902, 904, 906, and 908, and the partition designated as R allows recursive partitioning since the same partitioning options indicated in FIG. 9 may be repeated at a lower scale down to the lowest level (e.g., 4×4 level). In some implementations, additional restrictions may be applied to the partitioning scheme of FIG. 9. In the implementation of FIG. 9, rectangular partitions (e.g., 1:2 / 2:1 rectangular partitions) may be allowed, but are not allowed to be recursive, whereas square partitions are allowed to be recursive. Partitioning according to FIG. 9 with regression, if necessary, produces a final set of coding blocks. A coding tree depth may be further defined to indicate the division depth from the root node or root block. For example, the coding tree depth of the root node or root block, such as a 64×64 block, may be set to 0, and after the root block is further divided once according to FIG. 9, the coding tree depth is increased by 1. The maximum or deepest level from the 64×64 base block to the 4×4 minimum partition should be 4 (starting from level 0) for the above scheme. Such a partitioning scheme may be applied to one or more of the color channels. Each color channel may be partitioned independently according to the scheme of FIG. 9 (e.g., a partitioning pattern or option from among predefined patterns may be determined independently for each of the color channels for each hierarchical level).Alternatively, two or more of the color channels may share the same hierarchical pattern tree of FIG. 9 (e.g., the same partitioning pattern or option from among predefined patterns may be chosen for two or more color channels per hierarchical level).

[0110]

[0137] FIG. 10 illustrates another example predefined partitioning pattern that allows for recursive partitioning to form a partitioning tree. As illustrated in FIG. 10, an example 10-way partitioning structure or pattern may be predefined. A root block may start at a predefined level (e.g., from a base block at a 128×128 level or a 64×64 level). The example partitioning structure of FIG. 10 includes various 2:1 / 1:2 and 4:1 / 1:4 rectangular partitions. A partition type having three sub-partitions, designated 1002, 1004, 1006, and 1008 in the second row of FIG. 10, may be referred to as a "T-type" partition. The "T-type" partitions 1002, 1004, 1006, and 1008 may be referred to as a left T-type, an upper T-type, a right T-type, and a lower T-type. In some implementations, none of the rectangular partitions of FIG. 10 are allowed to be further subdivided. A coding tree depth may be further defined to indicate the partition depth from the root node or root block. For example, the coding tree depth of the root node or root block, such as a 128×128 block, may be set to 0, and the coding tree depth increases by 1 after the root block is further partitioned once according to FIG. 10. In some implementations, only the all-square partitions of 1010 may be allowed to recursively partition to the next level of the partition tree according to the pattern of FIG. 10. In other words, recursive partitioning may not be allowed for the square partitions in the T-type patterns 1002, 1004, 1006, and 1008. The partitioning procedure according to FIG. 10 with recursion generates a final set of coding blocks, if necessary. Such a scheme may be applied to one or more of the color channels. In some implementations, more flexibility may be added to the use of partitions below the 8×8 level. For example, 2×2 chroma inter prediction may be used in certain cases.

[0111]

[0138] In some other implementations of coding block partitioning, a quadtree structure may be used to split a base block or an intermediate block into quadtree partitions. Such quadtree partitioning may be applied hierarchically and recursively to any square-shaped partition. Whether a base block or an intermediate block or partition is further quadtree partitioned may be adapted to various local characteristics of the base block or intermediate block / partition. The quadtree partitioning at the picture boundary may be further adapted. For example, an implicit quadtree partitioning may be implemented at the picture boundary such that a block continues to be quadtree partitioned until its size fits the picture boundary.

[0112]

[0139] In some other implementations, hierarchical binary partitioning from the base block may be used. For such schemes, the base block or mid-level block may be partitioned into two partitions. The binary partitioning may be horizontal or vertical. For example, horizontal binary partitioning may cause the base block or mid-block to be divided into equal partitions on the right and left. Similarly, vertical binary partitioning may cause the base block or mid-block to be divided into equal partitions on the top and bottom. Such binary partitioning may be hierarchical and recursive. At each of the base block or mid-block, a determination may be made as to whether the binary partitioning scheme should be continued and, if the scheme is continued further, whether horizontal or vertical binary partitioning should be used. In some implementations, further partitioning may stop at a predefined minimum partition size (in one or both dimensions). Alternatively, further partitioning may stop as soon as a predefined partitioning level or depth from the base block is reached. In some implementations, the aspect ratio of the partitions may be limited. For example, the aspect ratio of a section need not be less than 1:4 (or more than 4:1). Thus, a vertically elongated section with a vertical to horizontal aspect ratio of 4:1 may simply be further bisected vertically into an upper section and a lower section, each having a vertical to horizontal aspect ratio of 2:1.

[0113]

[0140] In some further examples, a three-partitioning scheme may be used to partition the base block or any intermediate blocks, as shown in FIG. 13. A three-partitioning may be implemented, either vertically, as shown in 1302 of FIG. 13, or horizontally, as shown in 1304 of FIG. 13. The example division ratio of FIG. 13 is shown as 1:2:1 vertically or horizontally, but other ratios may be predefined. In some implementations, two or more different ratios may be predefined. Such a three-partitioning scheme may be used to complement a quad-tree or bipartitioning structure, since such a ternary tree partitioning has the ability to capture an object at a block center in one continuous partition, while quad-trees and bipartitions always split along the block center, and thus should cause the object to be divided into separate partitions. In some implementations, the width and height of the partitions of the example ternary tree are always powers of two to avoid further transformations.

[0114]

[0141] The above partitioning schemes may be combined in any manner at different partitioning levels. As an example, the above-mentioned quadtree and bipartitioning schemes may be combined to partition a base block into a quadtree-binary-tree (QTBT) structure. In such a scheme, the base block or intermediate blocks / partitions may be quadtree-partitioned or bipartitioned, subject to a set of predefined conditions if specified. A specific example is illustrated in FIG. 14. In the example of FIG. 14, the base block is first quadtree-partitioned into four partitions, as shown by 1402, 1404, 1406, and 1408. Each of the resulting partitions is then either quadtree-partitioned into four further partitions (such as 1408), or bipartitioned into two further partitions at the next level (such as 1402 and 1406, which are either symmetrically divided either horizontally or vertically), or not divided (such as 1404). Bisection or quadtree partitioning may be allowed recursively for square shaped partitions, as shown by the overall example partition pattern in 1410 and the corresponding tree structure / representation in 1420, where solid lines represent quadtree partitioning and dashed lines represent bisection. A flag may be used for each bisection node (two leafless partitions) to indicate whether the bisection is horizontal or vertical. For example, a flag "0" may represent a horizontal bisection and a flag "1" may represent a vertical bisection, as shown in 1420 in agreement with the partitioning structure in 1410. In the case of quadtree partitioning, it is not necessary to indicate the partition type, since a quadtree partition always splits a block or partition both horizontally and vertically to yield four subblocks / partitions of equal size. In some implementations, a flag "1" may represent a horizontal bisection and a flag "0" may represent a vertical bisection.

[0115]

[0142] In some implementations of QTBT, the quadtree and bisection rule sets may be represented by the following predefined parameters and corresponding functions associated with them: - CTU size: Root node size of the quadtree (size of the base block) - MinQTSize: The minimum allowable quadtree leaf node size - MaxBTSize: Maximum allowed binary tree root node size - MaxBTDepth: Maximum allowed binary tree depth - MinBTSize: The minimum allowable binary tree leaf node size In some example implementations of the QTBT partitioning structure, the CTU size may be set as 128×128 luma samples with two corresponding 64×64 blocks of chroma samples (when the example chroma subsampling is considered and used), MinQTSize may be set as 16×16, MaxBTSize may be set as 64×64, MinBTSize (both width and height) may be set as 4×4, and MaxBTDepth may be set as 4. A quadtree partitioning may first be applied to the CTU to generate quadtree leaf nodes. The quadtree leaf nodes may have a size from its minimum allowed size of 16×16 (i.e., MinQTSize) to 128×128 (i.e., CTU size). If the node is 128×128, it will not be initially split by the binary tree since the size exceeds MaxBTSize (i.e., 64×64). Otherwise, the nodes that do not exceed MaxBTSize may be partitioned by a bitree. In the example of FIG. 14, the base block is 128×128. According to the predefined rule set, the basic block may only be quadtree partitioned. The base block has a partitioning depth of 0. Each of the resulting four partitions is 64×64, which does not exceed MaxBTSize, so it may be further quadtree or bitree partitioned at level 1. The process continues. When the bitree depth reaches MaxBTDepth (i.e., 4), no further partitions may be considered. When a bitree node has a width equal to MinBTSize (i.e., 4), no further horizontal partitions may be considered. Similarly, when a bitree node has a height equal to MinBTSize, no further vertical partitions are considered.

[0116]

[0143] In some implementations, the above QTBT scheme may be configured to support flexibility for luma and chroma to have the same or separate QTBT structures. For example, for P and B slices, the luma and chroma CTBs in one CTU may share the same QTBT structure. However, for an I slice, the luma CTB may be partitioned into CBs by a QTBT structure, and the chroma CTB may be partitioned into chroma CBs by another QTBT structure. This means that CUs may be used to refer to different color channels in an I slice; for example, an I slice may consist of a coding block of a luma component, or a coding block of two chroma components, and a CU in a P or B slice may consist of coding blocks of all three color components.

[0117]

[0144] In some other implementations, the QTBT scheme may be supplemented with the ternary scheme described above. Such implementations may be referred to as multi-type tree (MTT) structures. For example, in addition to the bipartitioning of the nodes, one of the ternary partitioning patterns of FIG. 13 may be chosen. In some implementations, only square nodes may undergo ternary partitioning. An additional flag may be used to indicate whether the ternary partitioning is horizontal or vertical.

[0118]

[0145] Two-level or multi-level tree designs, such as the QTBT implementation, and the QTBT implementation supplemented by a three-way split, may be primarily motivated by complexity reduction. Logically, the complexity of traversing a tree is T D where T represents the number of split types and D is the depth of the tree. Trade-offs may be made by using multiple types (T) while reducing the depth (D).

[0119]

[0146] In some implementations, the CB may be further partitioned. For example, the CB may be further partitioned into multiple prediction blocks (PBs) for intra- or inter-frame prediction during the encoding and decoding process. In other words, the CB may be further divided into different sub-partitions, and individual prediction decisions / configurations may be made in each sub-partition. At the same time, the CB may be further partitioned into multiple transform blocks (TBs) to accurately represent the level at which the transformation or inverse transformation of the video data is performed. The partitioning scheme of the CB into PBs and TBs may be the same or different. For example, each partitioning scheme may be implemented using a unique procedure based on, for example, various characteristics of the video data. The PB and TB partitioning schemes may be unrelated in some implementations. The PB and TB partitioning schemes and boundaries may be correlated in some other implementations. In some implementations, for example, the TB may be partitioned after the PB partition, and in particular, each PB may be determined following the partitioning of the coding block, and then further partitioned into one or more TBs. For example, in some implementations, the PB may be divided into one, two, four, or some other number of TBs.

[0120]

[0147] In some implementations, for partitioning of base blocks into coding blocks and further into predictive and / or transform blocks, the luma and chroma channels may be treated separately. For example, in some implementations, partitioning of coding blocks into predictive and / or transform blocks may be allowed for the luma channel, while such partitioning of coding blocks into predictive and / or transform blocks may not be allowed for the chroma channels. In such implementations, transformation and / or prediction of luma blocks may therefore only be performed at the coding block level. As another example, the minimum transform block size of the luma and chroma channels may be different, e.g., coding blocks of the luma channel may be allowed to be partitioned into smaller transform and / or predictive blocks than the chroma channels. As yet another example, the maximum depth of partitioning of coding blocks into transform and / or predictive blocks may be different between the luma and chroma channels, e.g., coding blocks of the luma channel may be allowed to be partitioned into deeper transform and / or predictive blocks than the chroma channels. As a specific example, a luma coding block may be partitioned into transform blocks of multiple sizes, which can be represented by recursive partitions going up to two levels down, and transform block shapes such as square, 2:1 / 1:2, and 4:1 / 1:4, as well as transform block sizes from 4×4 to 64×64 may be allowed. However, for chroma blocks, only the largest possible transform block specified for the luma block may be allowed.

[0121]

[0148] In some implementations of partitioning of coding blocks into PBs, the depth, shape, and / or other characteristics of the PB partitioning may depend on whether the PB is intra-coded or inter-coded.

[0122]

[0149] Partitioning of the coding block (or prediction block) into transform blocks may be performed recursively or non-recursively in various exemplary schemes, including but not limited to quadtree partitioning and predefined pattern partitioning, and with further consideration of transform blocks at boundaries of the coding block or prediction block. In general, the resulting transform blocks may be of different partitioning levels, may not be the same size, and may not need to be square in shape (e.g., the resulting transform blocks can be rectangular with some allowable sizes and aspect ratios). Further examples are described in more detail below in connection with Figures 15, 16, and 17.

[0123]

[0150] However, in some other implementations, the CB obtained through any of the above partitioning schemes may be used as a basic or minimum coding block for prediction and / or transformation. In other words, no further division is performed for performing inter-prediction / intra-prediction and / or transformation. For example, the CB obtained from the above QTBT scheme may be directly used as a unit for performing prediction. Specifically, such a QTBT structure removes the concept of multiple partition types, that is, such a QTBT structure eliminates the distinction between CU, PU, ​​and TU, and supports more flexibility for CU / CB partition shapes as described above. In such a QTBT block structure, the CU / CB can have a square or rectangular shape. The leaf nodes of such a QTBT are used as units for prediction and transformation processing without any further partitioning. This means that in such an exemplary QTBT coding block structure, the CU, PU, ​​and TU have the same block size.

[0124]

[0151] The various CB compartmentalization schemes described above, as well as further compartmentalization of the CB into PB and / or TB (not including PB / TB compartmentalization), may be combined in any manner. The following specific implementations are provided as non-limiting examples.

[0125]

[0152] Specific implementation examples of coding block and transform block partitioning are described below. In such implementation examples, a base block may be divided into coding blocks using a recursive quadtree partitioning or the above-mentioned predefined partitioning patterns (such as those in Figures 9 and 10). For each level, whether further quadtree partitioning of a particular partition should be continued may be determined by local video data characteristics. The resulting CBs may be at various quadtree partitioning levels and of various sizes. The decision as to whether a picture area should be coded using inter-picture (temporal) prediction or intra-picture (spatial) prediction may be made at the CB level (or at the CU level for a total of three color channels). Each CB may be further divided into one, two, four, or other number of PBs according to the predefined PB partitioning type. Inside one PB, the same prediction process may be applied, and related information may be transmitted to the decoder for each PB. After obtaining the residual block by applying a prediction process based on the PB division type, the CB can be partitioned into TBs according to another quadtree structure similar to the coding tree of the CB. In this particular implementation, the CB or TB may be limited to a square shape, but need not be limited to it. Furthermore, in this particular example, the PB may be square or rectangular shaped for inter prediction, and only square for intra prediction. The coding block may be divided, for example, into four square shaped TBs. Each TB may be further divided recursively (using quadtree division) into smaller TBs called residual quadtrees (RQTs).

[0126]

[0153] Another implementation example for partitioning the base block into CB, PB, and / or TB is described further below. For example, instead of using multiple partition unit types such as those shown in FIG. 9 or FIG. 10, a quadtree with nested multi-type trees using 2- and 3-partition segmentation structures (e.g., QTBT or QTBT with 3-partition as described above) may be used. Separation of CB, PB, and TB (i.e., partitioning of CB into PB and / or TB, and partitioning of PB into TB) may be abandoned except when necessary for CBs that have a size that is too large for the maximum transform length (such CBs may require further partitioning). This example partitioning scheme may be designed to support more flexibility for CB partition shapes, such that both prediction and transformation can be performed for CB levels without further partitioning. In such a coding tree structure, the CB may have a square or rectangular shape. Specifically, the coding tree block (CTB) may be first partitioned by a quadtree structure. Then, the quadtree leaf nodes may be further partitioned by a nested multi-type tree structure. An example of a nested multi-type tree structure using 2 or 3 splits is shown in FIG. 11. Specifically, the example multi-type tree structure in FIG. 11 includes four split types, called vertical 2 split (SPLIT_BT_VER) (1102), horizontal 2 split (SPLIT_BT_HOR) (1104), vertical 3 split (SPLIT_TT_VER) (1106), and horizontal 3 split (SPLIT_TT_HOR) (1108). Thus, CB corresponds to the leaf of the multi-type tree. In this implementation example, this segmentation is used for both prediction and transformation processing without any further partitioning, as long as CB is not too large for the maximum transform length. This means that in most cases, in a quad-tree with a nested multi-type tree coding block structure, CB, PB, and TB have the same block size. An exception occurs when the maximum supported transform length is smaller than the width or height of the color components of CB. In some implementations, in addition to the 2 or 3 division, the nested pattern of FIG. 11 may further include a quadtree division.

[0127]

[0154] One specific example of a quadtree with nested multi-type tree coding block structure of block partitions (including quadtree, 2, and 3 split options) for one base block is shown in FIG. 12. More specifically, FIG. 12 shows a base block 1200 quadtree partitioned into four square partitions 1202, 1204, 1206, and 1208. The multi-type tree structure of FIG. 11 and the decision to further use the quadtree for further partitioning are made for each of the quadtree partition partitions. In the example of FIG. 12, partition 1204 is not further partitioned. Partitions 1202 and 1208 each adopt another quadtree partition. For partition 1202, the second level quadtree partitioned top-left, top-right, bottom-left, and bottom-right partitions adopt the third level partition of the quadtree, horizontal 2 partition 1104 of FIG. 11, no partition, and horizontal 3 partition 1108 of FIG. 11, respectively. Partition 1208 employs another quadtree partitioning, with the second level quadtree partitioned top-left, top-right, bottom-left, and bottom-right partitions employing a third level partitioning of vertical 3-partition 1106 in FIG. 11, no partitioning, no partitioning, and horizontal 2-partition 1104 in FIG. 11, respectively. Two of the sub-partitions of the top-left partition in the third level of 1208 are further partitioned according to horizontal 2-partition 1104 and horizontal 3-partition 1108 in FIG. 11, respectively. Partition 1206 employs a second level partitioning pattern that partitions into two partitions according to vertical 2-partition 1102 in FIG. 11, with each partition further partitioned according to horizontal 3-partition 1108 and vertical 2-partition 1102 in FIG. 11 at the third level. A fourth level partitioning is further applied to one of them according to horizontal 2-partition 1104 in FIG. 11.

[0128]

[0155] As a specific example of the above, the maximum luma transform size may be 64×64 and the maximum supported chroma transform size may be different from the luma, e.g., 32×32. Even if the above example CB of FIG. 12 is not generally further divided into smaller PBs and / or TBs, when the width or height of a luma coding block or a chroma coding block is greater than the maximum transform width or height, the luma coding block or the chroma coding block may be automatically divided in the horizontal and / or vertical directions to meet the transform size limitation in that direction.

[0129]

[0156] In the specific example of partitioning base blocks into CBs as described above and as described above, the coding tree scheme may support the ability for luma and chroma to have separate block tree structures. For example, for P and B slices, the luma and chroma CTBs in one CTU may share the same coding tree structure. For I slices, for example, the luma and chroma may have separate coding block tree structures. When separate block tree structures are applied, the luma CTB may be partitioned into luma CBs by one coding tree structure, and the chroma CTB is partitioned into chroma CBs by another coding tree structure. This means that a CU in an I slice may consist of a coding block of a luma component, or a coding block of two chroma components, and a CU in a P or B slice always consists of coding blocks of all three color components, unless the video is monochrome.

[0130]

[0157] When a coding block is further partitioned into multiple transform blocks, the transform blocks may be arranged in the bitstream according to various orders or scan manners. Implementation examples of partitioning of coding blocks or predictive blocks into transform blocks and the coding order of the transform blocks are described in further detail below. In some implementations, as described above, the transform partitioning may support multiple shapes of transform blocks, such as 1:1 (square), 1:2 / 2:1, and 1:4 / 4:1, with transform block sizes ranging from 4×4 to 64×64, for example. In some implementations, when the coding block is smaller than or equal to 64×64, the transform block partitioning may be applied only to the luma component, in which case, for chroma blocks, the transform block size is the same as the coding block size. Otherwise, if the coding block width or height is greater than 64, both the luma and chroma coding blocks may be implicitly partitioned into multiples of min(W,64)×min(H,64) and min(W,32)×min(H,32) transform blocks, respectively.

[0131]

[0158] In some implementations of transform block partitioning, for both intra and inter coded blocks, the coding blocks may be further partitioned into multiple transform blocks with partition depth up to a predefined number of levels (e.g., 2 levels). The transform block partition depth and size may be related. For some implementations, the mapping from the transform size of the current depth to the transform size of the next depth is shown below in Table 1.

[0132] [Table 1]

[0133]

[0159] Based on the example mapping of Table 1, for a 1:1 square block, the next level transform partition may produce four 1:1 square sub-transform blocks. The transform partition may stop at, for example, 4×4. Thus, a transform size of 4×4 at the current depth corresponds to the same size of 4×4 at the next depth. In the example of Table 1, for a 1:2 / 2:1 non-square block, the next level transform partition may produce two 1:1 square sub-transform blocks, while for a 1:4 / 4:1 non-square block, the next level transform partition may produce two 1:2 / 2:1 sub-transform blocks.

[0134]

[0160] In some implementations, additional restrictions on transform block partitioning may be applied to the luma components of intra-coded blocks. For example, at each level of transform partitioning, all sub-transform blocks may be restricted to be of equal size. For example, for a 32×16 coded block, level 1 transform partitioning creates two 16×16 sub-transform blocks, and level 2 transform partitioning creates eight 8×8 sub-transform blocks. In other words, the second level partitioning must be applied to all first level sub-blocks to keep the transform units of equal size. An example of transform block partitioning for an intra-coded square block according to Table 1 is shown in FIG. 15 with the coding order illustrated by the arrows. Specifically, 1502 shows a square coded block. The first level partitioning into four equal-sized transform blocks according to Table 1 is shown in 1504 with the coding order indicated by the arrows. All second level partitioning of the first level equal-sized blocks into 16 equal-sized transform blocks according to Table 1 is shown in 1506 with the coding order indicated by the arrows.

[0135]

[0161] In some implementations, the above restrictions on intra-coding may not apply for the luma components of an inter-coded block. For example, after the first level of transform division, any one of the sub-transform blocks may be further divided independently at another level. Thus, the resulting transform blocks may or may not be of the same size. An example division of an inter-coded block into transform blocks in its coding order is shown in FIG. 16. In the example of FIG. 16, an inter-coded block 1602 is divided into transform blocks at two levels according to Table 1. At the first level, the inter-coded block is divided into four transform blocks of equal size. Then, only one of the four transform blocks (but not all of the transform blocks) is further divided into four sub-transform blocks, resulting in a total of seven transform blocks having two different sizes, as shown by 1604. An example coding order of these seven transform blocks is shown by the arrows at 1604 in FIG. 16.

[0136]

[0162] In some implementations, for chroma components, some additional restrictions on transform blocks may be applied: for example, for chroma components, the transform block size can be as large as the coding block size, but cannot be smaller than a predefined size, e.g., 8×8.

[0137]

[0163] In some other implementations, for coding blocks with width (W) or height (H) greater than 64, both luma and chroma coding blocks may be implicitly divided into multiples of min(W,64)×min(H,64) and min(W,32)×min(H,32) transform units, respectively, where in this disclosure, "min(a,b)" may return the smaller of a and b.

[0138]

[0164] Figure 17 further illustrates another alternative exemplary scheme for partitioning a coding block or a predictive block into transform blocks. Instead of using a recursive transform partitioning as illustrated in Figure 17, a predefined set of partitioning types may be applied to a coding block depending on the transform type of the coding block. In the particular example illustrated in Figure 17, one of six exemplary partitioning types may be applied to cause the coding block to be divided into a varying number of transform blocks. Such a scheme to generate a transform block partitioning may be applied to a coding block or a predictive block.

[0139]

[0165] More specifically, the partitioning scheme of FIG. 17 provides up to six exemplary partition types for any given transform type (transform type refers to the type of the main transform, e.g., ADST). In this scheme, a transform partition type may be assigned for each coding block or predictive block, e.g., based on a rate-distortion cost. In an example, the transform partition type assigned to a coding block or predictive block may be determined based on the transform type of the coding block or predictive block. A particular transform partition type may correspond to a transform block partition size and pattern, as illustrated by the six transform partition types illustrated in FIG. 17. The correspondence between various transform types and various transform partition types may be predefined. An example is shown below, with capitalized labels indicating transform partition types that may be assigned to a coding block or predictive block based on a rate-distortion cost.

[0140]

[0166] · PARTITION_NONE: Allocates a transformation size equal to the block size.

[0141]

[0167] · PARTITION_SPLIT: Allocates a transformation size that is 1 / 2 the width of the block size, and 1 / 2 the height of the block size.

[0142]

[0168] · PARTITION_HORZ: Allocate a transformation size that has the same width as the block size and 1 / 2 the height of the block size.

[0143]

[0169] · PARTITION_VERT: Allocates a transformation size that has a width of 1 / 2 the block size and a height equal to the block size.

[0144]

[0170] · PARTITION_HORZ4: Allocates a transformation size that has the same width as the block size and 1 / 4 of the height of the block size.

[0145]

[0171] · PARTITION_VERT4: Allocates a transformation size that has a width of 1 / 4 of the block size and a height equal to the block size.

[0146]

[0172] In the above example, all transform partition types as shown in Figure 17 include a uniform transform size for the partitioned transform blocks. This is merely an example and not a limitation. In some other implementations, mixed transform block sizes may be used for the partitioned transform blocks of a particular partition type (or pattern).

[0147]

[0173] The PBs (or CBs, also called PBs when not further partitioned into predictive blocks) obtained from any of the above partitioning schemes can then become individual blocks for encoding via intra or inter prediction. In the case of inter prediction for the current PB, a residual between the current block and the predictive block may be generated, coded, and included in the coded bitstream.

[0148]

[0174] Inter prediction may be performed, for example, in single reference mode or mixed reference mode. In some implementations, a skip flag may be included initially in the bitstream for the current block (or at a higher level) to indicate whether the current block is inter-coded and should not be skipped. If the current block is inter-coded, another flag may be further included in the bitstream as a signal to indicate whether single reference mode or mixed reference mode is used for prediction of the current block. In the case of single reference mode, one reference block may be used to generate a prediction block for the current block. In the case of mixed reference mode, two or more reference blocks may be used to generate a prediction block, for example by weighted averaging. The mixed reference mode may also be referred to as two or more reference mode, two reference mode, or multiple reference mode. One or more reference blocks may be identified using one or more reference frame indexes and additionally using corresponding one or more motion vectors, which indicate a shift between the reference block and the current block in location relative to the frame, for example horizontal and vertical pixels. For example, an inter-predicted block for a current block may be generated from a single reference block identified by one motion vector in a reference frame as the predictive block in single reference mode, while in case of a mixed reference mode, the predictive block may be generated by a weighted average of two reference blocks in two reference frames indicated by two reference frame indexes and two corresponding motion vectors. The motion vectors may be coded and included in the bitstream in various manners.

[0149]

[0175] In some implementations, an encoding or decoding system may maintain a decoded picture buffer (DPB). Some images / pictures may be kept in the DPB waiting to be displayed (in the decoding system), and some images / pictures in the DPB may be used as reference frames to enable inter prediction (in the decoding or encoding system). In some implementations, reference frames in the DPB may be tagged as short-term or long-term references for the current image being encoded or decoded. For example, short-term reference frames may include frames used for inter prediction of blocks in the current frame or in a predefined number (e.g., two) of subsequent video frames closest to the current frame in decoding order. Long-term reference frames may include frames in the DPB that can be used to predict image blocks in frames that are more than a predefined number of frames away from the current frame in decoding order. Information about such tags for short-term and long-term reference frames may be referred to as a Reference Picture Set (RPS) and may be added to the header of each frame in the encoded bitstream. Each frame in the encoded video stream may be identified by a Picture Order Counter (POC), which is either absolutely numbered according to the playback sequence or with respect to a group of pictures starting with, for example, an I-frame.

[0150]

[0176] In some implementations, one or more reference picture lists including identification of short-term and long-term reference frames for inter prediction may be formed based on information in the RPS. For example, a single picture reference list may be formed for unidirectional inter prediction and denoted as L0 reference (or reference list 0), while two picture reference lists may be formed for bidirectional inter prediction and denoted as L0 (or reference list 0) and L1 (or reference list 1) for each of the two prediction directions. The reference frames included in the L0 and L1 lists may be ordered in various predetermined manners. The lengths of the L0 and L1 lists may be signaled in the video bitstream. The unidirectional inter prediction may be a single reference mode, or may be a combined reference mode when multiple references for generating a prediction block by weighted averaging in a combined prediction mode are on the same side of a frame where a block to be predicted is located. The bidirectional inter prediction may be only a combined mode, since bidirectional inter prediction involves at least two reference blocks.

[0151]

[0177] In some implementations, a merge mode (MM) for inter prediction may be performed. In general, for merge mode, one or more of the motion vectors in single reference prediction or the motion vectors in mixed reference prediction for the current PB may be derived from other motion vectors instead of being calculated and signaled independently. For example, in an encoding system, a current motion vector of a current PB may be represented by a difference between the current motion vector and one or more other already encoded motion vectors (called reference motion vectors). Such a difference of the motion vector, rather than the entire current motion vector, may be encoded and included in the bitstream and linked to the reference motion vector. Correspondingly, in a decoding system, a motion vector corresponding to the current PB may be derived based on the decoded motion vector difference and the decoded reference motion vector linked thereto. As an inherent aspect of the general merge mode (MM) inter prediction, such inter prediction based on the motion vector difference may be referred to as merge mode with motion vector difference (MMVD). Thus, MM in general, or MMVD in particular, may be implemented to exploit the correlation between the motion vectors associated with different PBs to improve coding efficiency. For example, adjacent PBs may have similar motion vectors, and therefore the MVD may be small and efficiently codeable. As another example, the motion vectors may be temporally correlated (between frames) for blocks that are similarly spatially / located.

[0152]

[0178] In some implementations, an MM flag may be included in the bitstream during the encoding process to indicate whether the current PB is in merge mode. Additionally or alternatively, an MMVD flag may be included in the encoding process and signaled in the bitstream to indicate whether the current PB is in MMVD mode. MM and / or MMVD flags or indicators may be provided at the PB level, CB level, CU level, CTB level, CTU level, slice level, frame level, picture level, sequence level, etc. As a specific example, both an MM flag and an MMVD flag may be included for the current CU, and the MMVD flag may be signaled immediately after the skip flag and the MM flag to specify whether the MMVD mode is used for the current CU.

[0153]

[0179] In some implementations of MMVD, a list of reference motion vectors (RMVs), or MV predictor candidates for motion vector prediction, may be formed for the block being predicted. The list of RMV candidates may include a predetermined number (e.g., two) of MV predictor candidate blocks, whose motion vectors may be used to predict the current motion vector. The RMV candidate blocks may include blocks selected from adjacent blocks in the same frame and / or from temporal blocks (e.g., blocks equivalently placed in frames preceding or following the current frame). These options represent blocks in a spatial or temporal location relative to the current block that are likely to have a similar or identical motion vector to the current block. The size of the list of MV predictor candidates may be predetermined. For example, the list may include two or more candidates. To be on the list of RMV candidates, a candidate block may be required, for example, to have the same reference frame (or multiple reference frames) as the current block, must exist (e.g., a boundary check needs to be performed when the current block is near an edge of a frame), must have already been encoded during the encoding process, and / or must have already been decoded during the decoding process. In some implementations, the list of merge candidates may be populated first with spatially adjacent blocks (scanned in a certain predefined order) if available and if the above conditions are met, and then with temporal blocks if space is still available in the list. Adjacent RMV candidate blocks may be selected, for example, from the left and top blocks of the current block. The list of RMV predictor candidates may be dynamically formed at different levels (sequence, picture, frame, slice, superblock, etc.) as a Dynamic Reference List (DRL). The DRL may be signaled in the bitstream.

[0154]

[0180] In some implementations, the actual MV predictor candidate used as the reference motion vector for predicting the motion vector of the current block may be signaled. In the case where the RMV candidate list includes two candidates, a one-bit flag called a merge candidate flag may be used to indicate the selection of the reference merge candidate. If the current block is predicted in a mixed mode, each of the multiple motion vectors predicted using the MV predictor may be associated with a reference motion vector from the merge candidate list. The encoder may determine which of the RMV candidates more closely predicts the MV of the current coding block and signal the selection as an index to the DRL.

[0155]

[0181] In some implementations of MMVD, after an RMV candidate is selected and used as a base motion vector predictor for predicting a motion vector, a motion vector difference (MVD or delta MV representing the difference between the motion vector to be predicted and the reference candidate motion vector) may be calculated in the encoding system. Such an MVD may include information representing the magnitude of the MV difference and the direction of the MV difference, and both the magnitude of the MV difference and the direction of the MV difference may be signaled in the bitstream. The magnitude of the motion difference and the direction of the motion difference may be signaled in various manners.

[0156]

[0182] In some implementations of MMVD, a distance index may be used to specify the magnitude information of the motion vector difference and to indicate one of a set of predefined offsets representing predefined motion vector differences from a start point (reference motion vector). The MV offset according to the signaled index may then be added to the horizontal or vertical component of the start (reference) motion vector. Whether the horizontal or vertical component of the reference motion vector should be offset may be determined by the direction information of the MVD. An example predefined relationship between the distance index and the predefined offsets is specified in Table 2.

[0157] [Table 2]

[0158]

[0183] In some implementations of MMVD, a direction index may be further signaled and used to indicate the direction of the MVD relative to the reference motion vector. In some implementations, the direction may be limited to one of horizontal and vertical. An example 2-bit direction index is shown in Table 3. In the example of Table 3, the interpretation of the MVD may differ depending on the information of the start / reference MV. For example, when the start / reference MV corresponds to a uni-prediction block or a bi-prediction block where both reference frame lists point to the same side of the current picture (i.e., when the POCs of the two reference pictures are both greater than or both less than the POC of the current picture), the code in Table 3 may specify the sign (direction) of the MV offset added to the start / reference MV. When the start / reference MV corresponds to a bi-predictive block with two reference pictures on different sides of the current picture (i.e., the POC of one reference picture is larger than that of the current picture and the POC of the other reference picture is smaller than that of the current picture) and when the difference between the reference POC in picture reference list 0 and the current frame is larger than the difference between the reference POC in picture reference list 1 and the current frame, the code in Table 3 may specify the sign of the MV offset added to the reference MV corresponding to the reference picture in picture reference list 0, and the sign of the offset of the MV corresponding to the reference picture in picture reference list 1 may have the opposite value (the opposite sign of the offset). Otherwise, if the difference between the reference POC in picture reference list 1 and the current frame is larger than the difference between the reference POC in picture reference list 0 and the current frame, the code in Table 3 may specify the sign of the MV offset added to the reference MV associated with picture reference list 1, and the sign of the offset to the reference MV associated with picture reference list 0 has the opposite value.

[0159] [Table 3]

[0160]

[0184] In some implementations, the MVD may be scaled according to the POC difference per direction. If the POC difference in both lists is the same, no scaling is required. Otherwise, if the POC difference in reference list 0 is greater than that of reference list 1, the MVD for reference list 1 is scaled. If the POC difference in reference list 1 is greater than list 0, the MVD for list 0 may be scaled in the same manner. If the start MV is uni-predicted, the MVD is added to the available MV or the reference MV.

[0161]

[0185] In some implementations of MVD coding and signaling for bidirectional composite prediction, in addition to or as an alternative to coding and signaling two MVDs separately, symmetric MVD coding may be implemented such that only one MVD needs to be signaled and the other MVD may be derived from the signaled MVD. In such implementations, motion information including reference picture indexes of both list 0 and list 1 is signaled. However, for example, only the MVD associated with reference list 0 is signaled, and the MVD associated with reference list 1 is derived without being signaled. Specifically, at the slice level, a flag called "mvd_l1_zero_flag" may be included in the bitstream to indicate whether reference list 1 is not signaled in the bitstream. If this flag is 1, indicating that reference list 1 is equal to zero (and therefore not signaled), a bidirectional prediction flag called "BiDirPredFlag" may be set to 0, meaning there is no bidirectional prediction. Otherwise, if mvd_l1_zero_flag is zero, then if the closest reference picture in list 0 and the closest reference picture in list 1 form a forward-backward pair of reference pictures or a backward-forward pair of reference pictures, then BiDirPredFlag may be set to 1, and the reference pictures in both list 0 and list 1 are short-term reference pictures. Otherwise, BiDirPredFlag is set to 0. BiDirPredFlag=1 may indicate that a symmetric mode flag is additionally signaled in the bitstream. The decoder may extract the symmetric mode flag from the bitstream when BiDirPredFlag is 1. The symmetric mode flag may be signaled, for example, at the CU level (if necessary) and may indicate whether a symmetric MVD coding mode is used for the corresponding CU.When the symmetric mode flag is 1, it indicates the use of a symmetric MVD encoding mode and that only the reference picture indices for both list 0 and list 1 (called "mvp_l0_flag" and "mvp_l1_flag") are signaled along with the MVD associated with list 0 (called "MVD0"), and that the other motion vector difference "MVD1" is derived rather than signaled. For example, MVD1 may be derived as -MVD0. Thus, in the example symmetric MVD mode, only one MVD is signaled.

[0162]

[0186] In some other implementations of MV prediction, a harmonized scheme may be used to implement general merge-mode MMVD and some other types of MV prediction for both single-reference mode MV prediction and mixed-reference mode MV prediction. Various syntax elements may be used to signal the manner in which the MV of the current block is predicted.

[0163]

[0187] For example, for single criterion mode, the following MV prediction modes may be signaled:

[0164]

[0188] NEARMV - Uses one of the motion vector predictors (MVP) in the list directly pointed to by the DRL (Dynamic Reference List) index without any MVD.

[0165]

[0189] NEWMV - Use one of the motion vector predictors (MVP) in the list signaled by the DRL index as a reference and apply a delta to the MVP (eg, using MVD).

[0166]

[0190] GLOBALMV - Use motion vectors based on frame-level global motion parameters.

[0167]

[0191] Similarly, if a mixed reference inter prediction mode uses two reference frames corresponding to the two MVs to be predicted, the following MV prediction modes may be signaled:

[0168]

[0192] NEAR_NEARMV - Use one of the motion vector predictors (MVPs) in the list signaled by the DRL index without the MVD for each of two of the MVs to be predicted.

[0169]

[0193] NEAR_NEWMV - to predict the first of two motion vectors, use one of the motion vector predictors (MVP) in the list signaled by the DRL index as a reference MV without MVD, and to predict the second of two motion vectors, use one of the motion vector predictors (MVP) in the list signaled by the DRL index as a reference MV with the delta MV (MVD) signaled as an additional signal.

[0170]

[0194] NEW_NEARMV - to predict the second of two motion vectors, use one of the motion vector predictors (MVP) in the list signaled by the DRL index as a reference MV without MVD, and to predict the first of two motion vectors, use one of the motion vector predictors (MVP) in the list signaled by the DRL index as a reference MV with the delta MV (MVD) signaled as an addition.

[0171]

[0195] NEW_NEWMV - Use one of the motion vector predictors (MVPs) in the list signaled by the DRL index as a reference MV and use this together with the delta MV signaled as an addition to predict for each of the two MVs.

[0172]

[0196] GLOBAL_GLOBALMV - Use the MV from each reference based on its frame-level global motion parameters.

[0173]

[0197] Thus, the term "NEAR" above refers to MV prediction using a reference MV without any MVD as a general merge mode, while the term "NEW" refers to MV prediction with the use of a reference MV and offsetting the reference MV with a signaled or derived MVD as in the MMVD mode. In the case of composite inter prediction, for example, the two MVDs may be correlated, and such correlation may be exploited to reduce the amount of information required to signal two motion vector deltas, but both the reference base motion vector and the motion vector delta may generally be different or unrelated between the two references or two MVDs. To exploit such correlation, joint signaling of the two MVDs may be implemented and indicated in the bitstream, as described in more detail below.

[0174]

[0198] In some implementations of the MVD, a predefined pixel resolution for the MVD may be allowed. For example, 1 / 8 pixel motion vector accuracy (or precision) may be allowed. The above MVDs for various MV prediction modes may be constructed and signaled in various manners. In some implementations, various syntax elements may be used to signal the above motion vector differences in the reference frame list 0 or list 1.

[0175]

[0199] For example, a syntax element called "mv_joint" may specify which components of the associated motion vector difference are non-zero. In the case of MVD, this is signaled for all non-zero components together. For example, mv_joint has the following values: A 0 may indicate no non-zero MVD along the horizontal or vertical directions. A 1 may indicate that there is non-zero MVD only along the horizontal direction. 2 may indicate that there is non-zero MVD only along the vertical direction. 3 may indicate that there is non-zero MVD along both the horizontal and vertical directions.

[0176]

[0200] When the "mv_joint" syntax element for MVD signals that there are no non-zero MVD components, no further MVD information may be signaled. However, if the "mv_joint" syntax signals that there are one or two non-zero components, additional syntax elements may be further signaled for each non-zero MVD component, as described below.

[0177]

[0201] For example, a syntax element called "mv_sign" may be used to additionally specify whether the corresponding motion vector difference component is positive or negative.

[0178]

[0202] As another example, a syntax element called "mv_class" may be used to specify a class of a motion vector difference from among a set of predefined classes for the corresponding non-zero MVD component. The predefined classes for the motion vector difference may be used, for example, to divide a continuous magnitude space of the motion vector difference into non-overlapping ranges, each range corresponding to an MVD class. Thus, the signaled MVD class indicates the magnitude range of the corresponding MVD component. In the implementation example shown in Table 4 below, higher classes correspond to motion vector differences having a larger magnitude range. In Table 4, the symbol (n,m] is used to represent a range of motion vector differences greater than n pixels and less than or equal to m pixels.

[0179] [Table 4]

[0180]

[0203] In some other examples, a syntax element called "mv_bit" may further be used to specify the integer part of the offset between the non-zero motion vector difference component and the start magnitude of the correspondingly signaled MV class magnitude range. Thus, mv_bit may indicate the magnitude or magnitude of the MVD. The number of bits required in "my_bit" to signal the full range per MVD class may vary depending on the MV class. As an example, MV_CLASS0 and MV_CLASS1 in the implementation of Table 4 may require only a single bit to indicate 1 or 2 integer pixel offsets from 0 of the start MVD, and each higher MV_CLASS in the implementation of Table 4 may require progressively one more bit for "mv_bit" than the previous MV_CLASS.

[0181]

[0204] In some other examples, a syntax element called "mv_fr" may be further used to specify the first two fractional bits of the motion vector difference for the corresponding non-zero MVD component, while a syntax element called "mv_hp" may be used to specify the third fractional bit (high resolution bit) of the motion vector difference for the corresponding non-zero MVD component. The two bits of "mv_fr" may essentially provide a ¼ pixel MVD resolution, while the "mv_hp" bits may further provide a ⅛ pixel resolution. In some other implementations, two or more "mv_hp" bits may be used to provide an MVD pixel resolution finer than ⅛ pixel. In some implementations, additional flags may be signaled at one or more of various levels to indicate whether ⅛ pixel or higher MVD resolution is supported. If an MVD resolution does not apply to a particular coding unit, the above syntax elements for the corresponding non-supported MVD resolution may not be signaled.

[0182]

[0205] In some implementations of the above, the fractional resolution may be independent of different MVD classes. In other words, a similar option for motion vector resolution may be provided using a predefined number of "mv_fr" and "mv_hp" bits to signal fractional MVD of non-zero MVD components regardless of the magnitude of the motion vector difference.

[0183]

[0206] However, in some other implementations, the resolution of the motion vector difference in various MVD magnitude classes may be differentiated or adaptive. Specifically, a high-resolution MVD for a large MVD magnitude of a higher MVD class may not bring a statistically significant improvement in compression efficiency or coding gain. Therefore, the MVD may be coded with a reduced resolution (integer pixel resolution or fractional pixel resolution) for a larger MVD magnitude range corresponding to a higher MVD magnitude class. In the same way, the MVD may be coded with a reduced resolution (integer pixel resolution or fractional pixel resolution) for a larger MVD value in general. Such MVD class-dependent or MVD magnitude-dependent MVD resolution may be generally referred to as an adaptive MVD resolution, a magnitude-dependent adaptive MVD resolution, or a magnitude-dependent MVD resolution. The term "resolution" may also be referred to as "pixel resolution". The adaptive MVD resolution may be realized in various manners, as described by the following implementations, to achieve better compression efficiency overall. In particular, due to the statistical observation that treating the MVD resolution of a large magnitude or high class MVD at a similar level to that of a small magnitude or low class MVD in a non-conforming manner may not significantly improve the inter-prediction residual coding efficiency of blocks with a large magnitude or high class MVD, the reduction in the number of signaling bits by aiming for a less precise MVD may be greater than the additional bits required to code the inter-prediction residual as a result of such a less precise MVD. In other words, using a higher MVD resolution for a large magnitude or high class MVD may not produce more coding gain than using a lower MVD resolution.

[0184]

[0207] In some typical implementations, the pixel resolution or precision of the MVD may decrease or not increase as the MVD class increases. Decreasing the pixel resolution of the MVD corresponds to a coarser MVD (or a larger step from one MVD level to the next MVD level). In some implementations, the correspondence between MVD pixel resolution and MVD class may be specified, predefined, or preconfigured, and thus may not need to be signaled in the encoded bitstream.

[0185]

[0208] In some implementations, the MV classes in Table 3 may each be associated with a different MVD pixel resolution.

[0186]

[0209] In some implementations, each MVD class may be associated with a single allowed resolution. In some other implementations, one or more MVD classes may each be associated with two or more optional MVD pixel resolutions. A signal in the bitstream for a current MVD component having such an MVD class may be followed by additional signaling to indicate the optional pixel resolution selected for the current MVD component. In some implementations, the adaptively allowed MVD pixel resolutions may include, but are not limited to, (in descending order of resolution) 1 / 64-pel (pixel), 1 / 32-pel, 1 / 16-pel, 1 / 8-pel, 1-4-pel, 1 / 2-pel, 1-pel, 2-pel, 4-pel, etc. Thus, each one of the ascending MVD classes may be associated with one of these MVD pixel resolutions in non-ascending order. In some implementations, an MVD class may be associated with two or more of the above resolutions, with the higher resolution being lower than or equal to the lower resolution of the preceding MVD class. For example, if MV_CLASS_3 in Table 4 is associated with optional 1-pel and 2-pel resolutions, then the highest resolution with which MV_CLASS_4 in Table 4 may be associated should be 2-pel. In some other implementations, the highest allowable resolution of an MV class may be higher than the lowest allowable resolution of the preceding (lower) MV class. However, the average of the allowable resolutions of ascending MV classes may simply be in non-ascending order.

[0187]

[0210] In some implementations, when fractional pixel resolution higher than 1 / 8 pel is allowed, the "mv_fr" and "mv_hp" signaling may be correspondingly expanded to a total of more than 3 fractional bits.

[0188]

[0211] In some implementations, fractional pixel resolution may be allowed only for MVD classes below or equal to the threshold MVD class. For example, fractional pixel resolution may be allowed only for MVD_class_0 and not for all other MV classes in Table 4. Similarly, fractional pixel resolution may be allowed only for MVD classes below or equal to any one of the other MV classes in Table 4. For other MVD classes above the threshold MVD class, only integer pixel resolution of the MVD is allowed. In this manner, fractional resolution signaling, such as one or more of the "mv-fr" and / or "mv-hp" bits, may not need to be signaled for MVDs signaled with MVD classes above or equal to the threshold MVD class. For MVD classes having a resolution lower than 1 pixel, the number of bits of the "mv-bit" signaling may be further reduced. For example, for MV_CLASS_5 in Table 4, the range of MVD pixel offsets is (32,64], and therefore 5 bits are required to signal the entire range with 1-pel resolution. However, if MV_CLASS_5 is associated with a 2-pel MVD resolution (a lower resolution than 1-pel resolution), then 4 bits instead of 5 bits may be required for "mv-bit", and none of "mv-fr" and "mv-hp" need to be signaled following the signaling of "mv_class" as MV-CLASS_5.

[0189]

[0212] In some implementations, fractional pixel resolution may only be allowed for MVDs with integer values ​​below a threshold integer pixel value. For example, fractional pixel resolution may only be allowed for MVDs smaller than 5 pixels. Corresponding to this example, fractional resolution may be allowed for MV_CLASS_0 and MV_CLASS_1 in Table 4, and not allowed for all other MV classes. As another example, fractional pixel resolution may only be allowed for MVDs smaller than 7 pixels. Corresponding to this example, fractional resolution may be allowed for MV_CLASS_0 and MV_CLASS_1 in Table 4 (having a range below 5 pixels), and not allowed for MV_CLASS_3 and above (having a range above 5 pixels). For MVDs belonging to MV_CLASS_2, whose pixel range encompasses 5 pixels, fractional pixel resolution of the MVD may or may not be allowed depending on the "mv-bit" value. Fractional pixel resolution may be allowed if the "m-bit" value is signaled as 1 or 2 (so the integer part of the signaled MVD is 5 or 6 calculated as the start of the pixel range of MV_CLASS_2 with offset 1 or 2 as indicated in "m-bit"). Otherwise, if the "mv-bit" value is signaled as 3 or 4 (so that the integer part of the signaled MVD is 7 or 8), fractional pixel resolution may not be allowed.

[0190]

[0213] In some other implementations, for MV classes equal to or greater than the threshold MV class, only a single MVD value may be allowed. For example, such a threshold MV class may be MV_CLASS2. Thus, MV_CLASS_2 and above may only be allowed to have a single MVD value and may not have fractional pixel resolution. The single allowed MVD value for these MV classes may be predefined. In some examples, the single allowed value may be the high end value of the respective range for these MV classes in Table 4. For example, MV_CLASS_2 through MV_CLASS_10 may be greater than or equal to the threshold class of MV_CLASS2, and the single allowed MVD values ​​for these classes may be predefined as 8, 16, 32, 64, 128, 256, 512, 1024, and 2048, respectively, as shown in Table 5. In some other examples, the single allowed value may be the center value of the respective range for these MV classes in Table 4. For example, MV_CLASS_2 through MV_CLASS_10 may be above the class threshold, and the single allowed MVD values ​​for these classes may be predefined as 3, 6, 12, 24, 48, 96, 192, 384, 768, and 1536, respectively. Any other value within the range may also be defined as the single allowed resolution for the respective MVD class.

[0191] [Table 5]

[0192]

[0214] In the above implementation, when the signaled "mv_class" is equal to or exceeds a predefined MVD class threshold, the "mv_class" signaling alone is sufficient to determine the MVD value. The magnitude and direction of the MVD should then be determined using "mv_class" and "mv_sign".

[0193]

[0215] Thus, when MVD is signaled for only one reference frame (from reference frame list 0 or list 1, but not from both), or when MVD is signaled for two reference frames together, the precision (or resolution) of the MVD may be a function of the associated motion vector difference class in Table 3 and / or the magnitude of the MVD. Various other adaptable MVD resolution schemes depending on the MVD magnitude or class are envisioned.

[0194]

[0216] Returning to the various composite inter prediction modes in which each MV may be predicted by a reference motion vector and coded by an MVD, the two MVDs may be signaled separately or together in the bitstream as described above. Thus, in some implementations, in addition to the above-mentioned NEAR_NEARMV, NEAR_NEWMV, NEW_NEARMV, NEW_NEWMV, and GLOBAL_GLOBALMV modes, another inter prediction mode called JOINT_NEWMV may be introduced for the mode in which the MVDs of reference list 0 and reference list 1 are signaled together. Specifically, when the inter prediction mode is indicated as NEW_NEWMV, the MVDs of reference list 0 and reference list 1 are signaled separately, while when the inter prediction mode is indicated as JOINT_NEWMV mode, the MVDs of reference list 0 and reference list 1 are signaled together. In particular, for joint MVD, only one MVD, called joint_delta_mv, may need to be signaled and transmitted in the bitstream, and the MVDs of reference list 0 and reference list 1 may be derived from joint_delta_mv. The derived MVD may then be combined with the reference motion vectors in reference list 0 or reference list 1 to generate two motion vectors for locating reference blocks for hybrid inter prediction.

[0195]

[0217] In some implementations of composite inter prediction, the JOINT_NEWMV mode may be signaled together with the NEAR_NEARMV, NEAR_NEWMV, NEW_NEARMV, NEW_NEWMV, and GLOBAL_GLOBALMV modes. In such implementations, syntax for an indication of any one of these alternative composite inter prediction modes may be included in the bitstream at any one of various signaling levels (e.g., sequence level, picture level, frame level, slice level, tile level, superblock level, etc.). Alternatively, the JOINT_NEWMV mode may be implemented as a sub-mode of the NEW_NEWMV mode. In other words, during the NEW_NEWMV mode, the two MVDs of the two reference blocks are either signaled together (thus being a JOINT_NEWMV sub-mode) or not signaled (being another sub-mode of the NEW_NEWMV mode). In such an implementation, a first syntax element may be included in the bitstream to indicate any one of the NEAR_NEARMV, NEAR_NEWMV, NEW_NEARMV, NEW_NEWMV, and GLOBAL_GLOBALMV modes, and when the first syntax element indicates that the NEW_NEWMV mode has been selected for a coding block, a second syntax element may be further included in the bitstream and extractable by a decoder to indicate whether the MVDs of the coding block are signaled separately or together.

[0196]

[0218] For implementations of joint MVD in composite inter prediction, the MVD associated with the reference MV may be derived from a signaled joint MVD from the bitstream, such as the joint_delta_mv described above. Such derivation may involve, for example, scaling of the signaled joint MVD to obtain one or both of the two MVDs. In other words, the signaled joint MVD may be scaled before being added to the motion vector predictor (MVP) or the reference MV. As a result of the scaling, the precision or pixel resolution of the scaled MVD may differ from the allowed precision of the motion vector difference. In some implementations, such MVDs scaled from the jointly signaled MVDs may first be quantized to the allowed precision of the MVD of the current picture or slice or tile or superblock or coded block before being added to the reference MVD for generating the motion vector.

[0197]

[0219] In some implementations, the frame index of the reference frame in the composite inter prediction mode may be signaled in the bitstream. The frame index may correspond to a picture order counter (POC) associated with the reference frame. The distance between the reference frame and the current frame may be defined and expressed as the difference between the corresponding POCs. The direction of the reference frame (before or after the current frame) may be expressed by a code. Thus, the signed distance may be used to represent the position of the reference frame relative to the current frame. The reference frames for the composite inter prediction mode may be referred to as a first reference frame and a second reference frame.

[0198]

[0220] In some implementations, when JOINT_NEWMV mode is signaled and the POC distances between the two reference frames and the current frame are different, the MVD may be scaled for reference list 0 or reference list 1 based on the POC distance. Specifically, the distance between reference frame list 0 and the current frame may be represented as td0, and the distance between reference frame list 1 and the current frame may be represented as td1. When td0 is equal to or greater than td1, joint_mvd may be used directly for reference list 0, and the MVD of reference list 1 may be derived from joint_mvd based on equation (1).

[0199]

number

[0200]

[0221] Otherwise, if td1 is equal to or greater than td0, then joint_mvd is used directly for base list 1, and the MVD for base list 0 is derived from joint_mvd based on equation (2).

[0201]

number

[0202]

[0222] Returning to the above-mentioned motion vector predictor (MVP) and MVP list, in some implementations, such a motion vector predictor list may be established for a group of coding blocks (e.g., a superblock) that includes motion vector candidates for predicting a motion vector for each predictive block of the superblock. The maximum size of the MVP list may be predefined or configured. The candidates in the MVP list may be established using a predefined set of rules. For example, these candidates may be selected from already reconstructed motion vectors that belong to blocks that are spatially close to the current coding block or superblock (referred to as spatial motion vector predictors, or SMVPs). Alternatively or additionally, these candidates may belong to blocks in a reference frame of the current coding block or superblock (referred to as temporal motion vector predictors, or TMVPs). The spatial motion vector predictor may be an adjacent or non-adjacent SMVP. An adjacent SMVP may refer to a motion vector predictor that belongs to a predictive block that is adjacent to the current coding block or superblock. A non-adjacent SMVP may refer to a motion vector predictor that belongs to a predictive block that is not directly adjacent to the current coding block or superblock. Other types of MVP candidates may also be derived from the reconstructed motion vectors. As another example, one or more additional MVP banks may be maintained as one of the sources for establishing the MVP list, as described in more detail below.

[0203]

[0223] The MVP list may be constructed to hold a predetermined number of reconstructed MVP candidates (SMVP, TMVP, or other derived MVPs, or other types of MVP candidates) on both the encoder side and the decoder side of the current coding block or superblock. When encoding the current prediction block in inter prediction mode, the encoder should select an MVP that brings optimal coding efficiency as a predictor of a motion vector for the current prediction block from the candidates in the MVP candidate list. The index of the selected MVP in the MVP list may be signaled in the bitstream. Correspondingly, the decoder should update the MVP list for the current coding block or superblock at the same time that the bitstream is reconstructed, extract the MVP index of the current inter predicted prediction block, obtain the MVP from the MVP candidate list according to the extracted MVP index in the MVP list, and use the MVP as a predictor of the motion vector of the current prediction block to reconstruct the motion vector of the current prediction block (e.g., by combining the motion vector predictor extracted from the MVP list with the corresponding MVD). For example, the MVP list may represent a stack with a predetermined fixed size.

[0204]

[0224] For example, the SMVP may be derived from spatially proximate prediction blocks, including spatially proximate and adjacent prediction blocks and spatially proximate and non-adjacent blocks, where the spatially proximate and adjacent prediction blocks are directly adjacent to the current block or superblock at the top or left of the block or superblock (assuming that these are previous blocks that have already been reconstructed), and the spatially proximate and non-adjacent blocks are close to but not directly adjacent to the current block or superblock. An example of a set of spatially proximate prediction blocks for a luma block or superblock is illustrated in Figure 18.

[0205]

[0225] FIG. 18 illustrates a superblock 1802, which in this example includes 16 predictive blocks (each predictive block is inter-predicted). For example, each predictive block may be an 8×8 block. The superblock 1802 may be associated with various adjacent and non-adjacent predictive blocks, which are illustrated by various smaller squares around the superblock 1802 in FIG. 18. Only the adjacent predictive blocks at the top and left are illustrated, since these predictive blocks represent predictive blocks that have already been reconstructed (e.g., in a decoder or in a loop decoding unit of an encoder). The adjacent predictive blocks shaded in FIG. 18 represent adjacent and adjacent predictive blocks, while the other predictive blocks represent adjacent and non-adjacent predictive blocks of the current superblock 1802.

[0206]

[0226] In some implementations, to determine whether any of the motion vectors of the spatially adjacent blocks of FIG. 18 should be considered for entry in the MVP candidate list, these adjacent predictive blocks may be examined or searched against a predefined set of rules. For example, these adjacent predictive blocks may be examined or searched to determine whether they are associated with the same reference frame index (for inter prediction) as the current predictive block in the superblock 1802. If they do not share the same reference frame index as the current predictive block, these motion vectors may not be eligible for entry in the MVP candidate list. Since the size of the MVP candidate list is limited (e.g., 4 or other number of candidates), these adjacent predictive blocks are examined / searched and ranked in a predefined order. The search order may be predefined. An example predefined search order is illustrated in FIG. 18, where the adjacent and adjacent predictive blocks at the top are examined first from left to right as indicated by arrow 1, and then the adjacent and adjacent predictive blocks at the left are examined from top to bottom as indicated by arrow 2. The neighboring prediction block in the upper right corner for the superblock 1801 is next examined, as indicated by arrow 3. After this, the neighboring prediction block in the upper left corner for the current superblock 1802 is examined, as indicated by arrow 4, followed by the neighboring prediction block in the second row on the top, the neighboring prediction block in the second column on the left, the neighboring prediction block in the third row on the top, and the neighboring prediction block in the third column on the left, in this example search order. The search order within the neighboring rows of prediction blocks may be from left to right, while the search order within the neighboring columns of prediction blocks may be from top to bottom. During the check and search process, the neighboring prediction blocks associated with the same reference frame as the current prediction block may be flagged as eligible for an entry in the MVP candidate list if there is still space available in the list. The terms MP candidate list and MP list are used interchangeably.

[0207]

[0227] In some implementations, adjacent SMVP candidates are first put into the MVP list before TMVP if the corresponding adjacent predictive blocks share the same reference frame as the current block. Non-adjacent SMVP candidates are put into the MVP list after TMVP if there is still space and if the corresponding adjacent predictive blocks share the same reference frame as the current block. Thus, in some implementations, all SMVP candidates have the same reference picture as the current block. For example, if the current predictive block has a single reference picture (single reference inter prediction), MVP candidates with a single reference picture that is the same as the reference picture of the current block are eligible to be put into the MVP candidate list, or MVP candidates with a composite reference picture, one of the two reference pictures being the same as the reference picture of the current block with a single reference picture, are eligible to be put into the MVP candidate list. As another example, for a current block that has a composite reference picture (e.g., two reference pictures), a candidate block may be placed in the MVP candidate list (if there is still space) only if the candidate block is also predicted under the composite reference and the two reference pictures of the candidate block are the same as the two reference pictures of the current block.

[0208]

[0228] In some other implementations, more or fewer rows or columns of adjacent and non-adjacent blocks than illustrated in FIG. 18 may be included in the SMVP candidate search.

[0209]

[0229] In some implementations, as described above and in more detail below, a TMVP may further be derived from a reference picture or frame of the current prediction block and included in the MVP candidate list. To generate a temporal MV predictor, the MVs of multiple frames are first stored together with the frame index associated with each frame. Then, for each prediction block of the current frame (e.g., for each 8×8 block), the MVs of multiple frames whose spatial trajectories pass through the current block are identified and stored in the temporal MV buffer together with the frame indexes of the corresponding multiple frames. In the case of inter prediction using a single reference frame, regardless of whether the reference frame is a forward reference frame or a backward reference frame, the MVs are stored in the 8×8 unit to perform temporal motion vector prediction of future frames. In the case of hybrid inter prediction, only the forward MVs are stored in the 8×8 unit to perform temporal motion vector prediction of future frames.

[0210]

[0230] An example is shown in Figure 19. In this example, MV ref The MV of frame 1906 (R1, right hand side of FIG. 19) for the block pointed to in frame 1906, called MV, points from R1 to its reference frame 1908 (left hand side of FIG. 19). In this example, the MV ref passes through an 8x8 block 1910 of the current frame 1902. MV ref is stored in the temporal MV buffer associated with this 8x8 block 1910. In some implementations, the motion vector MV passing through a particular current prediction block is ref To identify , multiple frames such as R1 are scanned in a predefined order, for example, LAST_FRAME, BWDREF_FRAME, ALTREF_FRAME, ALTREF2_FRAME, and LAST2_FRAME in reference frame list 0 and reference frame list 1. In some implementations, an MV from a higher indexed reference frame (in scan order) does not replace a previously identified MV assigned by a lower indexed reference frame (in scan order).

[0211]

[0231] Then, given predefined block coordinates of a current block in a current frame, the associated MVs stored in the temporal MV buffer are used to derive a TMVP pointing from the current block to the reference frame 1904 in FIG. 19, e.g., MV0 in FIG. 19. ref may be identified and projected to the current block. Such TMVPs have space in the list and may be considered to be MVP candidates in the MVP candidate list, e.g., after a contiguous SMVP search as described above or in other orders.

[0212]

[0232] In some implementations, TMVP may be searched or determined at the granularity of a group of predictive blocks (e.g., a superblock). In other words, an entire superblock may be associated with the same TMVP candidate or candidates. A superblock may include multiple predictive blocks. For example, a superblock may be 16×16 or 32×32, while a predictive block may be 8×8. Thus, a superblock includes 4 or 16 predictive blocks, correspondingly. For example, motion vectors may be derived and reconstructed at the predictive block level. Thus, a search for TMVP for a superblock may involve scanning multiple predictive blocks to identify their motion vectors and to determine which of these motion vectors or scaled versions of these motion vectors should be considered as TMVP candidates and in what order to make the MVP candidate list for the superblock. The multiple predictive blocks to be scanned in the search for TMVP candidates may include blocks both within the superblock (referred to as inner TMVP blocks) or outside the superblock (referred to as outer or outer TMVP blocks). In some implementations, the set of predictive blocks to be scanned and the order in which the set of predictive blocks are scanned may be predefined, preconfigured, or adaptively configured. The block positions of the set of predictive blocks relative to the superblock may be correspondingly redefined, preconfigured, or adaptively configured.

[0213]

[0233] An example is shown in FIG. 20. In FIG. 20, a 16×16 superblock includes four predictive blocks B0, B1, B2, and B3. These predictive blocks thus form an inner block. A set of outer predictive blocks may also be predefined, e.g., B4 through B6. These predictive blocks, both inner and outer TMVP blocks, may be searched / tested to determine and identify valid TMVP candidates for the superblock including B0 through B3. Thus, in this example, up to seven blocks are checked for valid temporal MV predictors. The search order may be predefined among these inner and outer predictive blocks. For example, the scan may proceed from inner blocks to outer blocks. During the scan process, the TMVP for each scanned block, if identified, is placed in the MV predictor list if there is still space after other scans (including but not limited to higher-ranked SMVP scans). For example, and as described above, the temporal MV predictor may be checked after adjacent spatial MV predictors but before non-adjacent spatial MV predictors. In one example, the TMVP search order may be B0→B1→B2→B3→B4→B5→B6. Various other example search implementations under various configurations are provided in further detail below in conjunction with Figures 25-30.

[0214]

[0234] In some implementations, for the derivation of the MV predictor, all spatial and temporal MV candidates are pooled and each predictor may be assigned a weight determined during the scanning of spatially and temporally adjacent prediction blocks. Based on the associated weights, the MVP candidates are sorted and ranked, and up to a predetermined number of candidates (e.g., four) are identified and added to the MVP list. This list of MV predictors is also called the Dynamic Reference List (DRL) and is used in the dynamic MV prediction mode, as described above. The DRL may be implemented as a list of MVP indexes.

[0215]

[0235] In some implementations, after pooling TMVPs and SMVPs, if the MVP list is not yet full, an additional search for MVP candidates may be performed, and such additional MVP candidates may be used to make the MVP list full. The additional MVP candidates may include, for example, whole MVs, zero MVs, composite MVs combined without scaling, etc. In some implementations, only adjacent TMVPs and SMVPs may be pooled before the additional search. In other words, in these implementations, non-adjacent TMVPs may be filtered out.

[0216]

[0236] In some implementations, the adjacent SMVP candidates, TMVP candidates, and non-adjacent SMVP candidates (if allowed) added to the MVP list may be further sorted. An example of such a sorting process may be based on a weight for each candidate. The weight of a candidate may be predefined, for example, depending on the overlap area between the current block and the candidate block in space.

[0217]

[0237] In some implementations, some types of MVPs may not be considered and therefore may not be affected during the reordering process. For example, external / non-adjacent and TMVP candidates may not be considered during the reordering process, which means that the reordering process only affects spatially adjacent candidates.

[0218]

[0238] In some other implementations, further MVP candidates may be derived. MVP candidates may be derived for a single reference picture and in mixed mode. For example, the above restriction on SMVP, which requires that a neighboring block share a reference frame with the current block to be considered an SMVP candidate, may be relaxed. For example, in the case of single inter prediction, if the reference frame of the neighboring predicted block of the current block is not the same as the reference frame of the current block, but they are in the same direction (forward or backward), a temporal scaling algorithm may be utilized to scale the MV of the neighboring predicted block to the reference frame of the current block to form the MVP of the motion vector of the current block.

[0219]

[0239] An example is shown in Figure 21. In Figure 21, a motion vector MV1 2110 is shown from a neighboring prediction block 2102 of a current super-block 2104 in a current frame 2105 to a reference prediction block 2106 in a reference frame 2108 of the neighboring prediction block 2102, and such motion vector MV1 2110 may be scaled according to the frame position of the current frame 2105, the reference frame 2106 of the neighboring prediction block 2102, and the reference frame 2120 of the current super-block 2104 to generate MV0 2130 as shown in Figure 21 as an MVP candidate for the current super-block 2104.

[0220]

[0240] As another example, as shown in Figure 22, for a situation with a current super-block 2202 of a current frame 2204 under hybrid inter prediction, MVs constructed from different neighboring prediction blocks of the current super-block 2202 are utilized to derive an MVP for the current block, for example, when the reference frames of the constructed MVs are the same as the current block 2202. In Figure 22, the constructed MVs (MV2, MV3) have the same reference frames 2210 and 2212 as the current block 2202, but they may be from different neighboring blocks 2206 and 2208.

[0221]

[0241] The above implementation example may be expanded to implement a motion vector candidate bank mechanism. For example, multiple MV bank buffers may be implemented, and each bank buffer may be associated with a unique reference frame type, which corresponds to a single or pair of reference frames covering single and composite inter modes, respectively. All bank buffers may be implemented with the same size. When a new MV is added to a full bank buffer, existing MVs are evicted to make room for the new MV.

[0222]

[0242] A coding block (e.g., a superblock) can refer to the MV candidate bank to collect reference MV candidates for the MV list in addition to those obtained for the MV list in other manners described above. After coding a superblock, the MV bank is updated with the MVs used by the prediction blocks of the superblock.

[0223]

[0243] In some implementations, encoding may be performed in tiles. Each tile may be associated with an independent MV reference bank that is utilized by all superblocks in the tile. At the beginning of encoding each tile, the corresponding bank is emptied. Then, while encoding each superblock in this tile, MVs from the bank may be used as MV reference candidates. At the end of encoding a superblock, the bank is updated.

[0224]

[0244] An example bank update process based on superblocks in a tile is illustrated in FIG. 23. In FIG. 23, a tile 2302 includes multiple superblocks (SBs). A current superblock being coded is shown as 2304. The superblocks above 2304 or to the left of the same row of 2304 have been coded (or have been reconstructed at the decoder), while the superblocks below 2304 or to the right of the same row of 2304 are to be coded (or to be reconstructed at the decoder), as indicated in FIG. 23. As indicated by the arrows in FIG. 23, after the superblock 2304 is coded, the first (e.g., up to 64) candidate MVs used by each coding block inside the superblock 2304 are added to the bank 2306. During the update, a pruning process may also be included.

[0225]

[0245] In some implementations, after the above-mentioned MV candidate scan of the superblock is performed to use the MV candidate bank, if there are still free slots in the MV candidate list, the codec may refer to the MV candidate bank (in the buffer that matches the reference frame type) for additional MV candidates. For example, MVs in the bank may be added to the MVP candidate list, if they are not already in the list, by proceeding from the back of the bank back to the beginning until the MVP list is full.

[0226]

[0246] According to various implementations described above, an exemplary construction of the MVP list may follow the search / processing order illustrated in FIG. 24: adjacent SMVPs (2402), a sorting process for existing candidates (2404), TMVPs (2406), non-adjacent SMVPs (2408), derived candidates (2410), reserve or additional MVP candidates (2412), and candidates from the reference MV candidate bank (2414) until a predefined number of MVP candidates are identified.

[0227]

[0247] As mentioned above, the space or number of MVP candidates in the MVP list may be limited. For example, the MVP list may be limited to four MVP candidates. In many cases, there may be a large number of TMVPs identifiable according to the above implementation or other aspects, and therefore, after searching for adjacent SMVPs, for example, according to the procedure of FIG. 24, the MVP list should be filled almost all the time. Therefore, other MVP candidates (e.g., MVP candidates belonging to non-adjacent SMVPs, other MVPs, additional MVPs, or MVPs from the MVP bank) may almost always be deprived of the opportunity to be evaluated for inclusion in the MVP list. Therefore, in some implementations described below, a limit may be imposed on the number of TMVP candidates that may be included in the MVP candidate list. Such a limit may be predefined or configured at various coding levels. Such a predefined or configured limit on TMVPs may be based on video type and statistics to facilitate the achievement of improved coding gain.

[0228]

[0248] In some implementations, the number of TMVP candidates that may be inserted into an MVP candidate list may be limited to N, where N is a positive integer. For example, N may be 1 or 2. In such an embodiment, the resulting MVP list may not be overly dominated by TMVP, and may be made more balanced by including other MVP types.

[0229]

[0249] In some further implementations, the process for constructing the MVP list may be designed to allow various MVP types, including the SMVPs, TMVPs, and other types of derived and / or additional MVPs described above, to complement each other to together result in improved coding efficiency.

[0230]

[0250] For example, considering that the adjacent and adjacent or adjacent and non-adjacent blocks scanned to find SMVP candidates are above or to the left of the current superblock, the complementary TMVP may be more likely if the scan of intra-predicted blocks of the current superblock for TMVP starts from an intra-predicted block in the current superblock that is further away from the above and to the left of the current superblock. In such a case, the TMVP of the predicted block that is further away from and therefore more complementary to those scanned for SMVP is provided with an earlier opportunity to create the MVP list.

[0231]

[0251] Such intra-predicted block scan orders may be distinct from those described above with respect to FIG. 20. There may be various example implementations for such scanning. For example, some orders may be better than others in terms of improving overall coding efficiency depending on the type and / or characteristics of the video frame being encoded. The scan order for such purposes among intra-predicted blocks may be predefined or may be configured and signaled in the bitstream. For example, there may be a predefined order with a predefined order index. The selection of the scan order from among multiple predefined scan orders may be done by the encoder, and an index of the selected scan order may be signaled for the superblock in the bitstream to use.

[0232]

[0252] In some implementations, intra prediction blocks may be scanned for a superblock to identify candidate TMVPs in the order illustrated in Figures 25 and 26, where the intra blocks are scanned from right to left and bottom to top. Specifically, Figure 25 illustrates an example superblock of size 16x16, including four prediction blocks, each of which is 8x8. The example scanning order of the intrablocks proceeds from right to left and bottom to top, e.g., B3→B2→B1→B0. Similarly, Figure 26 illustrates an example superblock of size 32x32, including 16 prediction blocks, each of which is 8x8. The example scanning order of the intrablocks proceeds from right to left and top to bottom, e.g., B15→B14→B13→B12→B11→B10→B9→B8→B7→B6→B5→B4→B3→B2→B1→B0. These implementations, for example, scan for TMVP candidates starting with interior blocks that are further away (at least horizontally) from and therefore complementary to adjacent and non-adjacent blocks that may provide SMVP candidates.

[0233]

[0253] In some alternative implementations, intra prediction blocks may be scanned for a superblock to identify candidate TMVPs in the order illustrated in Figures 27 and 28. Specifically, Figure 27 illustrates an example superblock of size 16x16, including four prediction blocks, each of 8x8. The example scanning order of the intrablocks proceeds from bottom to top and left to right, for example, in the order B3→B1→B2→B0. Similarly, Figure 28 illustrates an example superblock of size 32x32, including 16 prediction blocks, each of 8x8. The example scanning order of the intrablocks proceeds from right to left and top to bottom, for example, B15→B11→B7→B3→B14→B10→B6→B2→B13→B9→B5→B1→B12→B8→B4→B0. These implementations, for example, scan for TMVP candidates starting with interior blocks that are further away (at least vertically) from and therefore complementary to nearby and adjacent non-adjacent blocks that may provide SMVP candidates.

[0234]

[0254] In some other alternative implementations, intra prediction blocks may be scanned for a superblock to identify candidate TMVPs in the order illustrated in Figures 29 and 30. Specifically, Figure 29 illustrates an example superblock of size 16x16, including four prediction blocks, each of which is 8x8. The example scanning order of the intrablocks proceeds from bottom to top and left to right, for example, in the order B3→B2→B1→B0. Similarly, Figure 30 illustrates an example superblock of size 32x32, including 16 prediction blocks, each of which is 8x8. The example scanning order of the intrablocks proceeds from right to left and top to bottom, for example, B15→B14→B11→B13→B10→B7→B12→B9→B6→B3→B8→B5→B2→B4→B1→B0. These implementations, for example, scan for TMVP candidates starting with interior blocks that are further away (in both vertical and horizontal directions) from and thus complementary to adjacent and non-adjacent blocks that may provide SMVP candidates.

[0235]

[0255] Following the example scan order for intra predicted blocks above, up to a limited number N of TMVPs may be identified and included in the MVP list, so long as there is still space in the MVP list when the intra blocks are scanned. If fewer than N TMVPs are identified when scanning the intra blocks, there is still space in the MVP list, and the next MVP candidate to scan is an exterior block, then such exterior block will be scanned as long as the total TMVPs are less than or equal to N (if N is configured for all TMVPs as a whole) and there is still space in the MVP list.

[0236]

[0256] In some other variations on the above implementation examples for Figures 25 to 30, M candidates may be skipped during scanning of an intra-predicted block, where M can be a non-negative value, and if M is equal to 0, this is identical to the implementation of Figures 25 to 30.

[0237]

[0257] Candidate intra-predicted blocks to be skipped in a TMVP scan may be, for example, closer to the top and / or left side of a superblock to facilitate the generation of TMVPs in an MVP list that are less complementary to other MVP candidates such as SMVPs, as described above. The intra-blocks to be skipped may be in units of rows or columns of predicted blocks, for example, one or more rows or columns of an intra-block may be skipped in a TMVP scan.

[0238]

[0258] For example, if more than one internal TMVP block row is included, the first internal TMVP block row (the top row) may be skipped. Specifically, as shown in Figures 25, 27, and 29, B0 and B1 may be skipped and only the internal TMVP blocks B2 and B3 are checked in the respective orders in Figures 25, 27, and 29. For example, as shown in Figures 26, 28, and 30, B0 through B3 may be skipped and the remaining internal blocks are checked in the respective orders shown in Figures 26, 28, and 30.

[0239]

[0259] As another example, if more than one internal TMVP sequence is included, the first internal TMVP block sequence may be skipped. For example, as shown in Figures 25, 27, and 29, B0 and B2 may be skipped and only the internal TMVP blocks B1 and B3 are checked in the respective orders shown in Figures 25, 27, and 29. For example, as shown in Figures 26, 28, and 30, B0, B4, B8, and B12 may be skipped and the remaining internal blocks are checked in the respective orders shown in Figures 26, 28, and 30.

[0240]

[0260] As another example, if more than one internal TMVP column and more than one internal TMVP row are included, the first internal TMVP block column and the first internal TMVP row may both be skipped. For example, as shown in Figures 25, 27, and 29, B0, B1, and B2 may be skipped and only the internal TMVP block B3 is checked. For example, as shown in Figures 26, 28, and 30, B0 through B3, B4, B8, and B12 may be skipped and the remaining internal blocks B5, B6, B7, B9, B10, B11, B13, B14, and B15 are checked in the respective orders shown in Figures 26, 28, and 30.

[0241]

[0261] In some implementations, in addition to skipping some of the intra-predicted blocks of the current superblock during a TMVP scan, the outer TMVP blocks may be skipped and are not checked for insertion into the MVP list, regardless of the size of the current superblock (e.g., 16x16, or 32x32). For example, in Figure 20, the outer blocks such as B4, B5, and B6 may be skipped and they may not be checked or inserted into the MVP list.

[0242]

[0262] The upper limit of N motion vector predictors of TMVP candidates in the MVP list may apply to only inner blocks, or only outer blocks, or to both inner and outer blocks as a whole. In some implementations, at most one TMVP candidate from an inner TMVP block may be inserted into the MVP list. In some other implementations, at most one TMVP candidate from an outer TMVP block may be inserted into the MVP list. In some other implementations, at most one TMVP candidate from inner and outer TMVP blocks may be inserted into the MVP list.

[0243]

[0263] In some implementations, the number of predicted blocks to be scanned to identify TMVP candidates may be limited to K, where K is a positive integer. For example, K may be limited to 4. Thus, only K blocks may be scanned for TMVP candidates. Such a limit may apply to scanning inner TMVP blocks, or outer TMVP blocks, or inner and outer TMVP blocks as a whole. In some implementations, a separate K may be specified for inner and outer blocks. These limits may be predefined or fixed. They may be independent of the superblock size. Alternatively, these limits may be adaptively configured and indicated in the bitstream. For example, K inner blocks may be checked to identify TMVPs for insertion into the MVP list. The K intra predicted blocks to be checked may be uniformly distributed within a region populated with TMVP blocks, for example, according to some predefined or configured pattern. For example, the intra-predicted blocks to be checked may be uniformly distributed between B0 and B3 in Figures 25, 27, and 29, or between B0 and B15 in Figures 26, 28, and 30. As another example, K may be applied to the intra-predicted blocks and set to 1. In other words, only one intra block is checked. For example, the intra-predicted block to be checked may be block B2 in Figures 25, 27, and 29, or block B15 in Figures 16, 28, and 30.

[0244]

[0264] An upper limit on the number of TMVPs to be inserted into the MVP list (the integer N above) and an upper limit on the number of predicted blocks to check for TMVP candidates (the integer K above) may both be specified. These may be predefined or configured. They may be interdependent. For example, N and P may be specified as the difference between N and P and N. In some constraints, P and N may be specified as the same integer.

[0245]

[0265] In some implementations, the upper limit value of N motion vector predictors of TMVP candidates in the MVP list may be applied conditionally. For example, the upper limit value N may be applied to the current super-block when the current super-block is coded in bidirectional inter prediction mode and two reference frames point to the current frame in different directions (the POC of one reference frame is smaller than the POC of the current frame and the POC of the other reference frame is larger than the POC of the current frame), and may not be applied otherwise.

[0246]

[0266] In some implementations, the conditional application of the upper limit N to TMVP candidates in the MVP list may be based on the number of SMVP candidates already added to the MVP list. For example, the upper limit N2 of TMVP candidates that can be added to the MVP list may be in effect when there are already N2 SMVP candidates already added to the MVP list, otherwise the N2 limit does not apply. For example, N1 may be set to 2 and N2 may be set to 1. N1 and N2 may be predefined or configured. N1 and N2 may be superblock size dependent or independent of the size of the superblock.

[0247]

[0267] Turning to some aspects for signaling MVPs in a bitstream, in some implementations, the selection of an MVP from an MVP list or a DRL index may be correlated with the number of TMVPs inserted in the MVP list, so the context for signaling the DRL index and the inter-prediction mode may depend on whether at least L TMVPs have been inserted in the MVP list, where L is any positive integer. In one example, L may be set to 1. In other words, when there are at least L TMVPs in the MVP list, the first context may be used to signal the DRL index and the inter-prediction mode of a particular prediction block, and otherwise the second context may be used to signal the DRL index.

[0248]

[0268] In some implementations, the position of a TMVP candidate block in the MVP list may be implicitly signaled, for example, via other coded information, including but not limited to the motion vectors of spatially adjacent blocks or the number of spatial MVPs already inserted in the MVP list.

[0249]

[0269] FIG. 31 shows a flowchart 3100 of an example method according to the principles underlying the above implementation for searching TMVP candidates and for constructing an MVP list. The example method flow starts at S3101. At S3110, after receiving a video bitstream, it is determined that a current prediction block should be inter-predicted by a reference block in a reference frame, and that a motion vector of the current prediction block should be predicted by a reference motion vector. At S3120, a set of candidate prediction blocks in the current frame is identified as a search pool of TMVP candidates for the current prediction block. At S3130, the set of candidate prediction blocks is searched according to a search order to identify up to N TMVP candidates for the current prediction block, and the search ends in response to N TMVP candidates being identified, where N is a positive integer. At S3140, an MVP list is constructed to indicate a set of MVP candidates, the set of MVP candidates including one or more of a set of SMVP candidates and up to N TMVP candidates. At S3150, an MVP index of the current prediction block is extracted from the video stream. At S3160, a reference motion vector for inter predicting the current prediction block is identified according to the extracted MVP index and the MVP list. The example method stops at S3199.

[0250]

[0270] In the embodiments and implementations of the present disclosure, any steps and / or operations may be combined or arranged in any quantity or order as desired. Two or more of the steps and / or operations may be performed in parallel. The embodiments and implementations in the present disclosure may be used separately or in combination in any order. Furthermore, each of the method (or embodiment), the encoder, and the decoder may be implemented by a processing circuit device (e.g., one or more processors or one or more integrated circuits). In one example, the one or more processors execute a program stored in a non-transitory computer-readable medium. The embodiments in the present disclosure may be applied to a luma block or a chroma block. The term block may be interpreted as a prediction block, a coding block, or a coding unit or CU. The term block here may also be used to refer to a transform block. In the following items, when a block size is mentioned, the block size may refer to the block width or height, or the maximum value of the width and height, or the minimum value of the width and height, or the area size (width*height), or the aspect ratio of the block (width:height, or height:width).

[0251]

[0271] The techniques described above can be implemented as computer software physically stored on one or more computer-readable media using computer-readable instructions. For example, Figure 32 illustrates a computer system (3200) suitable for implementing certain embodiments of the disclosed subject matter.

[0252]

[0272] Computer software may be coded using any suitable machine or computer language, which may be subjected to mechanisms such as assembling, compiling, linking, etc. to produce code comprising instructions that can be executed by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., either directly, or through interpretation, microcode execution, etc.

[0253]

[0273] The instructions may be executable by various types of computers or components thereof including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet of Things devices, and the like.

[0254]

[0274] The components illustrated in Figure 32 for the computer system (3200) are exemplary in nature and are not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing the embodiments of the present disclosure. The arrangement of components should not be interpreted as having any dependency or requirement regarding any one or combination of components illustrated in the exemplary embodiment of the computer system (3200).

[0255]

[0275] The computer system (3200) may include certain human interface input devices. Such human interface input devices may respond to input by one or more human users through, for example, tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, clapping), visual input (e.g., gestures), and olfactory input (not shown). Human interface devices may also be used to capture certain media that are not necessarily directly related to conscious human input, such as audio (e.g., speech, music, ambient sounds), images (e.g., scanned images, photographic images obtained from a still image camera), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0256]

[0276] The input human interface devices may include one or more (only one of each is depicted) of a keyboard (3201), a mouse (3202), a trackpad (3203), a touch screen (3210), a data glove (not shown), a joystick (3205), a microphone (3206), a scanner (3207), and a camera (3208).

[0257]

[0277] The computer system (3200) may further include certain human interface output devices. Such human interface output devices may stimulate one or more of the senses of a human user, for example, through haptic output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touch screen (3210), data-gloves (not shown), or joystick (3205), but may also be haptic feedback devices that do not function as input devices), audio output devices (speakers (3209), headphones (not shown), etc.), visual output devices (screens (3210), to include CRT screens, LCD screens, plasma screens, OLED screens, etc., each with or without touch screen input capability, each with or without haptic feedback capability - some of which may be capable of outputting two-dimensional visual output or three or more dimensional output through means such as stereoscopic output, virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), as well as printers (not shown).

[0258]

[0278] The computer system (3200) may also include human-accessible storage devices and their associated media, such as optical media including CD / DVD ROM / RW (3220), including media (3221) such as CDs / DVDs, thumb drives (3222), removable hard drives or solid state drives (3223), older style magnetic media such as tape and floppy disks (not shown), and specialized ROM / ASIC / PLD based devices (not shown) such as security dongles.

[0259]

[0279] Moreover, those skilled in the art will understand that the term "computer-readable medium" as used in conjunction with the subject matter of this disclosure does not encompass transmission media, carrier waves, or other transitional signals.

[0260]

[0280] The computer system (3200) may also include an interface (3254) to one or more communication networks (3255). For example, the network may be wireless, wireline, optical. Furthermore, the network may be local, wide area, metropolitan, vehicular and industrial, real-time, delay tolerant, etc. Examples of networks include local area networks such as Ethernet, wireless LAN, cellular networks to include GSM, 3G, 4G, 5G, LTE, etc., TV wireline or wireless wide area digital networks to include cable TV, satellite TV, and terrestrial television broadcast, vehicular and industrial to include CAN bus, etc. Certain networks generally require an external network interface adapter attached to a particular general-purpose data port or peripheral bus (3249) (e.g., a USB port of the computer system (3200)), while others are generally integrated into the core of the computer system (3200) by attachment to a system bus as described below (e.g., an Ethernet interface to a PC computer system, or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system (3200) can communicate with other entities. Such communication can be one-way receive-only (e.g., television broadcast), one-way transmit-only (e.g., CANbus to a specific CANbus device), or bidirectional, for example, to other computer systems using local or wide area digital networks. Specific protocols and protocol stacks are available for use with each of these networks and network interfaces as described above.

[0261]

[0281] The aforementioned human interface devices, human accessible storage devices, and network interfaces may be attached to a core (3240) of the computer system (3200).

[0262]

[0282] The cores (3240) may include one or more central processing units (CPUs) (3241), graphics processing units (GPUs) (3242), specialized programmable processing units (3243) in the form of field programmable gate areas (FPGAs), hardware accelerators (3244) for specific tasks, graphics adapters (3250), etc. These devices may be connected through a system bus (3248), along with read only memory (ROM) (3245), random access memory (3246), internal mass storage (3247) such as internal user inaccessible hard drives, SSDs, etc. In some computer systems, the system bus (3248) may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may be attached directly to the core's system bus (3248) or through a peripheral bus (3249). In one example, the screen (3210) can be connected to a graphics adapter (3250). Peripheral bus architectures include PCI, USB, etc.

[0263]

[0283] The CPU (3241), GPU (3242), FPGA (3243), and accelerator (3244) can execute certain instructions that can combine to make up the aforementioned computer code. This computer code can be stored in ROM (3245) or RAM (3246). Temporary data can also be stored in RAM (3246), while permanent data can be stored, for example, in internal mass storage (3247). Rapid storage and retrieval to and from any of the memory devices can be enabled through the use of cache memory, which can be closely associated with one or more of the CPU (3241), GPU (3242), mass storage (3247), ROM (3245), RAM (3246), etc.

[0264]

[0284] The computer-readable medium can bear computer code for performing various computer-implemented operations. The medium and computer code can be those specially designed and constructed for the present disclosure, or they can be of the kind well known and available to those skilled in the computer software arts.

[0265]

[0285] As a non-limiting example, a computer system (3200) having an architecture, and specifically a core (3240), can provide functionality as a result of a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media can be user-accessible mass storage as introduced above, as well as media associated with specific storage of the core (3240) of a non-transitory nature, such as the core internal mass storage (3247) or ROM (3245). Software implementing various embodiments of the present disclosure can be stored in such devices and executed by the core (3240). The computer-readable media can include one or more memory devices or chips, depending on the particular needs. The software can cause the core (3240) and specifically the processor (including a CPU, GPU, FPGA, etc.) of the core to execute certain processes described herein, or certain parts of certain processes, including defining data structures stored in RAM (3246) and modifying such data structures in response to processes defined by the software. Additionally or alternatively, the computer system may provide functionality as a result of logic hardwired or otherwise embodied in circuitry (e.g., accelerator (3244)) operable in place of or together with software to perform certain processes, or certain portions of certain processes, described herein. References to software may encompass logic, and vice versa, where appropriate. References to computer-readable media may encompass circuitry (such as integrated circuits (ICs)) that store software for execution, circuitry that encompasses logic for execution, or both, where appropriate. The present disclosure encompasses any appropriate combination of hardware and software.

[0266]

[0286] While this disclosure has described several exemplary embodiments, there are modifications, permutations, and various substitute equivalents that are within the scope of this disclosure. It will thus be recognized that those skilled in the art will be capable of devising numerous systems and methods that, although not explicitly shown or described herein, embody the principles of the present disclosure and are therefore within the spirit and scope of the present disclosure.

[0267] Addendum A: Acronyms JEM: Joint Exploration Model VVC: Versatile Video Coding BMS: Benchmark Set MV: Motion Vector HEVC: High Efficiency Video Coding SEI:Supplementary Enhancement Information VUI: Video Usability Information GOP: Group of Pictures TU: conversion unit PU: Prediction unit CTU: Coding Tree Unit CTB: coding tree block PB: Predicted block HRD: Hypothetical Reference Decoder SNR: Signal to Noise Ratio CPU: Central Processing Unit GPU: Graphics Processing Unit CRT: cathode ray tube LCD: Liquid crystal display OLED: Organic Light Emitting Diode CD:Compact Disc DVD: Digital Video Disc ROM: Read-only memory RAM: Random Access Memory ASIC: Application Specific Integrated Circuit PLD: Programmable Logic Device LAN: Local Area Network GSM: Global System for Mobile Communications LTE: Long Term Evolution CANBus: Controller Area Network Bus USB: Universal Serial Bus PCI: Peripheral Component Interconnect FPGA: Field Programmable Gate Area SSD: Solid State Drive IC: Integrated Circuit HDR: High Dynamic Range SDR: Standard Dynamic Range JVET: Joint Video Exploration Team MPM: Most Probable Mode WAIP: Wide-angle Intra Prediction CU: Coding Unit PU: Prediction unit TU: conversion unit CTU: Coding Tree Unit PDPC: Position Dependent Prediction Combination ISP: Intra Subpartition SPS: Sequence Parameter Setting PPS: Picture Parameter Set APS: Calibration parameter set VPS: Video Parameter Set DPS: Decoding Parameter Set ALF: Adaptive Loop Filter SAO: Sample Adaptive Offset CC-ALF: Cross-Component Adaptive Loop Filter CDEF: Constrained Directional Enhancement Filter CCSO: Cross component sample offset LSO: Local Sample Offset LR: Loop Restoration Filter AV1:AOMedia Video 1 AV2:AOMedia Video 2 MVD: Motion Vector Difference CfL: Chroma from Luma SDT: Semi-Decoupled Tree SDP: Semi-Decoupled Partitioning SST: Semi-separate tree SB: Super Block IBC (or IntraBC): Intrablock copy CDF: Cumulative density function SCC: Screen Content Coding GBI: Generalized Biprediction BCW: Bi-prediction with CU-level weights CIIP: Combined Intra-Inter Prediction POC: Picture Order Count RPS: Reference Picture Set DPB: Decoded Picture Buffer MMVD: Merge with motion vector difference mode

Claims

1. 1. A method for processing a current predicted block of a current frame in a video stream, comprising the steps of: determining that the current prediction block is to be inter-predicted by a reference block in a reference frame and that a motion vector of the current prediction block is to be predicted by a reference motion vector; identifying a set of candidate prediction blocks within the current frame as a search pool for temporal motion vector predictor (TMVP) candidates for the current prediction block; searching the set of candidate prediction blocks according to a search order to identify up to N TMVP candidates for the current predicted block, the searching step terminating in response to N TMVP candidates being identified, where N is a positive integer; constructing a motion vector predictor (MVP) list indicating a set of MVP candidates, the set of MVP candidates including one or more of a set of spatial MVP (SMVP) candidates and the maximum N TMVP candidates; extracting an MVP index of the current predicted block from the video stream; identifying the reference motion vector for inter predicting the current prediction block based on the extracted MVP index and the MVP list; A method comprising:

2. The current prediction block belongs to a current super-block; the current super-block comprises a plurality of predicted blocks; the set of candidate predictive blocks includes at least a subset of the plurality of predictive blocks of the current superblock. The method of claim 1.

3. the plurality of prediction blocks of the current superblock form a prediction block array having a column dimension and a row dimension; a reconstructed adjacent prediction block of the current superblock is above the current superblock in the row dimension and / or to the left of the current superblock in the column dimension; the search order for searching the set of candidate prediction blocks for the maximum N TMVP candidates includes starting away from and moving closer to the reconstructed adjacent prediction block of the current superblock along at least one of the column dimension and the row dimension of the prediction block array. The method of claim 2.

4. the search order for searching the set of candidate prediction blocks for the maximum N TMVP candidates comprises: starting from the bottom row of the array of predictive blocks to the top row, and proceeding row by row from the rightmost predictive block to the leftmost predictive block; or starting from the rightmost column of the array of predictive blocks and proceeding to the leftmost column, and proceeding column by column from the bottom predictive block to the top predictive block; or searching the array of prediction blocks along a diagonal direction based on the column dimension and the row dimension, starting with the rightmost and bottommost prediction block of the array of prediction blocks and proceeding toward the leftmost and topmost prediction block; The method of claim 3 , comprising:

5. The method of claim 3 , wherein the set of candidate predictive blocks comprises a subset of the plurality of predictive blocks having M predictive blocks skipped from the plurality of predictive blocks, where M is a non-negative integer.

6. The method of claim 5 , wherein the M skipped predictive blocks include at least the leftmost column of the predictive block array.

7. The method of claim 5 , wherein the M skipped predictive blocks include at least a top row of the predictive block array.

8. The method of claim 5 , wherein the M skipped predictive blocks include at least a top row and a leftmost column of the predictive block array.

9. The method of claim 3 , wherein the set of candidate predictive blocks excludes any additional predictive blocks outside the current superblock.

10. The method of claim 3 , wherein the set of candidate predictive blocks further comprises at least one additional predictive block outside the current superblock.

11. searching the plurality of predicted blocks of the current super-block to identify up to N1 TMVP candidates, where N1 is a positive integer less than or equal to N; ceasing to search the plurality of predicted blocks of the current super-block in response to N1 TMVP candidates being identified from the plurality of predicted blocks; The method of claim 10 further comprising:

12. 11. The method of claim 10, wherein up to one TMVP candidate is identified from the at least one additional predictive block outside the current superblock, the method further comprising: ceasing to search the at least one additional predictive block outside the current superblock in response to one TMVP candidate being identified from the at least one additional predictive block outside the current superblock.

13. The method of claim 3 , wherein the set of candidate predictive blocks includes up to L predictive blocks from the plurality of predictive blocks, where L is a positive integer less than a total number of predictive blocks in the current superblock.

14. The method of claim 13 , wherein L=1 and the set of candidate predictive blocks includes only the rightmost and bottommost predictive blocks of the predictive block array.

15. The method of claim 13 , wherein the up to L prediction blocks are uniformly distributed within the plurality of prediction blocks of the current super-block.

16. 4. The method of claim 3, wherein the position of the set of candidate prediction blocks is determined by coded information including motion vectors of spatially neighboring blocks of the current superblock or the number of SMVP candidates already included in the MVP list.

17. 17. The method of claim 1, further comprising the step of further limiting the number of TMVP candidates in the MVP list to N2 in response to there being N1 SMVP candidates in the MVP list, where N1 and N2 are positive integers and N2 is less than or equal to N.

18. 17. The method of claim 1, wherein a context for signaling an index in the MVP list and an inter-prediction mode associated with the current predicted block depends on whether at least N TMVPs are included in the MVP list.

19. 17. The method of claim 1, wherein limiting the TMVP candidates in the MVP list to a maximum of N is responsive to the current prediction block being coded under a bidirectional inter-prediction mode.

20. 1. An electronic device for processing a current predicted block of a current frame of a video stream, comprising: a memory for storing instructions; and a processor, the processor comprising: determining that the current prediction block is to be inter predicted by a reference block in a reference frame and that a motion vector of the current prediction block is to be predicted by a reference motion vector; identifying a set of candidate prediction blocks within the current frame as a search pool for a temporal motion vector predictor (TMVP) of the current predictive block; searching the set of candidate prediction blocks according to a search order to identify up to N TMVP candidates for the current predicted block, the searching terminating in response to N TMVP candidates being identified, where N is a positive integer; constructing a motion vector predictor (MVP) list indicating a set of MVP candidates, the set of MVP candidates including one or more of a set of spatial MVP (SMVP) candidates and the up to N TMVP candidates; extracting an MVP index of the current predicted block from the video stream; and identifying the reference motion vector for inter predicting the current prediction block based on the extracted MVP index and the MVP list; and executing the instructions to cause the electronic device to perform the following:

21. 17. An electronic device for processing a current predicted block of a current frame of a video stream, comprising: a memory for storing instructions; and a processor, the processor executing the instructions to cause the electronic device to perform any one of the methods of claims 2 to 16.

22. A non-transitory computer-readable medium for storing instructions, the instructions being configured, when executed by a processor of an electronic device, to cause the electronic device to perform any one of the methods of claims 1 to 16.

Images