VIDEO DECODING METHOD AND APPARATUS, AND VIDEO ENCODING METHOD - Patent application

Abstract

The present application discloses signaling of motion vectors or motion vector differentials depending on whether a magnitude-dependent adaptive motion vector differential pixel resolution is used in video encoding and decoding. An exemplary method for processing video blocks of a video stream is disclosed. The method may include receiving a video stream, determining that a video block is inter-coded based on a predictive block and a motion vector (MV) to be derived from a motion vector differential (MVD) of the video block and a reference motion vector (RMV), extracting or deriving a data item related to at least one of the MVD or RMV from the video stream in a manner that depends at least on whether the MVD is coded with the magnitude-dependent adaptive MVD pixel resolution, extracting the MVD from the video stream, deriving the MV based on the extracted RMV and the MVD, and reconstructing the video block based at least on the MV and the predictive block.

Description

[Technical field]

[0001] [Incorporated by reference] This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 300,433, filed January 18, 2022 and entitled "Improvement for Adaptive MVD resolution," and U.S. Patent Application No. 17 / 824,168, filed May 25, 2022 and entitled "Interdependence Between Adaptive Resolution of Motion Vector Difference and Signaling / Derivation of Motion Vector-Related Parameters," all of which are incorporated herein by reference in their entireties.

[0002] [Technical field] FIELD This disclosure relates generally to video coding, and more particularly to methods and systems that provide adaptive resolution of motion vector differentials in inter prediction of video blocks. [Background technology]

[0003] This background description provided herein is intended to generally present the context of the present disclosure. The work of the presently named inventors is not expressly or impliedly admitted as prior art to the present disclosure, to the extent that that work is described in this background section, and any aspects of the description that may not otherwise qualify as prior art at the time of filing.

[0004] Video encoding and decoding can be performed using inter-picture prediction with motion compensation. Uncompressed digital video can include a sequence of pictures, each having spatial dimensions of, for example, 1920x1080 luminance samples and associated full or subsampled chrominance samples. The sequence of pictures can have a fixed or variable picture rate (alternatively called frame rate), for example, 60 pictures per second or 60 frames per second. Uncompressed video has specific bit rate requirements for streaming or data processing. For example, a video with a pixel resolution of 1920x1080, a frame rate of 60 frames / second, and 4:2:0 chroma subsampling with 8 bits / pixel per color channel requires a bandwidth close to 1.5 Gbit / s. One hour of such video requires more than 600 Gbytes of storage space.

[0005] One goal of video encoding and decoding can be the reduction of redundancy in an uncompressed input video signal by compression. Compression can help reduce the above bandwidth and / or storage space requirements by more than two orders of magnitude in some cases. Both lossless and lossy compression, as well as combinations thereof, can be used. Lossless compression refers to techniques where an exact copy of the original signal can be reconstructed from the compressed original signal by the decoding process. Lossy compression refers to an encoding / decoding process where the original video information is not fully preserved during encoding and is not fully recoverable during decoding. When using lossy compression, the reconstructed signal may not be the same as the original signal, but the distortion between the original and reconstructed signals is made small enough to make the reconstructed signal useful for the intended application despite some information loss. In the case of video, lossy compression is widely used in many applications. The amount of acceptable distortion depends on the application. For example, a user of a particular consumer video streaming application may tolerate higher distortion than a user of a movie or television broadcast application. The compression ratio achievable by a particular coding algorithm can be selected or adjusted to reflect different distortion tolerances, with higher tolerable distortion generally allowing for coding algorithms that result in higher losses and higher compression ratios.

[0006] Video encoders and decoders can utilize techniques from a number of broad categories and steps, including, for example, motion compensation, Fourier transform, quantization, and entropy coding.

[0007] Video codec techniques may include a technique known as intra-coding, in which sample values ​​are represented without reference to samples or other data from previously reconstructed reference pictures. In some video codecs, a picture is spatially subdivided into blocks of samples. If an entire block of samples is coded in intra mode, the picture may be called an intra-picture. Intra-pictures and their derivatives, such as independent decoder refresh pictures, can be used to reset the decoder state and thus may be used as the first picture in a coded bitstream and video session or as still images. The samples of the block after intra prediction may then undergo a transform to the frequency domain, and the transform coefficients so produced may be quantized before entropy coding. Intra prediction may represent a technique that minimizes sample values ​​in the pre-transform domain. In some cases, the smaller the DC value after the transform and the smaller the AC coefficients, the fewer bits are needed for a given quantization step size to represent the block after entropy coding.

[0008] Conventional intra-coding, such as that known from MPEG-2 generation coding techniques, does not use intra-prediction. However, some newer video compression techniques include techniques that attempt to code / decode a block based on surrounding sample data and / or metadata obtained during the coding and / or decoding of a block of data that is spatially adjacent to the data and that precedes it in the decoding order. Such techniques are hereafter referred to as "intra-prediction" techniques. It should be noted that, at least in some cases, intra-prediction uses only reference data from the current picture being reconstructed and not from other reference pictures.

[0009] There may be many different forms of intra-prediction. When more than one such technique is available for a given video coding technique, the technique in use may be referred to as an intra-prediction mode. One or more intra-prediction modes may be provided in a particular codec. In certain cases, a mode may have sub-modes and / or may be associated with various parameters, and the mode / sub-mode information and intra-coding parameters of a block of video may be coded independently or collectively included in a mode codeword. Since which codeword to use for a given mode, sub-mode, and / or parameter combination may affect the coding efficiency gain through intra-prediction, entropy coding techniques may be used to convert the codeword into a bitstream.

[0010] Certain modes of intra prediction were introduced by H.264, refined in H.265, and further refined in newer coding techniques such as Joint Exploration Model (JEM), Versatile Video Coding (VVC), and Benchmark Set (BMS). In general, for intra prediction, a predictor block may be formed using neighboring sample values ​​that are available. For example, the available values ​​of a particular set of neighboring samples along a particular direction and / or line may be copied into the predictor block. The reference of the direction in use may be coded into the bitstream or may itself be predicted.

[0011] Referring to FIG. 1A, at the bottom right, a subset of nine predictor directions specified in the 33 possible intra predictor directions of H.265 (corresponding to the 33 angle modes of the 35 intra modes specified in H.265) is shown. The point where the arrows meet (101) corresponds to the sample being predicted. The arrows represent the direction in which neighboring samples are used to predict the sample at 101. For example, arrow (102) indicates that sample (101) is predicted from one or more neighboring samples to the upper right at an angle of 45 degrees from the horizontal. Similarly, arrow (103) indicates that sample (101) is predicted from one or more neighboring samples to the lower left of sample (101) at an angle of 22.5 degrees from the horizontal.

[0012] Still referring to FIG. 1A, at the top left is depicted a square block (104) of 4×4 samples (indicated by thick dashed lines). The square block (104) contains 16 samples, each labeled with "S", its position in the Y dimension (e.g., row index), and its position in the X dimension (e.g., column index). For example, sample S21 is the second sample (from the top) in the Y dimension and the first sample (from the left) in the X dimension. Similarly, sample S44 is the fourth sample in the block (104) in both the Y and X dimensions. S44 is at the bottom right, since the block is 4×4 samples in size. Also shown are exemplary reference samples that follow a similar numbering scheme. The reference samples are labeled with "R" and their Y position (e.g., row index) and X position (column index) relative to the block (104). In both H.264 and H.265, predicted samples that are adjacent to the block being reconstructed are used.

[0013] Intra-picture prediction of block 104 may start by copying reference sample values ​​from neighboring samples according to the signaled prediction direction. For example, suppose that the coded video bitstream includes signaling for this block 104 indicating the prediction direction of the arrow (102), i.e., the sample is predicted from one or more prediction samples located to the upper right at an angle of 45 degrees from the horizontal. In such a case, samples S41, S32, S23, and S14 are predicted from the same reference sample R05. Then, sample S44 is predicted from reference sample R08.

[0014] In certain cases, values ​​of multiple reference samples may be combined, for example through interpolation, to calculate a reference sample, particularly when the orientation is not equally divisible by 45 degrees.

[0015] The number of possible directions increases as video coding technology continues to develop. In H.264 (2003), for example, nine different directions are available for intra prediction. That increases to 33 in H.265 (2013), and JEM / VVC / BMS can support up to 65 directions at the time of this disclosure. Empirical studies have been conducted to help identify the most suitable intra prediction directions, and certain techniques in entropy coding have been used to encode those most suitable directions with a small number of bits while accepting a certain bit penalty for the direction. Furthermore, the direction itself can sometimes be predicted from neighboring directions used in intra prediction of decoded neighboring blocks.

[0016] FIG. 1B shows a schematic diagram (180) representing 65 intra prediction directions according to JEM to illustrate the increasing number of prediction directions in various coding techniques that have developed over time.

[0017] The method of mapping bits representing intra-prediction directions to prediction directions in the coded video bitstream may vary from one video coding technique to another and may range, for example, from a simple direct mapping of prediction directions to intra-prediction modes to codewords, to complex adaptation schemes including most-probable modes, and similar techniques. In all cases, however, there may be certain directions for intra-prediction that are statistically less likely to occur in the video content than certain other directions. Since a goal of video compression is to reduce redundancy, these less likely directions may be represented by a greater number of bits than more likely directions in a well-designed video coding technique.

[0018] Inter-picture prediction or inter-prediction may be based on motion compensation. In motion compensation, sample data from a previously reconstructed picture or part thereof (reference picture) may be used for prediction of a newly reconstructed picture or picture part (e.g., block) after being spatially shifted in a direction indicated by a motion vector (hereinafter MV). In some cases, the reference picture may be the same as the picture currently being reconstructed. The MV may have two dimensions X and Y, or it may have three dimensions, with the third dimension being an indication of the reference picture in use (akin to the temporal dimension).

[0019] In some video compression techniques, the current MV applicable to a particular area of ​​sample data can be predicted from other MVs, e.g., from other MVs related to other areas of sample data that are spatially adjacent to the area being reconstructed and that precede the current MV in decoding order. Doing so can significantly reduce the overall amount of data required to code the MV by relying on correlated MVs to remove redundancy, thereby increasing compression efficiency. For example, when coding an input video signal obtained from a camera (known as natural video), MV prediction can work because there is a statistical possibility that areas larger than the area to which a single MV is applicable move in similar directions in the video sequence, and thus can be predicted in some cases using similar motion vectors derived from MVs of neighboring areas. As a result, the actual MV of a given area is similar or the same as the MV predicted from the surrounding MVs. Such an MV can then be represented after entropy coding with fewer bits than would be used if the MV were directly coded instead of predicted from neighboring MVs. In some cases, MV prediction can be an example of lossless compression of a signal (i.e., an MV) derived from an original signal (i.e., a sample stream). In other cases, the MV prediction itself may be lossy, for example due to rounding errors when computing a predictor from several surrounding MVs.

[0020] Various MV prediction mechanisms are described in H.265 / HEVC (ITU-T Rec. H265, “High Efficiency Video Coding”, December 2016). Among the many MV prediction mechanisms specified by H.265, a technique hereafter referred to as “spatial merge” is described in this specification.

[0021] Specifically, referring to Figure 2, a current block (201) has samples that are found by the encoder during the motion search process to be predictable from a previous block of the same size that is spatially shifted. Instead of coding its MV directly, the MV can be derived from metadata associated with one or more reference pictures, for example, from the most recent reference picture (in decoding order), using the MV associated with any one of five surrounding samples denoted as A0, A1 and B0, B1, B2 (202 to 206, respectively). In H.265, MV prediction can use predictors from the same reference picture that neighboring blocks use. Summary of the Invention

[0022] The present disclosure relates generally to video coding, and more particularly to methods and systems for signaling syntax related to different motion vector differentials based on whether magnitude-dependent adaptive resolution for motion vector differentials in inter prediction is used.

[0023] In an example implementation, a method for processing a video block of a video stream is disclosed. The method may include receiving a video stream, determining that a video block is inter-coded based on a predictive block and a motion vector (MV), where the MV is derived from a motion vector difference (MVD) and a reference motion vector (RMV) of the video block, extracting or deriving a data item related to at least one of the MVD or the RMV from the video stream in a manner that depends at least on whether the MVD is coded with a magnitude-dependent adaptive MVD pixel resolution, extracting the MVD from the video stream, deriving the MV based on the extracted RMV and MVD, and reconstructing the video block based at least on the MV and the predictive block.

[0024] In the above implementations, the data item may include syntax elements associated with at least one of the MVD or RMV.

[0025] In any one of the above implementations, the data item may include the RMV.

[0026] In any one of the above implementations, the data item may include an RMV index of the video block that maps to a Dynamic Reference List (DRL), the DRL being configured to identify a plurality of ordered candidate RMVs.

[0027] In any one of the above implementations, the step of extracting a data item in response at least to whether the MVD of the video block is coded with a size-dependent adaptive MVD pixel resolution may include a step of determining an RMV index range N in response at least to whether the MVD of the video block is coded with a size-dependent adaptive MVD pixel resolution, where N is a positive integer, and a step of parsing the video stream based on the RMV index range to extract an RMV index for the video block.

[0028] In any one of the above implementations, the RMV indices 1 through N may map to a predefined set of positions within the DRL.

[0029] In any one of the above implementations, RMV indexes 1 through N may map to the first N candidate RMVs among a plurality of ordered candidate RMVs identified by the DRL.

[0030] In any one of the above implementations, N can be 1 or 2.

[0031] In any one of the above implementations, N may be signaled in the video stream, and the method further comprises extracting N from the video stream.

[0032] In any one of the above implementations, N may be signaled in a syntax element at the sequence level, frame level, slice level, title level, or superblock level.

[0033] In any one of the above implementations, N=1 and the RMV index may not be present in the video stream and is derived in response to determining N=1.

[0034] In any one of the above implementations, the method for extracting or deriving the RMV index may further depend on whether the video block is predicted in single-reference mode in addition to whether the MVD is coded with a magnitude-dependent adaptive MVD pixel resolution.

[0035] In any one of the above implementations, the RMV index is extracted from the video stream, and the context for signaling the RMV index in the video stream may depend on whether the MVD is coded with a magnitude-dependent adaptive MVD pixel resolution.

[0036] In any one of the above implementations, if the MVD is coded with a magnitude-dependent adaptive MVD pixel resolution, a first context may be used to signal the RMV in the video stream, while if the MVD is not coded with a magnitude-dependent adaptive MVD pixel resolution, a second context separate from the first context may be used to signal the RMV in the video stream.

[0037] In any one of the above implementations, the method may further include, in response to the video block being coded with a magnitude-dependent adaptive MVD pixel resolution, extracting from the video stream an information item indicating whether Overlapped Block Motion Compensation (OBMC) or Warped Motion is used when the video block is predicted in a single reference mode.

[0038] In any one of the above implementations, the method may further include, in response to the video block being coded with a magnitude-dependent adaptive MVD pixel resolution, extracting from the video stream an information item indicating whether a compound inter-intra prediction mode is used when the video block is predicted in a single reference mode.

[0039] In any one of the above implementations, the context derivation for signaling at least one syntax element related to MVD may depend on whether the video block is coded with a magnitude-dependent adaptive MVD pixel resolution.

[0040] In any one of the above implementations, the at least one syntax element related to the MVD includes at least one of a first MVD syntax element indicating which components of the MVD are non-zero, a second MVD syntax element specifying the sign of the MVD, a third MVD syntax element specifying a magnitude range of the MVD, a fourth MVD syntax element specifying an integer magnitude offset within the magnitude range of the MVD, or a fifth MVD syntax element specifying a pixel resolution of the MVD.

[0041] In any one of the above implementations, if the video block is coded with a size-dependent adaptive MVD pixel resolution, a first context may be derived to decode at least one syntax element related to the MVD, while if the video block is not coded with a size-dependent adaptive MVD pixel resolution, a second context separate from the first context may be derived to decode at least one syntax element related to the MVD.

[0042] In another implementation, a method for decoding a video block of a video stream is disclosed that includes receiving a video stream, determining that the video block is inter-coded based on a predictive block and a motion vector (MV), where the MV is derived from a motion vector differential (MVD) and a reference motion vector (RMV) of the video block, extracting an RMV index of the video block that maps to a dynamic reference list (DRL) configured to identify a plurality of ordered candidate RMVs, and determining, based on the value of the RMV index, whether the MVD is coded with a magnitude-dependent adaptive MVD pixel resolution.

[0043] In the above implementation, the method further includes the steps of: extracting a flag from the video stream if the value of the RMV index indicates one of the first N RMV candidates among the multiple ordered candidate RMVs identified by the DRL, where N is a positive integer; determining whether the MVD is coded with a magnitude-dependent adaptive MVD pixel resolution based on the flag; and determining that the MVD is not coded with a magnitude-dependent adaptive MVD pixel resolution if the value of the RMV index does not indicate any of the first N RMV candidates among the multiple ordered candidate RMVs.

[0044] In any of the above implementations, N may be predefined as 1 or 2. In any of the above implementations, N is signaled separately in the video stream. In any of the above implementations, N may be signaled in a syntax element at the sequence level, frame level, slice level, title level, or superblock level.

[0045] Aspects of the disclosure also provide a video encoding or decoding device or apparatus including circuitry configured to perform any of the implementations of the above methods.

[0046] Aspects of the disclosure also provide a non-transitory computer-readable medium storing instructions that, when executed by a computer for video decoding and / or encoding, cause the computer to perform a method of video decoding and / or encoding.

[0047] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings. [Brief description of the drawings]

[0048] [Figure 1A] 1 illustrates a schematic diagram of an exemplary subset of intra-prediction direction modes. [Figure 1B] 1 shows an illustrative diagram of an exemplary intra-prediction direction. [Diagram 2] 1 illustrates a schematic diagram of a current block and its surrounding spatial merging candidates for motion vector prediction in an example. [Diagram 3] 1 shows a simplified block diagram of a communication system (300) according to an example embodiment. [Figure 4] 1 shows a simplified block diagram of a communication system (400) according to an example embodiment. [Diagram 5] FIG. 2 shows a schematic block diagram of a decoder according to an example embodiment; [Figure 6] FIG. 2 shows a schematic block diagram of an encoder according to an example embodiment; [Figure 7] 1 shows a block diagram of a video encoder according to another example embodiment; [Figure 8] 1 shows a block diagram of a video decoder according to another exemplary embodiment; [Figure 9] 1 illustrates a coding block partitioning scheme according to an exemplary embodiment of the disclosure. [Figure 10] 13 illustrates another coding block partitioning scheme according to an exemplary embodiment of the disclosure. [Figure 11] 13 illustrates another coding block partitioning scheme according to an exemplary embodiment of the disclosure. [Figure 12] 4 illustrates an exemplary division of a base block into coding blocks according to an example partitioning scheme. [Figure 13] An example ternary partitioning scheme is shown below. [Figure 14] 1 illustrates an example quadtree / binary tree coding block partitioning scheme. [Figure 15] 1 illustrates a scheme for partitioning a coding block into multiple transform blocks and a coding order of the transform blocks according to an exemplary embodiment of the disclosure. [Figure 16] 13 illustrates another scheme for partitioning a coding block into multiple transform blocks and the coding order of the transform blocks according to an exemplary embodiment of the disclosure. [Figure 17] 13 illustrates another scheme for partitioning a coding block into multiple transform blocks according to an exemplary embodiment of the disclosure. [Figure 18] 1 shows a flowchart of a method according to an exemplary embodiment of the disclosure. [Figure 19] 13 illustrates another flowchart of a method according to an exemplary embodiment of the disclosure. [Figure 20] 1 shows a schematic diagram of a computer system according to an example embodiment of the disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0049] Throughout the specification and claims, terms may have subtle meanings that are suggested or implied in the context beyond the meaning explicitly stated. The phrases "in one embodiment" or "in some embodiments" used herein do not necessarily refer to the same embodiment, and the phrases "in another embodiment" or "in other embodiments" used herein do not necessarily refer to different embodiments. Similarly, the phrases "in one implementation" or "in some implementations" used herein do not necessarily refer to the same implementation, and the phrases "in another implementation" or "in other implementations" used herein do not necessarily refer to different implementations. For example, claimed subject matter is intended to include the example embodiment / implementation in whole or in part.

[0050] Generally, terms may be understood, at least in part, from their use in context. For example, terms such as "and" and "or" and "and / or" as used herein may include various meanings that may depend, at least in part, on the context in which such terms are used. Typically, "or" when used to link a list, such as A, B, or C, is intended to mean A, B, and C, as used herein in an inclusive sense, as well as A or B or C, as used in an exclusive sense. Additionally, the terms "one or more" or "at least one" as used herein may be used to describe any feature, structure, or characteristic in the singular sense, or may be used to describe a combination of features, structures, or characteristics in the plural sense, depending, at least in part, on the context. Similarly, terms such as "a" or "an" and "the" may be understood to convey a singular use or a plural use, again depending, at least in part, on the context. Additionally, the terms "based on" or "determined by" may be understood as not necessarily intended to convey an exclusive set of factors, but instead may acknowledge the existence of additional factors not necessarily explicitly recited, again depending at least in part on the context.

[0051] FIG. 3 illustrates a simplified block diagram of a communication system (300) according to an embodiment of the present disclosure. The communication system (300) includes a plurality of terminal devices that can communicate with each other, for example, via a network (350). For example, the communication system (300) includes a first pair of terminal devices (310) and (320) that are interconnected via the network (350). In the example of FIG. 3, the first pair of terminal devices (310) and (320) may perform unidirectional transmission of data. For example, the terminal device (310) may code video data (e.g., of a stream of video pictures captured by the terminal device (310)) for transmission to another terminal device (320) via the network (350). The encoded video data may be transmitted in the form of one or more coded video bitstreams. The terminal device (320) may receive the coded video data from the network (350), decode the coded video data to recover the video pictures, and display the video pictures according to the recovered video data. One-way data transmission may be implemented, for example, in media serving applications.

[0052] In another example, the communication system (300) includes a second pair of terminal devices (330) and (340) performing bidirectional transmission of coded video data, such as may be performed during video conferencing applications. For the bidirectional transmission of data, in the example, each of the terminal devices (330) and (340) may code video data (e.g., of a stream of video pictures captured by that terminal device) for transmission to the other of the terminal devices (330) and (340) over the network (350). Each of the terminal devices (330) and (340) may also receive coded video data transmitted by the other of the terminal devices (330) and (340), decode the coded video data to recover the video pictures, and display the video pictures on an accessible display device in accordance with the recovered video data.

[0053] In the example of FIG. 3, the terminal devices (310), (320), (330), and (340) may be implemented as a server, a personal computer, and a smartphone, although the applicability of the underlying principles of the present disclosure may not be so limited. The embodiments of the present disclosure may be implemented in a desktop computer, a laptop computer, a tablet computer, a media player, a wearable computer, a dedicated video conferencing device, and / or the like. The network (350) represents any number or type of network that conveys coded video data between the terminal devices (310), (320), (330), and (340), including, for example, wireline and / or wireless communication networks. The communication network (350) may exchange data in circuit-switched, packet-switched, and / or other types of channels. Exemplary networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For purposes of this discussion, the architecture and topology of the network (350) may be irrelevant to the operation of the present disclosure unless expressly described herein.

[0054] 4 illustrates the arrangement of a video encoder and a video decoder in a video streaming environment as an example application of the disclosed subject matter. The disclosed subject matter may be similarly applicable to other video-enabled applications including, for example, video conferencing, digital TV broadcasting, gaming, virtual reality, storage of compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0055] A video streaming system may include a video capture subsystem (413) that may include a video source (401), e.g., a digital camera, that generates a stream of uncompressed video pictures or images (402). In an example, the stream of video pictures (402) includes samples recorded by the digital camera of the video source (401). The stream of video pictures (402) is represented by a bold line to emphasize its high data volume compared to the encoded video data (404) (or coded video bitstream), and may be processed by electronics (420) that includes a video encoder (403) coupled to the video source (401). The video encoder (403) may include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter, as described in more detail below. The encoded video data (404) (or encoded video bitstream (404)), represented with thin lines to emphasize its lower data volume compared to the stream of uncompressed video pictures (402), may be stored on a streaming server (405) for future use or for direct downstream video device (not shown). One or more streaming client subsystems, such as the client subsystems (406) and (408) of FIG. 4, may access the streaming server (405) to retrieve copies (407) and (409) of the encoded video data (404). The client subsystem (406) may include a video decoder (410), for example in an electronic device (430). The video decoder (410) decodes the incoming copy of the encoded video data (407) and generates an outgoing stream of video pictures (411) that is uncompressed and can be rendered on a display (412) (e.g., a display screen) or other rendering device (not shown). The video decoder (410) may be configured to perform some or all of the various functions described in this disclosure.In some streaming systems, the encoded video data 404, 407, and 409 (e.g., a video bitstream) may be encoded according to a particular video coding / compression standard. An example of such a standard is ITU-T Recommendation H.265. In an example, the video coding standard under development is colloquially known as Versatile Video Coding (VVC). The disclosed subject matter may be used in conjunction with VVC and other video coding standards.

[0056] The electronics 420 and 430 may include other components (not shown). For example, the electronics 420 may include a video decoder (not shown), and the electronics 430 may also include a video encoder (not shown).

[0057] 5 shows a block diagram of a video decoder (510) according to any of the following embodiments of the present disclosure. The video decoder (510) may be included in an electronic device (530). The electronic device (530) may include a receiver (531) (e.g., a receiving circuit). The video decoder (510) may be used in place of the video decoder (410) of the example of FIG. 4.

[0058] The receiver (531) may receive one or more coded video sequences to be decoded by the video decoder (510). In the same or other embodiments, one coded video sequence at a time may be decoded, with the decoding of each coded video sequence being independent of the other coded video sequences. Each video sequence may be associated with multiple video frames or images. The coded video sequences may be received from a channel (501), which may be a storage device that stores the coded video data or a hardware / software link to a streaming source that transmits the coded video data. The receiver (531) may receive the coded video data along with other data, such as coded audio data and / or auxiliary data streams, which may be forwarded to their respective processing circuits (not shown). The receiver (531) may separate the coded video sequences from the other data. To combat network jitter, a buffer memory (515) may be placed between the receiver (531) and the entropy decoder / parser (520) (hereafter "parser (520)"). In certain applications, the buffer memory (515) may be implemented as part of the video decoder (510). In other applications, it may be separate (not shown) outside the video decoder (510). In still other applications, there may be a buffer memory (not shown) outside the video decoder (510), for example, to combat network jitter, and there may be another additional buffer memory (515) in the video decoder (510), for example, to manipulate the playback timing. When the receiver (531) is receiving data from a storage / forwarding device with sufficient bandwidth and controllability, or from an isosynchronous network, the buffer memory (515) may not be needed or may be small.For use with best-effort packet networks such as the Internet, a buffer memory (515) of sufficient size may be required, which may be relatively large in size. Such a buffer memory may be implemented with an adaptive size and may be at least partially implemented in an operating system or similar element (not shown) external to the video decoder (510).

[0059] The video decoder (510) may include a parser (520) for reconstructing symbols (521) from the coded video sequence. These categories of symbols include information used to manage the operation of the video decoder (510) and information for controlling a rendering device such as a display (512) (e.g., a display screen) that may or may not be an integral part of the electronic device (530) but may be coupled to the electronic device (530) as shown in FIG. 5. The control information for the rendering device may take the form of a Supplemental Enhancement Information (SEI) message or a Video Usability Information (VUI) parameter set fragment (not shown). The parser (520) may parse / entropy decode the coded video sequence received by the parser (520). The coding of the coded video sequence may follow a video coding technique or standard and may follow various principles including variable length coding, Huffman coding, context-dependent or non-context-dependent arithmetic coding, etc. The parser (520) may extract from the coded video sequence a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder based on at least one parameter corresponding to the subgroup. The subgroups may include a Group of Pictures (GOP), a picture, a tile, a slice, a macroblock, a coding unit (CU), a block, a transform unit (TU), a prediction unit (PU), etc. The parser (520) may also extract information from the coded video sequence, such as transform coefficients (e.g., Fourier transform coefficients), quantizer parameter values, motion vectors, etc.

[0060] The parser (520) may perform an entropy decoding / parsing operation on the video sequence received from the buffer memory (515) to generate symbols (521).

[0061] The reconstruction of the symbols (521) can involve a number of different processing or functional units depending on the type of coded video picture or portion thereof (e.g., inter and intra pictures, inter and intra blocks) and other factors. The units involved, and how they are involved, can be controlled by subgroup control information parsed by the parser (520) from the coded video sequence. The flow of such subgroup control information between the parser (520) and the following processing or functional units is not depicted for simplicity.

[0062] Beyond the functional blocks already mentioned, the video decoder (510) may be conceptually subdivided into a number of functional units, which are described below. In an actual implementation operating under commercial constraints, many of these functional units may closely interact with each other and may be at least partially integrated with each other. However, for purposes of clearly describing the various functions of the disclosed subject matter, a conceptual subdivision into functional units is adopted in the remainder of this disclosure.

[0063] The first unit may include a scalar / inverse transform unit (551), which may receive quantized transform coefficients as symbols (521) from the parser (520) along with control information including information indicating what inverse transform to use, block size, quantization coefficients / parameters, quantization scaling matrices, etc. The scalar / inverse transform unit (551) may output blocks containing sample values ​​that may be input to an aggregator (555).

[0064] In some cases, the output samples of the scaler / inverse transformer (551) may relate to intra-coded blocks, i.e., blocks that do not use prediction information from a previously reconstructed picture, but can use prediction information from a previously reconstructed portion of the current picture. Such prediction information may be provided by an intra-picture prediction unit (552). In some cases, the intra-picture prediction unit (552) may generate a block of the same size and shape as the block being reconstructed using surrounding block information that has already been reconstructed and stored in the current picture buffer (558). The current picture buffer (558) may, for example, buffer the partially reconstructed and / or fully reconstructed current picture. In some implementations, the aggregator (555) may add, on a sample-by-sample basis, the prediction information generated by the intra-prediction unit (552) to the output sample information provided by the scaler / inverse transformer unit (551).

[0065] In other cases, the output samples of the scaler / inverse transform unit (551) may relate to an inter-coded and potentially motion-compensated block. In such cases, the motion compensated prediction unit (553) may access the reference picture memory (557) to fetch samples used for inter-picture prediction. After motion compensating the fetched samples according to the symbols (521) related to the block, the samples may be added by the aggregator (555) to the output of the scaler / inverse transform unit (551) (the output of unit 551 may be referred to as a residual sample or residual signal) to generate output sample information. The address in the reference picture memory (557) from which the motion compensated prediction unit (553) fetches the prediction samples may be controlled by a motion vector available to the motion compensated prediction unit (553) in the form of a symbol (521) that may have, for example, an X component and a Y component (shift) as well as a reference picture component (time). Motion compensation may also include interpolation of sample values ​​fetched from a reference picture memory (557) when sub-sample accurate motion vectors are used, and may also be associated with a motion vector prediction mechanism, etc.

[0066] The output samples of the aggregator (555) may be subjected to various loop filtering techniques in the loop filter unit (556). Video compression techniques may include in-loop filter techniques that are controlled by parameters contained in the coded video sequence (also called the coded video bitstream) and made available to the loop filter unit (556) as symbols (521) from the parser (520), but may also be responsive to meta-information obtained during the decoding of a previous portion (in decoding order) of the coded picture or coded video sequence, or may even be responsive to previously constructed loop filtered sample values. Several types of loop filters may be included as part of the loop filter unit 556 in various orders, as described in more detail below.

[0067] The output of the loop filter unit (556) can be a sample stream that can be output to a rendering device (512) and further stored in a reference picture memory (557) for use in future inter-picture prediction.

[0068] Once a particular coded picture is fully reconstructed, it can be used as a reference picture for future inter-picture prediction. For example, once a coded picture corresponding to a current picture is fully reconstructed and the coded picture is identified as a reference picture (e.g., by the parser (520)), the current picture buffer (558) can become part of the reference picture memory (557), and any unused current picture buffer can be reallocated before beginning reconstruction of a subsequent coded picture.

[0069] The video decoder (510) may perform decoding operations according to a given video compression technique adopted in a standard, such as ITU-T Recommendation H.265. The coded video sequence may conform to a syntax prescribed by the video compression technique or standard in use, in the sense that the coded video sequence conforms to both the syntax of the video compression technique or standard and a profile documented in the video compression technique or standard. In particular, a profile may select a particular tool from all tools available in the video compression technique or standard as the only tool available for use under that profile. For standard compliance, the complexity of the coded video sequence may be within the bounds defined by the level of the video compression technique or standard. In some cases, the level limits the maximum picture size, the maximum frame rate, the maximum reconstruction sample rate (e.g., measured in megasamples / second), the maximum reference picture size, etc. The limits set by the levels may in some cases be further restricted through Hypothetical Reference Decoder (HRD) specifications and metadata for HRD buffer management signaled in the coded video sequence.

[0070] In some example embodiments, the receiver (531) may receive additional (redundant) data along with the encoded video. The additional data may be included as part of the coded video sequence. The additional data may be used by the video decoder (510) to properly decode the data and / or to more accurately reconstruct the original video data. The additional data may take the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0071] 6 shows a block diagram of a video encoder (603) according to an exemplary embodiment of the present disclosure. The video encoder (603) may be included in an electronic device (620). The electronic device (620) may further include a transmitter (640) (e.g., a transmission circuit). The video encoder (603) may be used in place of the video encoder (403) of the example of FIG. 4.

[0072] The video encoder (603) may receive video samples from a video source (601) (which is not part of the electronics (620) in the example of FIG. 6) that may capture video images to be coded by the video encoder (603). In other examples, the video source (601) may be implemented as part of the electronics (620).

[0073] The video source (601) may provide a source video sequence to be coded by the video encoder (603) in the form of a digital video sample stream that can be of any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, etc.), any color space (e.g., BT.601 YCrCb, RGB, XYZ, etc.), and any suitable sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media serving system, the video source (601) may be a storage device that can store pre-prepared video. In a video conferencing system, the video source (601) may be a camera that captures local image information as a video sequence. The video data may be provided as multiple individual pictures or images that, when viewed in sequence, impart motion. The picture itself may be organized as a spatial array of pixels, each of which may have one or more samples depending on the sampling structure, color space, etc. in use. Those skilled in the art can easily understand the relationship between pixels and samples. The remainder of this specification focuses on samples.

[0074] According to some example embodiments, the video encoder (603) may code and compress pictures of a source video sequence into a coded video sequence (643) in real-time or under any other time constraint required by the application. Imposing an appropriate coding rate constitutes one function of the controller (650). In some embodiments, the controller (650) may be operatively coupled to and control other functional units as described below. Coupling is not shown for simplicity. Parameters set by the controller (650) may include parameters related to rate control (picture skip, quantizer, lambda value for rate-distortion optimization techniques, etc.), picture size, group-of-picture (GOP) layout, maximum motion vector search range, etc. The controller (650) may be configured with other appropriate functions related to the video encoder (603) optimized for a particular system design.

[0075] In some example embodiments, the video encoder (603) may be configured to operate in a coding loop. As an oversimplified description, in an example, the coding loop may include a source coder (630) (e.g., responsible for generating symbols, such as a symbol stream, based on an input picture to be coded and a reference picture) and a (local) decoder (633) embedded in the video encoder (603). The decoder (633) reconstructs the symbols to generate sample data in a similar manner as a (remote) decoder would generate it, even if the embedded decoder (633) processes the video stream coded by the source coder (630) without entropy coding (in the sense that any compression between the symbols and the coded video stream can be lossless in the video compression techniques contemplated in the disclosed subject matter). The reconstructed sample stream (sample data) is input to a reference picture memory (634). Since the decoding of the symbol stream produces bit-exact results independent of the location of the decoder (local or remote), the contents in the reference picture memory (634) are also bit-perfect between the local and remote encoders. That is, the prediction part of the encoder "sees" exactly the same sample values ​​as the reference picture samples that the decoder will "see" when using the prediction during decoding. This basic principle of reference picture synchronicity (and the resulting drift when synchronicity cannot be maintained, e.g., due to channel errors) is used to improve coding quality.

[0076] The operation of the "local" decoder (633) can be the same as a "remote" decoder, such as the video decoder (510), already described in detail above in conjunction with Figure 5. Referring also momentarily to Figure 5, however, because symbols are available and the encoding / decoding of symbols into a coded video sequence by the entropy coder (645) and parser (520) can be lossless, the entropy decoding portion of the video decoder (510), including the buffer memory (515) and parser (520), may not be implemented entirely in the local decoder (633) within the encoder.

[0077] At this point, it can be observed that any decoder technique, except for parsing / entropy decoding, which may only be present in the decoder, may necessarily need to be present in roughly the same functional form in the corresponding encoder. For this reason, the disclosed subject matter may at times focus on decoder operations related to the decoding portion of the encoder. Thus, descriptions of encoder techniques may be omitted, since they are the inverse of the decoder techniques described generically. Only in certain scopes or aspects are more detailed descriptions of the encoders given below.

[0078] In operation, in some embodiments, the source coder (630) may perform motion-compensated predictive coding, which predictively codes an input picture with reference to one or more previously coded pictures from a video sequence, designated as "reference pictures." In this manner, the coding engine (632) codes color channel differences (or residuals) between pixel blocks of the input picture and pixel blocks of a reference picture that may be selected as a predictive reference for the input picture. The term "residue" or its adjective "residual" may be used interchangeably.

[0079] The local video decoder (633) may decode the coded video data of pictures that may be designated as reference pictures based on the symbols generated by the source coder (630). The operation of the coding engine (632) may advantageously be a lossy process. When the coded video data may be decoded in a video decoder (not shown in FIG. 6), the reconstructed video sequence may usually be a copy of the source video sequence with some errors. The local video decoder (633) may reproduce the decoding process that may be performed by the video decoder on the reference pictures, causing the reconstructed reference pictures to be stored in the reference picture cache (634). In this way, the video encoder (603) may locally store copies of reconstructed reference pictures that have common content with the reconstructed reference pictures that would be obtained by the far-end (remote) video decoder (without transmission errors).

[0080] The predictor (635) may perform a prediction search for the coding engine (632). That is, for a new picture to be coded, the predictor (635) may look for certain metadata, such as reference picture motion vectors, block shapes, or sample data (as candidate reference pixel blocks) from the reference picture memory (634) that may be suitable prediction references for the new picture. The predictor (635) may operate on a sample block-by-pixel block basis to find a suitable prediction reference. In some cases, as determined by the search results obtained by the predictor (635), the input picture may have prediction references derived from multiple reference pictures stored in the reference picture memory (634).

[0081] The controller (650) may manage the coding operations of the source coder (630), including, for example, setting the parameters and subgroup parameters used to encode the video data.

[0082] The output of all the above functional units may undergo entropy coding in an entropy coder (645), which converts the symbols produced by the various functional units into a coded video sequence by lossless compression of the symbols according to techniques such as Huffman coding, variable length coding, arithmetic coding, etc.

[0083] The transmitter (640) may buffer the coded video sequence produced by the entropy coder (645) to prepare it for transmission over a communication channel (660), which may be a hardware / software link to a storage device that stores the encoded video data. The transmitter (640) may merge the coded video data from the video coder (603) with other data to be transmitted, such as coded audio data and / or auxiliary data streams (sources not shown).

[0084] A controller (650) may manage the operation of the video encoder (603). During coding, the controller (650) may assign a particular coded picture type to each coded picture, which may affect the coding technique that may be applied to each picture. For example, pictures may often be assigned as one of the following picture types:

[0085] An Intra Picture (I-Picture) may be a picture that can be coded and decoded without using any other picture in a sequence as a source of prediction. Some video codecs allow various types of Intra pictures, including, for example, Independent Decoder Refresh (IDR) pictures. Those skilled in the art are aware of such variations of I-pictures and their respective applications and characteristics.

[0086] A Predictive Picture (P-picture) may be a picture that can be encoded and decoded by intra- or inter-prediction using at most one motion vector and reference index to predict the sample values ​​of each block.

[0087] A Bi-directionally Predictive Picture (B-picture) may be a picture that can be coded and decoded by intra- or inter-prediction using at most two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, multiple-predictive picture(s) can use more than two reference pictures and associated metadata for the reconstruction of a single block.

[0088] A source picture may generally be spatially subdivided into a number of sample coding blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples, respectively) and coded block by block. Blocks may be predictively coded with reference to other (already coded) blocks as determined by the coding assignment applied to each picture of the 'blocks'. For example, blocks of I pictures may be non-predictively coded or they may be predictively coded with reference to already coded blocks of the same picture (spatial or intra prediction). Pixel blocks of P pictures may be predictively coded by spatial prediction or by temporal prediction with reference to one previously coded reference picture. Blocks of B pictures may be predictively coded by spatial prediction or by temporal prediction with reference to one or two previously coded reference pictures. Source pictures or intermediate processed pictures may be subdivided into other types of blocks for other purposes. The division of coding blocks and other types of blocks may or may not follow the same method, as described in more detail below.

[0089] The video encoder (603) may perform coding operations according to a given video coding technique or standard, such as ITU-T Recommendation H.265. During its operation, the video encoder (603) may perform various compression operations, including predictive coding operations that exploit temporal and spatial redundancy in the input video sequence. Accordingly, the coded video data may conform to a syntax defined by the video coding technique or standard being used.

[0090] In some example embodiments, the transmitter (640) may transmit additional data along with the encoded video. The source coder (630) may include such data as part of the coded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

[0091] Video may be captured as multiple source pictures (video pictures) in a time sequence. Intra-picture prediction (often abbreviated as intra-prediction) exploits spatial correlation in a given picture, while inter-picture prediction exploits temporal or other correlation between pictures. For example, a particular picture being coded / decoded, called the current picture, may be partitioned into blocks. A block in the current picture may be coded by a vector, called a motion vector, if it is similar to a reference block in a reference picture that was previously coded in the video and is still buffered. A motion vector points to a reference block in the reference picture and may have a third dimension that identifies the reference picture if multiple reference pictures are used.

[0092] In some example embodiments, bi-prediction techniques may be used for inter-picture prediction. According to such bi-prediction techniques, two reference pictures are used, e.g., a first reference picture and a second reference picture, both preceding the current picture in the video in decoding order (but which may be in the past or future, respectively, in display order). A block in the current picture may be coded with a first motion vector that points to a first reference block in the first reference picture and a second motion vector that points to a second reference block in the second reference picture. The block is jointly predictable by a combination of the first and second reference blocks.

[0093] Furthermore, merge mode techniques may be used in inter-picture prediction to improve coding efficiency.

[0094] According to some example embodiments of the present disclosure, predictions such as inter-picture prediction and intra-picture prediction are performed on a block-by-block basis. For example, a picture in a sequence of video pictures is partitioned into coding tree units (CTUs) for compression, and the CTUs in a picture may have the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. In general, a CTU may include three parallel coding tree blocks (CTBs), i.e., one luma CTB and two chroma CTBs. Each CTU may be recursively quad-tree partitioned into one or more coding units (CUs). For example, a CTU of 64×64 pixels may be partitioned into one CU of 64×64 pixels or four CUs of 32×32 pixels. One or more of the 32×32 blocks may be further partitioned into four CUs of 16×16 pixels. In some example embodiments, each CU may be analyzed during encoding to determine a prediction type for the CU among various prediction types, such as an inter prediction type or an intra prediction type. A CU may be divided into one or more Prediction Units (PUs) according to temporal and / or spatial predictability. In general, each PU includes one luma prediction block (PB) and two chroma PBs. In embodiments, prediction operations in coding (encoding / decoding) are performed in units of prediction blocks. The division of a CU into PUs (or PBs of different color channels) may be performed in various spatial patterns. A luma or chroma PB may include a matrix of sample values ​​(e.g., luma values), such as, for example, 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, etc.

[0095] 7 shows a diagram of a video encoder (703) according to another exemplary embodiment of the disclosure. The video encoder (703) is configured to receive a processed block of sample values ​​(e.g., a predictive block) in a current video picture included in a sequence of video pictures and to encode the processed block into a coded picture that is part of a coded video sequence. The example video encoder (703) may be used in place of the example video encoder (403) of FIG. 4.

[0096] For example, the video encoder (703) receives a matrix of sample values ​​for a processing block, such as a prediction block of 8x8 samples. The video encoder (703) then determines whether the processing block is best coded in intra-mode, inter-mode, or bi-predictive mode, for example using rate-distortion optimization (RDO). If the processing block is determined to be coded in intra-mode, the video encoder (703) may use intra-prediction techniques to encode the processing block into a coded picture, and if the processing block is determined to be coded in inter-mode or bi-predictive mode, the video encoder (703) may use inter-prediction or bi-prediction techniques, respectively, to encode the processing block into a coded picture. In some exemplary embodiments, the merge mode may be used as a sub-mode of inter-picture prediction in which motion vectors are derived from one or more motion vector predictors without the benefit of coded motion vector components outside the predictors. In some other exemplary embodiments, there may be motion vector components applicable to the current block. Accordingly, the video encoder (703) may include components not explicitly shown in FIG. 7, such as a mode decision module, to determine the prediction mode of a processing block.

[0097] In the example of FIG. 7, the video encoder (703) includes an inter-encoder (730), an intra-encoder (722), a residual calculator (723), a switch (726), a residual encoder (724), a general controller (721), and an entropy encoder (725), coupled as shown in the exemplary arrangement in FIG.

[0098] The inter encoder (730) is configured to receive samples of a current block (e.g., a processing block), compare the block to one or more reference blocks in a reference picture (e.g., blocks in previous and subsequent pictures in a standard location), generate inter prediction information (e.g., a description of redundant information according to an inter coding technique, motion vectors, merge mode information), and calculate an inter prediction result (e.g., a predicted block) based on the inter prediction information using any suitable technique. In some examples, the reference picture is a decoded reference picture that has been decoded based on video information encoded using a decoding unit (633) embedded in the example encoder (620) of FIG. 6 (shown as a residual decoder 728 of FIG. 7, as described in more detail below).

[0099] The intra encoder (722) is configured to receive samples of a current block (e.g., a processing block), compare the block to previously coded blocks in the same picture, and generate transformed quantized coefficients, and in some cases, further generate intra prediction information (e.g., intra prediction direction information according to one or more intra encoding techniques). The intra encoder (722) may calculate an intra prediction result (e.g., a prediction block) based on the intra prediction information and a reference block in the same picture.

[0100] The generic controller (721) may be configured to determine generic control data and control other components of the video encoder (703) based on the generic control data. In an example, the generic controller (721) determines a prediction mode of a block and provides a control signal to the switch (726) based on the prediction mode. For example, if the prediction mode is an intra mode, the generic controller (721) controls the switch (726) to select an intra mode result for use by the residual calculator (723), and controls the entropy encoder (725) to select intra prediction information and include the intra prediction information in the bitstream. If the prediction mode of the block is an inter mode, the generic controller (721) controls the switch (726) to select an inter prediction result for use by the residual calculator (723), and controls the entropy encoder (725) to select inter prediction information and include the inter prediction information in the bitstream.

[0101] The residual calculator (723) is configured to calculate a difference (residual data) between a received block and a prediction result of the block selected from the intra-encoder (722) or the inter-encoder (730). The residual encoder (724) may be configured to encode the residual data to generate transform coefficients. For example, the residual encoder (724) may be configured to transform the residual data from the spatial domain to the frequency domain to generate the transform coefficients. The transform coefficients then undergo a quantization process to obtain quantized transform coefficients. In various exemplary embodiments, the video encoder (703) also includes a residual decoder (728). The residual decoder (728) is configured to perform an inverse transform and generate decoded residual data. The decoded residual data may be used by the intra-encoder (722) and the inter-encoder (730) as appropriate. For example, the inter-encoder (730) can generate decoded blocks based on the decoded residual data and the inter-prediction information, and the intra-encoder (722) can generate decoded blocks based on the decoded residual data and the intra-prediction information. The decoded blocks are appropriately processed to generate decoded pictures, which may be buffered in a memory circuit (not shown) and used as reference pictures.

[0102] The entropy encoder (725) may be configured to format a bitstream to include the encoded block and perform entropy coding. The entropy encoder (725) may be configured to include various information in the bitstream. For example, the entropy encoder (725) may be configured to include general control data, selected prediction information (e.g., intra-prediction information or inter-prediction information), residual information, and other suitable information in the bitstream. Residual information may not be present when coding a block in a merged sub-mode of either an inter mode or a bi-prediction mode.

[0103] 8 shows a diagram of an example video decoder (810) according to another embodiment of the disclosure. The video decoder (810) is configured to receive coded pictures that are part of a coded video sequence and decode the coded pictures to generate reconstructed pictures. In an example, the video decoder (810) may be used in place of the example video decoder (410) of FIG. 4.

[0104] In the example of FIG. 8, the video decoder (810) includes an entropy decoder (871), an inter decoder (880), a residual decoder (873), a reconstruction module (874), and an intra decoder (872), coupled as shown in the exemplary arrangement in FIG. 8.

[0105] The entropy decoder (871) may be configured to reconstruct from the coded picture certain symbols representing syntax elements from which the coded picture is constructed. Such symbols may include, for example, prediction information (e.g., intra- or inter-prediction information) that may identify the mode in which the block is coded (e.g., intra-, inter-, bi-prediction, merged, or other sub-modes), certain samples or metadata used for prediction by the intra decoder (872) or the inter decoder (880), residual information, e.g., in the form of quantized transform coefficients, etc. In an example, if the prediction mode is an inter or bi-prediction mode, the inter prediction information is provided to the inter decoder (880), and if the prediction type is an intra prediction type, the intra prediction information is provided to the intra decoder (872). The residual information may undergo inverse quantization and is provided to the residual decoder (873).

[0106] The inter decoder (880) may be configured to receive the inter prediction information and generate inter prediction results based on the inter prediction information.

[0107] The intra decoder (872) may be configured to receive intra prediction information and generate a prediction result based on the intra prediction information.

[0108] The residual decoder (873) may be configured to perform inverse quantization to retrieve inverse quantized transform coefficients and process the inverse quantized transform coefficients to transform the residual from the frequency domain to the spatial domain. The residual decoder (873) may also utilize certain control information (to include quantization parameters (QPs)), which may be provided by the entropy decoder (871) (datapath not shown since this is only low volume control information).

[0109] The reconstruction module (874) may be configured to combine the residual output by the residual decoder (873) and the prediction results (possibly output by an inter- or intra-prediction module) in the spatial domain to form reconstructed blocks that form part of the reconstructed picture as part of the reconstructed video. Note that other suitable operations, such as deblocking operations, may be performed to improve visual quality.

[0110] It should be noted that the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) may be implemented using any suitable technology. In some exemplary embodiments, the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) may be implemented using one or more integrated circuits. In other embodiments, the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) may be implemented using one or more processors executing software instructions.

[0111] Referring to block partitioning for encoding and decoding, a general partitioning may start from a base block and may follow a predefined set of rules, a specific pattern, a partition tree, or any partition structure or scheme. The partitioning may be hierarchical and recursive. After splitting or partitioning the base block according to any of the exemplary partitioning procedures described below or other procedures or a combination thereof, a final set of partitions or coding blocks may be obtained. Each of these partitions may be at one of various partitioning levels in the partitioning hierarchy and may be of various shapes. Each of the partitions may be referred to as a coding block (CB). For various exemplary partitioning implementations described further below, each resulting CB may be of any of the allowed sizes and partitioning levels. Such partitions are referred to as coding blocks in that they may form the units within which some basic encoding / decoding decisions may be made and encoding / decoding parameters may be optimized, determined, and signaled in the encoded video bitstream. The highest or deepest level of the final partition represents the depth of the coding block partitioning structure of the tree. A coding block may be a luma coding block or a chroma coding block. The CB tree structure for each color may be called a coding block tree (CBT).

[0112] The coding blocks of all color channels may be collectively referred to as a coding unit (CU). The hierarchical structure of all color channels may be collectively referred to as a coding tree unit (CU). The partitioning pattern or structure of various color channels in a CTU may or may not be the same.

[0113] In some implementations, the partition tree scheme or structure used for the luma channel and the chroma channel need not be the same. In other words, the luma channel and the chroma channel may have separate coding tree structures or patterns. Furthermore, whether the luma channel and the chroma channel use the same or different coding partition tree structures and the actual coding partition tree structure to be used may depend on whether the slice being coded is a P, B or I slice. In the case of an I slice, the chroma channel and the luma channel may have separate coding partition tree structures or coding partition tree structure modes, while in the case of a P or B slice, the luma channel and the chroma channel may share the same coding partition tree scheme. When separate coding partition tree structures or modes are applied, the luma channel may be partitioned into CBs by one coding partition tree structure and the chroma channel may be partitioned into chroma CBs by another coding partition tree structure.

[0114] In some implementations, a predefined partitioning pattern may be applied to the base block. As shown in FIG. 9, an example four-way partition tree may start at a first predefined level (64×64 block level or other size as the base block size), and the base block may be partitioned hierarchically down to a predefined lower level (e.g., 4×4 level). For example, the base block may follow four predefined partitioning options or patterns shown at 902, 904, 906, and 908, and the partitions labeled R allow for recursive partitioning in that the same partitioning options shown in FIG. 9 may be repeated at a lower scale, down to the lowest level (e.g., 4×4 level). In some implementations, additional restrictions may be applied to the partitioning scheme of FIG. 9. In the implementation of FIG. 9, rectangular partitions (e.g., 1:2 / 2:1 rectangular partitions) may be allowed, but they may not be recursive, whereas square partitions can be recursive. Partitioning according to Fig. 9 by recursion produces a final set of coding blocks, if necessary. A coding tree depth may be further defined to indicate the partition depth from a root node or root block. For example, the coding tree depth of a root node or root block, e.g., a 64x64 block, may be set to 0, and after the root block is further divided once according to Fig. 9, the coding tree depth is increased by 1. The maximum or deepest level from the 64x64 base block to the 4x4 smallest partition is 4 (starting from level 0) in the above scheme. Such a partitioning scheme may be applied to one or more of the color channels. Each color channel may be partitioned independently according to the scheme of Fig. 9 (e.g., a partitioning pattern or option among predefined patterns may be determined independently for each color channel at each hierarchical level).Alternatively, two or more of the color channels may share the same hierarchical pattern tree of FIG. 9 (e.g., the same partitioning pattern or option in the predefined patterns may be selected for two or more color channels at each hierarchical level).

[0115] FIG. 10 illustrates another exemplary predefined partitioning pattern that allows recursive partitioning to form a partitioning tree. As illustrated in FIG. 10, ten exemplary partitioning structures or patterns may be predefined. A root block may start at a predefined level (e.g., from a base block at a 128×128 level or a 64×64 level). The exemplary partitioning structure of FIG. 10 includes various 2:1 / 1:2 and 4:1 / 1:4 rectangular partitions. A partition type including three subpartitions, illustrated at 1002, 1004, 1006, and 1008 in the second row of FIG. 10, may be referred to as a “T-type” partition. The “T-type” partitions 1002, 1004, 1006, and 1008 may be referred to as Left T-Type, Top T-Type, Right T-Type, and Bottom T-Type. In some embodiments, none of the rectangular partitions of FIG. 10 may be further subdivided. The coding tree depth may be further defined to indicate the partition depth from the root node or root block. For example, the coding tree depth of the root node or root block, e.g., a 128x128 block, may be set to 0, and after the root block is further partitioned once according to FIG. 10, the coding tree depth is increased by 1. In some implementations, only the full square partitions of 1010 may be allowed to be recursively partitioned to the next level of the partitioning tree according to the pattern of FIG. 10. In other words, recursive partitioning may not be allowed for the square partitions in the T-type patterns 1002, 1004, 1006, and 1008. The partitioning procedure according to FIG. 10 by recursion generates a final set of coding blocks as necessary. Such a scheme may be applied to one or more of the color channels. In some implementations, further flexibility may be added to the use of partitions below the 8x8 level. For example, 2x2 chroma inter prediction may be used in certain cases.

[0116] In some other embodiments for coding block partitioning, a quadtree structure may be used to divide a base block or an intermediate block into quadtree partitions. Such quadtree division may be applied hierarchically and recursively to any square partitions. Whether a base block or an intermediate block or partition is further quadtree divided may be adapted to various local characteristics of the base block or intermediate block / partition. The quadtree division at picture boundaries may be further adapted. For example, an implicit quadtree division may be performed at picture boundaries such that a block continues to be quadtree divided until its size fits the picture boundary.

[0117] In some other embodiments, a hierarchical binary partitioning from the base block may be used. In such a scheme, the base block or mid-level block may be divided into two partitions. The binary partitioning may be either horizontal or vertical. For example, a horizontal binary partitioning may divide the base block or mid-block into equal left and right partitions. Similarly, a vertical binary partitioning may divide the base block or mid-block into equal top and bottom partitions. Such binary partitioning may be hierarchical and recursive. It may be determined for each base block or mid-block whether the binary partitioning scheme should continue, and if the scheme continues further, whether horizontal or vertical binary partitioning should be used. In some implementations, the further partitioning may stop at a predefined minimum partition size (in one or both dimensions). Alternatively, the further partitioning may stop when a predefined partitioning level or depth from the base block is reached. In some implementations, the aspect ratio of the partitions may be limited. For example, the aspect ratio of a partition cannot be less than 1:4 (or greater than 4:1). As such, a vertical strip partition with a vertical-to-horizontal aspect ratio of 4:1 can only be further bipartitioned vertically into upper and lower partitions each having a vertical-to-horizontal aspect ratio of 2:1.

[0118] In yet another example, a ternary partitioning scheme may be used to partition the base block or any intermediate block, as shown in FIG. 13. The ternary pattern may be implemented vertically, as shown at 1302 in FIG. 13, or horizontally, as shown at 1304 in FIG. 13. The exemplary split ratio in FIG. 13 in the vertical or horizontal direction is shown as 1:2:1, but other ratios may be predefined. In some implementations, two or more different ratios may be predefined. Such a ternary partitioning scheme may be used to complement a quadtree or binary tree partitioning structure in that such ternary tree partitioning can capture objects located at block centers in one continuous partition, while quadtrees and binary trees always split along block centers, resulting in objects being separated into separate partitions. In some implementations, the width and height of the partitions of the exemplary ternary tree are always powers of two to avoid additional transformations.

[0119] The above partitioning schemes may be combined in any manner at different partitioning levels. As an example, the above-mentioned quadtree partitioning scheme and binary partitioning scheme may be combined to partition the base block into a quadtree-binary-tree (QTBT) structure. In such a scheme, the base block or intermediate blocks / partitions, if specified, may be quadtree-split or binary-split according to a set of predefined conditions. A concrete example is depicted in FIG. 14. In the example of FIG. 14, the base block is first quadtree-split into four partitions, as indicated by 1402, 1404, 1406 and 1408. Thereafter, each of the resulting partitions is quadtree partitioned at the next level into four further partitions (e.g., 1408), or bisected (e.g., either horizontally or vertically) into two further partitions (e.g., 1402 or 1406, both symmetric), or not partitioned (e.g., 1404). Binary or quadtree partitioning may be allowed recursively for square-shaped partitions, as shown by the overall exemplary partitioning pattern of 1410 and the corresponding tree structure / representation of 1420, where solid lines represent quadtree partitioning and dashed lines represent bisecting partitioning. A flag may be used for each bisecting partition node (non-leaf bisecting partition) to indicate whether the bisecting partition is horizontal or vertical. For example, as shown at 1420, a flag "0" may represent a horizontal bisecting partition and a flag "1" may represent a vertical bisecting partition, consistent with the partitioning structure of 1410. In the case of a quadtree split partition, there is no need to indicate the split type, since a quadtree split always splits a block or partition both horizontally and vertically to generate four sub-blocks / partitions of equal size. In some implementations, a flag "1" may represent a horizontal binary split, and a flag "0" may represent a vertical binary split.

[0120] In some implementations of QTBT, the quadtree and binary partitioning rule sets may be represented by the following predefined parameters and their associated corresponding functions: CTU size: Root node size of the quadtree (size of the base block) MinQTSize: The minimum allowable quadtree leaf node size. ·MaxBTSize: Maximum allowed binary tree root node size · MaxBTDepth: Maximum allowed binary tree depth ·MinBTSize: The minimum allowable binary tree leaf node size.

[0121] In some implementations of the QTBT partitioning structure, the CTU size may be set as 128×128 luma samples with two corresponding 64×64 blocks of chroma samples (if an example chroma subsampling is considered and used), MinQTSize may be set as 16×16, MaxBTSize may be set as 64×64, MinBTSize (for both width and height) may be set as 4×4, and MaxBTDepth may be set as 4. Quadtree partitioning may first be applied to the CTU to generate quadtree leaf nodes. A quadtree leaf node may have a size from its minimum allowed size of 16×16 (i.e., MinQTSize) to 128×128 (i.e., CTU size). If the node is 128×128, the size exceeds MaxBTSize (i.e., 64×64), so the node is not initially split by the binary tree. Otherwise, nodes that do not exceed MaxBTSize may be partitioned by a bitree. In the example of FIG. 14, the base block is 128×128. The base block can only be quadtree partitioned according to a predefined set of rules. The base block has a partitioning depth of 0. Each of the resulting four partitions is 64×64, which does not exceed MaxBTSize, and may be further quadtree or bitree partitioned at level 1. The process continues. When the bitree depth reaches MaxBTDepth (i.e., 4), no further splits are possible. If a bitree node has a width equal to MinBTSize (i.e., 4), no further horizontal splits are possible. Similarly, if a bitree node has a height equal to MinBTSize, no further vertical splits are possible.

[0122] In some embodiments, the above QTBT scheme may be configured to support flexibility for luma and chroma to have the same QTBT structure or separate QTBT structures. For example, in the case of P slices and B slices, the luma CTB and chroma CTB in one CTU may share the same QTBT structure. However, in the case of I slices, the luma CTB may be partitioned into CBs by a QTBT structure, and the chroma CTB may be partitioned into chroma CBs by another QTBT structure. This means that a CU may be used to refer to different color channels in an I slice, for example, an I slice may consist of a coding block of a luma component and a coding block of two chroma components, and a CU in a P or B slice may consist of coding blocks of all three color components.

[0123] In some other implementations, the QTBT scheme may be supplemented by the ternary scheme described above. Such implementations may be referred to as multi-type-tree (MTT) structures. For example, in addition to the binary partitioning of nodes, one of the ternary partition patterns of FIG. 13 may be selected. In some implementations, only square nodes may undergo ternary partitioning. An additional flag may be used to indicate whether the ternary partitioning is horizontal or vertical.

[0124] The design of two-level or multi-level trees, such as the QTBT implementation and the QTBT implementation supplemented by a ternary partition, may be motivated primarily by reducing complexity. In theory, the complexity of traversing a tree is T D where T represents the number of split types and D is the depth of the tree. Trade-offs may be made by using multiple types (T) while reducing the depth (D).

[0125] In some implementations, the CB may be further partitioned. For example, the CB may be further partitioned into multiple prediction blocks (PBs) for intra- or inter-frame prediction during the encoding and decoding processes. In other words, the CB may be further divided into different sub-partitions, where separate prediction decisions / configurations may be made. In parallel, the CB may be further partitioned into multiple transform blocks (TBs) to delineate the level at which the transformation or inverse transformation of the video data is performed. The partitioning scheme of the CB into PBs and TBs may be the same or different. For example, each partitioning scheme may be performed using its own procedure, for example, based on various characteristics of the video data. The PB and TB partitioning schemes may be independent in some embodiments. The PB and TB partitioning schemes and boundaries may be correlated in some implementations. In some implementations, for example, the TB may be partitioned after the PB partition, and in particular, each PB may be determined following the partitioning of the coding blocks, and then further partitioned into one or more TBs. For example, in some implementations, the PB may be divided into one, two, four, or some other number of TBs.

[0126] In some implementations, for the partitioning of base blocks into coding blocks, and further into predictive and / or transform blocks, the luma and chroma channels may be treated differently. For example, in some implementations, partitioning of coding blocks into predictive and / or transform blocks may be allowed for the luma channel, while such partitioning of coding blocks into predictive and / or transform blocks may not be allowed for the chroma channels. In such implementations, transformation and / or prediction of luma blocks may thus only be performed at the coding block level. As another example, the minimum transform block size of the luma and chroma channels may be different, e.g., coding blocks of the luma channel may be allowed to be partitioned into smaller transform and / or predictive blocks than the chroma channels. As yet another example, the maximum depth of partitioning of coding blocks into transform and / or predictive blocks may be different between the luma and chroma channels, e.g., coding blocks of the luma channel may be allowed to be partitioned into deeper transform and / or predictive blocks than the chroma channels. As a specific example, a luma coding block may be partitioned into transform blocks of multiple sizes that can be repeated by recursive partitioning down to a maximum of two levels, allowing transform block shapes such as square, 2:1 / 1:2, and 4:1 / 1:4, and transform block sizes from 4×4 to 64×64. In the case of a chroma block, however, the maximum number of transform blocks specified for the luma block may be allowed.

[0127] In some implementations for partitioning a coding block into PBs, the depth, shape, and / or other characteristics of the PB partitioning may depend on whether the PB is intra- or inter-coded.

[0128] Partitioning of coding blocks (or prediction blocks) into transform blocks may be performed recursively or non-recursively in various exemplary schemes, including but not limited to quadtree partitioning and predefined pattern partitioning, with further consideration of transform blocks at coding or prediction block boundaries. In general, the resulting transform blocks may be at different partitioning levels, may not be the same size, and may not need to be square in shape (e.g., they can be rectangular with any allowed size and aspect ratio). Further examples are described in more detail below in connection with Figures 15, 16, and 17.

[0129] In some other implementations, however, the CB obtained by any of the above partitioning schemes may be used as a basic or minimum coding block for prediction and / or transformation. In other words, no further division is performed for inter / intra prediction and / or transformation. For example, the CB obtained by the above QTBT scheme may be directly used as a unit for performing prediction. Specifically, such a QTBT structure removes the concept of multiple partition types, e.g., it removes the separation of CU, PU, ​​and TU, and supports more flexibility of the CU / CB partition shape described above. In such a QTBT block structure, the CU / CB can have either a square or rectangular shape. The leaf nodes of such a QTBT are used as units for prediction and transformation processing without further partitioning. This means that the CU, PU, ​​and TU have the same block size in such an exemplary QTBT coding block structure.

[0130] The various CB partitioning schemes described above, as well as further partitioning of the CB into PB and / or TB (including no PB / TB partitioning), may be combined in any manner. The following specific implementations are provided as non-limiting examples.

[0131] Specific examples of partitioning of coding blocks and transform blocks are described below. In such examples, a base block may be partitioned into coding blocks using recursive quadtree partitioning or the above predefined partitioning patterns (e.g., the partitioning patterns of Figs. 9 and 10). At each level, whether further quadtree partitioning of a particular partition should be followed may be determined by local video data characteristics. The resulting CBs may be at different quadtree partitioning levels and different sizes. The decision on whether to code a picture area by inter-picture (temporal) or intra-picture (spatial) prediction may be made at the CB level (or at the CU level for all three color channels). Each CB may be further partitioned into one, two, four, or other number of PBs according to a predefined PB partition type. Within one PB, the same prediction process may be applied, and related information may be conveyed to the decoder on a PB basis. After obtaining the residual blocks by applying the prediction process based on the PB partition type, the CBs may be partitioned into TBs according to another quadtree structure similar to the coding tree of the CB. In this particular implementation, however, the CB or TB need not be restricted to a square shape. Furthermore, in this particular example, the PB may be square or rectangular shaped for inter prediction, and only square for intra prediction. A coding block may be divided, for example, into four square shaped TBs. Each TB may be further divided recursively (by quadtree partitioning) into smaller TBs called Residual Quadtrees (RQTs).

[0132] Other embodiments for partitioning the base block into CB, PB and / or TB are described further below. For example, instead of using multiple partition unit types such as those shown in FIG. 9 or FIG. 10, a quadtree segmentation structure (e.g., QTBT as described above or QTBT with ternary partitioning) including nested multi-type trees with bipartite and ternary partitioning may be used. The separation of CB, PB and TB (i.e., partitioning of CB into PB and / or TB and partitioning of PB into TB) may be abandoned except when required for CBs whose size is too large for the maximum transform length. However, such CBs may need to be further divided. This exemplary partitioning scheme may be designed to support further flexibility of CB partition shape such that both prediction and transformation can be performed at the CB level without further partitioning. In such coding tree structure, the CB may have either a square or rectangular shape. Specifically, the coding tree block (CTB) may be first partitioned by a quadtree structure. The quadtree leaf nodes may then be further partitioned by nested multi-type trees. An example of a nested multi-type tree structure using bisection or trisection is shown in FIG. 11. Specifically, the example multi-type tree structure of FIG. 11 includes four split types called vertical bisection (SPLIT_BT_VER) (1102), horizontal bisection (SPLIT_BT_HOR) (1104), vertical trisection (SPLIT_TT_VER) (1106), and horizontal trisection (SPLIT_TT_VER) (1108). CB then corresponds to the leaf of the multi-type tree. In this embodiment, as long as CT is not too large relative to the maximum transform length, this segmentation is used for both prediction and transform processing without further partitioning. This means that in most cases, CB, PB, and TB have the same block size in the quadtree structure containing the nested multi-type tree coding block. An exception occurs when the maximum supported transform length is smaller than the width or height of the color components of CB.In some implementations, in addition to the binary or ternary division, the nested pattern of FIG. 11 may further include a quadtree division.

[0133] One example of a quadtree structure including nested multi-type tree coding block of block partitions (including quadtree, bisection, and ternary partition options) on a block basis is shown in FIG. 12. More specifically, FIG. 12 shows a base block 1200 being quadtree partitioned into four square partitions 1202, 1204, 1206, and 1208. A decision to further use the multi-tree structure and quadtree of FIG. 11 for further partitioning is made for each of the quadtree partitioned partitions. In the example of FIG. 12, partition 1204 is not further partitioned. Partitions 1202 and 1208 each employ another quadtree partitioning. For partition 1202, the top-left, top-right, bottom-left, and bottom-right partitions quadtree partitioned at the second level employ quadtree partitioning at the third level, horizontal bisection 1104 of FIG. 11, no partition, and horizontal thirds partition 1108 of FIG. 11, respectively. Partition 1208 employs another quadtree partitioning, with the top-left, top-right, bottom-left, and bottom-right partitions quadtree partitioned at the second level employing vertical thirds partition 1106 of FIG. 11, no partition, no partition, and horizontal bisection 1104 of FIG. 11, respectively, at the third level. Two of the third level subpartitions of the top-left partition at the second level of 1208 are further partitioned according to horizontal bisection 1104 of FIG. 11 and horizontal thirds partition 1108 of FIG. 11, respectively. Partition 1206 is divided into two partitions employing a second level division pattern according to vertical bisection 1102 of Figure 11, and the two partitions are further divided at a third level according to horizontal third section 1108 and vertical bisection 1102 of Figure 11. To one of the two partitions, a fourth level division is further applied according to horizontal bisection 1104 of Figure 11.

[0134] For the above example, the maximum luma transform size may be 64 x 64, and the maximum supported chroma transform size may be different from the luma, e.g., 32 x 32. Even though the CB illustrated in Figure 12 is not generally further divided into smaller PB and / or TB, if the width or height of the luma coding block or chroma coding block is larger than the maximum transform width or height, the luma coding block or chroma coding block may be automatically divided in the horizontal and / or vertical direction to satisfy the transform size constraints in that direction.

[0135] In the specific example for partitioning base blocks into CBs above, as described above, the coding tree scheme may support the ability for luma and chroma to have separate block tree structures. For example, in the case of P slices and B slices, the luma CTB and chroma CTB in one CTU may share the same coding tree structure. In the case of I slices, for example, luma and chroma may have separate coding block tree structures. When separate block tree structures are applied, the luma CTB may be partitioned to the luma CB by one coding tree structure, and the chroma CTB is partitioned to the chroma CB by the other coding tree structure. This means that a CU in an I slice can consist of a coding block of the luma component or a coding block of two chroma components, and a CU in a P or B slice always consists of coding blocks of all three color components unless the video is monochrome.

[0136] When a coding block is further partitioned into multiple transform blocks, the transform blocks therein may be ordered in the bitstream according to various orders or scanning schemes. Examples of partitioning coding or prediction blocks into transform blocks and the coding order of the transform blocks are described further below. In some embodiments, as described above, the transform partitioning may support transform blocks of multiple shapes, e.g., 1:1 (square), 1:2 / 2:1, and 1:4 / 4:1, with transform block sizes ranging from, e.g., 4×4 to 64×64. In some implementations, when the coding block is 64×64 or smaller, the transform block partitioning may be applied only to the luma component, such that for chroma blocks, the transform block size is the same as the coding block size. Otherwise, if the width or height of the coding block is greater than 64, both the luma coding block and the chroma coding block may be implicitly split into multiple transform blocks of min(W,64)×min(H,64) and min(W,32)×min(H,32), respectively.

[0137] In some implementations of transform block partitioning, for both intra-coded and inter-coded blocks, the coding blocks may be further partitioned into multiple transform blocks with a partitioning depth up to a predefined number of levels (e.g., two levels). The depth and size of the transform block partitioning may be related. For some implementations, the mapping from the transform size of the current depth to the transform size of the next depth is shown below in Table 1. [Table 1]

[0138] Based on the example mapping of Table 1, in the case of a 1:1 square block, the next level transform partitioning may generate four 1:1 square sub-transform blocks. The transform partition may stop at, for example, 4×4. Thus, a transform size of the current depth of 4×4 corresponds to the same size of 4×4 in the next depth. In the example of Table 1, in the case of a 1:2 / 2:1 non-square block, the next level transform partitioning may generate two 1:1 square sub-transform blocks, while in the case of a 1:4 / 4:1 non-square block, the next level transform partitioning may generate two 1:2 / 2:1 sub-transform blocks.

[0139] In some embodiments, additional restrictions may be applied regarding transform block partitioning in the case of the luma component of an intra-coded block. For example, for each level of transform partitioning, all sub-transform blocks may be restricted to have equal size. For example, for a 32×16 coding block, level 1 transform partitioning generates two 16×16 sub-transform blocks, and level 2 transform partitioning generates eight 8×8 sub-transform blocks. In other words, the second level partitioning should be applied to all first level sub-blocks to keep the transform units equal in size. An example of transform block partitioning for an intra-coded square block according to Table 1 is shown in FIG. 15 with the coding order represented by the arrows. Specifically, 1502 denotes a square coding block. The first level partitioning into four equal-sized transform blocks according to Table 1 is shown at 1504 with the coding order indicated by the arrows. All second level partitioning of the first level equal-sized blocks into 16 equal-sized transform blocks according to Table 1 is shown at 1506 with the coding order indicated by the arrows.

[0140] In some embodiments, the above restrictions on intra-coding may not apply in the case of the luma component of an inter-coded block. For example, after the first level of transform partitioning, any one of the sub-transform blocks may be further partitioned independently at one further level. Thus, the resulting transform blocks may or may not be of the same size. An example of partitioning of an inter-coded block into transform blocks is shown in FIG. 16 together with their coding order. In the example of FIG. 16, an inter-coded block 1602 is partitioned into transform blocks at two levels according to Table 1. At the first level, the inter-coded block is partitioned into four transform blocks of equal size. Then, only one of the four transform blocks (but not all of them) is further partitioned into four sub-transform blocks, resulting in a total of seven transform blocks with two different sizes, as shown by 1604. An example of the coding order of these seven transform blocks is shown by the arrows at 1604 in FIG. 16.

[0141] In some embodiments, for chroma components, some additional restrictions on transform blocks may be applied, e.g., for chroma components, the transform block size can be as large as the coding block size, but cannot be smaller than a predefined size, e.g., 8×8.

[0142] In some other implementations, for coding blocks with either width (W) or height (H) greater than 64, both the luma coding block and the chroma coding block may be implicitly divided into multiple transform units of min(W,64)×min(H,64) and min(W,32)×min(H,32), respectively, where in this disclosure, "min(a,b)" may return the smaller value of a and b.

[0143] Figure 17 further illustrates another alternative example of a scheme for partitioning coding blocks or predictive blocks into transform blocks. As shown in Figure 17, rather than using recursive transform partitioning, a predefined set of partitioning types may be applied to a coding block depending on the transform type of the coding block. In the example shown in Figure 17, one of six example partitioning types may be applied to split the coding block into a varying number of transform blocks. Such a scheme to generate transform block partitioning may be applied to either coding blocks or predictive blocks.

[0144] More specifically, the partitioning scheme of FIG. 17 provides up to six example partition types for any given transform type (transform type refers to the type of primary transform, e.g., ADST and others). In this scheme, every coding block or predictive block may be assigned a transform partition type, e.g., based on rate-distortion cost. In one example, the transform partition type assigned to a coding block or predictive block may be determined based on the transform type of the coding block or predictive block. A particular transform partition type may correspond to a transform block division size and pattern, as illustrated by the six transform partition types depicted in FIG. 17. The correspondence between various transform types and various transform partitions may be predefined. Examples are illustrated below, using capitalized labels to indicate transform partition types that may be assigned to a coding block or predictive block based on rate-distortion cost: ·PARTITION_NONE: Allocate the transformation size equal to the block size. ·PARTITION_SPLIT: Allocates a transformation size that is 1 / 2 the width of the block size and 1 / 2 the height of the block size. ·PARTITION_HORZ: Allocates a transform size with width equal to the block size and 1 / 2 the height of the block size. ·PARTITION_VERT: Allocates a transformation size that has a width of 1 / 2 the block size and a height equal to the block size. ·PARTITION_HORZ4: Allocates a transformation size with width equal to the block size and 1 / 4 of the block size height. · PATRITION_VERT4: Assigns a transformation size that has a width of 1 / 4 of the block size and a height equal to the block size.

[0145] In the above example, all of the transform partition types shown in Figure 17 include a uniform transform size for the partitioned transform blocks. This is by way of example only and not limitation. In some other implementations, mixed transform block sizes may be used for the partitioned transform blocks in a particular partition type (or pattern).

[0146] The PBs (or CBs, also called PBs when not further partitioned into predictive blocks) obtained from any of the above partitioning schemes can then become individual blocks for coding by either intra-prediction or inter-prediction. In the case of inter-prediction for the current PB, a residual between the current block and the predictive block may be generated, coded, and included in the coded bitstream.

[0147] Inter prediction may be performed, for example, in a single reference mode or a compound-reference mode. In some implementations, a skip flag may be included initially in the bitstream of the current block (or at a higher level) to indicate whether the current block is inter-coded and should not be skipped. If the current block is inter-coded, another flag may be further included in the bitstream as a signal to indicate whether a single reference mode or a compound reference mode is used for prediction of the current block. In the case of a single reference mode, one reference block may be used to generate a prediction block for the current block. In the case of a compound reference mode, two or more reference blocks may be used to generate a prediction block, for example, by weighted averaging. A compound reference mode may be referred to as a more-than-one reference mode, a two-reference mode, or a multiple-reference mode. One or more reference blocks may be identified using a reference frame index and also using a corresponding motion vector that indicates a shift in position between the reference block and the current block, for example, in horizontal and vertical pixels. For example, in single reference mode, the inter-predicted block of the current block may be generated from a single reference block identified as the predictive block by one motion vector in a reference frame, while in mixed reference mode, the predictive block may be generated by a weighted average of two reference blocks in two reference frames indicated by two reference frame indexes and two corresponding motion vectors. The motion vectors may be coded in various ways and included in the bitstream.

[0148] In some implementations, the encoding or decoding system may maintain a decoded picture buffer (DPB). Some images / pictures may be maintained in the DPB waiting to be displayed (in decoding order), and some images / pictures in the DPB may be used as reference frames to enable inter-prediction (in the decoding system or the encoding system). In some implementations, the reference frames in the DPB may be tagged as either short-term or long-term references for the current image being encoded or decoded. For example, short-term reference frames may include frames used for inter-prediction of blocks in the current frame or in a predefined number (e.g., 2) of subsequent video frames that are closest to the current frame in decoding order. Long-term reference frames are used to predict image blocks in frames that are more than a predefined number of frames away from the current frame in decoding order. Information regarding such tags of short-term and long-term reference frames may be referred to as a Reference Picture Set (RPS) and may be added to the header of each frame in the encoded bitstream. Each frame in the encoded video stream may be identified by a Picture Order Counter (POC), which is numbered according to playback order in an absolute manner, or relative to a group of pictures starting, for example, from an I-frame.

[0149] In some embodiments, one or more reference picture lists including identification of short-term and long-term reference frames for inter prediction may be formed based on information of the RPS. For example, a single picture reference list may be formed for unidirectional inter prediction and denoted as L0 reference (or reference list 0), while two picture reference lists may be formed for bidirectional inter prediction and denoted as L0 (or reference list 0) and L1 (or reference list 1) for each of the two prediction directions. The reference frames included in the L0 list and the L1 list may be ordered in various predefined manners. The length of the L0 list and the L1 list may be signaled in the video bitstream. In the case where multiple references for generating a prediction block by weighted averaging in a combined prediction mode are on the same side of the block to be predicted, the combined prediction mode may be either a single reference mode or a combined reference mode. Bidirectional inter prediction is only a combined mode in that bidirectional inter prediction requires at least two reference blocks.

[0150] In some implementations, a merge mode (MM) for inter prediction may be implemented. In general, for the merge mode, one or more of the motion vectors in single reference prediction or the motion vectors in mixed reference prediction for the current PB may be derived from other motion vectors, rather than being calculated and signaled independently. For example, in an encoding system, a current motion vector of a current PB may be represented by a difference between the current motion vector and one or more other already coded motion vectors (referred to as reference motion vectors). Such a difference of the motion vector, rather than the entire current motion vector, may be coded and included in the bitstream and may be associated with the reference motion vector. Correspondingly, in a decoding system, a motion vector corresponding to the current PB may be derived based on the decoded motion vector difference and the reference motion vector associated therewith. As a specific form of the general merge mode (MM) inter prediction, such inter prediction based on the motion vector difference may be referred to as a merge mode with motion vector difference (MMVD). MM in general, or MMVD in particular, may thus be implemented to exploit correlations between motion vectors associated with different PPBs to improve coding efficiency. For example, adjacent PBs may have similar motion vectors, so that the MVD may be small and can be coded efficiently. As another example, motion vectors may be temporally correlated (between frames) for similarly located / positioned blocks in space.

[0151] In some embodiments, an MM flag may be included in the bitstream during encoding to indicate whether the current PB is in merge mode. Additionally or alternatively, an MMVD flag may be included in the bitstream and signaled during encoding to indicate whether the current PB is in MMVD mode. The MM and / or MMVD flags or indicators may be provided at the PB level, CB level, CU level, CTB level, CTU level, slice level, picture level, etc. For a particular example, both the MM flag and the MMVD flag may be included for the current CU, and the MMVD flag may be signaled immediately after the skip flag and the MM flag to specify whether the MMVD mode is used for the current CU.

[0152] In some implementations of MMVD, a list of reference motion vector (RMV) or MV predictor candidates for motion vector prediction may be formed for the block being predicted. The list of RMV candidates may include a predefined number (e.g., 2) of MV predictor candidate blocks whose motion vectors may be used to predict the current motion vector, and the RMV candidate blocks may include blocks selected from adjacent blocks in the same frame and / or temporal blocks (co-located blocks in preceding or subsequent frames of the current frame). These options represent blocks in a spatial or temporal position relative to the current block that are likely to have a similar or the same motion vector as the current block. The size of the list of MV predictor candidates may be pre-determined. For example, the list may include two or more candidates. To be on the RMV candidate list, a candidate block may be required, for example, to have the same reference frame (or frames) as the current block, must exist (e.g., a boundary check needs to be performed if the current block is close to the edge of the frame), must have been already coded during the coding process, and / or must have been already decoded during the decoding process. In some implementations, the list of merge candidates may be first populated with spatially neighboring blocks (scanned in a certain predefined order) if available and satisfy the above conditions, and then with temporal blocks if space is still available in the list. Neighboring RMV candidate blocks may be selected, for example, from the blocks to the left and above the current block. The list of RMV predictor candidates may be dynamically formed at different levels (sequence, picture, frame, slice, superblock, etc.) as a Dynamic Reference List (DRL). The DRL may be signaled in the bitstream.

[0153] In some implementations, the actual MV predictor candidate used as the reference motion vector to predict the motion vector of the current block may be signaled. When the RMV candidate list includes two candidates, a one-bit flag called a merge candidate flag may be used to indicate the selection of the reference merge candidate. When the current block is predicted in a mixed mode, each of the multiple motion vectors predicted by the MV predictor may be associated with a reference motion vector from the merge candidate list. The encoder may determine which of the RMV candidates more closely predicts the current coding block and signal the selection as an index in the DRL.

[0154] In some implementations of MMVD, after an RMV candidate is selected and used as a base motion vector prediction for predicting a motion vector, a motion vector differential (MVD or delta MV representing the difference between the motion vector to be predicted and the reference candidate motion vector) may be calculated in the coding system. Such MVD may include information representing the magnitude of the MV difference and the direction of the MV difference, both of which may be signaled in the bitstream. The magnitude of the motion difference and the direction of the motion difference may be signaled in various ways.

[0155] In some implementations of the MMVD, the distance index may be used to specify the magnitude information of the motion vector differential and to indicate one of a set of predefined offsets that represent a predefined motion vector differential from a starting point (reference motion vector). The MV offset according to the signaled index may then be added to either the horizontal or vertical component of the starting (reference) motion vector. Whether the horizontal or vertical component of the reference motion vector should be offset may be determined by the direction information of the MVD. An example of a predefined relationship between the distance index and the predefined offsets is specified in Table 2. [Table 2]

[0156] In some embodiments of MMVD, the distance index may be further signaled and used to indicate the direction of the MVD relative to the reference motion vector. In some implementations, the direction may be restricted to either the horizontal or vertical direction. As an example, a 2-bit direction index is shown in Table 3. In the example of Table 3, the interpretation of the MVD may change depending on the information of the start / reference MV. For example, when the start / reference MV corresponds to a uni-predictive block or a bi-predictive block, and both reference frame lists point to the same side of the current picture (i.e., the POCs of the two reference pictures are both greater than the POC of the current picture, or both less than the POC of the current picture), the code in Table 3 may specify the sign (direction) of the MV offset added to the start / reference MV. If the start / reference MV corresponds to a bi-predictive block with two reference pictures on different sides of the current picture (i.e., the POC of one reference picture is larger than the POC of the current picture and the POC of the other reference picture is smaller than the POC of the current picture), and the difference between the reference POC in picture reference list 0 and the current frame is larger than the difference between the reference POC in picture reference list 1 and the current frame, then the code in Table 3 may specify the sign of the MV offset added to the reference MV corresponding to the reference picture in picture reference list 0, and the sign of the offset of the MV corresponding to the reference picture in picture reference list 1 may have an inverse value (inverse sign of the offset). Otherwise, if the difference between the reference POC in picture reference list 1 and the current frame is larger than the difference between the reference POC in picture reference list 0 and the current frame, then the code in Table 3 may specify the sign of the offset added to the reference MV associated with picture reference list 1, and the sign of the offset for the reference MV associated with picture reference list 0 has an inverse value. [Table 3]

[0157] In some embodiments, the MVD may be scaled according to the difference in POC in each direction. If the difference from the POC in both lists is the same, no scaling is necessary. Otherwise, if the difference in POC in reference list 0 is larger than that of reference list 1, the MVD of reference list 1 is scaled. If the POC difference of reference list 1 is larger than list 0, the MVD of list 0 may be scaled in the same way. If the starting MV is uni-predicted, the MVD is added to the available MV or reference MV.

[0158] In some embodiments of MVD coding and signaling for bidirectional mixed prediction, in addition to or instead of coding and signaling two MVDs separately, symmetric MVD coding may be implemented, so that only one MVD needs to be signaled, and the other MVD can be derived from the signaled MVD. In such implementations, motion information including reference picture indexes of both list 0 and list 1 is signaled. However, for example, only the MVD associated with reference list 0 is signaled, and the MVD associated with reference list 1 is derived without being signaled. Specifically, a flag called "mvd_l1_zero_flag" may be included in the bitstream at slice level to indicate whether reference list 1 is not signaled in the bitstream. If this flag is 1, indicating that reference list 1 is equal to zero (and thus not signaled), a bidirectional prediction flag called "BiDirPredFlag" may be set to 0, meaning there is no bidirectional prediction. Otherwise, if mvd_l1_zero_flag is zero, then if the closest reference picture in list 0 and the closest reference picture in list 1 form a forward and backward pair of reference pictures or a backward and forward pair of reference pictures, then BiDirPredFlag may be set to 1, and the reference pictures in both list 0 and list 1 are short-term reference pictures. Otherwise, BiDirPredFlag is set to 0. A BiDirPredFlag of 1 may indicate that a symmetric mode flag is further signaled in the bitstream. A decoder may extract the symmetric mode flag from the bitstream when BiDirPredFlag is 1. The symmetric mode flag may be signaled, for example, at the CU level (optionally), which may indicate whether a symmetric MVD coding mode is used for the corresponding CU. When the symmetric mode flag is 1, it enables the use of symmetric MVD coding mode and the reference picture indices of both list 0 and list 1 (called "mvp_l0_flag" and "mvp_l1_flag").) is signaled along with the MVD associated with list 0 (referred to as "MVD0"), indicating that the other motion vector differential "MVD1" should be derived rather than signaled. For example, MVD1 may be derived as -MVD0. Thus, in the exemplary symmetric MVD mode, only one MVD is signaled. In some other embodiments of MV prediction, harmonized schemes may be used to implement general merge mode, MMVD, and other types of MV prediction, for both single reference mode and mixed reference mode MV prediction. Various syntax elements may be used to signal how the MV of the current block is predicted.

[0159] For example, in case of a single reference mode, the following MV prediction modes may be signaled: · NEARMV: Uses one of the motion vector predictors (MVP) in the list pointed to by the DRL (Dynamic Reference List) index directly without any MVD. · NEWMV: Use one of the motion vector predictors (MVP) in the list informed by the DRL index as a reference and apply a delta to the MVP (eg, using MVD). ·GLOBALMV: Uses motion vectors based on frame-level global motion parameters.

[0160] Similarly, for a mixed reference inter prediction mode using two reference frames corresponding to the two MVs to be predicted, the following MV prediction modes may be signaled: · NEAR_NEARMV: Use one of the motion vector predictors (MVPs) in the list signalled by the DRL index without the MVD for each of the two MVs to be predicted. NEAR_NEWMV: To predict the first of the two motion vectors, one of the motion vector predictors (MVP) in the list signalled by the DRL index is used as a reference MV without the MVD, and to predict the second of the two motion vectors, one of the motion vector predictors (MVP) in the list signalled by the DRL index is used as a reference MV together with the additionally signalled delta MV (MVD). · NEW_NEARMV: To predict the second of the two motion vectors, one of the motion vector predictors (MVP) in the list signalled by the DRL index is used as a reference MV without an MVD, and to predict the first of the two motion vectors, one of the motion vector predictors (MVP) in the list signalled by the DRL index is used as a reference MV together with the additionally signalled delta MV (MVD). · NEW_NEWMV: Use one of the motion vector predictors (MVPs) in the list signalled by the DRL index as a reference MV, which is used together with the further signalled delta MV to make a prediction for each of the two MVs. · GLOBAL_GLOBALMV: Use the MV from each reference based on their frame-level global motion parameters.

[0161] The above term "NEAR" refers to MV prediction using a reference MV without MVD as a general merge mode, while "NEW" refers to MV prediction using a reference MV and offsetting it with a signaled MVD as seen in MMVD mode. In the case of composite inter prediction, both the above reference-based motion vector and motion vector delta may be generally different or independent between the two references, even if they are correlated and such correlation is exploited to reduce the amount of information required to signal the two motion vector deltas. In such a situation, joint signaling of the two MVDs may be implemented and indicated in the bitstream.

[0162] The dynamic reference list (DRL) mentioned above may be dynamically maintained and used to hold the set of indexed motion vectors that are considered as candidate motion vector predictors.

[0163] In some embodiments, a predefined resolution of the MVD may be allowed. For example, a motion vector precision (or accuracy) of 1 / 8 pixel may be allowed. The above MVDs in different MV prediction modes may be configured and signaled in different ways. In some implementations, different syntax elements may be used to signal the above motion vector differentials in reference frame list 0 or list 1.

[0164] For example, a syntax element called "mv_joint" may specify which components of its associated motion vector differential are non-zero. For MVD, this is signaled for all non-zero components together. For example, mv_joint has the following values: 0 may indicate that there is no non-zero MVD in either the horizontal or vertical direction. A 1 may indicate that there is non-zero MVD only along the horizontal direction. 2 may indicate that there is non-zero MVD only along the vertical direction. 3 may indicate that there is non-zero MVD in both the horizontal and vertical directions.

[0165] When the "mv_joint" syntax element of the MVD indicates that no non-zero MVD is present, no further MVD information may be signaled. However, if the "mv_joint" syntax element indicates that one or two non-zero components are present, additional syntax elements may be further signaled for each of the non-zero MVD components, as described below.

[0166] For example, a syntax element called "mv_sign" may be used to further specify whether the corresponding motion vector differential component is positive or negative.

[0167] As another example, a syntax element called "mv_class" may be used to specify a class of a motion vector difference from among a set of predefined classes for the corresponding non-zero MVD component. The predefined classes of motion vector difference may be used, for example, to divide a continuous magnitude space of motion vector difference into non-overlapping ranges, each range corresponding to an MVD class. Thus, the signaled MVD class indicates the magnitude range of the corresponding MVD component. In the example shown below in Table 4, higher classes correspond to motion vector differences having a larger magnitude range. In Table 4, the symbol (n,m] is used to represent a range of motion vector difference that is greater than n pixels and less than or equal to m pixels. [Table 4]

[0168] In some other examples, a syntax element called "mv_bit" may further be used to specify the integer part of the offset between a non-zero motion vector differential component and the starting magnitude of the MV class magnitude range that is signaled accordingly. The number of bits required in "mv_bit" to signal the full range of each MVD class may vary depending on the MV class. For example, MV_CLASS0 and MV_CLASS1 in the implementation of Table 4 require only a single bit to indicate an integer pixel offset of 1 or 2 from a starting MVD of 0, and each higher MV_CLASS in the example of Table 4 requires one more bit in "mv_bit" than the previous MV_CLASS.

[0169] In some other examples, a syntax element called "mv_fr" may be further used to specify the first two fractional bits of the motion vector differential for the corresponding non-zero MVD component, while a syntax element called "mv_hp" may be used to specify the third fractional bit (high resolution bit) of the motion vector differential for the corresponding non-zero MVD component. The one bit "mv_fr" essentially provides ¼ pixel MVD resolution, while the "mv_hp" bit may further provide ⅛ pixel resolution. In some other implementations, more than one "mv_hp" may be used to provide MVD pixel resolution finer than ⅛ pixel. In some embodiments, additional flags may be signaled at one or more of various levels to indicate whether ⅛ pixel or higher MVD resolution is supported. If MVD resolution is not applied to a particular coding unit, the above syntax elements may not be signaled for the corresponding unsupported MVD resolution.

[0170] In some of the above embodiments, the fractional resolution can be independent of different classes of MVD. In other words, regardless of the magnitude of the motion vector differential, similar options of motion vector resolution may be provided using a predefined number of "mv_fr" and "mv_hp" bits to signal fractional MVD of non-zero MVD components.

[0171] However, in some other embodiments, the resolution of the motion vector difference at various MVD magnitude classes may be differentiated. Specifically, a high-resolution MVD for a large MVD magnitude of a higher MVD class may not bring a statistically significant improvement in comparison efficiency. Therefore, the MVD may be coded with a lower resolution (integer pixel resolution or fractional pixel resolution) for a larger MVD magnitude range corresponding to a higher MVD magnitude class. Similarly, the MVD may be coded with a lower resolution (integer pixel resolution or fractional pixel resolution) for a larger MVD value in general. Such MVD class-dependent or MVD magnitude-dependent MVD resolution may be generally referred to as adaptive MVD resolution, amplitude-dependent adaptive MVD resolution, or magnitude-dependent MVD resolution. The term "resolution" may further be referred to as "pixel resolution". The adaptive MVD resolution may be implemented in various ways described by the following embodiments to achieve better overall compression efficiency. In particular, due to the statistical observation that treating the MVD resolution of a large-scale or high-class MVD at a similar level to that of a low-scale or low-class MVD in an unadapted manner cannot significantly improve the inter-prediction residual coding efficiency of a block with a large-scale or high-class MVD, the reduction in the number of signaling bits by aiming for a less accurate MVD may be greater than the additional bits required to code the inter-prediction residual as a result of such a less accurate MVD. In other words, using a higher MVD resolution for a large-scale or high-class MVD may not bring much coding gain over using a lower MVD resolution.

[0172] In some typical implementations, the pixel resolution or precision of the MVD may or may not decrease with increasing MVD class. Reducing the pixel resolution of the MVD makes the MVD coarser (i.e., the step from one MVD level to the next MVD level is larger). In some implementations, the correspondence between MVD pixel resolution and MVD class may be specified, predefined, or preset, and thus may not need to be signaled in the encoded bitstream.

[0173] In some embodiments, the MV classes in Table 3 may each be associated with a different MVD pixel resolution.

[0174] In some implementations, each MVD class may be associated with a single permitted resolution. In some other implementations, one or more MVD classes may be associated with two or more arbitrary MVD pixel resolutions. A signal in the bitstream for a current MVD component having such an MVD class may then be followed by additional signaling indicating which arbitrary pixel resolution is selected for the current MVD component.

[0175] In some embodiments, the adaptively allowed MVD pixel resolutions may include, but are not limited to, 1 / 64pel (pixels), 1 / 32pel, 1 / 16pel, 1 / 8pel, 1 / 4pel, 1 / 2pel, 1pel, 2pel, 4pel, etc. (in ascending order of resolution). Thus, each one of the ascending MVD classes may be associated with one of these resolutions in non-ascending order. In some implementations, an MVD class may be associated with two or more of the above resolution classes, where the higher resolution may be lower than or equal to the lower resolution of the previous MVD class. For example, if MV_CLASS_3 in Table 4 may be associated with any 1pel and 2pel resolution, then the highest resolution that MV_CLASS_4 in Table 4 may be associated with would be 2pel. In some other implementations, the highest allowable resolution of an MV class may be higher than the lowest allowable resolution of the previous (lower) MV class. However, the average allowed resolutions of the ascending MV classes may only be in non-ascending order.

[0176] In some implementations, if fractional pixel resolution higher than 1 / 8 pel is allowed, then the "mv_fr" and "mv_hp" signaling may be extended accordingly to a total of more than three fractional bits.

[0177] In some embodiments, fractional pixel resolution may only be allowed for MVD classes below the threshold MVD class. For example, fractional pixel resolution may only be allowed for MVD_CLASS_0 and not for all other classes in Table 4. Similarly, fractional pixel resolution may only be allowed for MVD classes below any one of the other MV classes in Table 4. For the remaining MVD classes above the threshold MVD class, only integer pixel resolution of the MVD is allowed. In that way, fractional resolution signaling such as one or more of the "mv_fr" and / or "mv_hp" bits may not need to be signaled for MVDs signaled in MVD classes above the threshold MVD class. In case of MVD classes having a resolution lower than one pixel, the number of bits in the "mv_bit" signaling can be further reduced. For example, for MV_CLASS_5 in Table 4, the range of MVD pixel offset is (32,649], so 5 bits are needed to signal the full range with 1 pel resolution. However, if MV_CLASS_5 is associated with 2 pel MVD resolution (lower than 1 pixel resolution), then 4 bits are needed for "mv_bit" instead of 5 bits, and neither "mv_fr" nor "mv_hp" need to be signaled after signaling "mv_class" as MV_CLASS_5.

[0178] In some embodiments, fractional pixel resolution may only be allowed for MVDs with integer values ​​below a threshold integer pixel value. For example, fractional pixel resolution may only be allowed for MVDs smaller than 5 pixels. Corresponding to this example, fractional resolution may be allowed for MV_CLASS_0 and MV_CLASS_1 in Table 4, and not allowed for all other MV classes. As another example, fractional pixel resolution may only be allowed for MVDs smaller than 7 pixels. Corresponding to this example, fractional resolution may be allowed for MV_CLASS_0 and MV_CLASS_1 in Table 4 (having a range below 5 pixels), and not allowed for MV_CLASS_3 and higher classes (having a range above 5 pixels). For MVDs belonging to MV_CLASS_2, whose pixel range encompasses 5 pixels, fractional pixel resolution of the MVD may or may not be allowed depending on the value of "mv_bit". Fractional resolution may be allowed if the "mv_bit" value is signaled as 1 or 2 (such that the integer part of the signaled MVD is 5 or 6, calculated as the start of the pixel range of MV_CLASS_2 with the offset 1 or 2 indicated by "mv_bit"). Otherwise, if the "mv_bit" value is signaled as 3 or 4 (such that the integer part of the signaled MVD is 7 or 8), fractional pixel resolution may not be allowed.

[0179] In some other implementations, only a single MVD value may be allowed for MV classes equal to or greater than the threshold MV class. For example, such a threshold MV class may be MV_CLASS_2. Thus, MV_CLASS_2 and above may only be allowed to have a single MVD value and do not have fractional pixel resolution. The single allowed MVD value for these MV classes may be predefined. In some examples, the single allowed value may be the upper limit of the range of each of these MV classes in Table 4. For example, MV_CLASS_2 through MV_CLASS_10 may be equal to or greater than the threshold class of MV_CLASS_2, and the single allowed MVD values ​​for these classes may be predefined as 8, 16, 32, 64, 128, 256, 512, 1024, and 2048, respectively. In some other examples, the single allowed value may be the median of the range of each of these MV classes in Table 4. For example, MV_CLASS_2 through MV_CLASS_10 may be above the class threshold, and the single allowed MVD values ​​for these classes may be predefined as 6, 12, 24, 48, 96, 192, 384, 768, and 1536, respectively. Any other value within the range may also be defined as the single allowed resolution for each MVD class.

[0180] In the above implementation, if the signaled "mv_class" is equal to or greater than the predefined MVD class threshold, then only "mv_class" is sufficient to determine the MVD value. The magnitude and manner of the MVD is then determined using "mv_class" and "mv_sign".

[0181] Therefore, when an MVD is reported for only one reference frame (either from reference frame list 0 or list 1, but not both) or for two reference frames together, the accuracy (or resolution) of the MVD may depend on the associated class of motion vector differential in Table 3 and / or the magnitude of the MVD.

[0182] In some other implementations, the pixel resolution or precision of the MVD may or may not decrease with increasing MVD class. For example, the pixel resolution may depend on the integer part of the MVD magnitude. In some embodiments, fractional pixel resolution may only be allowed for MVD magnitudes below an amplitude threshold. In the case of a decoder, the integer part of the MVD magnitude may first be extracted from the bitstream. Then, the pixel resolution may be determined, and then it may be determined whether any fractional MVD is present in the bitstream and needs to be parsed (e.g., if fractional pixel resolution is not allowed for a particular extracted MVD integer magnitude, then the fractional MVD bits may not be included in the bitstream that requires extraction). The above embodiment regarding adaptive MVD pixel resolution depending on the MVD class applies to adaptive MVD pixel resolution depending on the MVD magnitude. For a particular example, MVD classes above or including a magnitude threshold may be allowed to have only one predefined value.

[0183] The various embodiments described above apply to single reference modes. These implementations also apply to the exemplary NE_NEARMV, NEAR_NEWMV, and / or NEW_NEWMV modes in joint prediction under MMVD. These implementations generally apply to any MVD adaptation resolution.

[0184] When adaptive (more specifically, magnitude-adaptive) pixel resolution of the MVD is used, various parameters related to the MV and MVD of a coding block may be interdependent. Parameters considered related to MV or MVD refer broadly to items of information that may affect how RMV is selected and detected, how RMV and MVD are signaled, calculated or derived. These MV or MVD related parameters include, but are not limited to:

[0185] A Dynamic Reference List (DRL) as above, which identifies an ordered list of candidate reference motion vectors (RMVs) for predicting the MV of the current coding block. For example, the DRL may identify a set of spatial or temporal neighboring block locations that are likely to have similar motion vectors as the current block. These locations correspond to RMVs that may be used as candidates for predicting the current motion vector. From those candidate RMVs, the encoder may select the RMV that most closely matches that of the current coding block, and then use that RMV to derive the corresponding MVD. The selected RMV may be represented or identified, for example, by a corresponding position or index in the DRL.

[0186] · The DRL index corresponding to the selected candidate RMV of the current coding block.

[0187] Support for adaptive MVD pixel resolution adoption.

[0188] · MVD information, including but not limited to mv_joint, mv_sign, mv_class, mv_bit, mv_fr, mv_hp as described above.

[0189] ·Instruction to use motion compensation modes such as overlap block motion compensation mode.

[0190] - Instructions for using advanced motion compensation modes such as Warp Motion mode.

[0191] Since these information items or parameters may be interdependent, especially when adaptive MVD pixel resolution is used, whether they are signaled or derived, the order in which they are signaled, the number of syntaxes to use for signaling, the context derivation for encoding / signaling those syntax elements may all be taken into consideration when designing a more efficient encoding / decoding scheme.

[0192] In some embodiments, whether one or more of the parameters related to MV or MVD are signaled in the video stream or derived from other signaled information may depend on whether adaptive MVD resolution is applied. Alternatively, or additionally, if such a particular parameter is signaled in the video stream, the manner in which it is signaled may depend on whether adaptive MVD resolution is applied.

[0193] For example, it may be specified that only the first N RMV candidates in the RL are allowed to be used by the encoder when the D-adaptive MVD pixel resolution is applied. In other words, the encoder may be required to select one of the first N entries from the DRL of the current coding block inter-coded with the adaptive MVD pixel resolution. In such an implementation, the encoder is free to select from the entire DRL list for prediction of the current MV when the adaptive MVD pixel resolution is not applied. The basis for such an implementation may be based on the observation that the RMV with a lower index in the DRL is statistically more likely to predict the motion vector of the current block more closely. In this way, the DRL index signaled in the bitstream for a block using the adaptive MVD pixel resolution requires a smaller number of bits and may use, for example, a separate syntax element for signaling.

[0194] Here, N is a positive number. N may be smaller than the total indexing space of the DRL. In some specific implementations, N may be 1 or 2, which means that when the adaptive MVD pixel resolution is used for the current block, the predictor MV or RMV is always the first RMV or the first or second RMV in the DRL.

[0195] The value of N may be predefined or may be signaled in the bitstream. For example, N may be signaled at various coding levels, including but not limited to the sequence level, frame level, slice level, title level, or superblock level. The value N applies to various blocks that use adaptive MVD pixel resolution within the signaled level.

[0196] In some implementations, for blocks using adaptive MVD pixel resolution, when the value of N is specified or signaled as 1, the DRL index does not need to be signaled in the bitstream. The index for extracting the RMV based on the DRL is automatically derived at the decoder as 1, referring to the first RMV candidate in the DRL. As another example, when the value of N is specified or signaled as 2, a single bit may be used to signal the DRL index of the coding block.

[0197] In some other implementations, the above method for signaling or deriving the DRL index may be applied only when inter prediction is a single reference mode. In other words, when adaptive MVD pixel resolution is applied and when inter prediction is a single reference mode rather than a mixed reference mode, it may be specified or signaled that only the first N MVP candidates in the DRL are allowed to be used by the encoder. The derivation or signaling of the range of the DRL index under such conditions may be similar to that described above, e.g., N may be limited and may be signaled or predefined. When the inter prediction mode is a mixed reference mode, or when the inter prediction mode is a single reference mode but adaptive MVD pixel resolution is not used, the RMV of the current coding block may be selected by the encoder from the candidate RMVs corresponding to the entire indexing range of the DRL.

[0198] Therefore, the DRL indexes signaled in the bitstream for various coding blocks may have different ranges (or number of bits). For coding blocks using adaptive MVD pixel resolution, the DRL indexes, when signaled, may be from 1 to N, while for coding blocks not using adaptive MVD pixel resolution, the DRL indexes may be from 1 to the full indexing range of the DRL. Therefore, the DRL indexes under those different circumstances may follow different probability models, and thus the entropy coding context of the signaled DRL index may be used / derived differently depending on whether adaptive MVD pixel resolution is used or not. In other words, if adaptive MVD pixel resolution is used, one set of contexts may be used to signal the DRL index. Otherwise, one set of contexts may be used to signal the DRL index. The derivation of the context at the decoder for decoding the DRL index will depend accordingly on whether adaptive MVD pixel resolution is used or not.

[0199] In some embodiments, the signaling of a motion compensation mode, such as OBMC mode and / or warped motion, may depend on whether an adaptive MVD pixel resolution is used. For example, a flag for whether OBMC mode and / or warped motion is used may be signaled only if an adaptive MVD pixel resolution is used for the current coding block. Otherwise, such a flag may not be signaled (e.g., the decoder may infer that OBMC mode and / or warped motion is not used).

[0200] In some embodiments, the above signaling of OBMC mode and / or warped motion may also be conditioned on the inter prediction mode of the current coding block being a single reference mode rather than a mixed reference mode. In other words, the OBMC mode and / or warped motion flag may be signaled only if the current coding block is associated with a single reference mode and adaptive MVD pixel resolution is used.

[0201] In some other embodiments, the adaptive MVD pixel resolution may be used only in single reference inter prediction mode. Thus, if the adaptive MVD pixel resolution is used for the current coding block determined by signaling or by derivation, it will indicate that inter prediction is not a mixed reference mode. In that situation, no flag needs to be signaled in the bitstream to indicate whether a single reference mode or a mixed reference mode is used for the current block.

[0202] In some other embodiments, the flag indicating whether the inter-intra mixed mode is used may depend on whether the adaptive MVD pixel resolution is applied and whether the current block is a single reference mode. For example, the inter-intra mixed mode for inter and intra mixed prediction of the current block can potentially only be used with single reference modes and only if the inter prediction is based on using the adaptive MVD pixel resolution. Therefore, a flag signaling whether the inter-intra mixed mode is used is not necessary, and it is inferred that the inter-intra mixed mode is not used if the current block is not predicted by a single reference mode and does not depend on the adaptive MVD pixel resolution.

[0203] In some other embodiments, the context derivation for signaling other MVD related syntax may depend on whether an adaptive MVD pixel resolution is applied. Those MVD related syntax may again include the mv_joint, mv_class, mv_bit, mv_fr, mv_hp, etc. described above. For example, mv_joint and / or mv_class may be statistically correlated with the adaptive MVD pixel resolution and may follow different probability models depending on whether the adaptive MVD pixel resolution is applied. Specifically, one context may be used / derived to signal mv_joint (or mv_class) if the adaptive MVD resolution is applied. Otherwise, one or more other different contexts may be used / derived to signal mv_joint (or mv_class). The context dependency of mv_joint and / or mv_class on the use of adaptive MVD pixel resolution is merely an example. Other MVD-related syntax elements may also be associated with context derivation that depends on whether adaptive MVD pixel resolution is applied.

[0204] In various embodiments above, it is assumed that whether or not an adaptive MVD pixel resolution is used for the current coding block may be first extracted (if signaled) or derived from the bitstream before determining any other MV or MVD related information items or parameters, e.g., DRL index, etc. In some other embodiments, such other information items may instead be signaled or derived before determining whether or not an adaptive MVD pixel resolution is used for the current coding block. Due to a correlation or interrelationship between such other information items and whether or not an adaptive MVD pixel resolution should be used, the signaling or derivation of whether or not an adaptive MVD pixel resolution is used may depend on such other information already extracted or derived from the bitstream.

[0205] For example, in some specific implementations, whether adaptive MVD resolution is applied may depend on the value of the DRL index. In such implementations, the DRL index used to determine the RMV of the current coding block may be signaled in the bitstream first (or derived from the bitstream in some other way). If the DRL index of the current coding block is signaled to be a value in the range of 1 to N, it indicates that the selection of the RMV candidate in the DRL may be performed in a finite DRL index range. In that situation, a flag may be further included in the bitstream by the encoder to indicate whether adaptive MVD pixel resolution is used for the current coding block. However, if the signaled DRL index is outside the range of 1 to n, it may be indicated that adaptive MVD pixel resolution is not used for the current coding block, so that a flag to indicate whether adaptive MVD pixel resolution is used does not need to be included in the bitstream. In that situation, the decoder will simply determine by derivation that adaptive MVD pixel resolution is not used. Here, N is a positive number less than the full index range of the DRL. For example, N may be 1 or 2. In some implementations, N may be predefined. In some other embodiments, N may be signaled. For example, N may be signaled at various signaling levels, such as the sequence level, the frame level, the slice level, the title level, or the superblock level.

[0206] FIG. 18 shows a flowchart 1800 of an exemplary method according to the principles underlying the above implementation for adaptive MVD resolution and its signaling. The exemplary decoding method starts at S1801. At S1810, a video stream is received. At S1820, it is determined that a video block is inter-coded based on a predictive block and a motion vector (MV), where the MV is derived from a motion vector differential (MVD) and a reference motion vector (RMV) of the video block. At S1830, a data item related to at least one of the MVD or RMV is extracted or derived from the video stream in a manner that depends at least on whether the MVD is coded with a magnitude-dependent adaptive MVD pixel resolution. At S1840, the MVD is extracted from the video stream, the MV is derived based on the extracted RMV and MVD, and the video block is reconstructed based at least on the MV and the predictive block. The exemplary method stops at S1899.

[0207] FIG. 19 shows a flowchart 1900 of an exemplary method according to the principles underlying the above implementation for adaptive MVD resolution and its signaling. The exemplary decoding method starts at S1901. At S1910, a video stream is received. At S1920, it is determined that a video block is inter-coded based on a prediction block and a motion vector (MV), where the MV is derived from a motion vector differential (MVD) and a reference motion vector (RMV) of the video block. At S1930, an RMV index of the video block is extracted that maps to a dynamic reference list (DRL), the DRL being configured to identify a plurality of ordered candidate RMVs. At S1940, it is determined whether the MVD is coded with a magnitude-dependent adaptive MVD pixel resolution based on the value of the RMV index. The exemplary method stops at S1999.

[0208] In the embodiments and implementations of the present disclosure, any steps and / or operations may be combined or arranged in any quantity or order as desired. Two or more of the steps and / or operations may be performed in parallel. The embodiments and implementations of the present disclosure may be used separately or combined in any order. Furthermore, each of the method (or embodiment), the encoder, and the decoder may be implemented by a processing circuit (e.g., one or more processors or one or more integrated circuits). In one example, the one or more processors execute a program stored in a non-transitory computer-readable medium. The embodiments of the present disclosure may be applied to a luma block or a chroma block. The term block may be interpreted as a prediction block, a coding block, or a coding unit, i.e., a CU. The term block here may also be used to refer to a transform block. In the following items, when referring to a block size, it may refer to either the width or height of the block, or the maximum value of the width and height, or the minimum value of the width and height, or the area size (width x height), or the aspect ratio of the block (width:height or height:width).

[0209] The above techniques may be implemented as computer software using computer readable instructions and physically stored on one or more computer readable media. For example, Figure 20 illustrates a computer system (2000) suitable for implementing certain embodiments of the disclosed subject matter.

[0210] Computer software can be coded in any suitable machine code or computer language that can be subject to mechanisms such as assembly, compilation, linking, etc. to generate code including instructions that can be executed by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., directly, or through interpretation, microcode execution, etc.

[0211] The instructions may be executable by various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smart phones, gaming consoles, Internet of Things devices, and the like.

[0212] The components illustrated in Figure 20 for the computer system (2000) are exemplary in nature and are not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing the embodiments of the present disclosure. The arrangement of components should not be interpreted as having any dependency or requirement regarding any one or combination of components illustrated in the exemplary embodiment of the computer system (2000).

[0213] The computer system (2000) may include certain human interface input devices. Such human interface input devices may be responsive to input by one or more users through, for example, tactile input (e.g., keystrokes, swipes, dataglove movements), audio input (e.g., voice, clapping), visual input (e.g., gestures), and olfactory input (not shown). Human interface devices may also be used to capture certain media that are not necessarily directly related to conscious human input, such as audio (e.g., speech, music, ambient sounds), images (e.g., scanned images, photographic images obtained from a still camera), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0214] The input human interface devices may include one or more of a keyboard (2001), a mouse (2002), a trackpad (2003), a touch screen (2010), a data glove (not shown), a joystick (2005), a microphone (2006), a scanner (2007), and a camera (2008) (only one of each is shown).

[0215] The computer system (2000) may also include certain human interface output devices that may stimulate one or more of the user's senses through, for example, tactile output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touchscreen (2010), data gloves (not shown), or joystick (2005), although there may also be haptic feedback devices that do not function as input devices), audio output devices (e.g., speakers (2009), headphones (not shown)), visual output devices (e.g., screens (2010) including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touchscreen input capability, each with or without haptic feedback capability, some of which are capable of outputting two-dimensional visual output or output in more than three dimensions by means of stereoscopic output, virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).

[0216] The computer system (2000) may also include human accessible storage devices and their associated media, such as CD / DVD ROM / RW (2020) with CD / DVD or similar media (2021), thumb drives (2022), removable hard disks or solid state drives (2023), legacy magnetic media, such as tapes and floppy disks (not shown), dedicated ROM / ASIC / PLD based devices, such as security dongles (not shown), and the like.

[0217] Those skilled in the art will also understand that the term "computer-readable medium" as used in connection with the presently disclosed subject matter does not include transmission media, carrier waves, or other transitory signals.

[0218] The computer system (2000) may also include an interface (2054) to one or more communication networks (2055). The networks may be, for example, wireless, wireline, optical. The networks may further be local, wide area, metropolitan, vehicular and industrial, real-time, delay tolerant, etc. Examples of networks include local area networks such as Ethernet, wireless LAN, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., TV wireline or wireless wide area digital networks including cable TV, satellite TV, and terrestrial broadcast TV, vehicular and factory networks including CANBus, etc. Certain networks generally require an external network interface adapter attached to a particular general purpose data port or peripheral bus (2049) (e.g., a USB port of the computer system (2000)). Others are generally built into the core of the computer system (2000) by attachment to a system bus as described below (e.g., an Ethernet network to a PC computer system, or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system (2000) can communicate with other entities. Such communications can be one-way receive-only (e.g., broadcast TV) or one-way transmit-only (e.g., the CANBus to a specific CANBus device), or can be bidirectional to other computer systems, for example, using local or wide area digital networks. Specific protocols or protocol stacks can be used with each of the networks and network interfaces as described above.

[0219] The above human interface devices, human accessible storage devices, and network interfaces may be attached to a core (2040) of the computer system (2000).

[0220] The cores (2040) may include one or more central processing units (CPUs) (2041), graphics processing units (GPUs) (2042), dedicated programmable processing units in the form of field programmable gate areas (FPGAs) (2043), hardware accelerators (2044) for specific tasks, graphics adapters (2050), etc. These devices may be connected through a system bus (2048), along with read only memory (ROM) (2045), random access memory (RAM) (2046), internal mass storage devices such as internal non-user accessible hard drives, SSDs, etc. (2047). In some computer systems, the system bus (2048) may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripherals may be attached directly to the core's system bus (2048) or through a peripheral bus (2049). In one example, a screen (2010) may be connected to a graphics adapter (2050). Architectures for peripheral buses include PCI, USB, etc.

[0221] The CPU (2041), GPU (2042), FPGA (2043), and accelerator (2044) can execute certain instructions that can be combined to constitute the above computer code. The computer code can be stored in ROM (2045) or RAM (2046). Temporary data can also be stored in RAM (2046), while persistent data can be stored, for example, in an internal mass storage device (2047). Rapid storage and retrieval from any of the memory devices is made possible by the use of cache memory. Cache memory can be closely associated with one or more of the CPU (2041), GPU (2042), mass storage device (2047), ROM (2045), RAM (2046), etc.

[0222] The computer-readable medium can bear computer code for performing various computer-implemented operations. The medium and computer code can be those specially designed and constructed for the purposes of the present disclosure, or they can be of the kind well known and available to those of ordinary skill in the computer software arts.

[0223] As a non-limiting example, a computer system having the architecture (2000), and in particular the core (2040), can provide functionality as a result of a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media can be media associated with the user-accessible mass storage devices introduced above, in addition to specific storage of the core (2040) that is non-transitory in nature, such as the core's internal mass storage (2047) or ROM (2045). Software implementing various embodiments of the present disclosure can be stored in such devices and executed by the core (2040). The computer-readable media can include one or more memory devices or chips, depending on the particular needs. The software can cause the core (2040), and in particular the processors therein (including CPUs, GPUs, FPGAs, etc.), to execute certain processes or certain portions of certain processes described herein, including defining data structures stored in RAM (2046) and modifying such data structures according to processes defined by the software. Additionally or alternatively, the computer system can provide functionality as a result of logic hardwired or otherwise embodied in circuitry (e.g., accelerators (2044)) that can operate in place of or in conjunction with software to execute certain processes or certain portions of certain processes described herein. References to software can encompass logic, where appropriate, and vice versa. References to computer-readable media can encompass circuitry (e.g., integrated circuits (ICs)) storing software for execution, circuitry embodying logic for execution, or both, where appropriate. The present disclosure encompasses any suitable combination of hardware and software.

[0224] While this disclosure has described several illustrative embodiments, there are alterations, permutations, and various substitute equivalents that are within the scope of the disclosure. Thus, it should be understood that those skilled in the art will be able to devise numerous systems and methods that embody the principles of the present disclosure and are thus within its spirit and scope, even if not explicitly shown and described herein.

[0225] Appendix A: Acronyms JEM: Joint Exploration Model VVC: Versatile Video Coding BMS:Benchmark Set MV: Motion Vector HEVC:High Efficiency Video Coding SEI:Supplementary Enhancement Information VUI:Video Usability Information GOP: Group of Picture(s) TU: Transform Unit(s) PU: Prediction Unit(s) CTU: Coding Tree Unit(s) CTB: Coding Tree Block(s) PB: Prediction Block(s) HRD:Hypothetical Reference Decoder SNR: Signal Noise Ratio CPU:Central Processing Unit(s) GPU:Graphics Processing Unit(s) CRT: Cathode Ray Tube LCD: Liquid Crystal Display OLED:Organic Light-Emitting Diode CD:Compact Disc DVD:Digital Video Disc ROM:Read-Only Memory RAM:Random Access Memory ASIC:Application-Specific Integrated Circuit PLD:Programmable Logic Device LAN:Local Area Network GSM:Global System for Mobile communications LTE:Long-Term Evolution CANBus:Controller Area Network Bus USB:Universal Serial Bus PCI:Peripheral Component Interconnect FPGA:Field Programmable Gate Area(s) SSD:Solid-State Drive IC:Integrated Circuit HDR:High Dynamic Range SDR:Standard Dynamic Range JVET:Joint Video Exploration Team MPM:Most Probable Mode WAIP:Wide-Angle Intra Prediction CU:Coding Unit PU:Prediction Unit TU:Transform Unit CTU:Coding Tree Unit PDPC:Position Dependent Prediction Combination ISP: Intra Sub-Partitions SPS: Sequence Parameter Setting PPS: Picture Parameter Set APS: Adaptation Parameter Set VPS: Video Parameter Set DSP: Decoding Parameter Set ALF: Adaptive Loop Filter SAO: Sample Adaptive Offset CC-ALF: Cross-Component Adaptive Loop Filter CDEF: Constrained Directional Enhancement Filter CCSO: Cross-Component Sample Offset LSO: Local Sample Offset LR: Loop Restoration Filter AV1: AOMedia Video 1 AV2: AOMedia Video 2 MVD: Motion Vector difference CfL: Chroma from Luma SDT: Semi Decoupled Tree SDP: Semi Decoupled Partitioning SST: Semi Separate Tree SB: Super Block IBC(or Intra BC): Intra Block Copy CDF: Cumulative Density Function SCC: Screen Content Coding GBI: Generalized Bi-prediction BCW:Bi-prediction with CU-level Weights CIIP:Combined intra-inter prediction POC:Picture Order Count RPS:Reference Picture Set DPB:Decoded Picture Buffer MMVD:Merge Mode with Motion Vector Difference

Claims

1. 1. A method of video decoding performed by a video decoder, comprising: receiving coded information for a current block in a current picture from a coded video bitstream; partitioning the current block into a plurality of sub-blocks; obtaining a first flag included in the coded information, the first flag indicating whether a cross-component linear model prediction (CCLM) is applied to the current block, in which chroma samples of the current block are predicted based on reconstructed luma samples of the current block; determining a predicted sample value for each of the chroma samples in each of the plurality of sub-blocks of the current block based on the CCLM in response to the first flag indicating that the CCLM is applied to the current block; reconstructing the current block based on the respective predicted sample values ​​of the chroma samples in each of the plurality of sub-blocks of the current block; The method according to claim 1,

2. The step of partitioning the current block comprises: partitioning the current block into the sub-blocks along a width direction based on a width of the current block being equal to or greater than a height of the current block; The method of claim 1.

3. The step of partitioning the current block comprises: partitioning the current block into the sub-blocks along a height direction based on a width of the current block being smaller than a height of the current block; The method of claim 1.

4. The step of partitioning the current block comprises: partitioning the current block along both a height direction and a width direction into the plurality of sub-blocks of a minimum sub-block size; The method of claim 1.

5. obtaining a syntax element in the coded information; determining the minimum sub-block size based on the syntax elements; Further comprising: The syntax element is in one of a sequence parameter set (SPS), a picture parameter set (PPS), a slice, and a tile. The method according to claim 4.

6. The determining step includes: determining the predicted sample values ​​of the chroma samples in a second sub-block of the plurality of sub-blocks based on reconstructed samples of a first sub-block of the plurality of sub-blocks; The second sub-block is adjacent to the first sub-block. The method of claim 1.

7. The determining step includes: in response to the first flag indicating that the CCLM applies to the current block, obtaining a second flag included in the coded information, the second flag indicating whether the CCLM is applied to each of the plurality of sub-blocks; determining the respective predicted sample values ​​of the chroma samples in each of the plurality of sub-blocks of the current block based on the CCLM in response to the second flag indicating that the CCLM is applied to each of the plurality of sub-blocks; Further comprising The method of claim 1.

8. The step of determining each predicted sample value comprises: determining a predicted sample value of the chroma sample in the first sub-block of the plurality of sub-blocks based on a first mode of the CCLM in response to a reconstructed neighboring sample being adjacent to a left side of a first sub-block of the plurality of sub-blocks, the first mode of the CCLM indicating that the predicted sample value of the chroma sample in the first sub-block is determined based on the reconstructed neighboring sample being adjacent to a left side of the first sub-block; determining a predicted sample value of the chroma sample in the second sub-block of the plurality of sub-blocks based on a second mode of the CCLM in response to the reconstructed neighboring samples being adjacent to a left side and an upper side of a second sub-block of the plurality of sub-blocks, the second mode of the CCLM indicating that the predicted sample value of the chroma sample in the second sub-block is determined based on the reconstructed neighboring samples being adjacent to a left side and an upper side of the second sub-block; Further comprising The method according to claim 7.

9. The determining step includes: in response to the first flag indicating that the CCLM applies to the current block, obtaining a second flag included in the coded information, the second flag indicating whether the CCLM is applied to each of the plurality of sub-blocks; obtaining an index included in the coded information, the index indicating a CCLM mode of the CCLM, the CCLM mode indicating which reconstructed neighboring samples are applied by the CCLM to generate the respective predicted sample values ​​of the chroma samples in each of the plurality of sub-blocks; determining the respective predicted sample values ​​of the chroma samples in each of the sub-blocks of the current block based on the CCLM using the CCLM mode in response to the second flag indicating that the CCLM is applied to each of the sub-blocks and the index indicating the CCLM mode; Further comprising The method of claim 1.

10. The determining step includes: determining, in response to the index indicating a first CCLM mode, the predicted sample value of each of the chroma samples in each of the plurality of sub-blocks based on reconstructed neighboring samples adjacent to the left and above a respective one of the plurality of sub-blocks; determining, in response to the index indicating a second CCLM mode, the predicted sample value of each of the chroma samples in each of the plurality of sub-blocks based on a reconstructed left neighboring sample of a respective one of the plurality of sub-blocks; determining, in response to the index indicating a third CCLM mode, the predicted sample value of each of the chroma samples in each of the plurality of sub-blocks based on a reconstructed neighboring sample adjacent to an upper side of a respective one of the plurality of sub-blocks; Further comprising 10. The method of claim 9.

11. at least one memory configured to store program code; processing circuitry configured to read the program code and to operate as directed by the program code; having Apparatus, wherein the program code, when executed by the processing circuitry, causes the processing circuitry to perform a method according to any one of claims 1 to 10.

12. 1. A method of video encoding performed by a video encoder, comprising: partitioning a current block in a current picture of a video into a plurality of sub-blocks; determining predicted sample values ​​for each of the chroma samples in each of the plurality of sub-blocks of the current block based on a cross-component linear model prediction (CCLM) in which the chroma samples of the current block are predicted based on reconstructed luma samples of the current block; performing intra prediction on the current block based on the predicted sample values ​​of the chroma samples in each of the plurality of sub-blocks of the current block; generating a first flag indicating that the CCLM applies to the plurality of sub-blocks of the current block, and encoding information including the first flag into a coded video bitstream; The method according to claim 1,

Images