Method and electronic device for decoding inter-predicted video blocks of a video stream

Abstract

To provide a method for encoding and decoding a motion vector difference.SOLUTION: The method includes determining that a motion vector associated with an inter-predicted video block is encoded as a motion vector difference (MVD) between the motion vector and a reference motion vector, obtaining, from a video stream, an indication of a magnitude range of the MVD among a plurality of predetermined magnitude ranges for the motion vector difference, determining a pixel resolution for the MVD according to the magnitude range, identifying additional MVD information in the video stream based on the pixel resolution, extracting the additional MVD information from the video stream, and decoding the inter-predicted video block based on a reference frame associated with the motion vector, the reference motion vector, the additional MVD information, and the pixel resolution.SELECTED DRAWING: Figure 18

Description

[Technical Field]

[0001] Incorporation by Reference This international PCT application claims the benefit of U.S. Provisional Patent Application No. 63 / 270,397, filed October 21, 2021, entitled "Adaptive Resolution for Motion Vector Difference," and U.S. Provisional Patent Application No. 63 / 289,122, filed December 13, 2021, which is based on and claims priority to U.S. Non-Provisional Patent Application No. 17 / 708,801, filed March 30, 2022. These prior patent applications are incorporated herein by reference in their entireties.

[0002] Technical Field FIELD This disclosure relates generally to video coding, and more particularly to methods and systems for providing adaptive resolution for motion vector differentials in inter prediction of video blocks. [Background technology]

[0003] The background discussion provided herein is intended to generally present the context for the present disclosure. Aspects of the description that may not be admitted as prior art at the time of filing of this application, including the work of the presently named inventors, to the extent that they are described in this background section, are not admitted expressly or impliedly as prior art to the present disclosure.

[0004] Video coding and decoding can be performed using inter-picture prediction with motion compensation. Uncompressed digital video can include a series of images, each having spatial dimensions of, for example, 1920 x 1080 luminance samples and associated full or subsampled chrominance samples. The series of images can have a fixed or variable image rate (also called frame rate), for example, 60 images per second or 60 frames per second. Uncompressed video has specific bitrate requirements for streaming or data processing. For example, a video with a pixel resolution of 1920 x 1080, a frame rate of 60 frames per second, and 4:2:0 chroma subsampling with 8 bits per pixel per color channel requires a bandwidth of nearly 1.5 Gbit / s. One hour of such video requires more than 600 Gbytes of storage space.

[0005] One goal of video coding and decoding is to reduce redundancy in an uncompressed input video signal through compression. Compression helps reduce the aforementioned bandwidth and storage space requirements, sometimes by more than two orders of magnitude. Both lossless and lossy compression, as well as combinations of them, can be used. Lossless compression refers to techniques that allow an exact copy of the original signal to be reconstructed from the compressed original signal through the decoding process. Lossy compression refers to a coding / decoding process in which the original video information is not fully preserved during coding and cannot be fully recovered during decoding. When using lossy compression, the reconstructed signal may not be identical to the original signal, but the distortion between the original and reconstructed signal is small enough that the reconstructed signal is useful for the intended application, despite some information loss. For video, lossy compression is widely adopted in many applications. The amount of acceptable distortion varies depending on the application. For example, users of certain consumer video streaming applications may tolerate higher distortion than users of film and television broadcast applications. The compression ratio achievable by a particular coding algorithm can be selected or adjusted to reflect different distortion tolerances. In general, the more distortion that can be tolerated, the more lossy the coding algorithm and the higher the compression ratio.

[0006] Video encoders and decoders can utilize techniques in several broad categories and steps, such as motion compensation, Fourier transform, quantization, and entropy coding.

[0007] Video codec technology can include a technique known as intra-coding. In intra-coding, sample values ​​are represented without reference to samples or other data from a previously reconstructed reference image. In some video codecs, an image is spatially subdivided into blocks of samples. If all blocks of samples are coded in intra mode, the image can be referred to as an intra-image. Intra-images and their derivatives, such as independent decoder refresh images, can be used to reset the decoder state and can therefore be used as the first image in a coded video bitstream and video session or as still images. The samples of the block after intra-prediction can then undergo a transform to the frequency domain, and the transform coefficients so generated can be quantized before entropy coding. Intra-prediction refers to a technique that minimizes sample values ​​in the pre-transform domain. In some cases, the smaller the DC value and the smaller the AC coefficients after the transform, the fewer bits are required for a given quantization step size to represent the block after entropy coding.

[0008] Conventional intra-coding, such as that known from MPEG-2 generation coding techniques, does not use intra-prediction. However, some newer video compression techniques include techniques that attempt to code / decode a block of data that precedes it in decoding order, for example, by using surrounding sample data and / or metadata obtained during the encoding and / or decoding of spatially neighboring blocks. Such techniques are hereinafter referred to as "intra-prediction" techniques. Note that, at least in some cases, intra-prediction uses only reference data from the current picture being reconstructed, and not from other reference pictures.

[0009] There may be various forms of intra-prediction. If one or more of such techniques can be used in a given video coding technique, the technique in use may be referred to as an intra-prediction mode. A particular codec may provide one or more intra-prediction modes. In certain cases, a mode may have sub-modes and / or may be associated with various parameters, and the mode / sub-mode information and the intra-coding parameters of a video block may be coded individually or jointly in a mode codeword. The codeword used for a given mode, sub-mode, and / or parameter combination may affect the coding efficiency achieved by intra-prediction, and therefore may also affect the entropy coding technology used to convert the codeword into a bitstream.

[0010] Certain intra-prediction modes were introduced in H.264, refined in H.265, and further refined in newer coding techniques such as the joint exploration model (JEM), versatile video coding (VVC), and benchmark sets (BMS). Generally, in intra-prediction, a prediction block can be formed using available neighboring sample values. For example, available values ​​of a particular set of neighboring samples along a particular direction and / or line can be copied into the prediction block. A reference to the direction in use can be coded in the bitstream or can itself be predicted.

[0011] Referring to FIG. 1A, shown at the bottom right is a subset of nine predictor directions specified by the 33 possible intra-predictor directions of H.265 (corresponding to the 33 angle modes of the 35 intra-modes specified in H.265). The point where the arrows converge (101) represents the sample being predicted. The arrows represent the direction in which sample 101 is predicted using neighboring samples. For example, arrow (102) indicates that sample 101 is predicted to the upper right from neighboring samples at a 45-degree angle from the horizontal direction. Similarly, arrow (103) indicates that sample 101 is predicted to the lower left from neighboring samples at a 22.5-degree angle from the horizontal direction.

[0012] 1A, a square block (104) of 4x4 samples is shown in the upper left (indicated by a thick dashed line). The square block (104) contains 16 samples, each labeled with an "S" indicating its position in the Y dimension (e.g., row index) and its position in the X dimension (e.g., column index). For example, sample S21 is the second sample (from the top) in the Y dimension and the first sample (from the left) in the X dimension. Similarly, sample S44 is the fourth sample in both the Y and X dimensions within the block (104). Since the block size is 4x4 samples, S44 is located at the bottom right. Also shown are examples of reference samples following a similar numbering scheme. The reference samples are labeled R, their Y position (e.g., row index) and X position (column index) relative to the block (104). Both H.264 and H.265 use predicted samples that neighbor the block being reconstructed.

[0013] Intra-image prediction of block 104 can begin by copying reference sample values ​​from neighboring samples according to the signaled prediction direction. For example, assume that the coded video bitstream includes signaling for this block 104 indicating a prediction direction of arrow (102), i.e., the sample is predicted from one or more prediction samples to the upper right and at a 45-degree angle from horizontal. In such a case, samples S41, S32, S23, and S14 are predicted from the same reference sample R05. Sample S44 is then predicted from reference sample R08.

[0014] In certain cases, especially when the orientation is not evenly divisible by 45 degrees, the values ​​of multiple reference samples can be combined, for example by interpolation, to calculate the reference sample.

[0015] As video coding technology advances, the number of possible directions increases. For example, in H.264 (2003), nine different directions are available for intra prediction. This increased to 33 in H.265 (2013), and as of the time of this disclosure, JEM / VVC / BMS can support up to 65 directions. Experimental studies have been conducted to help identify optimal intra prediction directions, and specific techniques of entropy coding can be used to encode those optimal directions with a small number of bits, accepting a specific bit penalty for the direction. Furthermore, the direction itself may be predictable from neighboring directions used in intra prediction of decoded neighboring blocks.

[0016] Figure 1B shows a schematic diagram (180) illustrating 65 intra-prediction directions according to JEM to illustrate the increasing number of prediction directions in various encoding techniques developed over time.

[0017] The manner in which bits representing intra-prediction directions are mapped to prediction directions in the coded video bitstream may vary depending on the video coding technique: for example, it may range from a simple direct mapping of prediction directions to complex adaptive schemes including intra-prediction modes, codenames, most-likely modes, and similar techniques. However, in any case, there may be certain directions in intra-prediction that are statistically less likely to occur in the video content than certain other directions. Because the goal of video compression is to reduce redundancy, in a well-designed video coding technique, these less-likely directions may be represented by more bits than more-likely directions.

[0018] Inter-image prediction or inter-prediction may be based on motion compensation, in which sample data from a previously reconstructed image or part thereof (reference image) can be used to predict a newly reconstructed image or image part (e.g., block) after spatially shifting it in a direction indicated by a motion vector (hereinafter MV). In some cases, the reference image can be the same as the image currently being reconstructed. The MV can have two dimensions, X and Y, or three dimensions, with the third dimension being a representation of the reference image in use (similar to the temporal dimension).

[0019] In some video compression techniques, the current MV applicable to a particular area of ​​sample data can be predicted from other MVs, e.g., other MVs associated with other areas of sample data that are spatially adjacent to the area being reconstructed and precede the current MV in decoding order. In this way, relying on removing redundancy in correlated MVs can significantly reduce the overall amount of data required for MV coding, thereby increasing compression efficiency. MV prediction works effectively, for example, when coding input video signals derived from a camera (known as natural video) because there is a statistical possibility that areas larger than the area to which one MV applies will move in similar directions within the video sequence, and in some cases, can be predicted using similar motion vectors derived from MVs in neighboring areas. As a result, the actual MV for a given area will be similar or identical to the MV predicted from surrounding MVs. After entropy coding, such MVs may be represented with fewer bits than would be used if the MV were directly coded rather than predicted from neighboring MV(s). In some cases, MV prediction may be an example of lossless compression of a signal (i.e., an MV) derived from the original signal (i.e., a sample stream). In other cases, the MV prediction itself may be non-reversible, for example due to rounding errors in computing the predictor from some surrounding MVs.

[0020] Various MV prediction mechanisms are described in H.265 / HEVC (ITU-T Rec. H.265, High Efficiency Video Coding, December 2016). Among the many MV prediction mechanisms specified in H.265, the one described below is a technique hereafter referred to as "spatial merging".

[0021] 2, the current block (201) contains samples that the encoder found during the motion search process to be predictable from a spatially shifted previous block of the same size. Instead of coding its MV directly, it can derive the MV from metadata associated with one or more reference pictures, e.g., from the most recent reference picture (in decoding order) using the MV associated with any of five surrounding samples denoted A0, A1, and B0, B1, B2 (202 to 206, respectively). In H.265, MV prediction can use predictors from some of the same reference pictures as neighboring blocks. Summary of the Invention

[0022] summary

[0001] This disclosure relates generally to video coding, and more particularly to a method and system for providing adaptive resolution of motion vector differentials in inter-prediction of video blocks. In one embodiment, a method for decoding an inter-predicted video block of a video stream includes: receiving a video stream; determining that a motion vector differential (MVD) between a reference motion vector associated with the inter-predicted video block and a motion vector is signaled within the video stream, where the reference motion vector corresponds to only one reference picture from Reference Frame List 0 and Reference Frame List 1 unless the MVD is jointly signaled for two reference pictures; obtaining from the video stream an indication of a magnitude range of the MVD among a plurality of predetermined magnitude ranges for the motion vector differential; determining a pixel resolution for the MVD according to the magnitude range; identifying additional MVD information within the video stream based on the pixel resolution; extracting the additional MVD information from the video stream; and decoding the inter-predicted video block based on the reference frame associated with the motion vector, the reference motion vector, the additional MVD information, and the pixel resolution.

[0023] In the above exemplary embodiment, the pixel resolution is 2 n -pel(2 n pixels), where n is an integer between -6 and 11 inclusive.

[0024] In any of the above exemplary implementations, the plurality of predetermined magnitude ranges for motion vector differences are associated in a predetermined manner with pixel resolutions in non-ascending order, with higher pixel resolutions being associated with smaller pixel resolution values.

[0025] In any of the above exemplary embodiments, the step of obtaining an indication of the magnitude range of the MVD includes the steps of: extracting from the video stream a first predefined syntax element indicating an MVD class of the MVD from a predetermined set of MVD classes, wherein a lower MVD class corresponds to a smaller MVD magnitude range; and determining the magnitude range of the MVD according to the MVD class.

[0026] In any of the above exemplary embodiments, the step of determining a pixel resolution for MVD according to the magnitude range includes the steps of: determining whether the magnitude range is higher than a predetermined MVD range threshold level; and determining that the pixel resolution is an integer number of pixels by determining that the magnitude range is higher than the predetermined MVD range threshold level; and determining that the pixel resolution is a fraction of a pixel by determining that the magnitude range is not higher than the predetermined MVD range threshold level.

[0027] In any of the above exemplary embodiments, identifying additional MVD information for the video stream based on pixel resolution includes: parsing the video stream according to a second predetermined syntax element to obtain an integer pixel portion of the MVD; and upon determining that the pixel resolution is fractional pixels, further parsing the video stream according to at least a third predetermined syntax element to obtain a fractional pixel portion of the MVD.

[0028] In any of the above exemplary embodiments, the MVD range threshold level comprises the lowest or second lowest of a predetermined set of MVD classes.

[0029] In any of the above exemplary embodiments, each of a predetermined set of MVD classes having a magnitude range above a predetermined MVD range threshold level is associated with a single allowed integer MVD pixel value.

[0030] In any of the above exemplary implementations, the single allowed integer pixel value comprises a pixel value corresponding to a higher of the corresponding magnitude range.

[0031] In any of the above exemplary embodiments, the single allowed integer pixel value comprises a pixel value corresponding to the midpoint of the corresponding magnitude range.

[0032] In any of the above exemplary embodiments, the step of determining a pixel resolution for MVD according to the magnitude range may include the steps of: determining whether the magnitude range is below, including, or above a predetermined MVD threshold magnitude value; determining that the pixel resolution is an integer number of pixels by determining that the magnitude range is above the predetermined magnitude value of the MVD threshold; and determining that the pixel resolution is a fractional number of pixels by determining that the magnitude range is not above the predetermined magnitude value of the MVD threshold.

[0033] In any of the above exemplary embodiments, the method may further include the steps of: extracting a second predetermined syntax element from the video stream indicating an MVD magnitude offset of the MVD relative to a starting magnitude of the magnitude range of the MVD by determining that the magnitude range includes a predetermined magnitude value of the MVD threshold; obtaining an integer magnitude of the MVD based on the MVD magnitude offset and magnitude range of the MVD; determining that the pixel resolution is fractional if the integer magnitude of the MVD is not higher than the predetermined magnitude of the MVD threshold; and determining that the pixel resolution is non-fractional if the integer magnitude of the MVD is higher than the predetermined magnitude of the MVD threshold.

[0034] In any of the above exemplary embodiments, upon determining that the pixel resolution is fractional, identifying additional MVD information in the video stream based on the pixel resolution may include parsing the video stream according to a predetermined third syntax element to obtain the fractional portion of the MVD.

[0035] In any of the above exemplary embodiments, the MVD threshold magnitude value is less than 4 pixels.

[0036] In any of the above exemplary implementations, the MVD pixel resolution associated with the plurality of predetermined magnitude ranges of motion vector differentials is different for each magnitude range.

[0037] In some other example embodiments, a method for decoding an inter-predicted video block of a video stream is disclosed. The method can include receiving a video stream; determining that a motion vector difference (MVD) between a reference motion vector associated with the inter-predicted video block and a motion vector is signaled within the video stream, where the reference motion vector corresponds to a reference picture in only one of Reference Frame List 0 and Reference Frame List 1 unless the MVD is jointly signaled for two reference pictures; extracting an integer part of a magnitude of the MVD from the video stream; determining a pixel resolution for the MVD according to the integer part of the magnitude of the MVD; identifying additional MVD information within the video stream based on the pixel resolution; and decoding the inter-predicted video block based on the reference frame associated with the motion vector, the reference motion vector, the additional MVD information, the integer part of the magnitude of the MVD, and the pixel resolution.

[0038] In the above exemplary embodiment, the MVD pixel resolution depends on the MVD magnitude in a non-increasing order.

[0039] In any of the above exemplary embodiments, the step of determining a pixel resolution for the MVD according to the integer portion of the magnitude of the MVD may include the steps of: determining whether the integer portion of the magnitude of the MVD is higher than a predetermined MVD threshold magnitude value; determining that the pixel resolution is an integer number of pixels by determining that the integer portion of the magnitude of the MVD is higher than the predetermined MVD threshold magnitude value; and determining that the pixel resolution is a fractional pixel by determining that the integer portion of the magnitude of the MVD is not higher than the predetermined MVD threshold magnitude value.

[0040] Aspects of the present disclosure also provide a video encoding or decoding device or apparatus including circuitry configured to perform any of the above method embodiments.

[0041] Aspects of the present disclosure also provide a non-transitory computer-readable medium storing instructions that, when executed by a computer, cause the computer to perform a method for video decoding and / or encoding. [Brief explanation of the drawings]

[0042] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings. [Figure 1A] FIG. 1A is a diagram illustrating a schematic diagram of an exemplary subset of intra-prediction directional modes. [Figure 1B] FIG. 1B shows an example intra-prediction directionality diagram. [Figure 2] FIG. 2 is a diagram illustrating a current block and its surrounding spatial merge candidates for motion vector prediction in one embodiment. [Figure 3] FIG. 3 is a schematic diagram illustrating a simplified block diagram of a communication system (300) according to an example embodiment. [Figure 4] FIG. 4 is a schematic diagram illustrating a simplified block diagram of a communication system (400) according to an example embodiment. [Figure 5] FIG. 5 is a diagram illustrating a simplified block diagram of a video decoder according to an exemplary embodiment. [Figure 6] FIG. 6 is a diagram illustrating a simplified block diagram of a video encoder according to an example embodiment. [Figure 7] FIG. 7 is a block diagram of a video encoder according to another example embodiment. [Figure 8] FIG. 8 is a block diagram of a video decoder according to another example embodiment. [Figure 9]FIG. 9 is a diagram illustrating a coding block partitioning scheme according to an exemplary embodiment of the present disclosure. [Figure 10] FIG. 10 is a diagram illustrating another coding block partitioning scheme according to an exemplary embodiment of the disclosure. [Figure 11] FIG. 11 is a diagram illustrating another coding block partitioning scheme according to an exemplary embodiment of the disclosure. [Figure 12] FIG. 12 is a diagram illustrating an example of partitioning of a base block into coding blocks according to an exemplary partitioning scheme. [Figure 13] FIG. 13 is a diagram illustrating an example of a ternary partitioning scheme. [Figure 14] FIG. 14 is a diagram illustrating an example of a quadtree binary tree coding block partitioning scheme. [Figure 15] FIG. 15 is a diagram illustrating a scheme for dividing a coding block into multiple transform blocks and the coding order of the transform blocks, according to an exemplary embodiment of the disclosure. [Figure 16] FIG. 16 is a diagram illustrating another scheme for dividing a coding block into multiple transform blocks and the coding order of the transform blocks, according to an exemplary embodiment of the disclosure. [Figure 17] FIG. 17 is a diagram illustrating another scheme for dividing a coding block into multiple transform blocks, according to an exemplary embodiment of the disclosure. [Figure 18] FIG. 18 illustrates a flowchart of a method according to an exemplary embodiment of the disclosure. [Figure 19] FIG. 19 shows another flowchart of a method according to an exemplary embodiment of the disclosure. [Figure 20] FIG. 20 illustrates a schematic diagram of a computer system according to an exemplary embodiment of the disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0043] Throughout the specification and claims, terms may have a nuance suggested or implied in context beyond their explicitly stated meaning. The phrases "in one embodiment" or "in some embodiments" used herein do not necessarily refer to the same embodiment, and the phrases "in another embodiment" or "in another embodiment" used herein do not necessarily refer to different embodiments. Similarly, the phrases "in one implementation" or "in some implementations" used herein do not necessarily refer to the same implementation, and the phrases "in another implementation" or "in another implementation" used herein do not necessarily refer to different implementations. For example, it is intended that the claimed subject matter include combinations of example embodiments / implementations in whole or in part.

[0044] Generally, terms can be understood, at least in part, from their usage in context. For example, terms such as "and," "or," or "and / or" as used herein can include a variety of meanings that may depend, at least in part, on the context in which such terms are used. Typically, when "or" is used to relate a list, such as A, B, or C, it is intended to refer not only to A, B, and C, which are used herein in an inclusive sense, but also to A, B, or C, which are used herein in an exclusive sense. Furthermore, as used herein, the terms "one or more" or "at least one" may be used to describe any feature, structure, or characteristic in a singular sense, or may be used to describe a combination of features, structures, or characteristics in a plural sense, depending, at least in part, on the context. Similarly, terms such as "a," "an," or "the" may be understood to convey a singular or a plural sense, depending, at least in part, on the context. Furthermore, the terms "based on" or "determined by" may be understood not to necessarily convey an exclusive set of elements, but instead may allow for the presence of additional elements not necessarily explicitly described, depending at least in part on the context. FIG. 3 illustrates a simplified block diagram of a communication system 300 according to one embodiment of the present disclosure. The communication system 300 includes multiple terminal devices that can communicate with each other, e.g., via a network 350. For example, the communication system 300 includes a pair of terminal devices 310 and 320 interconnected via the network 350. In the example of FIG. 3, the pair of terminal devices 310 and 320 perform unidirectional transmission of data. For example, the terminal device 310 may code video data (e.g., a stream of video images captured by the terminal device 310) for transmission to the other terminal device 320 via the network 350.The encoded image data may be transmitted in the form of one or more coded video bitstreams. The terminal device (320) may receive the coded video data from the network (350), decode the coded video data to reconstruct the video images, and display the video images according to the reconstructed video data. Unidirectional data transmission may be common in media delivery applications, etc.

[0045] In another example, the communication system 300 includes a second pair of terminal devices 330 and 340 for bidirectional transmission of coded video data, which may be implemented, for example, in a video conferencing application. For bidirectional transmission of data, for example, each of the terminal devices 330 and 340 may code video data (e.g., a stream of video images captured by the terminal device) for transmission to the other of the terminal devices 330 and 340 over the network 350. Each of the terminal devices 330 and 340 may receive the coded video data transmitted by the other of the terminal devices 330 and 340, decode the coded video data to reconstruct the video image, and display the video image on an accessible display device in accordance with the reconstructed video data.

[0046] In the example of FIG. 3 , the terminal devices 310, 320, 330, and 340 may be implemented as servers, personal computers, and smartphones, although the applicability of the principles underlying the present invention is not limited thereto. Embodiments of the present disclosure may be implemented in desktop computers, laptop computers, tablet computers, media players, wearable computers, dedicated video conferencing equipment, and the like. The network 350 represents any number of networks that convey coded video data between the terminal devices 310, 320, 330, and 340, including, for example, wireline and / or wireless communication networks. The communication network 350 may exchange data over circuit-switched, packet-switched, and / or other types of channels. Exemplary networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For purposes of this description, the architecture and topology of the network 350 may not be important to the operation of the present disclosure unless explicitly described herein.

[0047] 4 illustrates the arrangement of a video encoder and a video decoder in a video streaming environment as an example application of the disclosed subject matter. The disclosed subject matter may be similarly applied to other video applications, such as video conferencing, digital TV broadcasting, gaming, virtual reality, and storing compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0048] A video streaming system may include, for example, a video source (401), such as a capture subsystem (413), which may include a digital camera, that generates a stream of uncompressed video images or images (402). In one embodiment, the stream of video images (402) includes samples captured by the digital camera of the video source (401). The stream of video images (402), depicted as bold lines to emphasize its high data volume when compared to the encoded video data (404) (or coded video bitstream), may be processed by an electronic device (420) that includes a video encoder (403) coupled to the video source (401). The video encoder (403) may include hardware, software, or a combination thereof, and may enable or implement aspects of the disclosed subject matter, as described in more detail below. The encoded video data 404 (or encoded video bitstream 404), shown as a thin line to emphasize its lower data volume compared to the stream of uncompressed video images 402, can be stored on a streaming server 405 for future use or directly on a downstream video device (not shown). One or more streaming client subsystems, such as the client subsystems 406 and 408 of FIG. 4, can access the streaming server 405 and retrieve copies 407 and 409 of the encoded video data 404. The client subsystem 406 can include a video decoder 410, for example, within an electronic device 430. The video decoder 410 decodes the input copy of the encoded video data 407 and generates an output stream of uncompressed video images 411 that can be rendered on a display 412 (e.g., a display screen) or other rendering device (not shown). Video decoder 410 may be configured to perform some or all of the various functions described in this disclosure.In some streaming systems, the encoded video data 404, 407, and 409 (e.g., video bitstreams) may be coded according to a particular video coding / compression standard. Examples of these standards include ITU-T Recommendation H.265. For example, a video coding standard under development is informally known as Versatile Video Coding (VVC). The disclosed subject matter may be used in the context of VVC and other video coding standards.

[0049] It should be noted that the electronic devices 420 and 430 may include other components (not shown). For example, the electronic device 420 may include a video decoder (not shown), and the electronic device 430 may also include a video encoder (not shown).

[0050] 5 shows a block diagram of a video decoder (510) according to any of the following embodiments of the present disclosure. The video decoder (510) can be included in an electronic device (530). The electronic device (530) can include a receiver (531) (e.g., receiving circuitry). In the example of FIG. 4, the video decoder (510) can be used in place of the video decoder (410).

[0051] The receiver (531) can receive one or more coded video sequences to be decoded by the video decoder (510), and in the same or another embodiment, can decode one coded video sequence at a time, with the decoding of each coded video sequence being independent of the other coded video sequences. Each video sequence can be associated with multiple video frames or images. The coded video sequences can be received from a channel (501), which can be a hardware / software link to a storage device that stores the encoded video data or a streaming source that transmits the encoded video data. The receiver (531) can receive the encoded video data along with other data, such as coded audio data and / or auxiliary data streams, which can be forwarded to respective processing circuitry (not shown). The receiver (531) can separate the coded video sequences from other data. To address network jitter, a buffer memory (515) can be located between the receiver (531) and the entropy decoder / parser (520) (hereinafter "parser (520)"). In certain applications, the buffer memory (515) may be implemented as part of the video decoder (510). In other applications, it may be external to and separate from the video decoder (510) (not shown). In yet other applications, there may be a buffer memory (not shown) external to the video decoder (510), for example, to deal with network jitter, and there may also be another additional buffer memory (515) internal to the video decoder (510), for example, to handle playout timing. If the receiver (531) is receiving data from a store / forward device with sufficient bandwidth and controllability or from an isochronous network, the buffer memory (515) may be unnecessary or small.For use with best-effort packet networks such as the Internet, a sufficiently sized buffer memory (515) may be required, which may be relatively large. Such a buffer memory may be implemented with an adaptable size and may be implemented, at least in part, in an operating system or similar element (not shown) external to the video decoder (510).

[0052] The video decoder (510) may include a parser (520) for reconstructing symbols (521) from the coded video sequence. These symbol categories may include information used to manage the operation of the video decoder (510) and potential information for controlling a rendering device, such as a display (512) (e.g., a display screen), which may or may not be an integral part of the electronic device (530) but may be coupled to the electronic device (530). The control information for the rendering device(s) may be in the form of supplemental enhancement information (SEI) messages or video usability information (VUI) parameter set fragments (not shown). The parser (520) may parse / entropy decode the coded video sequence received by the parser (520). The entropy coding of the coded video sequence may follow a video coding technique or standard and may follow various principles, including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser (520) may extract a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder from the coded video sequence based on at least one parameter corresponding to the subgroup. The subgroup may include a group of pictures (GOP), an image, a tile, a slice, a macroblock, a coding unit (CU), a block, a transform unit (TU), a prediction unit (PU), etc.

[0053] The parser (520) may also extract information from the coded video sequence, such as transform coefficients (e.g., Fourier transform coefficients), quantization parameter values, motion vectors, etc. The parser (520) may perform entropy decoding / parsing operations on the video sequence received from the buffer memory (515) to generate symbols (521).

[0054] The reconstruction of the symbols (521) can involve several different processing or functional units, depending on the type of video image or portion thereof being coded (e.g., inter- and intra-image, inter- and intra-block) and other factors. The units involved and how they participate can be controlled by subgroup control information parsed from the coded video sequence by the parser (320). The flow of such subgroup control information between the parser (520) and the following units is not shown for simplicity.

[0055] In addition to the functional blocks already mentioned, the video decoder (510) may be conceptually divided into several functional units, as described below. In a practical implementation operating under commercial constraints, many of these functional units will interact closely with each other and may be at least partially integrated with each other. However, for purposes of clearly describing the various functions of the disclosed subject matter, the following disclosure employs a conceptual division into functional units.

[0056] The first unit may be a scalar / inverse transform unit (551). The scalar / inverse transform unit (551) may receive quantized transform coefficients and control information from the parser (520), including information indicating the type of inverse transform to use, the block size, the quantization coefficients / parameters, the quantization scaling matrix, and the symbol(s) as false (521). The scalar / inverse transform unit (551) may output blocks containing sample values ​​that can be input to the aggregator (555).

[0057] In some cases, the output samples of the scaler / inverse transform (551) may relate to intra-coded blocks, i.e., blocks that do not use prediction information from a previously reconstructed image but can use prediction information from a previously reconstructed portion of the current image. Such prediction information may be provided by an intra-image prediction unit (552). In some cases, the intra-image prediction unit (552) may generate blocks of the same size and shape as the block being reconstructed using already reconstructed surrounding blocks stored in a current image buffer (558). The current image buffer (558), for example, buffers a partially reconstructed current image and / or a fully reconstructed current image. In some implementations, the aggregator (555) may add, on a sample-by-sample basis, the prediction information generated by the intra-prediction unit (552) to the output sample information provided by the scaler / inverse transform unit (551).

[0058] In other cases, the output samples of the scalar / inverse transform unit (551) may relate to inter-coding and potentially to motion-compensated blocks. In such cases, the motion-compensated prediction unit (553) may access a reference picture memory (557) to fetch samples used for inter-picture prediction. After motion-compensating the fetched samples according to the block-related symbols (521), these samples may be added by an aggregator (555) to the output of the scalar / inverse transform unit (551) (the output of unit 551 may be referred to as a residual sample or residual signal) to generate output sample information. The addresses in the reference picture memory (557) from which the motion-compensated prediction unit (553) fetches prediction samples may be controlled by motion vectors available to the motion-compensated prediction unit (553), for example, in the form of symbols (521) having X and Y components (shift), and a reference picture component (time). Motion compensation can also include interpolating sample values ​​as fetched from a reference picture memory (557) if sub-sample accurate motion vectors are used, and can also be associated with a motion vector prediction mechanism, etc.

[0059] The output samples of the aggregator (555) may be subjected to various loop filtering techniques in a loop filter unit (556). Video compression techniques may include in-loop filter technologies controlled by parameters contained in the coded video sequence (also referred to as the coded video bitstream) and made available to the loop filter unit (556) as symbols (521) from the parser (520), and may be responsive to meta-information obtained between the decoding of the coded image or coded video sequence and the previous portion (in decoding order), as well as to previously reconstructed and loop-filtered sample values. Several types of loop filters may be included as part of the loop filter unit 556, in various orders, as described below.

[0060] The output of the loop filter unit (556) can be a sample stream that can be output to a rendering device (512) or stored in a reference image memory (557) for use in future inter-image prediction.

[0061] Once a particular coded picture is fully reconstructed, it can be used as a reference picture for future inter-prediction. For example, once a coded picture corresponding to a current picture is fully reconstructed and the coded picture is identified as a reference picture (e.g., by the parser (520)), the current picture buffer (4558) can become part of the reference picture memory (557), and a new current picture buffer can be reallocated before starting the reconstruction of a subsequent coded picture.

[0062] The video decoder (510) may perform decoding operations according to a predetermined video compression technology adopted in a standard such as ITU-T Rec. H.265. A coded video sequence may conform to the syntax specified by the video compression technology or standard being used, in the sense that the coded video sequence conforms to both the syntax of the video compression technology or standard and the profile documented in the video compression technology or standard. Specifically, a profile may select specific tools from all tools available in the video compression technology or standard as the only tools available for that profile. Standard compliance may also require that the complexity of the coded video sequence be within a range defined by a level of the video compression technology or standard. In some cases, the level may constrain the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. The constraints set by the level may, in some cases, be further constrained by a Hypothetical Reference Decoder (HRD) specification and HRD buffer management metadata signaled in the coded video sequence.

[0063] In some exemplary embodiments, the receiver (531) can receive additional (redundant) data along with the encoded video. The additional data may be included as part of the coded video sequence(s). The additional data may be used by the video decoder (510) to properly decode the data and / or to more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0064] 6 shows a block diagram of a video encoder (603) according to an exemplary embodiment of the present disclosure. The video encoder (603) can be included in an electronic device (620). The electronic device (620) can further include a transmitter (640) (e.g., a transmitting circuit). The video encoder (603) can be used in place of the video encoder (403) in the example of FIG. 4.

[0065] The video encoder (603) may receive video samples from a video source (601) (not part of the electronic device (620) in the example of FIG. 6) that can capture the video footage(s) to be coded by the video encoder (603). In another example, the video source (601) can be implemented as part of the electronic device (620).

[0066] The video source (601) may provide a source video sequence to be coded by the video encoder (603) in the form of a digital video sample stream, which may have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 Y CrCB, RGB, ...), and any suitable sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media distribution system, the video source (601) may be a storage device capable of storing prepared video. In a video conferencing system, the video source (601) may be a camera that captures local video information as a video sequence. The video data may be provided as multiple individual images or pictures that, when viewed in sequence, produce motion. The image itself may be organized as a spatial array of pixels, each of which may contain one or more samples, depending on the sampling structure, color space, etc., in use. Those skilled in the art will readily understand the relationship between pixels and samples. The following discussion focuses on samples.

[0067] According to some example embodiments, the video encoder (603) may code and compress images of a source video sequence into a coded video sequence (643) in real time or under any other time constraint required by the application. Achieving an appropriate coding speed is one function of the controller (650). In some embodiments, the controller (650) may be functionally connected to and control other functional units, as described below. Couplings are not shown for simplicity. Parameters set by the controller (650) may include rate control-related parameters (picture skip, quantizer, lambda value for rate-distortion optimization techniques, ...), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (650) may be configured with other appropriate functions associated with the video encoder (603) optimized for a particular system design.

[0068] In some exemplary embodiments, the video encoder (603) may be configured to operate in a coding loop. As an oversimplified explanation, in one example, the coding loop may include a source coder (630) (e.g., responsible for generating symbols, such as a symbol stream, based on an input image to be coded and a reference image) and a (local) decoder (633) embedded in the video encoder (603). The decoder (633) reconstructs the symbols to create sample data in a manner similar to that created by a (remote) decoder, even when the embedded decoder 633 processes a video stream coded by the source coder 630 without entropy coding (because the compression between the symbols and the coded video bitstream in entropy coding can be lossless with the video compression techniques considered in the disclosed subject matter). The reconstructed sample stream (sample data) is input to a reference image memory (634). Because decoding the symbol stream produces bit-exact results independent of decoder location (local or remote), the contents in the reference picture memory (634) are also bit-exact between the local and remote encoders. In other words, the predictive portion of the encoder "sees" exactly the same sample values ​​as the reference picture samples that the decoder "sees" when using prediction during decoding. This basic principle of reference picture synchrony (and the resulting drift when synchrony cannot be maintained, for example, due to channel errors) is used to improve coding quality.

[0069] The operation of the "local" decoder (533) can be the same as a "remote" decoder, such as the video decoder (410), as already described in detail above in connection with Figure 5. However, briefly referring also to Figure 5, because symbols are available and the encoding / decoding of symbols into a coded video sequence by the entropy coder (645) and parser (520) can be lossless, the entropy decoding portion of the video decoder (510), including the buffer memory (515), and the parser (520), may not be fully implemented in the encoder's local decoder (633).

[0070] An observation that can be made in this regard is that any decoder technology, except for parsing / entropy decoding, may only exist in the decoder, and may need to exist in substantially the same functional form in the corresponding encoder. For this reason, the disclosed subject matter may focus on decoder operations related to the decoding portion of the encoder. Therefore, a description of the encoder technology may be omitted, since it is the reverse of the decoder technology that is comprehensively described. Only for specific areas or aspects, a more detailed description of the encoder is provided below.

[0071] During operation in some implementations, the source coder (630) may perform motion-compensated predictive coding, which predictively codes an input image with reference to one or more previously coded images from a video sequence designated as "reference images." In this manner, the coding engine (632) codes differences (residuals) between pixel blocks of the input image and pixel blocks of one or more reference images that may be selected as one or more prediction references for the input image. The terms "residue" and its adjective form "residual" are sometimes used interchangeably.

[0072] The local video decoder (633) may decode the coded video data of an image that may be designated as a reference image based on the symbols generated by the source coder (630). The operation of the coding engine (632) may advantageously be a lossy process. If the coded video data is decoded by a video decoder (not shown in FIG. 6), the reconstructed video sequence may typically be a copy of the source video sequence with some errors. The local video decoder (633) repeats the decoding process performed by the video decoder on the reference image, which may result in a reconstructed reference image to be stored in the reference image cache (634). In this way, the video encoder (603) can locally store a copy of the reconstructed reference image that has common content with the reconstructed reference image that would be obtained by the far-end (remote) video decoder (without transmission errors).

[0073] The predictor (635) may perform a prediction search for the coding engine (632). That is, for a new image to be coded, the predictor (635) may search the reference image memory (634) for specific metadata, such as reference image motion vectors, block shapes, or sample data (as candidate reference pixel blocks), that may serve as suitable prediction references for the new image. The predictor (635) may operate on a sample block-by-sample block basis to find suitable prediction references. In some cases, as determined by the search results obtained by the predictor (635), the input image may have prediction references drawn from multiple reference images stored in the reference image memory (634).

[0074] The controller (650) may manage the coding operations of the source coder (630), including, for example, setting the parameters and subgroup parameters used to encode the video data.

[0075] The outputs of all of the above-mentioned functional units may undergo entropy coding in an entropy coder (645), which converts the symbols produced by the various functional units into a coded video sequence by losslessly compressing the symbols according to techniques such as Huffman coding, variable length coding, or arithmetic coding.

[0076] The transmitter (640) can buffer the coded video sequence created by the entropy coder (645) and prepare it for transmission via a communication channel (660), which can be a hardware / software link to a storage device that will store the encoded video data. The transmitter (640) can merge the coded video data from the video coder (603) with other data to be transmitted, such as coded audio data and / or ancillary data streams (not shown).

[0077] The controller (650) may manage the operation of the video encoder (603). During coding, the controller (650) may assign each coded picture a particular coded picture type, which may affect the coding technique that may be applied to each picture. For example, pictures are often assigned as one of the following picture types:

[0078] Intra-pictures (I-pictures) can be coded and decoded without using other pictures in the sequence as prediction sources. Some video codecs allow for different types of intra-pictures, including, for example, Independent Decoder Refresh ("IDR") pictures. Those skilled in the art are aware of these variations of I-pictures and their respective uses and characteristics.

[0079] A predicted image (P-image) may be coded and decoded using inter- or intra-prediction, which uses at most one motion vector and reference index to predict the sample values ​​of each block.

[0080] Bi-directionally predicted images (B-pictures) may be coded and decoded using inter- or intra-prediction, which uses up to two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, multiple predicted images may use two or more reference images and associated metadata for the reconstruction of a block.

[0081] A source image is typically spatially subdivided into multiple sample coding blocks (e.g., 4x4, 8x8, 4x8, or 16x16 blocks of each sample) and coded block by block. Blocks may be predictively coded with reference to other (already coded) blocks, as determined by the coding assignment applied to each image of the block. For example, blocks of an I image may be non-predictively coded, or they may be predictively coded with reference to previously coded blocks of the same image (spatial prediction or intra-prediction). Pixel blocks of a P image may be predictively coded via spatial or temporal prediction with reference to one previously coded reference image. Blocks of a B image may be predictively coded via spatial or temporal prediction with reference to one or two previously coded reference images. A source image or an intermediate processed image may be subdivided into other types of blocks for other purposes. As will be explained in more detail below, the division of coding blocks and other types of blocks may or may not follow the same method.

[0082] The video encoder (603) may perform coding operations according to a predetermined video coding technique or standard, such as ITU-T Rec. H.265. In doing so, the video encoder (603) may perform various compression operations, including predictive coding operations that exploit temporal and spatial redundancy in the input video sequence. Thus, the coded video data may conform to a syntax specified by the video coding technique or standard being used.

[0083] In one embodiment, the transmitter (640) may transmit additional data along with the encoded video. The source coder (630) may include such data as part of the coded video sequence. The additional data may include temporal / spatial / SNR enhancement layers and other types of redundant data, such as redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

[0084] Video may be captured as multiple source images (video images) in a time sequence. Intra-picture prediction (often abbreviated as intra-prediction) uses spatial correlation within a given image, while inter-picture prediction uses temporal or other correlation between images. In one example, a particular image being coded / decoded, referred to as the current image, is partitioned into blocks. If a block in the current image is similar to a reference block in a previously coded and still buffered reference image in the video, the block in the current image may be coded by a vector referred to as a motion vector. The motion vector points to a reference block in the reference image and may have a third dimension that identifies the reference image if multiple reference images are used.

[0085] In some embodiments, a bi-prediction technique can be used for inter-image prediction. Bi-prediction techniques use two reference images, such as a first reference image and a second reference image, both of which precede a current image in decoding order (but may be past or future, respectively, in display order) in a video. A block in the current image can be coded with a first motion vector that points to a first reference block in the first reference image and a second motion vector that points to a second reference block in the second reference image. A block can be predicted by a combination of the first and second reference blocks.

[0086] Furthermore, merge mode techniques can be used in inter-image prediction to improve coding efficiency.

[0087] According to some embodiments of the present disclosure, prediction, such as inter-image prediction and intra-image prediction, is performed in units of blocks. For example, images in a video image sequence are partitioned into coding tree units (CTUs) for compression, and the CTUs in an image have the same size, such as 64x64 pixels, 32x32 pixels, or 16x16 pixels. Generally, a CTU includes three coding tree blocks (CTBs), one luma CTB and two chroma CTBs. Each CTU can be recursively quad-tree split into one or more coding units (CUs). For example, a 64x64 pixel CTU can be split into one 64x64 pixel CU and four 32x32 pixel CUs. One or more of the 32x32 blocks can be further split into four 16x16 pixel CUs. In an example, each CU is analyzed to determine the CU's prediction type, such as an inter prediction type or an intra prediction type. A CU is split into one or more prediction units (PUs) depending on temporal and / or spatial predictability. Generally, each PU includes one luma prediction block (PB) and two chroma PBs. In one embodiment, prediction operations in coding (encoding / decoding) are performed in units of prediction blocks. The split of a CU into PUs (or PBs of different color channels) may be performed in various spatial patterns. For example, a luma or chroma PB may include a matrix of values ​​(e.g., luma values) for samples of 8x8 pixels, 16x16 pixels, 8x16 pixels, 16x8 samples, etc.

[0088] 7 shows a diagram of a video encoder (703) according to another exemplary embodiment of the present disclosure. The video encoder (703) is configured to receive a processed block (e.g., a predictive block) of sample values ​​in a current video image in a video image sequence and encode the processed block into a coded image that is part of the coded video sequence. The video encoder (403) of the example of FIG. 4 can be replaced by a video encoder (603).

[0089] For example, the video encoder (703) receives a matrix of sample values ​​for a processing block, such as a predictive block of 8x8 samples. The video encoder (703) determines whether the processing block is best coded using intra-mode, inter-mode, or bi-predictive mode, e.g., using rate-distortion optimization. If the processing block is coded in intra-mode, the video encoder (703) may use intra-prediction techniques to encode the processing block into a coded image. If the processing block is to be coded in inter-mode or bi-predictive mode, the video encoder (703) may use inter-prediction techniques or bi-prediction techniques, respectively, to code the processing block into a coded image. In some exemplary embodiments, merge mode may be an inter-image prediction submode in which motion vectors are derived from one or more motion vector predictors without the benefit of motion vector components coded outside the predictors. In some exemplary embodiments, there may be motion vector components applicable to the current block. Therefore, the video encoder (703) may include components not explicitly shown in FIG. 7, such as a mode decision module, to determine the partition mode of the processing block.

[0090] In the example of Figure 7, the video encoder (703) includes an entropy encoder (725), an inter-encoder (730), an intra-encoder (722), a residual calculator (723), a switch (726), a residual encoder (724), and a general-purpose controller (721), coupled together as shown in the exemplary arrangement of Figure 7.

[0091] The inter-encoder (730) is configured to receive samples of a current block (e.g., a processing block), compare the block with one or more reference blocks in a reference image (e.g., blocks in a preceding image and a subsequent image), generate inter-prediction information (e.g., a description of redundant information due to an inter-encoding technique, motion vectors, merge mode information), and calculate an inter-prediction result (e.g., a predicted block) based on the inter-prediction information using any suitable technique. In some embodiments, the reference image is a decoded reference image that is decoded based on the encoded video information using a decoding unit 633 embedded in the example encoder 620 of FIG. 6 (shown as residual decoder 728 of FIG. 7 and described in further detail below).

[0092] The intra encoder (722) is configured to receive samples of a current block (e.g., a processing block), compare the block to previously coded blocks in the same image, generate transformed and quantized coefficients, and possibly also generate intra prediction information (e.g., intra prediction direction information according to one or more intra coding techniques). The intra encoder (722) can also calculate intra prediction results (e.g., predicted blocks) based on the intra prediction information and reference blocks in the same image.

[0093] The general-purpose controller (721) may be configured to determine general-purpose control data and control other components of the video encoder (703) based on the general-purpose control data. In one embodiment, the general-purpose controller (721) determines a prediction mode for a block and provides a control signal to the switch (726) based on the prediction mode. For example, if the prediction mode is intra-mode, the general-purpose controller (721) controls the switch (726) to select the intra-mode result used by the residual calculator (723) and controls the entropy encoder (725) to select intra-prediction information and include the intra-prediction information in the bitstream. If the prediction mode is inter-mode, the general-purpose controller (721) controls the switch (726) to select the inter-prediction result used by the residual calculator (723) and controls the entropy encoder (725) to select inter-prediction information and include the inter-prediction information in the bitstream.

[0094] The residual calculator (723) may be configured to calculate the difference (residual data) between a received block and a prediction result for the block selected from the intra-encoder (722) or inter-encoder (730). The residual encoder (724) may be configured to encode the residual data to generate transform coefficients. In one embodiment, the residual encoder (724) may be configured to convert the residual data from the spatial domain to the frequency domain to generate transform coefficients. The transform coefficients are then subjected to a quantization process to obtain quantized transform coefficients. In various exemplary embodiments, the video encoder (703) also includes a residual decoder (728). The residual decoder (728) is configured to perform an inverse transform to generate decoded residual data. The decoded residual data can be used by the intra-encoder (722) and inter-encoder (730) as appropriate. For example, the inter-encoder (730) can generate decoded blocks based on the decoded residual data and inter-prediction information, and the intra-encoder (722) can generate decoded blocks based on the decoded residual data and intra-prediction information. The decoded blocks are processed appropriately to generate decoded images, which can be buffered in a memory circuit (not shown) and used as reference images.

[0095] The entropy encoder (725) is configured to format a bitstream to include the encoded blocks. The entropy encoder (725) is configured to include various information in the bitstream. For example, the entropy encoder (725) is configured to include general control data, selected prediction information (e.g., intra-prediction information or inter-prediction information), residual information, and other appropriate information in the bitstream. When coding blocks in a merged sub-mode of either an inter mode or a bi-prediction mode, the residual information may not be present.

[0096] 8 shows a diagram of an exemplary video decoder (810) according to another embodiment of the present disclosure. The video decoder (810) is configured to receive coded images that are part of a coded video sequence and to decode the coded images to generate reconstructed images. In one embodiment, the video decoder (810) is used in place of the video decoder (410) of the embodiment of FIG. 4.

[0097] In the embodiment of FIG. 8, the video decoder (810) includes an intra decoder (872), an entropy decoder (871), an inter decoder (880), a residual decoder (873), and a reconstruction module (874), coupled together as shown in the exemplary arrangement of FIG.

[0098] The entropy decoder (871) can be configured to reconstruct, from the coded picture, specific symbols representing syntax elements that make up the coded picture. Such symbols can include, for example, the mode in which the block is coded (e.g., intra mode, inter mode, bidirectional prediction mode, inter mode in merged or another submode, bidirectional prediction mode, etc.), prediction information (e.g., intra prediction information or inter prediction information), which can identify specific samples or metadata used for prediction by the intra decoder (822) or inter decoder (880), such as residual information in the form of quantized transform coefficients. In an embodiment, when the prediction mode is inter mode or bidirectional prediction mode, the inter prediction information is provided to the inter decoder (880); and when the prediction type is intra prediction type, the intra prediction information is provided to the intra decoder (872). The residual information can undergo inverse quantization and be provided to the residual decoder (873).

[0099] The inter decoder (880) is configured to receive the inter prediction information and generate an inter prediction result based on the inter prediction information.

[0100] The intra decoder (872) is configured to receive intra prediction information and generate a prediction result based on the intra prediction information.

[0101] The residual decoder (873) is configured to perform inverse quantization to extract de-quantized transform coefficients and process the de-quantized transform coefficients to convert the residual from the frequency domain to the spatial domain. The residual decoder (873) may also utilize certain control information (including quantizer parameters (QP)), which may be provided by the entropy decoder (871) (data path not shown, as it is low data volume control information only).

[0102] The reconstruction module (874) may be configured to combine, in the spatial domain, the prediction results (possibly as output by an inter- or intra-prediction module) with the residuals (as output by the residual decoder (873)) to form reconstruction blocks that form part of the reconstructed images as part of the reconstructed video. Note that other suitable operations, such as deblocking operations, may be performed to improve visual quality.

[0103] It should be noted that the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) may be implemented using any suitable technology. In some exemplary embodiments, the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) may be implemented using one or more integrated circuits. In other embodiments, the video encoders (403), (603), and (603) and the video decoders (410), (510), and (810) may be implemented using one or more processors executing software instructions.

[0104] Turning to block division for coding and decoding, general division or partitioning starts with a base block and can follow a predetermined set of rules, a specific pattern, a partition tree, or any partition structure or scheme. The division is hierarchical and recursive. After dividing or partitioning the base block according to the following division procedures or other example procedures, or a combination thereof, a final set of partitions or coding blocks can be obtained. Each of these partitions can be at any of various partition levels in the partition hierarchy and can have various shapes. Each partition can be referred to as a coding block (CB). In various partitioning embodiments described further below, each resulting CB can be of any allowed size and partition level. Such partitions are referred to as coding blocks because they form the unit on which basic coding / decoding decisions are made and coding / decoding parameters are optimized, determined, and potentially signaled in the encoded video bitstream. The topmost or deepest level of the final partition represents the depth of the coding block partition structure in the tree. A coding block can be a luma coding block or a chroma coding block. The CB tree structure for each color is sometimes called a coding block tree (CBT).

[0105] The coding blocks of all color channels may be collectively referred to as a coding unit (CU). The hierarchical structure of all color channels may be collectively referred to as a coding tree unit (CTU). The partitioning pattern or structure of different color channels within a CTU may or may not be the same.

[0106] In some implementations, the partitioning tree scheme or structure used for the luma channel and the chroma channel may not need to be the same. That is, the luma channel and the chroma channel may have separate coding tree structures or patterns. Furthermore, whether the luma channel and the chroma channel use the same or different coding partition tree structures, and the actual coding partition tree structure used, may vary depending on whether the slice being coded is a P, B, or I slice. For example, for an I slice, the chroma channel and the luma channel may have separate coding partition tree structures or coding partition tree structure modes, while for a P or B slice, the luma channel and the chroma channel may share the same coding partition tree structure. When separate coding partition tree structures or modes are applied, the luma channel may be partitioned into CBs by one coding partition tree structure, and the chroma channel may be partitioned into chroma CBs by another coding partition tree structure.

[0107] In some exemplary embodiments, a predetermined partitioning pattern may be applied to the base block. As shown in FIG. 9 , an example of a 4-way partition tree may start from a first predetermined level (e.g., a 64×64 block level or other size as the basic block size) and hierarchically partition the base block down to a predetermined lowest level (e.g., a 4×4 level). For example, the base block may be subject to four predetermined partition options or patterns, shown as 902, 904, 906, and 908. Recursive partitioning is permitted, where a partition designated as R may be repeated at a lower scale down to the lowest level (e.g., a 4×4 level) with the same partition option as shown in FIG. 9 . In some implementations, additional restrictions may be applied to the partitioning scheme of FIG. 9 . In the implementation of FIG. 9 , rectangular partitions (e.g., 1:2 / 2:1 rectangular partitions) may be permitted but may not be permitted to be recursive, while square partitions may be recursive. Partitioning with recursion following FIG. 9 generates a final set of coding blocks, if necessary. A coding tree depth can be defined to indicate the splitting depth from a root node or root block. For example, the coding tree depth of a root node or root block, such as a 64x64 block, can be set to 0, and after splitting the root block one more time following FIG. 9, the coding tree depth can be increased by 1. The maximum or deepest level from the 64x64 base block to the smallest 4x4 partition is 4 (starting from level 0) in the above scheme. Such a partitioning scheme may be applied to one or more color channels. Each color channel can be partitioned independently according to the scheme of FIG. 9 (e.g., the partitioning pattern or options within a predetermined pattern can be determined independently for each of the color channels at each hierarchical level).Alternatively, two or more color channels may share the same hierarchical pattern tree of FIG. 9 (e.g., the same partitioning pattern or option from among the predefined patterns may be selected for two or more color channels at each hierarchical level).

[0108] FIG. 10 illustrates another exemplary predetermined partitioning pattern that enables recursive partitioning to form a partitioning tree. As shown in FIG. 10, ten exemplary partitioning structures or patterns can be predefined. The root block can start at a predetermined level (e.g., from a base block at a 128×128 level or a 64×64 level). The exemplary partitioning structure of FIG. 10 includes various 2:1 / 1:2 and 4:1 / 1:4 rectangular partitions. The type of partition with three subpartitions, shown as 1002, 1004, 1006, and 1008 in the second row of FIG. 10, may be referred to as a “T-type” partition. The “T-type” partitions 1002, 1004, 1006, and 1008 may be referred to as Left T-Type, Top T-Type, Right T-Type, and Bottom T-Type. In some implementations, none of the rectangular partitions in FIG. 10 can be further subdivided. The coding tree depth can be further defined to indicate the splitting depth from the root node or root block. For example, the coding tree depth of the root node or root block, e.g., a 128x128 block, can be set to 0, and after splitting the root block one more time according to FIG. 10, the coding tree depth can be increased by 1. In some implementations, only all-square partitions 1010 may be allowed for recursive partitioning to the next level of the partitioning tree according to the pattern of FIG. 10. That is, recursive partitioning may not be allowed for the square partitions in the T-shaped patterns 1002, 1004, 1006, and 1008. The partitioning procedure with recursion following FIG. 10 generates the final set of coding blocks as needed. Such a scheme can be applied to one or more color channels. In some implementations, more flexibility may be added to the use of partitions below the 8x8 level. For example, 2x2 chroma inter prediction may be used in certain cases.

[0109] In some other exemplary implementations for coding block partitioning, a quadtree structure can be used to split base or intermediate blocks into quadtree partitions. Such quadtree splitting can be applied hierarchically and recursively to any square partition. Whether a base or intermediate block or partition is further quadtree split can be adapted to various local characteristics of the base or intermediate block / partition. Quadtree partitioning at image boundaries can be further adapted. For example, implicit quadtree splitting can be performed at image boundaries, such that blocks maintain quadtree splitting until their size fits the image boundary.

[0110] Some other exemplary implementations may use hierarchical binary partitioning from a base block. In such a scheme, a base block or an intermediate-level block may be divided into two partitions. The binary partitioning may be either horizontal or vertical. For example, horizontal binary partitioning may split a base block or intermediate block into equal left and right partitions. Similarly, vertical binary partitioning may split a base block or intermediate block into equal top and bottom partitions. Such binary partitioning is hierarchical and recursive. A decision is made for each base block or intermediate block as to whether to continue the binary partitioning scheme and, if so, whether to use horizontal or vertical binary partitioning. In some implementations, further partitioning may stop at a predetermined minimum partition size (or at either or both time limits). Alternatively, further partitioning may stop once a predetermined partitioning level or depth from the base block is reached. In some implementations, the aspect ratio of a partition may be limited. For example, the aspect ratio of a partition cannot be smaller than 1:4 (or greater than 4:1). Thus, a vertical strip partition having a vertical to horizontal aspect ratio of 4:1 can be further binary partitioned vertically into upper and lower partitions each having a vertical to horizontal aspect ratio of 2:1.

[0111] In yet some other embodiments, a ternary partitioning scheme can be used to partition the base block or any intermediate blocks, as shown in FIG. 13 . The ternary pattern can be implemented vertically, as shown at 1302 in FIG. 13 , or horizontally, as shown at 1304 in FIG. 13 . The example split ratio in FIG. 13 is shown as 1:2:1 either vertically or horizontally, but other ratios may be predefined. In some implementations, two or more different ratios may be predefined. Such a ternary partitioning scheme can be used to complement a quadtree or binary partitioning structure; such triple tree partitioning can capture objects at the center of a block into one contiguous partition, while quadtrees and binary trees always split objects into separate partitions because they are always split along the center of the block. In some implementations, the width and height of the partitions in the example triple tree are always powers of two to avoid additional transformations.

[0112] The above partitioning schemes can be combined in any manner at different partitioning levels. As an example, the above quadtree and binary partitioning schemes can be combined to partition a base block into a quadtree binary (QTBT) structure. In such a scheme, the base block or intermediate blocks / partitions, if specified, are either quadtree split or binary split according to a set of defined conditions. A specific example is shown in FIG. 14. In the example of FIG. 14, the base block is first quadtree split into four partitions, as shown at 1402, 1404, 1406, and 1408. Each resulting partition is then either quadtree partitioned into four more partitions at the next level (as at 1408), binary split into two more partitions at the next level (e.g., either horizontally or vertically, as at 1402 or 1406, both symmetric), or not split at all (as at 1404). Binary or quadtree splitting may be recursively permitted for square partitions, as illustrated by the overall exemplary partitioning pattern in 1410 and the corresponding tree structure / representation in 1420, where solid lines represent quadtree splitting and dashed lines represent binary splitting. A flag may be used for each binary splitting node (non-leaf binary partition) to indicate whether the binary splitting is horizontal or vertical. For example, as shown in 1420, a flag "0" may represent horizontal binary splitting, and a flag "1" may represent vertical binary splitting, consistent with the partitioning structure in 1410. In a quadtree split partition, there is no need to indicate the type of splitting, since a quadtree split always splits a block or partition horizontally and vertically to generate four sub-blocks / partitions of equal size. In some implementations, a flag "1" may represent horizontal binary splitting, and a flag "0" may represent vertical binary splitting.

[0113] In some exemplary implementations of QTBT, the quadtree and binary splitting rule sets may be represented by the following predetermined parameters and their associated corresponding functions: - CTU size: Size of the root node of the quadtree (size of the base block) - MinQTSize: The minimum allowed quadtree leaf node size - MaxBTSize: Maximum allowed binary tree root node size - MaxBTDepth: Maximum allowed binary tree depth - MinBTSize: Minimum allowed binary tree root node size In some example implementations of the QTBT partitioning structure, the CTU size can be set as 128x128 luma samples containing two corresponding 64x64 blocks of chroma samples (when used in conjunction with the chroma subsampling example), MinQTSize can be set to 16x16, MaxBTSize can be set to 64x64, MinBTSize (both width and height) can be set to 4x4, and MaxBTDepth can be set to 4. Quadtree partitioning is first applied to the CTU to generate quadtree leaf nodes. Quadtree leaf nodes have a minimum allowable size ranging from 16x16 (i.e., MinQTSize) to 128x128 (i.e., the CTU size). If a leaf QT node is 128x128, it will not be split by the binary tree initially because its size exceeds MaxBTSize (i.e., 64x64). Otherwise, nodes that do not exceed MaxBTSize can be partitioned by the binary tree. In the example of Figure 14, the base block is 128x128. Basic blocks are only capable of quadtree splitting according to a predetermined set of rules. The partitioning depth for basic blocks is 0. The four resulting partitions are each 64x64, do not exceed MaxBTSize, and may undergo further quadtree or binary tree splitting at level 1. The process continues. If the binary tree depth reaches MaxBTDepth (i.e., 4), no further splitting is considered. If a binary tree node has a width equal to MinBTSize (i.e., 4), no further horizontal splitting is considered. Similarly, if a binary tree node has a height equal to MinBTSize, no further vertical splitting is considered.

[0114] In some example implementations, the upper QTBT scheme may be configured to support the flexibility of luma and chroma having the same or separate QTBT structures. For example, in P slices and B slices, the luma and chroma CTBs of a CTU share the same QTBT structure. However, for an I slice, the luma CTB is partitioned into CBs by a QTBT structure, and the chroma CTB is partitioned into chroma CUs by a different QTBT structure. This means that CUs can be used to refer to different color channels within an I slice; for example, an I slice may consist of a coding block for a luma component or a coding block for two chroma components, while a CU in a P or B slice may consist of coding blocks for all three color components.

[0115] In some other implementations, the QTBT scheme can be supplemented with the ternary scheme described above. Such implementations may be referred to as multi-type tree (MTT) structures. For example, in addition to binary splitting of nodes, one of the ternary partition patterns in FIG. 13 may be used. In some implementations, only square nodes may be subject to ternary splitting. An additional flag may be used to indicate whether the ternary partitioning is horizontal or vertical.

[0116] Two-level or multi-level tree designs, such as QTBT implementations and QTBT implementations complemented by turnarisplitting, can be motivated primarily by reduced complexity. In theory, the complexity of traversing a tree is reduced by T D where T denotes the number of split types and D denotes the depth of the tree. A trade-off can be made by using multiple types (T) while reducing the depth (D).

[0117] In some implementations, the CB may be further partitioned. For example, the CB may be further partitioned into multiple prediction blocks (PBs) for the purpose of intra-frame or inter-frame prediction during the coding and decoding process. In other words, the CB may be further divided into different sub-partitions, and individual prediction decisions / configurations may be made. In parallel, the CB may be further partitioned into multiple transform blocks (TBs) for the purpose of delineating the levels at which transform or inverse transform of video data is performed. The partitioning schemes for the CB into PBs and TBs may or may not be the same. For example, each partitioning scheme may be performed using its own procedure, for example, based on various characteristics of the video data. The PB and TB partitioning schemes may be independent in some exemplary implementations. The partitioning schemes and boundaries of the PBs and TBs may be correlated in some other exemplary implementations. In some implementations, for example, the TBs may be partitioned after the partitioning of the PBs, and in particular, each PB may be determined after the partitioning of the coding blocks and then further partitioned into one or more TBs. For example, in some implementations, a PB may be split into 1, 2, 4, or some other number of TBs.

[0118] In some implementations, when partitioning base blocks into coding blocks and further partitioning into prediction blocks and / or transform blocks, the luma channel and the chroma channels may be treated separately. For example, in some implementations, partitioning of coding blocks into prediction blocks and / or transform blocks may be allowed for a luma channel, but partitioning of such coding blocks into prediction blocks and / or transform blocks may not be allowed for one or more chroma channels. In such implementations, transforms or predictions of luma blocks may therefore be performed only at the coding block level. As another example, the minimum transform block sizes for one or more luma and chroma channels may be different, e.g., coding blocks for a luma channel may be allowed to be partitioned into smaller transform and / or prediction blocks than the chroma channels. As yet another example, the maximum depth of partitioning a coding block into transform blocks and / or prediction blocks may differ between luma and chroma channels, e.g., a coding block for a luma channel may be allowed to be partitioned into deeper transform blocks and / or prediction blocks than one or more chroma channel(s). As a specific example, a luma coding block may be partitioned into transform blocks of multiple sizes, which may be represented by a recursive partition up to two levels down, and transform block shapes such as square, 2:1 / 1:2, 4:1 / 1:4, etc., and transform block sizes from 4x4 to 64x64 may be allowed. However, for chroma blocks, only the largest possible transform block designated for the luma block may be allowed.

[0119] In some implementations of partitioning a coding block into PBs, the depth, shape, and / or other characteristics of the PB partitioning may differ depending on whether the PB is intra- or inter-coded.

[0120] The partitioning of coding blocks (or predictive blocks) into transform blocks may be implemented recursively or non-recursively, with further consideration of transform blocks at the boundaries of the coding or predictive blocks, in various example schemes including, but not limited to, quadtree splitting and predetermined pattern splitting. In general, the resulting transform blocks may be at different split levels, may be the same size, and may not necessarily be square in shape (e.g., they may be rectangular with some allowed size and aspect ratio). Further examples are described in more detail below in connection with Figures 15, 16, and 17.

[0121] However, in some other implementations, the CB obtained through any of the above partitioning schemes can be used as a basic or minimum coding block for prediction and / or transform. That is, no further splitting is performed for inter-prediction / intra-prediction and / or transform purposes. For example, the CB obtained from the above QTBT scheme may be used directly as a unit for performing prediction. Specifically, such a QTBT structure removes the concept of multiple partition types, i.e., the separation of CUs, PUs, and TUs, and supports more flexibility in CU / CB partition shapes as described above. In such a QTBT block structure, CUs / CBs can have either square or rectangular shapes. The leaf nodes of such a QTBT are used as units for prediction and transform processing without further partitioning. This means that in such an exemplary QTBT coding block structure, the block sizes of CUs, PUs, and TUs are the same.

[0122] The various CB partitioning schemes described above, and further partitioning of the CB into PB and / or TB (including no PB / TB partitioning), can be combined in any manner. The following specific implementations are provided as non-limiting examples.

[0123] A specific implementation example of partitioning coding blocks and transform blocks is shown below. In such an exemplary implementation, a base block can be split into coding blocks using recursive quadtree partitioning or the aforementioned predetermined splitting patterns (such as those in FIGS. 9 and 10). At each level, whether to continue further quadtree splitting of a particular partition may be determined by local video data characteristics. The resulting CBs may be of different sizes at different quadtree splitting levels. The decision of whether to code an image region using inter-picture (temporal) prediction or intra-picture (spatial) prediction can be made at the CU level (or CU level, all three color channels). Each CB is further split into one, two, four, or other number of PBs according to a predetermined PB splitting type. Within one PB, the same prediction process may be applied, and related information may be transmitted to the decoder on a PB-by-PB basis. After applying the prediction process based on the PB splitting type to obtain the residual block, the CBs can be partitioned into TBs according to another quadtree structure similar to the coding tree of the CB. In this particular implementation, the CB or TB need not be limited to a square. Furthermore, in this particular example, the PB may be square or rectangular in shape for inter prediction, and may only be square for intra prediction. A coding block may be split into, for example, four square TBs. Each TB may be further split recursively (using quadtree splitting) into smaller TBs, which are referred to as residual quadtrees (RQTs).

[0124] Another exemplary implementation of partitioning a base block into CBs, PBs, or TBs is further described below. For example, rather than using multiple partition unit types as shown in FIGS. 9 and 10, a quadtree with nested multiple-type trees using binary and ternary split segmentation structures (e.g., QTBT and QTBT with ternary splitting as described above) can be used. The separation of CBs, PBs, and TBs (i.e., partitioning CBs into PBs and / or TBs, and partitioning PBs into TBs) can be abandoned unless such CBs require further splitting and are required for CBs that are too large for the maximum transform length. This exemplary partitioning scheme can be designed to support more flexibility in CB partitioning shapes, so that both prediction and transform can be performed at the CB level without further partitioning. In such a coding tree structure, CBs can be either square or rectangular in shape. Specifically, coding tree blocks (CTBs) may be initially partitioned by a quadtree structure. The quadtree leaf nodes may then be further partitioned by nested multitype tree structures. An example of a nested multitype tree structure using binary or ternary splitting is shown in FIG. 11. Specifically, the exemplary multitype tree structure of FIG. 11 includes four splitting types, referred to as vertical binary splitting (SPLIT_BT_VER) (1102), horizontal binary splitting (SPLIT_BT_HOR) (1104), vertical ternary splitting (SPLIT_TT_VER) (1106), and horizontal ternary splitting (SPLIT_TT_HOR) (1108). CB then corresponds to the leaf of the multitype tree. In this exemplary implementation, this segmentation is used for both prediction and transformation processing, and no further partitioning is performed, unless CB is too large for the maximum transformation length.This means that in most cases, CB, PB, and TB have the same block size in a quadtree with a nested multi-type tree coding block structure. An exception occurs when the maximum supported transform length is smaller than the width or height of the color component of CB. In some implementations, in addition to binary or ternary splitting, the nested pattern of Figure 11 may further include quadtree splitting.

[0125] An example of a quadtree with nested multi-type tree coding block structure of block partitions (including quadtree, binary, and ternary splitting options) for a single base block is shown in Figure 12. More specifically, Figure 12 shows a base block 1200 as a quadtree split into four square partitions 1202, 1204, 1206, and 1208. The multi-type tree structure of Figure 11 and the decision to use a quadtree for further splitting are made for each quadtree split partition. In the example of Figure 12, partition 1204 is not further split. Partitions 1202 and 1208 each employ a different quadtree split. For partition 1202, the top-left, top-right, bottom-left, and bottom-right partitions of the second-level quadtree split employ quadtree third-level splitting, horizontal binary splitting 1104 of Figure 11, and no-splitting, horizontal ternary splitting 1108 of Figure 11, respectively. Partition 1208 employs another quadtree splitting, with the top-left, top-right, bottom-left, and bottom-right partitions of the second-level quadtree split employing third-level splitting, vertical ternary splitting 1106 of Figure 11, and no-splitting, no-partitioning, horizontal binary splitting 1104 of Figure 11, respectively. The two subpartitions of the top-left partition 1208 of the third level are further split according to horizontal binary splitting 1104 and horizontal ternary splitting 1108 of Figure 11, respectively. Partition 1206 adopts a second-level split pattern following vertical binary splitting 1102 in Figure 11 into two partitions that are further split at a third level according to horizontal ternary partitioning 1108 and vertical binary partitioning 1102 in Figure 11. Fourth-level splitting is further applied to one of them according to horizontal binary splitting 1104 in Figure 11.

[0126] In the above example, the maximum luma transform size is 64x64, and the maximum supported chroma transform size may differ from the luma, for example, 32x32. Although the CB in the above example of Figure 12 is not generally further split into smaller PBs and / or TBs, if the width or height of the luma coding block or chroma coding block is larger than the maximum transform width or height, the luma coding block or chroma coding block may be automatically split in the horizontal and / or vertical directions to meet the transform size restrictions in that direction.

[0127] In the specific example of partitioning a base block into the above CBs, as mentioned above, the coding tree scheme may support the ability for luma and chroma to have separate block tree structures. For example, in P slices and B slices, the luma and chroma CTBs of one CTU share the same coding tree structure. For example, in an I slice, luma and chroma can have separate coding block tree structures. When applying separate block tree structures, the luma CTB is partitioned into luma CUs by one coding tree structure, and the chroma CTB is partitioned into chroma CUs by another coding tree structure. This means that a CU in an I slice can include a coding block for the luma component or a coding block for two chroma components, and a CU in a P slice or B slice can always include coding blocks for all three color components unless the video is monochrome.

[0128] When a coding block is further partitioned into multiple transform blocks, the transform blocks therein may be ordered in the bitstream according to various orders or scanning methods. Exemplary methods for partitioning coding or prediction blocks into transform blocks and the coding order of the transform blocks are described in further detail below. In some exemplary implementations, as mentioned above, the transform partitioning may support multiple shapes of transform blocks, e.g., 1:1 (square), 1:2 / 2:1, and 1:4 / 4:1, with transform block sizes ranging from 4×4 to 64×64. In some implementations, when the coding block is 64×64 or smaller, the transform block partitioning may only be applied to the luma component, and for chroma blocks, the transform block size is the same as the coding block size. Otherwise, if the width or height of the coding block is greater than 64, both the luma block and the chroma coding block may be implicitly split into transform blocks that are multiples of min(W, 64) x min(H, 64) and min(W, 32) x min(H32), respectively.

[0129] In some example implementations of transform block partitioning, for both intra-coding and inter-coding blocks, the coding blocks may be further partitioned into multiple transform blocks with a partitioning depth up to a predetermined number of levels (e.g., two levels). The partitioning depth and size of the transform blocks may be related. In some example implementations, the mapping from the transform size of the current depth to the transform size of the next depth is shown in Table 1. [Table 1]

[0130] Based on the example mapping in Table 1, for a 1:1 square block, the next level transform split may create four 1:1 square sub-transform blocks. The transform partition may stop at, for example, 4 x 4, so that a transform size at the current depth of 4x4 corresponds to the same 4x4 size at the next depth. In the example of Table 1, for a 1:2 / 2:1 non-square block, the next level transform split may create two 1:1 square sub-transform blocks, and for a 1:4 / 4:1 non-square block, the next level transform partitioning may create two 1:2 / 2:1 sub-transform blocks.

[0131] In some example implementations, additional restrictions on transform block partitioning may be applied to the luma component of an intra-coding block. For example, at each level of transform partitioning, all sub-transform blocks may be constrained to have the same size. For example, for a 32x16 coding block, level 1 transform splitting creates two 16x16 sub-transform blocks, while level 2 transform splitting creates eight 8x8 sub-transform blocks. In other words, to keep the transform units the same size, the second-level splitting must be applied to all first-level sub-blocks. An example of transform block partitioning for an intra-coding square block according to Table 1 is shown in FIG. 15, with the coding order indicated by the arrows. Specifically, 1502 denotes a square coding block. The first level split into four equal-sized transform blocks according to Table 1 is shown in 1504, with the coding order indicated by the arrows. The second level split, in which all equal-sized blocks of the first level are split into 16 equal-sized transform blocks according to Table 1, is shown at 1506 in the coding order indicated by the arrows.

[0132] In some example implementations, the above intra-coding restrictions may not apply to the luma component of an inter-coding block. For example, after the first level of transform splitting, any one of the sub-transform blocks may be further split independently at one level. Thus, the resulting transform blocks may or may not be of the same size. An example of splitting an inter-coded block into transform blocks in its coding order is shown in Figure 16. In the example of Figure 16, an inter-coded block 1602 is split into transform blocks at two levels according to Table 1. At the first level, the inter-coded block is split into four transform blocks of the same size. Then, only one (but not all) of the four transform blocks is further split into four sub-transform blocks, resulting in a total of seven transform blocks with two different sizes, as shown at 1604. An example of the coding order of these seven transform blocks is indicated by the arrows at 1604 in Figure 16.

[0133] In some example implementations, additional restrictions on the transform blocks may apply for one or more chroma component(s). For example, for one or more chroma component(s), the size of the transform block may be as large as the size of the coding block, but may not be smaller than a predefined size, such as 8x8.

[0134] In some other example implementations, for coding blocks where either the width (W) or height (H) is greater than 64, both the luma coding block and the chroma coding block may be implicitly split into multiples of min(W, 64) × min(H, 64) and min(W, 32) × min(H, 32), respectively. Here, in this disclosure, "min(a, b)" may return the smaller value between a and b.

[0135] Figure 17 further illustrates another alternative embodiment scheme for partitioning coding blocks or prediction blocks into transform blocks. As shown in Figure 17, instead of using recursive transform partitioning, a set of predefined partitioning types can be applied to coding blocks depending on the transform type of the coding block. In the particular example shown in Figure 17, any of six exemplary partitioning types can be applied to split the coding block into various numbers of transform blocks. The scheme for generating such transform block partitioning can be applied to either coding blocks or prediction blocks.

[0136] More specifically, the partitioning scheme of FIG. 17 provides up to six exemplary partitioning types for any given transform type (transform type refers to, for example, the type of primary transform, such as ADST). In this scheme, every coding block or predictive block can be assigned a transform partition type based, for example, on rate-distortion cost. In an embodiment, the transform partition type assigned to a coding block or predictive block may be determined based on the transform type of the coding block or predictive block. A particular transform partition type may correspond to the size and pattern of transform block splits, as illustrated by the six transform partition types shown in FIG. 17. The correspondence between various transform types and various transform partition types may be predefined. Below are examples of capitalized labels indicating transform partition types that can be assigned to coding blocks or predictive blocks based on rate-distortion cost:

[0137] PARTITION_NONE: Allocate the same transformation size as the block size

[0138] PARTITION_SPLIT: Assigns a transformation size of 1 / 2 the block size in width and 1 / 2 the block size in height.

[0139] PARTITION_HORZ: Allocates a transformation size with the same width as the block size and half the height of the block size.

[0140] PARTITION_VERT: Allocates a transformation size that is half the width of the block size and the same height as the block size.

[0141] PARTITION_HORZ4: Assigns a transformation size with the same width as the block size and 1 / 4 of the block size in height.

[0142] PARTITION_VERT4: Assigns a transformation size that is 1 / 4 the width of the block size and the same height as the block size.

[0143] In the above example, all of the transform partition types shown in Figure 17 include uniform transform sizes for the partitioned transform blocks. This is not a limitation but merely an example. In some other implementations, mixed transform block sizes may be used for the partitioned transform blocks of a particular partition type (or pattern).

[0144] The PBs (or CBs, also referred to as PBs if not further partitioned into predictive blocks) obtained from any of the above partitioning schemes may become individual blocks for coding via intra- or inter-prediction. In the case of inter-prediction of the current PB, a residual between the current block and the predictive block may be generated, coded, and included in the coded bitstream.

[0145] Inter prediction can be implemented, for example, in single-reference mode or mixed-reference mode. In some implementations, a skip flag can first be included in the bitstream (or at a higher level) of the current block to indicate whether the current block is inter-coded and not skipped. If the current block is inter-coded, another flag can be included in the bitstream to signal whether single-reference mode or mixed-reference mode is used for the current block. In single-reference mode, one reference block can be used to generate a prediction block for the current block. In mixed reference mode, two or more reference blocks can be used to generate a prediction block, for example, by weighted averaging. Mixed reference mode is sometimes referred to as more-than-one-reference mode, two-reference mode, or multiple-reference mode. One or more reference blocks can be identified using one or more reference frame indices and one or more corresponding motion vectors indicating the displacement of positions between the reference block and the current block, for example, in horizontal and vertical pixels. For example, the inter-predicted block of the current block may be generated from a single reference block identified by one motion vector in a reference frame as the predictive block in a single reference mode, while in a mixed reference mode the predictive block may be generated by a weighted average of two reference blocks in two reference frames indicated by two motion vectors. The motion vector(s) may be coded and included in the bitstream in various ways.

[0146] In some implementations, an encoding or decoding system may maintain a decoded picture buffer (DPB). Some images / pictures may be maintained in the DPB waiting to be displayed (by the decoding system), and some images / pictures in the DPB may be used as reference frames to enable inter-prediction. Depending on the implementation, reference frames in the DPB may be tagged as either short-term or long-term references for the current picture being encoded or decoded. For example, short-term reference frames may include frames used for inter-prediction of blocks within the current frame or within a predetermined number (e.g., two) of subsequent video frames closest to the current frame in decoding order. Long-term reference frames may include frames in the DPB that can be used to predict image blocks in frames that are more than a predetermined number of frames away from the current frame in decoding order. Information regarding such tags of short-term and long-term reference frames is referred to as a Reference Picture Set (RPS) and may be added to the header of each frame of the encoded bitstream. Each frame of the encoded video stream may be identified by a Picture Order Counter (POC), which may be numbered according to the playback sequence in an absolute manner, or relative to a group of pictures, starting with, for example, an I-frame.

[0147] In some example implementations, one or more reference image lists including identification of short-term and long-term reference frames for inter prediction may be formed based on information in the RPS. For example, for unidirectional inter prediction, a single image reference list, denoted as L0 reference (or reference list 0), may be formed, while for bidirectional inter prediction, two image reference lists, denoted as L0 (or reference list 0) and L1 (or reference list 1), may be formed for each of the two prediction directions. The reference frames included in the L0 and L1 lists may be ordered in various predefined manners. The lengths of the L0 and L1 lists may be signaled in the video bitstream. Unidirectional inter prediction is either a combined prediction mode, where multiple references for generating a weighted average prediction block in combined prediction mode are on the same side of the block to be predicted, or a single reference mode. Bidirectional inter prediction is only possible in combined mode, in that bidirectional inter prediction involves at least two reference blocks.

[0148] In some implementations, a merge mode (MM) of inter prediction may be implemented. Generally, in the merge mode, a motion vector of single reference prediction or one or more motion vectors of mixed reference prediction of the current PB may be derived from other motion vectors rather than being calculated and signaled independently. For example, in an encoding system, a current motion vector(s) for the current PB may be reduced to a difference(s) between the current motion vector(s) and one or more other already coded motion vectors (referred to as reference motion vectors). Such difference(s) in the motion vector(s), rather than the entire current motion vector(s), may be encoded and included in the bitstream and linked to the reference motion vector(s). Correspondingly, in a decoding system, a motion vector(s) corresponding to the current PB may be derived based on the decoded motion vector difference(s) and the decoded reference motion vector(s) linked thereto. As a specific form of general merge mode (MM) inter prediction, such inter prediction based on motion vector difference may be referred to as merge mode with motion vector difference (MMVD). Therefore, MM in general, or MMVD in particular, may be implemented to exploit correlations between motion vectors associated with different PBs to improve coding efficiency. For example, adjacent PBs may have similar motion vectors. As another example, motion vectors may be temporally (inter-frame) correlated for blocks similarly located / positioned in space.

[0149] In some implementations, an MM flag may be included in the bitstream during the encoding process to indicate whether the current PB is in merge mode. Additionally or alternatively, an MMVD flag may be included during the encoding process and signaled in the bitstream to indicate whether the current PB is in MMVD mode. The MM and / or MMVD flags or indicators may be provided at the PB level, CB level, CU level, CTB level, CTU level, slice level, picture level, etc. As a specific example, both the MM flag and the MMVD flag may be included in the current CU, and the MMVD flag may be signaled immediately after the skip flag and the MM flag to specify whether to use MMVD mode for the current CU.

[0150] In some example implementations of MMVD, a list of merge candidates for motion vector prediction may be formed for the block being predicted. The merge candidate list may include a predetermined number (e.g., two) of MV prediction candidate blocks whose motion vectors may be used to predict the current motion vector. MVD candidate blocks may include blocks selected from neighboring blocks in the same frame and / or temporal blocks (e.g., blocks at the same location in frames before and after the current frame). These options represent blocks at a spatial or temporal location relative to the current block that are likely to have a similar or identical motion vector to the current block. The size of the MV prediction candidate list may be predetermined. For example, the list may include two candidates. To be included in the list of merge candidates, for example, a candidate block must have the same reference frame(s) as the current block, must exist (e.g., a boundary check must be performed if the current block is near the edge of a frame), must have been previously encoded during the encoding process, and / or must have been previously decoded during the decoding process. In some implementations, the list of merge candidates may be populated first with spatially adjacent blocks (scanned in a certain predetermined order) if they are available and satisfy the above conditions, followed by the primary block if space is still available in the list. The adjacent candidate blocks may be selected, for example, from the blocks to the left and above the current block. The list of merge MV prediction candidates may be signaled in the bitstream.

[0151] In some implementations, the actual merge candidate used as the reference motion vector for predicting the motion vector of the current block may be signaled. If the merge candidate list contains two candidates, a one-bit flag called a merge candidate flag may be used to indicate the selection of the reference merge candidate. For a current block predicted in combined mode, each of the multiple motion vectors predicted using the MV predictor may be associated with a reference motion vector from the merge candidate list.

[0152] In some exemplary implementations of MMVD, after a merge candidate is selected and used as a basis for motion vector prediction for a motion vector to be predicted, the encoding system may calculate a motion vector differential (MVD or delta MV representing the difference between the motion vector to be predicted and the motion vector of the reference candidate). Such MVD may include information representing the magnitude and direction of the MV difference, and may be signaled in the bitstream. The motion differential magnitude and motion differential direction may be signaled in various ways.

[0153] In some exemplary implementations of MMVD, a distance index may be used to specify magnitude information of the motion vector difference and indicate one of a set of predefined offsets that represent the difference of a given motion vector from a starting point (reference motion vector). The MV offset according to the signaled index may then be added to either the horizontal or vertical component of the starting (reference) motion vector. Either the horizontal or vertical component of the reference motion vector needs to be offset, which is determined by the exemplary direction information of the MVD. An example of a predetermined relationship between the distance index and the predefined offset is shown in Table 2. [Table 2]

[0154] In some example implementations of MMVD, a direction index may be further signaled and used to represent the direction of the MVD relative to the reference motion vector. In some implementations, the direction may be limited to either horizontal or vertical. An example of a 2-bit direction index is shown in Table 3. In the example of Table 3, the interpretation of the MVD may change according to the information of the start / reference MV. For example, if the start / reference MV corresponds to a uni-predicted block or a bi-predicted block where both reference frame lists point to the same side of the current image (i.e., the POCs of the two reference pictures are both greater than or both less than the POC of the current image), the code in Table 3 may identify the sign (direction) of the MV offset added to the start / reference MV. If the start / reference MV corresponds to a bi-predicted block with two reference pictures on different sides of the current picture (i.e., the POC of one reference picture is larger than the POC of the current picture and the POC of the other reference picture is smaller than the POC of the current picture), and the difference between the reference POC of picture reference list 0 and the current frame is larger than the difference between the reference POC of picture reference list 1 and the current frame, the code in Table 3 may specify the sign of the MV offset added to the reference MV corresponding to the reference picture of picture reference list 0, and the sign of the offset of the MV corresponding to the reference picture of picture reference list 1 may have the opposite value (opposite sign of the offset). Otherwise, when the difference between the reference POC of picture reference list 1 and the current frame is larger than the difference between the reference POC of picture reference list 0 and the current frame, then the code in Table 3 may specify the sign of the MV offset added to the reference MV associated with picture reference list 1, and the sign of the offset to the reference MV associated with picture reference list 0 has the opposite value. [Table 3]

[0155] In some example implementations, the MVD may be scaled according to the POC difference in each direction. If the POC difference in both lists is the same, no scaling is necessary. Otherwise, if the POC difference in reference list 0 is greater than the POC in reference list 1, the MVD for reference list 1 is scaled. If the POC difference in reference list 1 is greater than list 0, the MVD for list 0 is also scaled in the same way. If the starting MV is uni-predicted, the MVD is available or added to the reference MV.

[0156] In some example implementations of MVD coding and signaling for bidirectional mixed prediction, in addition to or instead of separately coding and signaling two MVDs, symmetric MVD coding can be implemented and derived from a single signaled MVD. In such implementations, motion information including reference image indices for both list 0 and list 1 is signaled. However, for example, only the MVD associated with reference list 0 is signaled, and the MVD associated with reference list 1 is not signaled and is derived. Specifically, at the slice level, a flag can be included in the bitstream, referred to as "mvd_l_1_0_flag," indicating whether reference list 1 is not signaled in the bitstream. If this flag is 1, indicating that reference list 1 is equal to 0 (and therefore not signaled), a bidirectional prediction flag, referred to as "BiDirPredFlag," can be set to 0, meaning there is no bidirectional prediction. Otherwise, if mvd_l_1_0_flag is 0, and the closest reference picture in list 0 and the closest reference picture in list 1 form a forward-backward or backward-forward pair of reference pictures, BiDirPredFlag may be set to 1, and the reference pictures in both list 0 and list 1 are short-term reference pictures. Otherwise, BiDirPredFlag is set to 0. When BiDirPredFlag is 1, it indicates that a symmetric mode flag is additionally signaled in the bitstream. When BiDirPredFlag is 1, the decoder can extract the symmetric mode flag from the bitstream. For example, the symmetric mode flag may be signaled at the CU level (optionally) to indicate whether the symmetric MVD coding mode is used for the corresponding CU. When the symmetric mode flag is 1, it indicates the use of symmetric MVD coding mode, and indicates that only the reference image indices for both list 0 and list 1 (referred to as "mvp_l 0_flag" and "mvp_l 1_flag") are signaled in the MVD associated with list 0 (referred to as "MVD0"), and that the other motion vector differential, "MVD1", is derived rather than signaled.For example, MVD1 may be derived as -MVD0. Thus, in an example of a symmetric MVD mode, only one MVD is signaled. Other implementations of MV prediction can use harmonized schemes to implement general merge mode, MMVD, and several other types of MV prediction for both single-reference and mixed-reference MV prediction. Various syntax elements can be used to signal how the MV of the current block is predicted.

[0157] For example, in the case of a single reference mode, the next MV prediction mode may be signaled.

[0158] NEARMV - Use one of the motion vector predictors (MVP) in the list indicated by the Dynamic Reference List (DRL) index directly without MVD.

[0159] NEWMV - Use one of the motion vector predictions (MVPs) in the list signaled by the DRL index as a reference and apply a delta to the MVP (e.g., use MVD).

[0160] GLOBALMV - Use motion vectors based on frame-level global motion parameters.

[0161] Similarly, in a mixed reference inter prediction mode that uses two reference frames corresponding to the two MVs to be predicted, the next MV prediction mode may be signaled.

[0162] NEAR_NEARMV - For each of the two MVs to be predicted, use one of the motion vector predictors (MVPs) in the list signaled by the DRL index without using the MVD.

[0163] NEAR_NEWMV - When predicting the first of two motion vectors, one of the motion vector predictors (MVP) in the list signaled by the DRL index is used as a reference MV without using the MVD. When predicting the second of two motion vectors, one of the motion vector predictors (MVP) in the list signaled by the DRL index is used as a reference MV in combination with an additionally signaled delta MV (MVD).

[0164] NEW_NEARMV - When predicting the second of two motion vectors, one of the motion vector predictors (MVP) in the list signaled by the DRL index is used as the reference MV without the MVD. When predicting the first of two motion vectors, one of the motion vector predictors (MVP) in the list signaled by the DRL index is used as the reference MV in combination with the additionally signaled delta MV (MVD).

[0165] NEW_NEWMV - Use one of the motion vector predictions (MVPs) in the list signaled by the DRL index as a reference MV, in combination with an additionally signaled delta MV, to predict each of the two MVs.

[0166] GLOBAL_GLOBALMV - Use MV from each reference based on frame-level global motion parameters.

[0167] Thus, the term "NEAR" above refers to MV prediction using a reference MV without using MVD as the overall merge mode, while the term "NEW" refers to MV prediction using a reference MV and offsetting with the signaled MVD as in the MMVD mode. In hybrid inter prediction, both the reference-based motion vector and the motion vector delta described above may be globally different or independent between the two references, but they are correlated, and such correlation can be exploited to reduce the amount of information required to signal the two motion vector deltas. In such situations, joint signaling of the two MVDs may be implemented and indicated in the bitstream.

[0168] The dynamic reference list (DRL) mentioned above may be dynamically maintained and used to hold a set of indexed motion vectors that are considered as candidates for motion vector prediction.

[0169] In some implementations, a predefined resolution of the MVD may be allowed. For example, a motion vector precision (or accuracy) of 1 / 8 pixel may be allowed. The aforementioned MVDs for different MV prediction modes may be constructed and signaled in different ways. In some implementations, different syntax elements may be used to signal the above motion vector differential(s) for reference frame list 0 or list 1.

[0170] For example, a syntax element called "mv_joint" can specify which components of its associated motion vector differential are non-zero. In the case of MVD, this is signaled jointly for all non-zero components. For example, the value of mv_joint 0 can indicate that there is no non-zero MVD along either the horizontal or vertical direction; 1 can show that there exists a non-zero MVD only along the horizontal direction; 2 can be shown that there exists a non-zero MVD only in the vertical direction; 3 can be shown to have a non-zero MVD along both the horizontal and vertical directions.

[0171] If the "mv_joint" syntax element of the MVD signals the absence of non-zero MVD components, no further MVD information may be signaled. However, if the "mv_joint" syntax signals the presence of one or two non-zero components, additional syntax elements may be further signaled for each non-zero MVD component, as shown below.

[0172] For example, a syntax element called "mv_sign" can be used to additionally specify whether the component of the corresponding motion vector differential is positive or negative.

[0173] As another example, a syntax element called "mv_class" can be used to specify the class of motion vector differences between a predetermined set of classes of corresponding non-zero MVD components. The predetermined class of motion vector differences can be used, for example, to separate a continuous magnitude space of motion vector differences into non-overlapping ranges, each corresponding to an MVD class. Thus, the signaled MVD class indicates a range of magnitudes of the corresponding MVD component. In the exemplary implementation shown in Table 4, a higher class corresponds to a motion vector difference having a larger magnitude range. In Table 4, the symbol (n,m) is used to represent a range of motion vector differences greater than n pixels and less than or equal to m pixels. [Table 4]

[0174] In some other examples, a syntax element called "mv_bit" may be further used to specify the integer portion of the offset between a non-zero motion vector differential component and the starting magnitude of the corresponding signaled MV class. The number of bits required in "my_bit" to signal the full range of each MVD class may vary as a function of the MV class. For example, MV_CLASS0 and MV_CLASS1 in the implementation of Table 4 may require only one bit to indicate an integer pixel offset of 1 or 2 from the start of the MVD of 0. Each higher MV_CLASS may require one more bit in "mv_bit" than the preceding MV_CLASS.

[0175] In some other embodiments, a syntax element called "mv_fr" may be further used to specify the first two fractional bits of the motion vector differential for the corresponding non-zero MVD component, while a syntax element called "mv_hp" may be used to specify the third fractional bit (high-resolution bit) of the motion vector differential for the corresponding non-zero MVD component. The two bits of "mv_fr" may essentially provide a quarter-pixel MVD resolution, and the "mv_hp" bits may provide an additional eighth-pixel resolution. In some other implementations, more than one "mv_hp" bit may be used to provide an MVD pixel resolution finer than one-eighth pixel. In some exemplary implementations, additional flags may be signaled at one or more levels to indicate whether an MVD resolution of one-eighth pixel or finer is supported. If an MVD resolution is not applied to a particular coding unit, the above syntax elements for the corresponding unsupported MVD resolution may not be signaled.

[0176] In some example implementations above, the fractional resolution may be independent of different MVD classes. In other words, a predefined number of "mv_fr" and "mv_hp" bits may be used to signal fractional MVD of non-zero MVD components, regardless of the magnitude of the motion vector differential, to provide similar options for motion vector resolution.

[0177] However, in some other implementations, the resolution of motion vector differences for various MVD magnitude classes may be differentiated. Specifically, a high-resolution MVD with a large MVD magnitude for a higher MVD class may not provide a statistically significant improvement in compression efficiency. Therefore, the MVD may be coded by reducing the resolution (integer pixel resolution or fractional pixel resolution) of a larger MVD magnitude range corresponding to the higher MVD magnitude class. Similarly, the MVD may generally be coded by reducing the resolution (integer pixel resolution or fractional pixel resolution) of larger MVD values. Such MVD class-dependent or MVD magnitude-dependent MVD resolution may generally be referred to as adaptive MVD resolution. Adaptive MVD resolution may be implemented in various ways, as described in the following implementation examples, to achieve better overall compression efficiency. In particular, the reduction in the number of signal bits by aiming for a lower-precision MVD may be greater than the additional bits required for the inter-prediction residual as a result of such a lower-precision MVD. This is due to the statistical observation that treating the MVD resolution of large magnitude or high class MVDs similar to that of low magnitude or low class MVDs in a non-adaptive manner may not significantly improve the residual coding efficiency of inter prediction.

[0178] In some general example implementations, the pixel resolution or precision of the MVD may decrease or not increase with increasing MVD class. Decreasing the pixel resolution of the MVD corresponds to a coarser MVD (or a larger step from one MVD level to the next). In some implementations, the correspondence between MVD pixel resolution and MVD class may be specified, predefined, or preset, and may not need to be signaled in the encoded bitstream.

[0179] In some implementations, each MV class in Table 3 may be associated with a different MVD pixel resolution.

[0180] In some example implementations, each MVD class may be associated with a single allowed resolution. In other implementations, one or more MVD classes may be associated with two or more optional MVD pixel resolutions. Thus, the signaling in the bitstream for such an MVD class may be followed by additional signaling indicating which optional pixel resolution has been selected for the current MVD component.

[0181] In some example implementations, adaptively allowed MVD pixel resolutions may include, but are not limited to, 1 / 64 pel (pixel), 1 / 132 pel, 1 / 16 pel, 1 / 8 pel, 1-4 pel, 1 / 2-pel, 1-pel, 2-pel, 4-pel,... (in descending order of resolution). Therefore, each ascending MVD class may be associated with one of these resolutions in a non-ascending manner. In some implementations, an MVD class may be associated with two or more of the above resolutions, with the higher resolution being equal to or less than the lower resolution of the preceding MVD class. For example, if MV_CLASS_3 in Table 4 is associated with optional 1-pel and 2-pel resolutions, then the highest resolution to which MV_CLASS_4 in Table 4 may be associated is 2-pel. In other implementations, the maximum allowed resolution of an MV class may be higher than the minimum allowed resolution of the preceding (lower) MV class. However, the average allowed resolutions of ascending MV classes may only be in non-ascending order.

[0182] In some implementations, if fractional pixel resolution higher than 1 / 8 pel is allowed, the "mv_fr" and "mv_hp" signaling may be correspondingly extended to a total of three or more fractional bits.

[0183] In some example implementations, fractional pixel resolution may be allowed only for MVD classes equal to or less than the threshold MVD class. For example, fractional pixel resolution may be allowed only for MVD-CLASS 0 and not for all other MV classes in Table 4. Similarly, fractional pixel resolution may be allowed only for MVD classes equal to or less than any one of the other MV classes in Table 4. For other MVD classes above the threshold MVD class, only integer pixel resolution for MVD is allowed. In this way, fractional resolution signaling, such as one or more of the "mv-fr" and / or "mv-hp" bits, may not need to be signaled for MVDs signaled in MVD classes equal to or greater than the threshold MVD class. For MVD classes with resolutions less than one pixel, the number of bits for "mv-bit" signaling may be further reduced. For example, for MV_CLASS_5 in Table 4, the MVD pixel offset range is (32,64], so 5 bits are required to signal the entire range at 1pel resolution. However, if MV_CLASS_5 is associated with a 2pel MVD resolution, 4 bits instead of 5 bits may be required for "mv-bit", and neither "mv-fr" nor "mv-hp" need to be signaled following the signaling of "mv_class" as MV-CLASS_5.

[0184] In some example implementations, fractional pixel resolution may be allowed only for MVDs whose integer values ​​are below a threshold integer pixel value. For example, fractional pixel resolution may be allowed only for MVDs less than 5 pixels. Corresponding to this example, fractional resolution may be allowed for MV_CLASS_0 and MV_CLASS_1 in Table 4, but not for all other MV classes. As another example, fractional pixel resolution may be allowed only for MVDs less than 7 pixels. Corresponding to this example, fractional resolution may be allowed for MV_CLASS_0 and MV_CLASS_1 in Table 4 (ranges less than 5 pixels), but not for MV_CLASS_3 and above (ranges 5 pixels and above). For MVDs belonging to MV_CLASS_2, whose pixel range includes 5 pixels, fractional pixel resolution for the MVD may or may not be allowed depending on the "m-bit" value. If the "m-bit" value is signaled as 1 or 2 (calculated as the start of the pixel range for MV_CLASS_2 with an offset of 1 or 2 indicated by "m-bit", such that the integer part of the signaled MVD is 5 or 6), fractional pixel resolution may be allowed. Otherwise, if the "m-bit" value is signaled as 3 or 4 (such that the integer part of the signaled MVD is 7 or 8), fractional pixel resolution may not be allowed.

[0185] In some other implementations, only one MVD value may be allowed for MV classes equal to or greater than a threshold MV class. For example, such a threshold MV class may be MV_CLASS 2. Thus, MV_CLASS_2 and above may only be allowed to have a single MVD value, and fractional pixel resolution may not be available. The single MVD value allowed for these MV classes may be predefined. In some examples, the single allowed value may be the upper limit of the range for each of these MV classes in Table 4. For example, MV_CLASS_2 through MV_CLASS_10 may be equal to or greater than threshold class_2, and the single allowed MVD values ​​for these classes may be predefined as 8, 16, 32, 64, 128, 256, 512, 1024, and 2048, respectively. In some other examples, the single allowed value may be the median of the range for each of these MV classes in Table 4. For example, MV_CLASS_2 through MV_CLASS_10 may exceed the class threshold, and the single allowed MVD values ​​for these classes may be predefined as 3, 6, 12, 24, 48, 96, 192, 384, 768, and 1536, respectively. Other values ​​within the range can also be defined as the single allowed resolution for each MVD class.

[0186] In the above implementation, if the signaled 'mv_class' is equal to or greater than a predetermined MVD class threshold, the 'mv_class' signaling alone is sufficient to determine the MVD value. The magnitude and direction of the MVD are then determined using 'mv_class' and 'mv_sign'.

[0187] When MVD is signaled for only one reference frame (either from reference frame list 0 or list 1, but not both), or when it is signaled for two reference frames jointly, the accuracy (or resolution) of the MVD may depend on the class of the associated motion vector difference in Table 3 and / or the magnitude of the MVD.

[0188] In some other implementations, the pixel resolution or precision of the MVD may decrease or not increase with increasing MVD magnitude. For example, the pixel resolution may depend on the integer portion of the MVD magnitude. In some implementations, fractional pixel resolution may be allowed only for MVD magnitudes below an amplitude threshold. For a decoder, the integer portion of the MVD magnitude may first be extracted from the bitstream. The pixel resolution may then be determined to determine whether any fractional MVD is present in the bitstream and needs to be parsed. (For example, if fractional pixel resolution is not allowed for a particular extracted MVD integer magnitude, the bitstream may not contain fractional MVD bits that require extraction.) The implementation examples above related to adaptive MVD pixel resolution depending on the MVD class apply to adaptive MVD pixel resolution depending on the MVD magnitude. In certain examples, MVD classes that exceed or include a magnitude threshold may be allowed to have only one predefined value.

[0189] The various example implementations described above apply to single-reference modes. These implementations also apply to the NEW_NEARMV, NEAR_NEWMV, and / or NEW_NEWMV modes of MMVD joint prediction. These implementations also apply to MVD signaling in general.

[0190] FIG. 18 shows a flowchart 1800 of an example method that follows the principles underlying the above implementation for adaptive MVD resolution. The flow of the example decoding method starts at 1801. At S1810, a video stream is received and it is determined that a motion vector difference (MVD) between a reference motion vector and a motion vector associated with an inter-predicted video block is signaled within the video stream, where the reference motion vector corresponds to only one reference picture from Reference Frame List 0 and Reference Frame List 1 unless MVD is jointly signaled for two reference pictures. At S1820, an indication of an MVD magnitude range among a plurality of predetermined magnitude ranges of the motion vector difference is obtained from the video stream. At S1830, a pixel resolution of the MVD is determined according to the magnitude range. At S1840, additional MVD information is identified within the video stream based on the pixel resolution. At S1850, the additional MVD information is extracted from the video stream. At S1860, the inter-predicted video block is decoded based on the pixel resolution, the additional MVD information, the reference motion vector, and the reference frame associated with the motion vector. Example method flow ends at S1899.

[0191] FIG. 19 shows another flowchart 1900 of an example method that follows the principles underlying the above implementation for adaptive MVD resolution. The example method flow begins at 1901. At S1910, a video stream is received and it is determined that a motion vector difference (MVD) between a reference motion vector and a motion vector associated with an inter-predicted video block is signaled within the video stream, where the reference motion vector corresponds to only one reference picture from Reference Frame List 0 and Reference Frame List 1 unless the MVD is jointly signaled for two reference pictures. At S1920, an integer portion of the MVD magnitude is extracted from the video stream. At S1930, a pixel resolution for the MVD is determined according to the integer portion of the MVD magnitude. At S1940, additional MVD information is identified within the video stream based on the pixel resolution. At S1950, the inter-predicted video block is decoded based on the pixel resolution, the integer portion of the MVD magnitude, the additional MVD information, the reference motion vector, and the reference frame associated with the motion vector. The example method flow ends at S1999.

[0192] In the embodiments and implementations of the present disclosure, any steps and / or operations may be combined or arranged in any quantity or order as needed. Two or more steps and / or operations may be performed in parallel. The disclosed embodiments and implementations may be used separately or combined in any order. Furthermore, each of the method (or embodiment), encoder, and decoder may be implemented by a processing circuit (e.g., one or more processors or one or more integrated circuits). In one example, the one or more processors execute a program stored on a non-transitory computer-readable medium. The disclosed embodiments may apply to luma blocks or chroma blocks. Hereinafter, the term "block" may be interpreted as a prediction block, a coding block, or a coding unit (CU). The term "block" herein may also be used to refer to a transform block. In the following sections, references to block size may refer to the width or height of the block, the maximum value of the width and height, the minimum value of the width and height, the area size (width * height), or the aspect ratio of the block (width:height, height:width).

[0193] The techniques described above can be implemented as computer software using computer-readable instructions and physically stored on one or more computer-readable media. For example, Figure 20 illustrates a computer system (2000) suitable for implementing certain embodiments of the disclosed subject matter.

[0194] Computer software can be coded using any suitable machine code or computer language that can be subject to assembly, compilation, linking, or similar mechanisms to create code including instructions that can be executed directly or via implementation, microcode execution, etc., by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc.

[0195] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smart phones, gaming devices, internet of things devices, and the like.

[0196] The components illustrated in Figure 20 for the computer system (2000) are exemplary in nature and are not intended to suggest any limitation on the scope of use or functionality of the computer software implementing the embodiments of the present disclosure, nor should the arrangement of components be interpreted as having any dependency or requirement relating to any one or combination of components illustrated in the exemplary embodiment of the computer system (2000).

[0197] The computer system 2000 may include certain human interface input devices that may respond to input by one or more human users, for example, via tactile input (e.g., keystrokes, flipping, data glove movements), audio input (e.g., voice, clapping), visual input (e.g., gestures), or olfactory input (not shown). Human interface devices may also be used to capture certain media that do not necessarily involve direct conscious human input, such as audio (e.g., voice, music, ambient sounds), images (e.g., scanned images, photographic images obtained from still image cameras), and video (e.g., two-dimensional video, three-dimensional video, including stereoscopic images).

[0198] The input human interface devices may include one or more of the following (only one of each is shown): a keyboard (2001), a mouse (2002), a trackpad (2003), a touchscreen (2010), a data glove (not shown), a joystick (2005), a microphone (2006), a scanner (2007), and a camera (2008).

[0199] The computer system (2000) may also include certain human interface output devices. Such human interface output devices may stimulate one or more of the human user's senses, for example, through tactile output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touchscreen (2010), data gloves (not shown), or joystick (2005)), which may also be haptic feedback devices that do not serve as input devices), audio output devices (e.g., speakers (2009), headphones (not shown)), visual output devices (e.g., screens (2010), including CRT screens, LCD screens, plasma screens, and OLED screens, each with or without haptic feedback capabilities, some of which may enable two-dimensional visual output or three-dimensional or higher output via means such as virtual reality glasses (not shown), holographic displays, and stereoscopic output such as smoke tanks (not shown)), and printers (not shown).

[0200] The computer system (2000) may also include human-accessible storage devices and their accessible media, such as an optical media drive (2020) including a CD / DVD ROM / RW with media 921 such as CD / DVD, a USB memory (2022), a removable head drive or solid state drive (2023), conventional magnetic media such as tape, floppy disks (not shown), specialized ROM / ASIC / PLD-based devices such as security dongles, etc.

[0201] Those skilled in the art will also understand that the term "computer-readable medium" as used in connection with the presently disclosed subject matter does not encompass transmission media, carrier waves, or other transitory signals.

[0202] The computer system 2000 may also include an interface 2054 to one or more communications networks 2055. Networks may be, for example, wireless, wired, or optical. Networks may further be local, wide-area, metropolitan, vehicular, industrial, real-time, delay-tolerant, etc. Examples of networks include local area networks such as Ethernet and wireless LAN; cellular networks including GSM, 3G, 4G, 5G, LTE, etc.; wired or TV-wireless wide-area digital networks including cable TV, satellite TV, and terrestrial broadcast TV; and industrial and vehicular networks including CAN Bus. Certain networks generally require an external network interface adapter connected to a particular general-purpose data port or peripheral bus 2049 (e.g., a USB port on the computer system 2000); others are generally integrated into the core of the computer system 2000 by connecting to a system bus, as described below (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system (2000) can communicate with other entities. Such communication can be unidirectional, receive-only (e.g., broadcast television), unidirectional transmit-only (e.g., a CAN bus to a specific CAN bus device), or bidirectional, e.g., to other computer systems using local or wide-area digital networks. Specific protocols and protocol stacks, as described above, can be used with each of these networks and network interfaces.

[0203] The aforementioned human interface devices, human-accessible storage devices, and network interfaces can be connected to the core (2040) of the computer system (2000).

[0204] The cores (2040) may include one or more central processing units (CPUs) (2041), graphics processing units (GPUs) (2042), specialized programmable processing devices in the form of field programmable gate arrays (FPGAs) (2043), hardware accelerators for specific tasks (2044), graphics adapters (2050), etc. These devices may be connected via a system bus (2048), along with read-only memory (ROM) (2045), random access memory (2046), and internal mass storage devices such as internal non-user-accessible hard drives, SSDs, etc. (2047). In some computer systems, the system bus (2048) is accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may be connected directly to the core's system bus (2048) or via a peripheral bus (2049). In some embodiments, a screen (2010) may be connected to the graphics adapter (2050). Peripheral bus architectures include PCI, USB, and the like.

[0205] The CPU (2041), GPU (2042), FPGA (2043), and accelerator (2044) can combine to execute specific instructions that may constitute the aforementioned computer code. The computer code may be stored in ROM (2045) or RAM (2046). Transient data may be stored in RAM (2046), while persistent data may be stored, for example, in an internal mass storage device (2047). High-speed storage and retrieval from any of the memory devices may be enabled through the use of cache memory, which may be closely associated with one or more of the CPU (2041), GPU (2042), mass storage device (2047), ROM (2045), RAM (2046), etc.

[0206] The computer-readable medium can have computer code thereon for performing various computer-implemented operations. The medium and computer code can be those specially designed and created for the present disclosure, or they can be of the type well known and available in the art of computer software technology.

[0207] As a non-limiting example, a computer system having the architecture (200), and specifically the core (2040), can provide functionality as a result of a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media can be media associated with a user-accessible mass storage device, as described above, as well as specific storage devices of the core (2040) that are non-transitory in nature, such as the core-internal mass storage device (2047) or ROM (2045). Software implementing various embodiments of the present disclosure can be stored in such devices and executed by the core (2040). The computer-readable media can include one or more memory devices or chips, depending on particular needs. The software can cause the core (2040), and specifically the processor (including a CPU, GPU, FPGA, etc.) therein, to perform specific processes or portions thereof described herein, including defining data structures stored in RAM (2046) and modifying such data structures according to processes defined by the software. Additionally or alternatively, the computer system may provide functionality as a result of logic hardwired or otherwise embodied in circuitry (e.g., accelerator (2044)), which may operate in place of or in conjunction with software to perform particular processes or portions of particular processes described herein. Reference to software includes logic, and vice versa, where appropriate. Reference to a computer-readable medium may include circuitry (such as an integrated circuit (IC)) that stores software for execution, circuitry that embodies logic for execution, or both, as appropriate. The present disclosure encompasses any suitable combination of hardware and software.

[0208] While this disclosure describes several exemplary embodiments, there are alterations, permutations, and various equivalents that fall within the scope of this invention. It will thus be appreciated that those skilled in the art will be able to create numerous systems and methods that, although not explicitly shown or described herein, embody the principles of the invention and are therefore within its concept and scope.

[0209] Appendix A: acronym JEM: Joint Search Model VVC: Universal Video Coding BMS: Benchmark Set MV: Motion Vector HEVC: High Efficiency Video Coding SEI: Additional reinforcement information VUI: Video Usability Information GOP: Group of Pictures TU: Conversion unit PU: Prediction Unit CTU: Coding Tree Unit CTB: coding tree block PB: Predicted Block HRD: Hypothetical Reference Decoder SNR: Signal to Noise Ratio CPU: Central Processing Unit GPU: Graphics Processing Unit CRT: cathode ray tube LCD: Liquid crystal display OLED: Organic Light Emitting Diode CD: Compact Disc DVD: Digital Video Disc ROM: Read-only memory RAM: Random Access Memory ASIC: Application Specific Integrated Circuit PLD: Programmable Logic Device LAN: Local Area Network GSM: Global System for Mobile Communications LTE: Long Term Evolution CAN bus: Controller Area Network bus USB: Universal Serial Bus PCI: Peripheral Component Interconnect FPGA: Field Programmable Gate Area SSD: Solid State Drive IC: Integrated Circuit HDR: High Dynamic Range SDR: Standard Dynamics Range JVET:Joint Video Exploration Team MPM: Most probable mode WAIP: Wide-Angle Intra Prediction CU: Coding Unit PU: Prediction Unit TU: Conversion unit CTU: Coding Tree Unit PDPC: Position-dependent prediction combination ISP: Intra-sub prediction SPS: Sequence Parameter Setting PPS: Image parameter settings APS: Adaptive Parameter Setting VPS: Video Parameter Settings DPS: Decode parameter settings ALF: Adaptive Loop Filter SAO:Sample Adaptation Offset CC-ALF: Cross-component adaptive loop filter CDEF: Constraint Directivity Enhancement Filter CCSO: Cross-Component Sample Offset LSO: Local Sample Offset LR: Loop Recovery Filter AV1:AO Media Video 1 AV2:AO Media Video 2 MVD: Motion Vector Difference CfL: Chroma from Luma SDT: Semi-Decoupled Tree SDP: Semi-Decoupled Partitioning SST: Semi-separate tree SB: Super Block IBC (or IntraBC): Intrablock copy CDF: cumulative density function SCC: Screen Content Coding GBI:GBI forecast BCW: Bi-prediction with CU-level weighting CIIP: Combined Intra-Inter Prediction POC: Image Sequence Count RPS: Reference Image Settings DPB: Decoded Picture Buffer MMVD: Merge Mode with Motion Vector Differentials

Claims

[Claim 1] 1. A method for decoding inter-predicted video blocks of a video stream, performed by a decoder, comprising: receiving the video stream; determining that a motion vector differential (MVD) between a reference motion vector associated with the inter-predicted video block and a motion vector is signaled in a video stream, the reference motion vector corresponding to a reference picture in only one of Reference Frame List 0 and Reference Frame List 1, unless MVD is jointly signaled for two reference pictures; obtaining from the video stream an indication of a magnitude range of the MVD among a plurality of predetermined magnitude ranges for the motion vector differential; determining a pixel resolution for the MVD according to the magnitude range; identifying additional MVD information within the video stream based on the pixel resolution; extracting the additional MVD information from the video stream; decoding the inter-predicted video block based on a reference frame associated with the motion vector, the reference motion vector, the additional MVD information, and the pixel resolution; A method comprising:

Images