Schemes for adjusting adaptive resolution for motion vector difference

Abstract

To provide methods and systems for adjusting adaptive resolution for motion vector difference.SOLUTION: A method includes the steps of: receiving a video stream; in response to determining that a current video block is inter-coded based on a prediction block and a motion vector, deriving the motion vector from a reference motion vector and a motion vector difference (MVD) for the current video block; in response to determining that the MVD is coded with an adaptive MVD pixel resolution, determining a reference MVD pixel precision for the current video block; identifying a maximum allowed MVD pixel precision; determining a set of allowable MVD levels for the current video block based on the reference MVD pixel precision and the maximum allowed MVD pixel precision; and deriving the MVD from the video stream according to at least one MVD parameter signaled in the video stream.SELECTED DRAWING: Figure 18

Description

Technical Field

[0001] Cross - Reference to Related Applications This application claims the benefit of priority under 35 U.S.C. § 119(e) to U.S. Provisional Application No. 63 / 302,518, filed on January 24, 2022, entitled "Further Improvement for Adaptive MVD Resolution", and to U.S. Non - Provisional Application No. 17 / 824,193, filed on May 25, 2022, entitled "Schemes for Adjusting Adaptive Resolution for Motion Vector Difference". These prior applications are hereby incorporated by reference in their entirety. This application claims the benefit of priority under 35 U.S.C. § 119(e) to U.S. Provisional Application No. 63 / 302,518, filed on January 24, 2022, entitled "Further Improvement for Adaptive MVD Resolution", and to U.S. Non - Provisional Application No. 17 / 824,193, filed on May 25, 2022, entitled "Schemes for Adjusting Adaptive Resolution for Motion Vector Difference". These prior applications are hereby incorporated by reference in their entirety.

[0002] The present disclosure generally relates to video coding, and more particularly, to methods and systems for providing schemes for setting different values of acceptable motion vectors when implementing adaptive resolution for motion vector differences.

Background Art

[0003] The description of the background art provided herein is for the purpose of generally presenting the context of the present disclosure. The inventors' research is not admitted as prior art to the present disclosure, either expressly or implicitly, to the extent that the research is described in this background art section and in aspects of the description that may not be regarded as prior art at the time of filing of the present application.

[0004] Video coding and decoding can be performed using inter-picture prediction with motion compensation. Uncompressed digital video can include a series of pictures, each picture having, for example, a spatial dimension of 1920×1080 luminance samples and associated full-sampled or subsampled chrominance samples. The series of pictures can have, for example, a fixed or variable picture rate (or frame rate, also called) of 60 pictures per second or 60 frames per second. Uncompressed video has specific bitrate requirements for streaming or data processing. For example, video having a pixel resolution of 1920×1080, a frame rate of 60 frames / second, and 4:2:0 chroma subsampling with 8 bits per pixel per color channel requires a bandwidth close to 1.5 Gbit / s. One hour of such video requires storage space exceeding 600 GByte.

[0005] One purpose of video coding and decoding can be the reduction of redundancy of the uncompressed input video signal by compression. Compression can help reduce the aforementioned bandwidth and / or storage space requirements, in some cases by more than two orders of magnitude. Both reversible compression and irreversible compression, as well as combinations thereof, can be employed. Reversible compression refers to a technique in which an exact copy of the original signal can be reconstructed from the compressed original signal via a decoding process. Irreversible compression refers to a coding / decoding process in which the original video information is not completely retained during coding and cannot be completely restored during decoding. When using irreversible compression, the reconstructed signal may not be the same as the original signal, but the distortion between the original signal and the reconstructed signal is small enough to be useful for the intended purpose of the reconstructed signal, even with some information loss. In the case of video, irreversible compression is widely adopted for many applications. The amount of tolerable distortion depends on the application. For example, users of certain consumer video streaming applications may tolerate higher distortion than users of movie or television broadcast applications. The compression ratio achievable by a particular coding algorithm can be selected or adjusted to reflect various distortion tolerances. That is, generally, the higher the distortion tolerance, the more possible it is to use a coding algorithm that results in high loss and a high compression ratio.

[0006] Video encoders and decoders can utilize techniques from several broad categories and steps, including, for example, motion compensation, Fourier transform, quantization, and entropy coding.

[0007] Video codec technology can include techniques known as intra coding. In intra coding, sample values are represented without reference to samples or other data from previously reconstructed reference pictures. In some video codecs, a picture is spatially subdivided into blocks of samples. If all blocks of samples are coded in an intra mode, that picture can be called an intra picture. Those derived pictures such as intra pictures and independent decoder refresh pictures can be used to reset the decoder state and thus can be used as the first picture in a coded video bitstream and video session or as a still image. Next, the samples of the block after intra prediction can be transformed to the frequency domain, and the transform coefficients so generated can be quantized before entropy coding. Intra prediction represents a technique for minimizing the sample values in the pre-transform region. In some cases, the smaller the DC value after transformation and the smaller the AC coefficients, the fewer bits are required at a given quantization step size to represent the block after entropy coding.

[0008] For example, conventional intra coding, such as known from MPEG-2 production coding technology, does not use intra prediction. However, some newer video compression technologies include techniques that attempt to code / decode blocks based on surrounding sample data and / or metadata that precede in decoding order the blocks of data that are intra-coded or intra-decoded and that are obtained, for example, during spatial adjacent encoding and / or decoding. Such techniques are hereinafter called "intra prediction" techniques. Note that in at least some cases, intra prediction uses reference data only from the current picture being reconstructed and does not use reference data from other reference pictures.

[0009] Intra prediction can have many different forms. If two or more of such techniques are available in a given video coding technology, the techniques used can be referred to as intra prediction modes. One or more intra prediction modes can be provided by a particular codec. In certain cases, a mode can have sub-modes and / or can be associated with various parameters. The mode / sub-mode information and the intra coding parameters of a video block can be coded individually or can be included together in the codeword of the mode. Which codeword to use for a given combination of mode, sub-mode, and / or parameters can affect the coding efficiency improvement via intra prediction and thus can also affect the entropy coding technology used to convert the codeword into the bitstream.

[0010] Certain modes of intra prediction were introduced in H.264, improved in H.265, and further improved in more recent coding technologies such as the Joint Exploration Model (JEM), Versatile Video Coding (VVC), and Benchmark Set (BMS). Generally, in intra prediction, a predictor block can be formed using the available adjacent sample values. For example, the available values of a particular set of adjacent samples along a particular direction and / or line can be copied into the predictor block. The reference to the direction in use can be coded in the bitstream or can itself be predicted.

[0011] Referring to FIG. 1A, shown in the lower right is a subset of nine predictor directions specified in 33 possible intra predictor directions of H.265 (corresponding to 33 of the 35 intra modes specified in H.265). The point (101) where the arrows converge represents the predicted sample. The arrows represent the direction in which adjacent samples are used therefrom to predict the sample at 101. For example, arrow (102) indicates that sample (101) is predicted from one or more adjacent samples at a 45-degree angle from the horizontal direction and to the upper right. Similarly, arrow (103) indicates that sample (101) is predicted from one or more adjacent samples at a 22.5-degree angle from the horizontal direction and to the lower left of sample (101).

[0012] Referring further to FIG. 1A, shown in the upper left is a square block (104) of 4×4 samples (indicated by the thick dashed line). The square block (104) contains 16 samples, each labeled with "S", its position in the Y dimension (e.g., row index), and its position in the X dimension (e.g., column index). For example, sample S21 is the second sample from the top in the Y dimension and the first sample from the left in the X dimension. Similarly, sample S44 is the fourth sample in both the Y and X dimensions within block (104). Since the block is 4×4 samples in size, S44 is in the lower right. Exemplary reference samples following a similar numbering scheme are further shown. The reference samples are labeled with R, its Y position (e.g., row index), and its X position (column index) with respect to block (104). In both H.264 and H.265, predicted samples adjacent to the block being reconstructed are used.

[0013] The intra-picture prediction of block 104 may start by copying the reference sample value from adjacent samples according to the signaled prediction direction. For example, the coded video bitstream may include signaling for this block 104 indicating the prediction direction of arrow (102), i.e., it is assumed that the samples are predicted from one or more prediction samples at an angle of 45 degrees from the horizontal direction towards the upper right. In such a case, samples S41, S32, S23, and S14 are predicted from the same reference sample R05. Then, sample S44 is predicted from reference sample R08.

[0014] In certain cases, in order to calculate the reference sample, especially when the direction is not evenly divisible by 45 degrees, the values of multiple reference samples may be combined, for example, by interpolation.

[0015] The number of possible directions has been increasing as video coding technology continues to evolve. In H.264 (2003), for example, 9 different directions were available for intra prediction. This increased to 33 in H.265 (2013), and JEM / VVC / BMS can support up to 65 directions at the time of this disclosure. Experimental studies have been conducted to help identify the most appropriate intra prediction direction, and using certain techniques of entropy coding, specific bit penalties for the directions can be accepted so that their most appropriate directions can be encoded with fewer bits. Additionally, the direction itself may sometimes be predicted from the adjacent directions used in the intra prediction of the decoded adjacent blocks.

[0016] FIG. 1B shows a schematic diagram (180) showing 65 intra prediction directions by JEM to illustrate the increasing number of prediction directions in various coding technologies that have evolved over time.

[0017] Methods for mapping bits representing the intra prediction direction in a coded video bitstream to a prediction direction can vary depending on the video coding technology and can range from a simple direct mapping of prediction direction to intra prediction mode, to complex adaptive schemes including codewords, most probable modes, and similar techniques. However, in all cases, there may be certain directions of intra prediction that are statistically less likely to occur in the video content than certain other directions. Since the goal of video compression is redundancy reduction, in well-designed video coding technologies, those less likely directions may be represented with more bits than the more likely directions.

[0018] Interpicture prediction, or inter prediction, may be based on motion compensation. In motion compensation, sample data from a previously reconstructed picture or a portion thereof (reference picture) is spatially shifted in the direction indicated by a motion vector (hereinafter, MV) and then used for prediction of a newly reconstructed picture or picture portion (e.g., block). In some cases, the reference picture may be the same as the picture currently being reconstructed. The MV may have two dimensions X and Y, or three dimensions, where the third dimension is an indication of the reference picture in use (similar to the temporal dimension).

[0019] In some video compression techniques, the current MV applicable to a particular area of sample data can be predicted from other MVs, for example, other MVs related to other areas of sample data that are spatially adjacent to the area being reconstructed and precede the current MV in the decoding order. By doing so, the overall data amount required to code the MV can be significantly reduced by relying on the removal of redundancy of correlated MVs, thereby increasing the compression efficiency. MV prediction can function effectively, for example, when coding an input video signal derived from a camera (known as natural video), because areas larger than the area to which a single MV is applicable have a statistical likelihood of moving in the same direction in the video sequence, and thus, in some cases, can be predicted using a similar motion vector derived from the MVs of adjacent areas. As a result, the actual MV of a given area becomes similar or identical to the predicted MV from surrounding MVs. Such an MV can further be represented with fewer bits than the number of bits that would be used if the MV were coded directly rather than predicted from adjacent MVs after entropy coding. In some cases, MV prediction can be an example of lossless compression of a signal (i.e., an MV) derived from the original signal (i.e., a sample stream). In other cases, MV prediction itself can be lossy, for example, due to rounding errors when calculating predictors from several surrounding MVs.

[0020] Various MV prediction mechanisms are described in H.265 / HEVC (ITU-T Recommendation H.265, "High Efficiency Video Coding", December 2016). Among the many MV prediction mechanisms specified by H.265, the one described below is a technique hereinafter referred to as "spatial merge".

[0021] Specifically, referring to FIG. 2, the current block (201) contains samples that have been detected by the encoder as being predictable from the previous block of the same size that has been spatially shifted during the motion search process. Instead of directly coding that MV, the MV can be derived from metadata associated with one or more reference pictures, for example, from the latest reference picture (in decoding order), using an MV associated with any one of five surrounding samples denoted as A0, A1, and B0, B1, B2 (202 to 206 respectively). In H.265, the MV prediction can use predictors from the same reference picture that the adjacent blocks are using.

Summary of the Invention

Means for Solving the Problems

[0022] The present disclosure generally relates to video coding, and more particularly, to methods and systems for signaling various motion vectors or motion vector difference-related syntax based on whether a size-dependent adaptive resolution for motion vector differences in inter prediction is used.

[0023] In an exemplary implementation, a method for processing a current video block of a video stream is disclosed. The method may include receiving the video stream and determining that the current video block is to be inter-coded based on a predicted block and a motion vector (MV), where the MV should be derived from a reference motion vector (RMV) and a motion vector difference (MVD) of the current video block. The method further includes determining a reference MVD pixel accuracy of the current video block in response to determining that the MVD is coded at an adaptive MVD pixel resolution, specifying a maximum allowable MVD pixel accuracy, determining a set of allowable MVD levels of the current video block based on the reference MVD pixel accuracy and the maximum allowable MVD pixel accuracy, and deriving the MVD from the video stream according to at least one MVD parameter signaled in the video stream for the current video block and the set of allowable MVD levels.

[0024] In the above implementation, the reference MVD pixel accuracy of the current video block is specified / signaled / derived at the sequence level, picture level, frame level, super-block level, or coding block level.

[0025] In any one of the above implementations, the reference MVD pixel accuracy of the current video block depends on an MVD class associated with the MVD of the current video block.

[0026] In any one of the above implementations, the reference MVD pixel accuracy of the current video block depends on the MVD magnitude of the MVD of the current video block. In any one of the above implementations, the maximum allowable MVD pixel accuracy is predefined.

[0027] In any one of the above implementation forms, the method may further include a step of determining the current MVD class from a pre-defined set of MVD classes. The step of determining a set of acceptable MVD levels of the MVD based on the reference MVD pixel accuracy and the maximum allowable MVD pixel accuracy may include excluding, from the set of reference MVD levels determined based on the reference MVD pixel accuracy and the current MVD class, the MVD levels associated with MVD pixel accuracies greater than or equal to the maximum allowable MVD pixel accuracy, in order to determine the set of acceptable MVD levels of the current video block.

[0028] In any one of the above implementation forms, the maximum allowable MVD pixel accuracy is 1 / 4 pixel. In any one of the above implementation forms, the MVD levels associated with accuracies of 1 / 8 pixel or higher are excluded from the set of acceptable MVD levels of the current video block.

[0029] In any one of the above implementation forms, the method may further include a step of determining the current MVD class from a pre-defined set of MVD classes. The MVD levels associated with fractional MVD accuracy may be included in the set of acceptable MVD levels regardless of the reference MVD accuracy, if the current MVD class is below the threshold MVD class.

[0030] In any one of the above implementation forms, the threshold MVD class may be the lowest MVD class in the pre-defined set of MVD classes.

[0031] In any one of the above implementation forms, the method may further include a step of determining the size of the MVD, and the MVD levels associated with MVD accuracies higher than the threshold MVD accuracy are allowed in the set of acceptable MVD levels only if the size of the MVD is below the threshold MVD size.

[0032] In any one of the above implementation forms, the threshold MVD size is 2 pixels or less. In any one of the above implementation forms, the threshold MVD accuracy is 1 pixel. In any one of the above implementation forms, the MVD level associated with an MVD accuracy of 1 / 4 pixel or more is only allowed when the size of the MVD is 1 / 2 pixel or less. In any one of the above implementation forms, the maximum allowable MVD pixel accuracy may not be greater than the reference MVD pixel accuracy.

[0033] In another exemplary implementation form, a method for processing the current video block of a video stream is disclosed. The method may include the steps of receiving a video stream, determining that the current video block is inter-coded and associated with a plurality of reference frames, and determining, based on the signaling in the video stream, whether an adaptive motion vector difference (MVD) pixel resolution is applied to at least one of the plurality of reference frames.

[0034] In the above implementation form, the signaling may include a single-bit flag for indicating whether the adaptive MVD pixel resolution is applied to all of the plurality of reference frames or not applied to any of them.

[0035] In any one of the above implementation forms, the signaling may include separate flags each corresponding to one of the plurality of reference frames to indicate whether the adaptive MVD pixel resolution is applied.

[0036] In any one of the above implementation forms, the signaling may include an implicit indication that the adaptive MVD pixel resolution is not applied when the MVD corresponding to each of the plurality of reference frames is zero, and a single-bit flag for indicating whether the adaptive MVD pixel resolution is applied when the MVD corresponding to each of the plurality of reference frames is non-zero.

[0037] In other exemplary embodiments, a method for processing a current video block of a video stream is disclosed. The method includes receiving a video stream and determining that the current video block is to be inter-coded based on a predicted block and a motion vector (MV), where the MV is to be derived from a reference motion vector (RMV) and a motion vector difference (MVD) for the current video block; determining a current MVD class of the MVD from a predefined set of MVD classes; deriving at least one context for entropy decoding at least one explicit signaling in the video stream based on the current MVD class, where the at least one explicit signaling is included in the video stream to specify an MVD pixel resolution for at least one component of the MVD; and entropy decoding at least one explicit signaling from the video stream using the at least one context to determine an MVD pixel resolution for at least one component of the MVD.

[0038] In the above embodiments, at least one component of the MVD may include a horizontal component and a vertical component of the MVD, the at least one context may include two separate contexts respectively associated with one of the horizontal component and the vertical component of the MVD, and the horizontal component and the vertical component are associated with separate MVD pixel resolutions.

[0039] Aspects of the present disclosure also provide a video encoding or decoding device or apparatus including a circuit configured to execute any of the above method embodiments.

[0040] Aspects of the present disclosure also provide a non-transitory computer-readable medium storing instructions that cause a computer to perform a method for video decoding and / or encoding when executed by the computer for video decoding and / or encoding.

[0041] Further features, properties, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings.

Brief Description of the Drawings

[0042]

Figure 1A

Figure 1B

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Embodiments for Carrying Out the Invention

[0043] Throughout this specification and the claims, terms may have subtle meanings suggested or implied within the context beyond their explicitly described meanings. The phrases "in one embodiment" or "in some embodiments" as used herein do not necessarily refer to the same embodiment, and the phrases "in other embodiments" or "in other embodiments" as used herein do not necessarily refer to different embodiments. Similarly, the phrases "in one implementation" or "in some implementations" as used herein do not necessarily refer to the same implementation, and the phrases "in other implementations" or "in other implementations" as used herein do not necessarily refer to different implementations. For example, the claimed subject matter is intended to include combinations of all or part of the exemplary embodiments / implementations.

[0044] In general, technical terms may be understood at least in part from their usage in context. For example, terms such as "and", "or", or "and / or" as used herein may include various meanings that may depend at least in part on the context in which such terms are used. Typically, "or" when used to associate a list such as A, B, or C is intended here to mean A, B, and C in an inclusive sense, as well as A, B, or C in an exclusive sense here. In addition, the terms "one or more" or "at least one" as used herein may, depending at least in part on the context, be used to describe any feature, structure, or property in a singular sense, or may be used to describe a combination of features, structures, or properties in a plural sense. Similarly, terms such as "a", "an", or "the" may also be understood, depending at least in part on the context, to convey a singular usage or a plural usage. In addition, the terms "based on" or "determined by" may not necessarily be intended to convey a set of exclusive factors, and instead may, depending at least in part on the context, also allow for the presence of additional factors that are not necessarily explicitly described. FIG. 3 shows a simplified block diagram of a communication system (300) according to an embodiment of the present disclosure. The communication system (300) includes, for example, a plurality of terminal devices that can communicate with each other via a network (350). For example, the communication system (300) includes a first pair of terminal devices (310) and (320) interconnected via a network (350). In the example of FIG. 3, the first pair of terminal devices (310) and (320) can perform unidirectional transmission of data. For example, the terminal device (310) can code video data (e.g., a stream of video pictures captured by the terminal device (310)) for transmission to the other terminal device (320) via the network (350). The encoded video data can be transmitted in the form of one or more encoded video bitstreams.The terminal device (320) may receive the encoded video data from the network (350), decode the encoded video data to restore the video picture, and display the video picture according to the restored video data. The unidirectional data transmission may be performed, for example, for media serving applications.

[0045] In another example, the communication system (300) includes a second pair of terminal devices (330) and (340) that perform a two-way transmission of encoded video data that may occur, for example, during a video conferencing application. For the two-way transmission of data, in one example, each of the terminal devices (330) and (340) may encode video data (e.g., a stream of video pictures captured by that terminal device) for transmission to the other of the terminal devices (330) and (340) via the network (350). Each of the terminal devices (330) and (340) may also receive the encoded video data transmitted by the other of the terminal devices (330) and (340), decode the encoded video data to restore the video picture, and display the video picture on an accessible display device according to the restored video data.

[0046] In the example of FIG. 3, the terminal devices (310), (320), (330), and (340) may be implemented as servers, personal computers, and smartphones, but the application of the principles underlying the present disclosure need not be so limited. Embodiments of the present disclosure may be implemented in desktop computers, laptop computers, tablet computers, media players, wearable computers, dedicated video conferencing devices, and the like. The network (350) represents any number or type of network that transmits coded video data between the terminal devices (310), (320), (330), and (340), including, for example, a wired (wired) and / or wireless communication network. The communication network (350) may exchange data over a circuit-switched channel, a packet-switched channel, and / or other types of channels. Representative networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For the purposes of this description, the architecture and topology of the network (350) may not be important for the operation of the present disclosure unless explicitly described herein.

[0047] FIG. 4 shows the arrangement of a video encoder and a video decoder in a video streaming environment as an example of an application for the disclosed subject matter. The disclosed subject matter may be equally applicable to other video applications, including, for example, video conferencing, digital television broadcasting, gaming, virtual reality, storage of compressed video on digital media including CDs, DVDs, memory sticks, and the like.

[0048] A video streaming system may include, for example, a video source (401) for creating a stream (402) of uncompressed video pictures or images, such as a video capture subsystem (413) that can include, for example, a digital camera. In one example, the stream (402) of video pictures includes samples recorded by the digital camera of the video source 401. The stream (402) of video pictures shown as a thick line to emphasize the large amount of data when compared to the encoded video data (404) (or encoded video bitstream) can be processed by an electronic device (420) that includes a video encoder (403) coupled to the video source (401). The video encoder (403) can include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter, as described in more detail below. The encoded video data (404) (or encoded video bitstream (404)) is shown as a thin line to emphasize the low amount of data when compared to the stream (402) of uncompressed video pictures and can be stored in the streaming server (405) for future use or directly in a downstream video device (not shown). One or more streaming client subsystems, such as client subsystems (406) and (408) of FIG. 4, can access the streaming server (405) to retrieve copies (407) and (409) of the encoded video data (404). The client subsystem (406) can include, for example, a video decoder (410) within an electronic device (430). The video decoder (410) decodes an input copy (407) of the encoded video data and creates an output stream (411) of video pictures that can be rendered uncompressed on a display (412) (e.g., display screen) or other rendering device (not shown). The video decoder 410 can be configured to perform some or all of the various functions described in this disclosure.In some streaming systems, the encoded video data (404), (407), and (409) (e.g., a video bitstream) can be encoded according to a particular video coding / compression standard. Examples of such standards include ITU-T Recommendation H.265. In one example, a video coding standard under development is informally known as Versatile Video Coding (VVC). The disclosed subject matter may be used in the context of VVC, and other video coding standards.

[0049] Note that the electronic devices (420) and (430) can include other components (not shown). For example, the electronic device (420) can include a video decoder (not shown), and the electronic device (430) can also include a video encoder (not shown).

[0050] FIG. 5 shows a block diagram of a video decoder (510) according to any embodiment of the present disclosure below. The video decoder (510) can be included in an electronic device (530). The electronic device (530) can include a receiver (531) (e.g., a receiving circuit). The video decoder (510) can be used instead of the video decoder (410) in the example of FIG. 4.

[0051] The receiver (531) may receive one or more coded video sequences to be decoded by the video decoder (510). In the same or other embodiments, one coded video sequence may be decoded at a time, and the decoding of each coded video sequence is independent of other coded video sequences. Each video sequence may be associated with a plurality of video frames or video images. The coded video sequence may be received from a channel (501), which may be a hardware / software link to a storage device storing the encoded video data, or a streaming source transmitting the encoded video data. The receiver (531) may receive the encoded video data together with other data such as coded audio data and / or auxiliary data streams that may be transferred to respective processing circuits (not shown). The receiver (531) may separate the coded video sequence from other data. To counter network jitter, a buffer memory (515) may be disposed between the receiver (531) and the entropy decoder / parser (520) (hereinafter, "parser (520)"). For certain applications, the buffer memory (515) may be implemented as part of the video decoder (510). For other applications, the buffer memory (515) may be separate and external to the video decoder (510) (not shown). In still other applications, for example, there may be a buffer memory (not shown) external to the video decoder (510) to counter network jitter, and for example, there may be other additional buffer memories (515) inside the video decoder (510) to handle playback timing. When the receiver (531) is receiving data from a storage / transfer device with sufficient bandwidth and controllability, or from an isochronous network, the buffer memory (515) may be unnecessary or can be made small. For use in a best-effort packet network such as the Internet, a buffer memory (515) of sufficient size may be required, and its size may be relatively large.Such a buffer memory may be implemented with an adaptable size and may be implemented at least partially in an operating system external to the video decoder (510) or a similar element (not shown).

[0052] The video decoder (510) may include a parser (520) to reconstruct symbols (521) from the coded video sequence. The categories of those symbols include information used to manage the operation of the video decoder (510) and, potentially, information for controlling a rendering device such as a display (512) (e.g., a display screen) that can be coupled to the electronic device (530), whether or not it is an essential part of the electronic device (530) as shown in FIG. 5. The control information for the rendering device may be in the form of supplementary enhancement information (SEI message) or a parameter set fragment of video user capability information (VUI) (not shown). The parser (520) may syntax analyze / entropy decode the coded video sequence received by the parser (520). The entropy coding of the coded video sequence can be in accordance with a video coding technology or standard and can follow various principles including variable length coding, Huffman coding, arithmetic coding with or without context dependency, etc. The parser (520) may extract a set of subgroup parameters of at least one subgroup of pixels in the video decoder based on at least one parameter corresponding to a subgroup from the coded video sequence. Subgroups can include picture groups (GOP), pictures, tiles, slices, macroblocks, coding units (CU), blocks, transform units (TU), prediction units (PU), etc. The parser (520) may also extract information such as transform coefficients (e.g., Fourier transform coefficients), quantization parameter values, motion vectors, etc. from the coded video sequence.

[0053] The parser (520) may perform an entropy decoding / syntax analysis operation on the video sequence received from the buffer memory (515) to create symbols (521).

[0054] The reconstruction of the symbol (521) may include a plurality of different processing units or functional units depending on the type of the coded video picture or a portion thereof (such as inter-picture and intra-picture, inter-block and intra-block, etc.), and other factors. Which units are included and how the units are included may be controlled by subgroup control information parsed from the video sequence coded by the parser (520). Such a flow of subgroup control information between the parser (520) and the following plurality of processing units or functional units is not illustrated for simplicity.

[0055] In addition to the functional blocks already described, the video decoder (510) can be conceptually subdivided into several functional units as described below. In actual implementations operating under commercial constraints, many of these functional units interact closely with each other and can be at least partially integrated with each other. However, for the purpose of clearly describing the various functions of the disclosed subject matter, a conceptual subdivision into functional units is adopted in the following disclosure.

[0056] The first unit may include a scaler / inverse transform unit (551). The scaler / inverse transform unit (551) may receive, as symbols (521) from the parser (520), quantization transform coefficients, and control information including information indicating which type of inverse transform to use, block size, quantization coefficient / parameters, quantization scaling matrix, etc. The scaler / inverse transform unit (551) may output a block including sample values that can be input to the aggregator (555).

[0057] In some cases, the output samples of the scaler / inverse transform (551) may relate to intra-coded blocks, i.e., blocks that do not use prediction information from previously reconstructed pictures but can use prediction information from previously reconstructed parts of the current picture. Such prediction information may be provided by the intra-picture prediction unit (552). In some cases, the intra-picture prediction unit (552) may use the information of surrounding blocks that have already been reconstructed and stored in the current picture buffer (558) to generate blocks of the same size and shape as the block being reconstructed. The current picture buffer (558) buffers, for example, the partially reconstructed current picture and / or the fully reconstructed current picture. The aggregator (555) may, in some implementations, add the prediction information generated by the intra prediction unit (552) to the output sample information provided by the scaler / inverse transform unit (551) for each sample.

[0058] In other cases, the output samples of the scaler / inverse transform unit (551) may be related to inter-coded and potentially motion-compensated blocks. In such cases, the motion compensation prediction unit (553) can access the reference picture memory (557) to fetch the samples used for inter-picture prediction. After motion-compensating the samples fetched according to the symbols (521) related to the block, these samples can be added to the output of the scaler / inverse transform unit (551) by the aggregator (555) to generate output sample information (the output of unit 551 may be referred to as residual samples or a residual signal). The address in the reference picture memory (557) from which the motion compensation prediction unit (553) fetches the prediction samples can be controlled by a motion vector and be in the form of a symbol (521) that can have, for example, an X component, a Y component (shift), and a reference picture component (time) and be available to the motion compensation prediction unit (553). Motion compensation may also include interpolation of the sample values fetched from the reference picture memory (557) when an exact motion vector of sub-samples is used and may be associated with a motion vector prediction mechanism or the like.

[0059] The output samples of the aggregator (555) can undergo various loop filtering techniques in the loop filter unit (556). The video compression technology is controlled by parameters included in the coded video sequence (also called the coded video bitstream) and can include in-loop filter techniques made available to the loop filter unit (556) as symbols (521) from the parser (520), but can respond not only to meta information obtained during the decoding of the previous part (in decoding order) of the coded picture or coded video sequence but also to previously reconstructed and loop-filtered sample values. As will be explained in more detail below, some types of loop filters can be included as part of the loop filter unit 556 in various orders.

[0060] The output of the loop filter unit (556) can be a sample stream that is not only output to the rendering device (512) but also stored in the reference picture memory (557) for use in future inter-picture prediction.

[0061] Once a particular coded picture is fully reconstructed, it can be used as a reference picture for future inter-picture prediction. For example, when the coded picture corresponding to the current picture is fully reconstructed and the coded picture is identified as a reference picture (e.g., by the parser (520)), the current picture buffer (558) can become part of the reference picture memory (557), and the unused current picture buffer can be reallocated before starting the reconstruction of the next coded picture.

[0062] The video decoder (510) may perform a decoding operation according to a predetermined video compression technology adopted in a standard such as ITU-T Recommendation H.265. The coded video sequence may conform to the syntax specified by the video compression technology or standard being used in the sense that the coded video sequence is faithful to both the syntax of the video compression technology or standard and the profile documented in the video compression technology or standard. Specifically, a profile can select specific tools from all the tools available in the video compression technology or standard as the only tools that can be used under that profile. To conform to the standard, the complexity of the coded video sequence can be within the range defined by the level of the video compression technology or standard. In some cases, the level limits, for example, the maximum picture size, the maximum frame rate, the maximum reconstructed sample rate (measured, for example, in megasamples per second), the maximum reference picture size, etc. The limits set by the level can, in some cases, be further restricted by the specifications of the Hypothetical Reference Decoder (HRD) and the metadata of the HRD buffer management signaled in the coded video sequence.

[0063] In some exemplary embodiments, the receiver (531) may receive additional (redundant) data along with the encoded video. The additional data may be included as part of the coded video sequence. The additional data may be used by the video decoder (510) to properly decode the data and / or to more accurately reconstruct the original video data. The additional data can be in the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0064] FIG. 6 shows a block diagram of a video encoder (603) according to an exemplary embodiment of the present disclosure. The video encoder (603) may be included in an electronic device (620). The electronic device (620) may further include a transmitter (640) (e.g., a transmission circuit). The video encoder (603) can be used instead of the video encoder (403) in the example of FIG. 4.

[0065] The video encoder (603) may receive video samples from a video source (601) (not part of the electronic device (620) in the example of FIG. 6) that can capture video images to be coded by the video encoder (603). In other examples, the video source (601) may be implemented as a part of the electronic device (620).

[0066] The video source (601) may provide a source video sequence to be coded by the video encoder (603) in the form of a digital video sample stream that can be of any suitable bit depth (e.g., 8 bits, 10 bits, 12 bits, …), any color space (e.g., BT.601 YCrCb, RGB, XYZ…), and any suitable sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media serving system, the video source (601) may be a storage device capable of storing previously prepared videos. In a video conferencing system, the video source (601) may be a camera that captures local image information as a video sequence. Video data may be provided as a plurality of individual pictures or images that give motion when viewed in sequence. Each picture itself may be organized as a spatial array of pixels, and each pixel can include one or more samples depending on the sampling structure, color space, etc. being used. One of ordinary skill in the art can easily understand the relationship between pixels and samples. The following description focuses on samples.

[0067] According to some exemplary embodiments, the video encoder (603) can code and compress pictures of a source video sequence into a coded video sequence (643) in real time or under any other arbitrary time constraints required by the application. Enforcing an appropriate coding speed constitutes one function of the controller (650). In some embodiments, the controller (650) can be functionally coupled to and control other functional units, as will be described below. For the sake of brevity, the couplings are not shown. Parameters set by the controller (650) can include rate control related parameters (such as picture skip, quantizer, lambda value of rate distortion optimization techniques), picture size, picture group (GOP) layout, maximum motion vector search range, and the like. The controller (650) can be configured to have other appropriate functions related to the video encoder (603) optimized for a particular system design.

[0068] In some exemplary embodiments, the video encoder (603) may be configured to operate in a coding loop. As an overly simplified explanation, in one example, the coding loop may include a source coder (630) (e.g., involved in generating symbols such as a symbol stream based on an input picture and a reference picture to be coded) and a (local) decoder (633) incorporated in the video encoder (603). The decoder (633) reconstructs symbols and creates sample data in a manner similar to that which would be created by a (remote) decoder, even if the incorporated decoder 633 processes a video stream coded by the source coder 630 without entropy coding (in the video compression techniques contemplated by the disclosed subject matter, any compression between the symbols and the coded video bitstream can be reversible). The reconstructed sample stream (sample data) is input into the reference picture memory (634). Since decoding of the symbol stream results in a bit-exact result regardless of the decoder's location (local or remote), the contents of the reference picture memory (634) are also bit-exact between the local encoder and the remote encoder. In other words, the prediction part of the encoder "sees" the same sample values as the decoder "sees" as reference picture samples when using prediction during decoding. This basic principle of reference picture synchronization (and the resulting drift if, for example, synchronization cannot be maintained due to channel errors) is used to improve coding quality.

[0069] The operation of the "local" decoder (633) can be the same as that of a "remote" decoder such as the video decoder (510), which has already been described in detail above in conjunction with FIG. 5. Referring briefly to FIG. 5, however, since symbols are available and the encoding / decoding of symbols to the coded video sequence by the entropy encoder (645) and the parser (520) can be reversible, the entropy decoding portion of the video decoder (510) including the buffer memory (515) and the parser (520) may not be fully implemented in the local decoder (633) within the encoder.

[0070] What can be said at this point is that any decoder technology other than syntax analysis / entropy decoding that can only exist within the decoder may also necessarily need to exist in the corresponding encoder in substantially the same functional form. For this reason, the disclosed subject matter may focus on decoder operations, which are similar to the decoding portion of the encoder. Thus, the description of encoder technology can be omitted since it is the reverse of the decoder technology that is comprehensively described. A more detailed description of the encoder is provided below only in certain areas or aspects.

[0071] During operation, in some exemplary implementations, the source coder (630) may perform motion-compensated predictive coding that predictively codes an input picture by referring to one or more previously coded pictures from the video sequence designated as "reference pictures". In this way, the coding engine (632) codes the color channel difference (or residual) between a pixel block of the input picture and a pixel block of a reference picture that can be selected as a predictive reference to the input picture. The terms "residual" and its adjective form "residual" may be used interchangeably.

[0072] The local video decoder (633) may decode the coded video data of a picture that can be designated as a reference picture based on the symbols created by the source coder (630). The operation of the coding engine (632) may advantageously be an irreversible process. When the coded video data can be decoded by a video decoder (not shown in FIG. 6), the reconstructed video sequence may typically be a replica of the source video sequence with some errors. The local video decoder (633) may replicate the decoding process that can be executed by the video decoder for the reference picture and cause the reconstructed reference picture to be stored in the reference picture cache (634). In this way, the video encoder (603) may locally store a copy of the reconstructed reference picture that has the same content as the reconstructed reference picture obtained by the remote video decoder (without transmission errors).

[0073] The predictor (635) may perform a prediction search for the coding engine (632). That is, for a new picture to be coded, the predictor (635) may seek specific metadata such as sample data (as candidate reference pixel blocks) or reference picture motion vectors, block shapes, etc., that can serve as appropriate prediction references for the new pixels and search the reference picture memory (634). The predictor (635) may operate on the sample blocks for each pixel block to find an appropriate prediction reference. In some cases, the input picture may have prediction references drawn from a plurality of reference pictures stored in the reference picture memory (634) as determined by the search results obtained by the predictor (635).

[0074] The controller (650) may manage the coding operation of the source coder (630), including, for example, setting the parameters and subgroup parameters used to encode the video data.

[0075] The outputs of all the aforementioned functional units can undergo entropy coding within the entropy coder (645). The entropy coder (645) converts the symbols generated by the various functional units into a coded video sequence by reversibly compressing the symbols according to techniques such as Huffman coding, variable-length coding, arithmetic coding, etc.

[0076] The transmitter (640) can buffer the coded video sequence created by the entropy coder (645) to prepare for transmission via the communication channel (660), and the communication channel (660) may be a hardware / software link to a storage device that stores the encoded video data. The transmitter (640) can merge the coded video data from the video coder (603) with other data to be transmitted, such as coded audio data and / or an auxiliary data stream (source not shown).

[0077] The controller (650) may manage the operation of the video encoder (603). During coding, the controller (650) may assign a specific coded picture type to each coded picture, which may affect the coding techniques applicable to each picture. For example, a picture is often assigned as one of the following picture types.

[0078] An intra picture (I picture) can be coded and decoded without using other pictures in the sequence as a prediction source. Some video codecs allow different types of intra pictures, including, for example, independent decoder refresh (「IDR」) pictures. Those skilled in the art are aware of those variants of I pictures, as well as their respective uses and characteristics.

[0079] A predicted picture (P picture) can be a picture that can be coded and decoded using intra prediction or inter prediction that uses at most one motion vector and a reference index to predict the sample values of each block.

[0080] A bi - directionally predicted picture (B picture) can be a picture that can be coded and decoded using intra prediction or inter prediction that uses at most two motion vectors and reference indices to predict the sample values of each block. Similarly, multiple predicted pictures can use three or more reference pictures and associated metadata for the reconstruction of a single block.

[0081] A source picture is generally spatially subdivided into a plurality of sample coding blocks (e.g., blocks of 4×4, 8×8, 4×8, or 16×16 samples each) and can be coded block - by - block. The blocks may be coded predictively by referring to other (already coded) blocks as determined by the coding assignment applied to each picture of the block. For example, blocks of an I picture may be coded non - predictively or may be coded predictively by referring to already coded blocks of the same picture (spatial prediction or intra prediction). Pixel blocks of a P picture may be coded predictively via spatial prediction or via temporal prediction by referring to one previously coded reference picture. Blocks of a B picture may be coded predictively via spatial prediction or via temporal prediction by referring to one or two previously coded reference pictures. The source picture or an intermediate - processed picture may be subdivided into other types of blocks for other purposes. The division of coding blocks and other types of blocks may or may not follow the same method as will be described in more detail below.

[0082] The video encoder (603) can perform a coding operation according to a predetermined video coding technology or standard such as ITU-T Recommendation H.265. In its operation, the video encoder (603) can perform various compression operations, including a predictive coding operation that utilizes the temporal and spatial redundancies in the input video sequence. Therefore, the coded video data can conform to the syntax specified by the video coding technology or standard being used.

[0083] In some exemplary embodiments, the transmitter (640) can transmit additional data along with the encoded video. The source coder (630) can include such data as part of the encoded video sequence. The additional data may include, for example, temporal / spatial / SNR extension layers, other forms of redundant data such as redundant pictures and slices, SEI messages, VUI parameter set fragments, and the like.

[0084] Video may be captured as a plurality of source pictures (video pictures) in a time series. Intra-picture prediction (often abbreviated as intra prediction) utilizes the spatial correlation within a given picture, and inter-picture prediction utilizes the temporal or other correlation between pictures. For example, a particular picture being encoded / decoded, called the current picture, can be divided into blocks. A block within the current picture can be coded by a vector called a motion vector if it is similar to a reference block within a reference picture that has been previously coded and buffered in the video. The motion vector points to the reference block within the reference picture and can have a third dimension that identifies the reference picture if multiple reference pictures are being used.

[0085] In some exemplary embodiments, a dual prediction technique may be used for inter-picture prediction. According to such a dual prediction technique, two reference pictures such as a first reference picture and a second reference picture are used, both of which advance the current picture in the video in decoding order (however, in display order, they may be in the past or future respectively). A block in the current picture can be coded by a first motion vector pointing to a first reference block in the first reference picture and a second motion vector pointing to a second reference block in the second reference picture. The block can be predicted cooperatively by a combination of the first reference block and the second reference block.

[0086] Furthermore, a merge mode technique may be used to improve coding efficiency in inter-picture prediction.

[0087] According to some exemplary embodiments of the present disclosure, predictions such as inter-picture prediction and intra-picture prediction are performed on a block-by-block basis. For example, pictures in a sequence of video pictures are divided into coding tree units (CTUs) for compression, and the CTUs within a picture can have the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. Generally, a CTU can include three parallel coding tree blocks (CTBs), namely, one luma CTB and two chroma CTBs. Each CTU can be recursively quadtree-divided into one or more coding units (CUs). For example, a 64×64 pixel CTU can be divided into one 64×64 pixel CU, or four 32×32 pixel CUs. Each of one or more of the 32×32 blocks can be further divided into four 16×16 pixel CUs. In some exemplary embodiments, each CU can be analyzed during encoding to determine the prediction type of that CU from various prediction types, such as an inter-prediction type or an intra-prediction type. A CU can be divided into one or more prediction units (PUs) according to its temporal and / or spatial predictability. Generally, each PU includes one luma prediction block (PB) and two chroma PBs. In one embodiment, the prediction operation in coding (encoding / decoding) is performed in units of prediction blocks. The division of a CU into PUs (or PBs of different color channels) can be performed in various spatial patterns. A luma PB or a chroma PB can include a matrix of sample values (e.g., luma values), such as 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, etc.

[0088] FIG. 7 shows a diagram of a video encoder (703) according to another exemplary embodiment of the present disclosure. The video encoder (703) is configured to receive a processing block (e.g., a prediction block) of sample values within a current video picture in a sequence of video pictures and encode the processing block into a coded picture that is part of a coded video sequence. The exemplary video encoder (703) may be used instead of the video encoder (403) of the example of FIG. 4.

[0089] For example, the video encoder (703) receives a matrix of sample values for a processing block, such as an 8×8 sample prediction block. The video encoder (703) then determines, for example, using rate distortion optimization (RDO), whether the processing block is best coded using an intra mode, an inter mode, or a bi-prediction mode. If it is determined that the processing block is to be coded in the intra mode, the video encoder (703) encodes the processing block into the coded picture using intra prediction techniques, and if it is determined that the processing block is to be coded in the inter mode or the bi-prediction mode, the video encoder (703) may encode the processing block into the coded picture using inter prediction techniques or bi-prediction techniques, respectively. In some exemplary embodiments, as a sub-mode of inter-picture prediction, a merge mode derived from one or more motion vector predictors without the benefit of coded motion vector components outside the predictor may be used. In some other exemplary embodiments, there may be motion vector components applicable to the target block. Thus, the video encoder (703) may include components not explicitly shown in FIG. 7, such as a mode decision module, to determine the prediction mode of the processing block.

[0090] In the example of FIG. 7, video encoder (703) includes inter-encoder (730), intra-encoder (722), residual calculator (723), switch (726), residual encoder (724), general-purpose controller (721), and entropy encoder (725) coupled to each other as shown in the exemplary arrangement of FIG. 7.

[0091] Inter-encoder (730) receives samples of the current block (e.g., processing block), compares the block with one or more reference blocks (e.g., blocks in the previous picture and the subsequent picture in display order) in the reference picture, generates inter-prediction information (e.g., description of redundant information, motion vectors, merge mode information by inter-encoding techniques), and is configured to calculate an inter-prediction result (e.g., predicted block) based on the inter-prediction information using any suitable technique. In some examples, the reference picture is a decoded reference picture decoded based on the encoded video information using a decoding unit 633 incorporated in the exemplary encoder 620 of FIG. 6 (shown as residual decoder 728 of FIG. 7 and described in more detail below).

[0092] Intra-encoder (722) receives samples of the current block (e.g., processing block), compares the block with already-coded blocks in the same picture, generates quantized coefficients after transformation, and may also generate intra-prediction information (e.g., intra-prediction direction information by one or more intra-encoding techniques). Intra-encoder (722) may calculate an intra-prediction result (e.g., predicted block) based on the intra-prediction information and reference blocks in the same picture.

[0093] The general-purpose controller (721) may be configured to determine general-purpose control data and control other components of the video encoder (703) based on the general-purpose control data. In one example, the general-purpose controller (721) determines the prediction mode of a block and provides a control signal to the switch (726) based on the prediction mode. For example, when the prediction mode is the intra mode, the general-purpose controller (721) controls the switch (726) to select the intra mode result for use by the residual calculator (723), controls the entropy encoder (725) to select the intra prediction information and include the intra prediction information in the bitstream. When the prediction mode of the block is the inter mode, the general-purpose controller (721) controls the switch (726) to select the inter prediction result for use by the residual calculator (723), controls the entropy encoder (725) to select the inter prediction information and include the inter prediction information in the bitstream.

[0094] The residual calculator (723) may be configured to calculate the difference (residual data) between the received block and the prediction result for the block selected from the intra encoder (722) or the inter encoder (730). The residual encoder (724) may be configured to encode the residual data to generate conversion coefficients. For example, the residual encoder (724) may be configured to convert the residual data from the spatial domain to the frequency domain to generate conversion coefficients. The conversion coefficients then undergo quantization processing to obtain quantized conversion coefficients. In various exemplary embodiments, the video encoder (703) also includes a residual decoder (728). The residual decoder (728) is configured to perform inverse conversion to generate decoded residual data. The decoded residual data can be appropriately used by the intra encoder (722) and the inter encoder (730). For example, the inter encoder (730) can generate a decoded block based on the decoded residual data and inter prediction information, and the intra encoder (722) can generate a decoded block based on the decoded residual data and intra prediction information. The decoded block is appropriately processed to generate a decoded picture, and the decoded picture can be buffered in a memory circuit (not shown) and used as a reference picture.

[0095] The entropy encoder (725) may be configured to format the bitstream to include the encoded blocks and perform entropy coding. The entropy encoder (725) is configured to include various information in the bitstream. For example, the entropy encoder (725) may be configured to include general control data, selected prediction information (e.g., intra prediction information or inter prediction information), residual information, and other appropriate information in the bitstream. When coding a block in either the inter mode or the merge submode of the bi-prediction mode, the residual information may not be present.

[0096] FIG. 8 shows a diagram of an exemplary video decoder (810) according to another embodiment of the present disclosure. The video decoder (810) is configured to receive a coded picture that is part of a coded video sequence and decode the coded picture to generate a reconstructed picture. In one example, the video decoder (810) may be used in place of the video decoder (410) of the example of FIG. 4.

[0097] In the example of FIG. 8, the video decoder (810) includes an entropy decoder (871), an inter decoder (880), a residual decoder (873), a reconstruction module (874), and an intra decoder (872) coupled to each other as shown in the exemplary arrangement of FIG. 8.

[0098] The entropy decoder (871) can be configured to reconstruct from the coded picture certain symbols that represent syntax elements from which the coded picture is composed. Such symbols can include, for example, the mode in which a block is coded (e.g., intra mode, inter mode, bi-prediction mode, merge sub-mode or other sub-modes), prediction information (e.g., intra prediction information or inter prediction information) that can identify certain samples or metadata used for prediction by the intra decoder (872) or the inter decoder (880), and residual information such as in the form of quantized transform coefficients. In one example, when the prediction mode is an inter prediction mode or a bi-prediction mode, the inter prediction information is provided to the inter decoder (880), and when the prediction type is an intra prediction type, the intra prediction information is provided to the intra decoder (872). The residual information can undergo inverse quantization and is provided to the residual decoder (873).

[0099] The inter decoder (880) may be configured to receive inter prediction information and generate an inter prediction result based on the inter prediction information.

[0100] The intra decoder (872) may be configured to receive intra prediction information and generate a prediction result based on the intra prediction information.

[0101] The residual decoder (873) may be configured to perform inverse quantization to extract inverse quantization transform coefficients, process the inverse quantization transform coefficients, and convert the residual from the frequency domain to the spatial domain. The residual decoder (873) may also utilize specific control information (for including quantization parameter (QP)), and that information may be provided by the entropy decoder (871) (since this may be only a small amount of control information, the data path is not shown).

[0102] The reconstruction module (874) may be configured to combine, in the spatial domain, the residual as the output by the residual decoder (873) and the prediction result (optionally, as the output by the inter prediction module or the intra prediction module) to form a reconstructed block that forms a part of the reconstructed picture as a part of the reconstructed video. Note that other appropriate operations such as a deblocking operation may be performed to improve visual quality.

[0103] Note that the video encoders (403), (603), and (703), and the video decoders (410), (510), and (810) can be implemented using any appropriate technique. In some exemplary embodiments, the video encoders (403), (603), and (703), and the video decoders (410), (510), and (810) can be implemented using one or more integrated circuits. In other embodiments, the video encoders (403), (603), and (603), and the video decoders (410), (510), and (810) can be implemented using one or more processors that execute software instructions.

[0104] Turning to block partitioning for coding and decoding, a common partitioning may start from a base block or may follow a predefined set of rules, a specific pattern, a partitioning tree, or any partitioning structure or scheme. The partitioning may be hierarchical and recursive. After separating or partitioning the base block according to any of the exemplary partitioning procedures or other procedures described below, or a combination thereof, a final set of partitions or coding blocks may be obtained. Each of these partitions may be at one of various partitioning levels within the partitioning hierarchy and may be partitions of various shapes. Each of the partitions may sometimes be referred to as a coding block (CB). In various exemplary partitioning implementations described further below, each resulting CB may be a CB of an acceptable size and partitioning level. Such a partition can make some basic coding / decoding decisions for it, optimize, determine, and signal in an encoded video bitstream the coding / decoding parameters, and thus form a unit that can be called a coding block. The highest or deepest level in the final partition represents the depth of the coding block partitioning tree structure. The coding block may be a luma coding block or a chroma coding block. The CB tree structure for each color may sometimes be referred to as a coding block tree (CBT).

[0105] The coding blocks for all color channels may sometimes be collectively referred to as a coding unit (CU). The hierarchical structure for all color channels may sometimes be collectively referred to as a coding tree unit (CTU). The partitioning patterns or partitioning structures of the various color channels within the CTU may or may not be the same.

[0106] In some implementations, the partitioning tree method or structure used for the luma channel and the chroma channel need not be the same. In other words, the luma channel and the chroma channel may have separate coding tree structures or patterns. Further, whether the luma channel and the chroma channel use the same coding partition tree structure or different coding partition tree structures, and the actual coding partition tree structure to be used may depend on whether the slice being coded is a P slice, a B slice, or an I slice. For example, in the case of an I slice, the chroma channel and the luma channel may have separate coding partition tree structures or coding partition tree structure modes, but in the case of a P slice or a B slice, the luma channel and the chroma channel may share the same coding partition tree method. When separate coding partition tree structures or modes are applied, the luma channel may be partitioned into CB by one coding partition tree structure, and the chroma channel may be partitioned into chroma CB by another coding partition tree structure.

[0107] In some exemplary implementations, a predetermined partitioning pattern may be applied to the base block. As shown in FIG. 9, an exemplary four-way partitioning tree may start from a first predefined level (e.g., as a base block size, a 64×64 block level or other size), and the base block may be hierarchically partitioned down to a predefined lowest level (e.g., a 4×4 level). For example, the base block can follow four predefined partitioning options or patterns indicated by 902, 904, 906, and 908, and the partition designated as R can be recursively partitioned in that the same partitioning option shown in FIG. 9 can be repeated at a lower scale down to the lowest level (e.g., the 4×4 level). In some implementations, additional restrictions may be imposed on the partitioning method of FIG. 9. In the implementation of FIG. 9, rectangular partitions (e.g., 1:2 / 2:1 rectangular partitions) may be allowed, but may not be allowed to be recursive, while square partitions are allowed to be recursive. Optionally, subsequent partitioning of FIG. 9 by recursion generates a final set of coding blocks. A coding tree depth may be further defined to indicate the depth of partitioning from the root node or root block. For example, the coding tree depth for a root node or root block of a 64×64 block may be set to 0, and after the root block is further partitioned once according to FIG. 9, the coding tree depth increases by only 1. The maximum or deepest level from a 64×64 base block to a 4×4 minimum partition is 4 in the above manner (starting from level 0). Such a partitioning method may be applied to one or more of the color channels. Each color channel may be independently partitioned according to the method of FIG. 9 (e.g., for each of the color channels at each hierarchical level, a partitioning pattern or option from a predefined pattern may be independently determined). Alternatively, two or more color channels may share the same hierarchical pattern tree of FIG. 9 (e.g., for two or more color channels at each hierarchical level, the same partitioning pattern or option from a predefined pattern may be selected).

[0108] FIG. 10 shows another exemplary pre - defined partitioning pattern that enables recursive partitioning to form a partitioning tree. As shown in FIG. 10, ten exemplary partitioning structures or patterns may be pre - defined. The root block may start from a pre - defined level (e.g., from a base block of 128×128 level or 64×64 level). The exemplary partitioning structures of FIG. 10 include various 2:1 / 1:2 and 4:1 / 1:4 rectangular partitions. The partition type having three sub - partitions shown at 1002, 1004, 1006, and 1008 in the second column of FIG. 10 may be called a "T - type" partition. The "T - type" partitions 1002, 1004, 1006, and 1008 may be called left T - type, up T - type, right T - type, and down T - type, respectively. In some exemplary implementations, none of the rectangular partitions of FIG. 10 can be further subdivided. To indicate the partitioning depth from the root node or root block, a coding tree depth may be further defined. For example, the coding tree depth for the root node or root block of a 128×128 block may be set to 0, and after the root block is further partitioned once according to FIG. 10, the coding tree depth increases by only 1. In some implementations, only all square partitions of 1010 may be recursively partitioned to the next level of the partitioning tree following the pattern of FIG. 10. In other words, recursive partitioning may not be possible for the square partitions within the T - type patterns 1002, 1004, 1006, and 1008. Optionally, a final set of coding blocks is generated by a recursive partitioning procedure following FIG. 10. Such a method may be applied to one or more of the color channels. In some implementations, more flexibility may be added to the use of partitions less than 8×8 level. For example, in some cases, 2×2 chroma - in - ter prediction may be used.

[0109] In some other exemplary implementations for coding block splitting, a quadtree structure may be used to split a base block or an intermediate block into quadtree partitions. Such quadtree splitting may be applied hierarchically and recursively to any square partition. Whether a base block or an intermediate block or partition is further split into quadtree partitions may conform to various local characteristics of the base block or intermediate block / partition. Quadtree splitting at the picture boundary may be further adapted. For example, implicit quadtree splitting may be performed at the picture boundary such that the block continues to be split into quadtree partitions until its size fits within the picture boundary.

[0110] In some other exemplary implementations, hierarchical binary splitting from a base block may be used. In such a case, a base block or an intermediate level block may be split into two partitions. The binary splitting may be either horizontal or vertical. For example, horizontal binary splitting may split a base block or an intermediate block into equal left and right partitions. Similarly, vertical binary splitting may split a base block or an intermediate block into equal upper and lower partitions. Such binary splitting may be hierarchical and recursive. Whether to continue the binary splitting method, and if the method continues, whether horizontal binary splitting should be used or vertical binary splitting should be used, may be determined for each of the base blocks or intermediate blocks. In some implementations, further splitting may stop at a predefined minimum partition size (in one or both dimensions). Alternatively, further splitting may stop when a predefined splitting level or depth from the base block is reached. In some implementations, the aspect ratio of the partitions may be limited. For example, the aspect ratio of the partitions may not be less than 1:4 (or greater than 4:1). Thus, a vertical strip partition having a vertical-to-horizontal aspect ratio of 4:1 may only be further vertically split into upper and lower partitions each having a vertical-to-horizontal aspect ratio of 2:1.

[0111] In some further examples, as shown in FIG. 13, a three-way splitting scheme may be used to split a base block or any intermediate block. The ternary pattern may be implemented vertically as shown at 1302 in FIG. 13 or horizontally as shown at 1304 in FIG. 13. The exemplary split ratios in FIG. 13 are shown as 1:2:1, either vertically or horizontally, although other ratios may be predefined. In some implementations, two or more different ratios may be predefined. Such a quadtree split is capable of capturing an object located at the block center within one continuous partition, while a quadtree and a binary tree always split along the block center, and thus such a three-way splitting scheme may be used to complement a quadtree or a binary split structure in that it splits an object into separate partitions. In some implementations, the width and height of an exemplary ternary partition are always a power of two to avoid further conversions.

[0112] The above-described splitting methods can be combined in any way at different splitting levels. As an example, the quadtree and binary splitting methods described above may be combined to split a base block into a quadtree-binary tree (QTBT) structure. In such a method, a base block or an intermediate block / partition may be either quadtree split or binary split according to a set of predefined conditions, if specified. A specific example is shown in FIG. 14. In the example of FIG. 14, the base block is first quadtree split into four partitions as shown by 1402, 1404, 1406, and 1408. Each of the resulting partitions is then either quadtree split into four further partitions (such as 1408) or binary split into two further partitions (either horizontal or vertical, such as 1402 or 1406 which are both symmetric) at the next level or not split (such as 1404). The binary or quadtree splitting may be made recursively possible for square partitions as shown by the overall exemplary splitting pattern of 1410 and the corresponding tree structure / representation of 1420, where solid lines represent quadtree splits and dashed lines represent binary splits. A flag may be used for each binary split node (non-leaf binary partition) to indicate whether the binary split is horizontal or vertical. For example, as shown in 1420 which corresponds to the splitting structure of 1410, the flag "0" may represent a horizontal binary split and the flag "1" may represent a vertical binary split. In the case of a quadtree split partition, since the quadtree split always splits a block or partition both horizontally and vertically to produce four sub-blocks / partitions of equal size, there is no need to indicate the split type. In some implementations, the flag "1" may represent a horizontal binary split and the flag "0" may represent a vertical binary split.

[0113] In some exemplary implementations of QTBT, the quadtree and binary splitting rule sets may be represented by the following predefined parameters and their corresponding associated functions. - CTU size: The size of the quadtree root node (the size of the base block) -MinQTSize: Minimum allowable quadtree leaf node size -MaxBTSize: Maximum allowable binary tree root node size -MaxBTDepth: Maximum allowable binary tree depth -MinBTSize: Minimum allowable binary tree leaf node size In some exemplary implementations of the QTBT splitting structure, the CTU size may be set as 128×128 luma samples having two corresponding 64×64 blocks of chroma samples (when exemplary chroma subsampling is considered and used), MinQTSize may be set as 16×16, MaxBTSize may be set as 64×64, MinBTSize (for both width and height) may be set as 4×4, and MaxBTDepth may be set as 4. Quadtree splitting may first be applied to the CTU to generate quadtree leaf nodes. The quadtree leaf nodes may have sizes ranging from its minimum allowable size of 16×16 (i.e., MinQTSize) to 128×128 (i.e., CTU size). If the node is 128×128, since its size exceeds MaxBTSize (i.e., 64×64), it will not be split first by the binary tree. Otherwise, nodes not exceeding MaxBTSize may be split by the binary tree. In the example of FIG. 14, the base block is 128×128. The base block allows only quadtree splitting according to a predefined set of rules. The base block has a splitting depth of 0. Each of the resulting four partitions is 64×64 and does not exceed MaxBTSize and may be further split by quadtree or binary tree at level 1. The process continues. When the binary tree depth reaches MaxBTDepth (i.e., 4), further splitting may not be considered. When the width of the binary tree node is equal to MinBTSize (i.e., 4), further horizontal splitting may not be considered. Similarly, when the height of the binary tree node is equal to MinBTSize, further vertical splitting is not considered.

[0114] In some exemplary implementations, the above QTBT scheme may be configured to support the flexibility for luma and chroma to have the same QTBT structure or separate QTBT structures. For example, in the case of P slices and B slices, the luma CTB and chroma CTB within one CTU may share the same QTBT structure. However, in the case of I slices, the luma CTB may be divided into CUs by the QTBT structure, and the chroma CTB may be divided into chroma CUs by another QTBT structure. This may be used for CUs to reference different color channels within the I slice. For example, the I slice may be composed of coding blocks of the luma component or coding blocks of two chroma components, meaning that the CUs within the P slice or B slice may be composed of coding blocks of all three color components.

[0115] In some other implementations, the QTBT scheme may be complemented by the ternary scheme described above. Such an implementation may be referred to as a multi-type tree (MTT) structure. For example, in addition to the binary split of nodes, one of the ternary split patterns in FIG. 13 may be selected. In some implementations, only square nodes can undergo a ternary split. An additional flag may be used to indicate whether the ternary split is horizontal or vertical.

[0116] The design of two-level or multi-level trees, such as QTBT implementations and QTBT implementations complemented by ternary splits, may be motivated mainly by complexity reduction. Theoretically, the complexity of traversing the tree is T D where T represents the number of split types and D represents the depth of the tree. A trade-off may be made by using multiple types (T) while reducing the depth (D).

[0117] In some implementations, the CB may be further divided. For example, the CB may be further divided into a plurality of prediction blocks (PBs) for the purpose of intra-frame prediction or inter-frame prediction during the coding process and the decoding process. In other words, the CB may be further divided into different sub-partitions where individual prediction decisions / configurations may be made. In parallel, the CB may be further divided into a plurality of transform blocks (TBs) for the purpose of describing the level at which video data is transformed or inverse-transformed. The division methods of the CB into PBs and TBs may or may not be the same. For example, each division method may be performed using a unique procedure based on various characteristics of the video data, for example. The division methods of the PBs and TBs may be independent in some exemplary implementations. The division methods and boundaries of the PBs and TBs may be correlated in some other exemplary implementations. In some implementations, for example, the TB may be divided after the PB division, and in particular, each PB may be further divided into one or more TBs after being determined following the division of the coding block. For example, in some implementations, the PB may be divided into one, two, four, or other numbers of TBs.

[0118] In some implementations, the luma and chroma channels may be processed differently in order to split a base block into coding blocks, and further into prediction blocks and / or transform blocks. For example, in some implementations, the splitting of a coding block into prediction blocks and / or transform blocks may be allowed for the luma channel, but such splitting of a coding block into prediction blocks and / or transform blocks may not be allowed for the chroma channel. In such implementations, thus, the transformation and / or prediction of luma blocks can be performed only at the coding block level. In other examples, the minimum transform block sizes of the luma and chroma channels may be different. For example, a coding block of the luma channel may be allowed to be split into smaller transform blocks and / or prediction blocks than the chroma channel. In yet other examples, the maximum depth of the splitting of a coding block into transform blocks and / or prediction blocks may be different between the luma and chroma channels. For example, a coding block of the luma channel may be allowed to be split into deeper transform blocks and / or prediction blocks than the chroma channel. As a specific example, a luma coding block may be split into transform blocks of multiple sizes that can be represented by a recursive split that goes down a maximum of two levels, and transform block shapes such as square, 2:1 / 1:2, and 4:1 / 1:4, as well as transform block sizes from 4×4 to 64×64 may be allowed. However, in the case of chroma blocks, only the maximum possible transform blocks specified for luma blocks may be allowed.

[0119] In some exemplary implementations for splitting a coding block into PBs, the depth, shape, and / or other characteristics of the PB split may depend on whether the PB is intra-coded or inter-coded.

[0120] The splitting of a coding block (or prediction block) into transform blocks may be performed recursively or non - recursively in various exemplary ways including, but not limited to, quadtree splitting and predefined pattern splitting, further considering the transform blocks at the boundaries of the coding block or prediction block. Generally, the resulting transform blocks may be at different splitting levels, may not be of the same size, and need not be square in shape (e.g., they can be rectangles having some allowed sizes and aspect ratios). Further examples are described in more detail below in connection with FIGS. 15, 16, and 17.

[0121] However, in some other implementations, the CBs obtained via any of the above - mentioned splitting methods may be used as the basic or minimum coding blocks for prediction and / or transformation. In other words, no further splitting is performed for the purpose of inter - prediction / intra - prediction and / or for the purpose of transformation. For example, the CBs obtained from the above - mentioned QTBT method may be used as is as a unit for prediction. Specifically, such a QTBT structure removes the concept of multiple splitting types, i.e., it removes the separation of CU, PU, and TU, and supports more flexibility for the CU / CB splitting shape as described above. In such a QTBT block structure, the CU / CB can have either a square or a rectangular shape. Such leaf nodes of the QTBT are used as units for prediction and transformation processing without further splitting. This means that the CU, PU, and TU have the same block size in such an exemplary QTBT coding block structure.

[0122] The various CB splitting methods described above, as well as further splitting of the CBs into PBs and / or TBs (including no PB / TB splitting), can be combined in any way. The following specific implementations are provided as non - limiting examples.

[0123] Specific and exemplary implementations of the division of coding blocks and transform blocks are described below. In such exemplary implementations, the base block may be divided into coding blocks using recursive quadtree division or a predefined division pattern as described above (such as the division patterns in FIGS. 9 and 10). At each level, whether to continue the further quadtree division of a particular partition may be determined by local video data characteristics. The resulting CBs are at various quadtree division levels and may be CBs of various sizes. The decision of whether to code the picture area using inter-picture (temporal) prediction or intra-picture (spatial) prediction may be made at the CB level (or, in the case of all three color channels, the CU level). Each CB may be further divided into one, two, four, or other numbers of PBs according to a predefined PB division type. Within one PB, the same prediction process may be applied, and the related information may be sent to the decoder on a PB basis. After obtaining the residual block by applying the prediction process based on the PB division type, the CB can be divided into TBs according to another quadtree structure similar to the coding tree for CBs. In this particular implementation, the CB or TB may not be limited to being square. Further in this particular example, the PB may be square or rectangular for inter prediction and square only for intra prediction. The coding block may be divided, for example, into four square TBs. Each TB may be further divided recursively (using quadtree division) into smaller TBs called residual quadtree (RQT).

[0124] Other exemplary implementations for splitting a base block into CB, PB, and / or TB are further described below. For example, instead of using multiple partition unit types such as those shown in FIG. 9 or FIG. 10, a quadtree having a nested multi-type tree using a binary and ternary segmentation structure (e.g., QTBT or QTBT by the ternary segmentation described above) may be used. The separation of CB, PB, and TB (i.e., the splitting of CB into PB and / or TB, and the splitting of PB into TB) may be waived except when such a CB has a size that is too large for the maximum transform length when the CB requires further splitting. This exemplary splitting method may be designed to support further flexibility in the CB splitting shape so that both prediction and transformation can be performed at the CB level without further splitting. In such a coding tree structure, the CB may have either a square or rectangular shape. Specifically, a coding tree block (CTB) may first be split by a quadtree structure. Then, the quadtree leaf nodes may be further split by a nested multi-type tree structure. An example of a nested multi-type tree structure using binary or ternary splitting is shown in FIG. 11. Specifically, the exemplary multi-type tree structure of FIG. 11 includes four splitting types called vertical binary split (SPLIT_BT_VER) (1102), horizontal binary split (SPLIT_BT_HOR) (1104), vertical ternary split (SPLIT_TT_VER) (1106), and horizontal ternary split (SPLIT_TT_HOR) (1108). Then, the CB corresponds to the leaf of the multi-type tree. In this exemplary implementation, this segmentation is used for both prediction and transformation processing without further splitting as long as the CB is not too large for the maximum transform length. This means that in most cases, CB, PB, and TB have the same block size in a quadtree having a nested multi-type tree coding block structure. An exception occurs when the maximum supported transform length is smaller than the width or height of the color components of the CB.In some implementations, in addition to binary or ternary partitioning, the nested pattern of FIG. 11 may further include quadtree partitioning.

[0125] One specific example of a quadtree having a nested multi-type tree coding block structure of block partitioning (including options of quadtree partitioning, binary partitioning, and ternary partitioning) for one base block is shown in FIG. 12. More specifically, FIG. 12 shows that the base block 1200 is quadtree partitioned into four square partitions 1202, 1204, 1206, and 1208. The decision to further use the multi-type tree structure and quadtree of FIG. 11 for further partitioning is made for each of the quadtree-partitioned partitions. In the example of FIG. 12, partition 1204 is not further partitioned. Partition 1202 and partition 1208 each adopt another quadtree partitioning. In partition 1202, the second-level quadtree-partitioned upper-left, upper-right, lower-left, and lower-right partitions respectively adopt the third-level partitions of a quadtree, the horizontal binary partition 1104 of FIG. 11, non-partitioning, and the horizontal ternary partition 1108 of FIG. 11. Partition 1208 adopts another quadtree partitioning, and the second-level quadtree-partitioned upper-left, upper-right, lower-left, and lower-right partitions respectively adopt the third-level partitions of the vertical ternary partition 1106, non-partitioning, non-partitioning, and the horizontal binary partition 1104 of FIG. 11. Two of the sub-partitions of the third-level upper-left partition of 1208 are further partitioned according to the horizontal binary partition 1104 and the horizontal ternary partition 1108 of FIG. 11, respectively. Partition 1206 adopts a second-level partitioning pattern into two partitions following the vertical binary partition 1102 of FIG. 11, and the two partitions are further partitioned at the third level according to the horizontal ternary partition 1108 and the vertical binary partition 1102 of FIG. 11. A fourth-level partition is further applied to one of them according to the horizontal binary partition 1104 of FIG. 11.

[0126] In the above specific example, the maximum luma transform size may be 64×64, and the maximum chroma transform size supported may be different from that of luma, for example, 32×32. Even when the exemplary CB in FIG. 12 is generally not further divided into smaller PBs and / or TBs, when the width or height of a luma coding block or chroma coding block is greater than the maximum transform width or maximum transform height, the luma coding block or chroma coding block may be automatically divided in that direction so as to satisfy the transform size limit in the horizontal and / or vertical directions.

[0127] In the specific example regarding the division of the above base block into CBs, as described above, the coding tree method may support the ability for luma and chroma to have separate block tree structures. For example, in the case of P slices and B slices, the luma CTB and chroma CTB within one CTU may share the same coding tree structure. In the case of I slices, for example, luma and chroma may have separate coding block tree structures. When separate block tree structures are applied, the luma CTB may be divided into luma CBs by one coding tree structure, and the chroma CTB may be divided into chroma CBs by another coding tree structure. This means that a CU within an I slice may be composed of a coding block of the luma component or coding blocks of two chroma components, and a CU within a P slice or B slice is always composed of coding blocks of all three color components unless the video is monochrome.

[0128] When a coding block is further divided into a plurality of transform blocks, the transform blocks therein may be ordered in the bitstream according to various orders or scanning methods. Exemplary implementations for dividing a coding block or a prediction block into transform blocks, and the coding order of the transform blocks, are described in more detail below. In some exemplary implementations, as described above, the transform division may support a plurality of shapes of transform blocks having transform block sizes in the range from, for example, 4×4 to 64×64, such as 1:1 (square), 1:2 / 2:1, and 1:4 / 4:1. In some implementations, when the coding block is smaller than or equal to 64×64, the transform block division may be applied only to the luma component such that the transform block size is the same as the coding block size in the case of the chroma block. Otherwise, when the width or height of the coding block is greater than 64, both the luma coding block and the chroma coding block may be implicitly divided into transform blocks that are multiples of min(W,64)×min(H,64) and min(W,32)×min(H,32), respectively.

[0129] In some exemplary implementations of transform block division, for both intra-coded blocks and inter-coded blocks, the coding block may be further divided into a plurality of transform blocks having a division depth up to a predefined number of levels (e.g., 2 levels). The division depth and size of the transform blocks may be related. In the case of some exemplary implementations, the mapping from the transform size at the current depth to the transform size at the next depth is shown in Table 1 below.

[0130] [Table 1]

[0131] Based on the exemplary mapping in Table 1, for a 1:1 square block, the next-level transform split can create four 1:1 square sub-transform blocks. The transform split may stop, for example, at 4×4. Thus, the transform size of the current depth of 4×4 corresponds to the same size of 4×4 at the next depth. In the example of Table 1, for a 1:2 / 2:1 non-square block, the next-level transform split can create two 1:1 square sub-transform blocks, while for a 1:4 / 4:1 non-square block, the next-level transform split can create two 1:2 / 2:1 sub-transform blocks.

[0132] In some exemplary implementations, further restrictions may be applied to the transform block splitting for the luma component of the intra-coded block. For example, for each level of transform split, all sub-transform blocks may be restricted to have equal sizes. For example, for a coding block of 32×16, the level-1 transform split creates two 16×16 sub-transform blocks, and the level-2 transform split creates eight 8×8 sub-transform blocks. In other words, in order to keep the transform units of equal size, the level-2 split must be applied to all first-level sub-blocks. An example of transform block splitting for a square block intra-coded according to Table 1 is shown in FIG. 15 together with the coding order indicated by the arrows. Specifically, 1502 shows the square coding block. The level-1 split into four equal-size transform blocks according to Table 1 is shown in 1504 together with the coding order indicated by the arrows. The level-2 split of all first-level equal-size blocks into 16 equal-size transform blocks according to Table 1 is shown in 1506 together with the coding order indicated by the arrows.

[0133] In some exemplary implementations, the above restrictions on intra - coding may not apply to the luma components of inter - coded blocks. For example, after the first level of transform splitting, any one of the sub - transform blocks can be further independently split at another level. Thus, the resulting transform blocks may or may not be blocks of the same size. An exemplary splitting of an inter - coded block into transform locks according to their coding order is shown in FIG. 16. In the example of FIG. 16, the inter - coded block 1602 is split into transform blocks at two levels according to Table 1. At the first level, the inter - coded block is split into four transform blocks of equal size. Then, only one of the four transform blocks (not all of them) is further split into four sub - transform blocks, resulting in a total of seven transform blocks having two different sizes, as shown by 1604. An exemplary coding order of these seven transform blocks is indicated by the arrows at 1604 in FIG. 16.

[0134] In some exemplary implementations, for the chroma components, some additional restrictions on the transform blocks may apply. For example, for the chroma components, the transform block size can be the same size as the coding block size, but cannot be made smaller than a predefined size, e.g., smaller than 8×8.

[0135] In some other exemplary implementations, for coding blocks where either the width (W) or the height (H) is greater than 64, both the luma coding block and the chroma coding block may be implicitly split into transform units that are multiples of min(W,64)×min(H,64) and min(W,32)×min(H,32), respectively. Here, in the present disclosure, "min(a,b)" can return the smaller value between a and b.

[0136] FIG. 17 further shows another alternative exemplary manner for splitting a coding block or a prediction block into transform blocks. As shown in FIG. 17, instead of using recursive transform splitting, a set of predefined splitting types may be applied to the coding block according to the transform type of the coding block. In the specific example shown in FIG. 17, one of six exemplary splitting types may be applied to split the coding block into various numbers of transform blocks. Such a manner of generating transform block splitting may be applied to either a coding block or a prediction block.

[0137] More specifically, the splitting manner of FIG. 17 provides up to six exemplary splitting types for any given transform type (the transform type refers to a type of primary transform such as ADST, for example). In this manner, all coding blocks or prediction blocks may have a transform splitting type assigned based on, for example, rate distortion cost. In one example, the transform splitting type assigned to a coding block or a prediction block may be determined based on the transform type of the coding block or prediction block. As shown by the six transform splitting types illustrated in FIG. 17, a particular transform splitting type may correspond to the splitting size and pattern of the transform blocks. A correspondence between various transform types and various transform splitting types may be predefined. An example where the transform splitting types that may be assigned to a coding block or a prediction block based on rate distortion cost are indicated by capital labels is shown below.

[0138] ·PARTITION_NONE: Assigns a transform size equal to the block size.

[0139] ·PARTITION_SPLIT: Assigns a transform size of half the width of the block size and half the height of the block size.

[0140] ·PARTITION_HORZ: Assigns a transform size of the same width as the block size and half the height of the block size.

[0141] ·PARTITION_VERT: Allocate a conversion size with a width of half the block size and a height equal to the block size.

[0142] ·PARTITION_HORZ4: Allocate a conversion size with a width equal to the block size and a height of one - quarter of the block size.

[0143] ·PARTITION_VERT4: Allocate a conversion size with a width of one - quarter of the block size and a height equal to the block size.

[0144] In the above example, the conversion split types shown in FIG. 17 include a uniform conversion size for all the split conversion blocks. This is not a limitation but merely an example. In some other implementations, mixed conversion block sizes may be used for the split conversion blocks in a particular split type (or pattern).

[0145] A PB (or CB, which is also called PB if it is not further split into prediction blocks) obtained from any of the above - mentioned splitting methods can become individual blocks for coding via either intra - prediction or inter - prediction. For inter - prediction in the current PB, a residual between the current block and the prediction block is generated, coded, and can be included in the coded bitstream.

[0146] Inter prediction can be performed, for example, in a single-reference mode or a multiple-reference mode. In some implementations, a skip flag may first be included in the bitstream of the current block (or at a higher level) to indicate whether the current block is inter-coded and not skipped. If the current block is inter-coded, another flag may be further included in the bitstream as a signal indicating whether a single-reference mode or a multiple-reference mode is used for predicting the current block. In the single-reference mode, one reference block may be used to generate the prediction block of the current block. In the multiple-reference mode, for example, two or more reference blocks may be used to generate the prediction block by weighted averaging. The multiple-reference mode may be referred to as a multi-reference mode, a two-reference mode, or a multiple-reference mode. One or more reference blocks may be identified using one or more reference frame indices and further using one or more corresponding motion vectors indicating a shift between the reference block and the current block at positions within, for example, horizontal and vertical pixels. For example, the inter prediction block of the current block may be generated from a single reference block identified by one motion vector in a reference frame as a prediction block in the single-reference mode, but in the multiple-reference mode, the prediction block may be generated by weighted averaging of two reference blocks in two reference frames indicated by two reference frame indices and two corresponding motion vectors. Motion vectors may be coded in various ways and may be included in the bitstream.

[0147] In some implementations, an encoding or decoding system may maintain a decoded picture buffer (DPB). Some images / pictures may be maintained in the DPB waiting to be displayed (in the decoding system), and some images / pictures in the DPB may be used as reference frames to enable inter prediction (in the decoding system or encoding system). In some implementations, the reference frames in the DPB may be tagged as either short-term or long-term references to the current image being encoded or decoded. For example, the short-term reference frames may include, in decoding order, the frames used for inter prediction of blocks within the current frame or within a predefined number (e.g., two) of subsequent video frames closest to the current frame. The long-term reference frames may include the frames within the DPB that can be used to predict image blocks within frames that are more than a predefined number of frames from the current frame in decoding order. Information regarding such tags for short-term and long-term reference frames is referred to as a reference picture set (RPS) and may be added to the header of each frame in the encoded bitstream. Each frame in the encoded video stream may be identified by a picture order counter (POC), which is numbered according to the playback sequence, either in an absolute manner or, for example, in relation to a picture group starting from an I-frame.

[0148] In some exemplary implementations, one or more reference picture lists, including identification of short-term and long-term reference frames for inter prediction, may be formed based on information within the RPS. For example, a single picture reference list may be formed for uni-directional inter prediction denoted as L0 reference (or reference list 0), and two picture reference lists may be formed for bi-directional inter prediction denoted as L0 (or reference list 0) and L1 (or reference list 1) for each of the two prediction directions. The reference frames included in the L0 list and the L1 list may be ordered in various predetermined ways. The lengths of the L0 list and the L1 list may be signaled in the video bitstream. Uni-directional inter prediction may be either in a single-reference mode or a multiple-reference mode if the multiple references for generation of a predicted block by weighted averaging in a combined prediction mode are on the same side of the block being predicted. Bi-directional inter prediction may be only in a combined mode in that the bi-directional inter prediction includes at least two reference blocks.

[0149] In some implementations, a merge mode (MM) for inter prediction may be implemented. Generally, in the case of the merge mode, one or more of the motion vectors in the single-reference prediction of the current PB or the motion vectors in the multiple-reference prediction may not be independently calculated and signaled, but may be derived from other motion vectors. For example, in an encoding system, the current motion vector of the current PB may be represented by the difference between the current motion vector and one or more other already-encoded motion vectors (referred to as reference motion vectors). Such a difference of the motion vector rather than the entire current motion vector may be encoded and included in the bitstream and linked to the reference motion vector. Correspondingly, in a decoding system, the motion vector corresponding to the current PB may be derived based on the decoded motion vector difference and the decoded reference motion vector linked thereto. As a specific form of general merge mode (MM) inter prediction, such inter prediction based on the motion vector difference is sometimes referred to as merge mode with motion vector difference (MMVD). Therefore, general MM or particularly MMVD can be implemented to improve coding efficiency by leveraging the correlation between motion vectors associated with different PBs. For example, adjacent PBs may have similar motion vectors, and thus the MVD may be small and can be efficiently coded. In other examples, the motion vectors may be temporally (between frames) correlated for blocks that are similarly positioned / arranged in space.

[0150] In some exemplary implementations, an MM flag may be included in the bitstream during the encoding process to indicate whether the current PB is in merge mode. Additionally or alternatively, an MMVD flag may be included and signaled in the bitstream during the encoding process to indicate whether the current PB is in MMVD mode. The MM and / or MMVD flag or indicator may be provided at the PB level, CB level, CU level, CTB level, CTU level, slice level, picture level, etc. In a particular example, both an MM flag and an MMVD flag may be included for the current CU, and the MMVD flag may be signaled immediately after the skip flag and the MM flag to specify whether the MMVD mode is used for the current CU.

[0151] In some exemplary implementations of MMVD, for a predicted block, a list of reference motion vectors (RMVs) or MV predictor candidates for motion vector prediction may be formed. The list of RMV candidates can include a predetermined number (e.g., two) of MV predictor candidate blocks whose motion vectors can be used to predict the current motion vector. The RMV candidate blocks may include blocks selected from adjacent blocks and / or temporal blocks (e.g., blocks located identically in the current frame during progress or in subsequent frames) within the same frame. These options represent blocks at spatial or temporal positions relative to the current block that are likely to have a motion vector similar or identical to that of the current block. The size of the list of MV predictor candidates may be predetermined. For example, the list may include two or more candidates. In order to be on the list of RMV candidates, a candidate block may, for example, need to have the same reference frame (or multiple frames) as the current block, must exist (e.g., boundary checks need to be performed if the current block is near the edge of the frame), must already have been encoded during the encoding process, and / or must already have been decoded during the decoding process. In some implementations, a list of merge candidates is available and, when the above conditions are met, is first filled with spatially adjacent blocks (scanned in a particular predefined order) and then, if space is still available within the list, may be filled with temporal blocks. Adjacent RMV candidate blocks may be selected, for example, from the blocks to the left and above the current block. The list of RMV predictor candidates may be dynamically formed at various levels (sequence, picture, frame, slice, superblock, etc.) as a dynamic reference list (DRL). The DRL may be signaled in the bitstream.

[0152] In some implementations, the actual MV predictor candidates that are used as reference motion vectors for predicting the motion vector of the current block may be signaled. If two candidates are included in the RMV candidate list, a 1-bit flag called the merge candidate flag may be used to indicate the selection of the reference merge candidate. For a current block being predicted in a composite mode, each of the multiple motion vectors predicted using the MV predictor may be associated with a reference motion vector from the merge candidate list. The encoder may determine which RMV candidate predicts the current coding block more accurately and signal the selection as an index to the DRL.

[0153] In some exemplary implementations of MMVD, after an RMV candidate is selected and used as the base motion vector predictor of the motion vector to be predicted, a motion vector difference (MVD or delta MV, representing the difference between the motion vector to be predicted and the reference candidate motion vector) may be calculated in the encoding system. Such an MVD may include information representing the magnitude and direction of the MV difference, and both of them may be signaled in the bitstream. The magnitude of the motion difference and the direction of the motion difference may be signaled in various ways.

[0154] In some exemplary implementations of MMVD, a distance index may be used to specify the magnitude information of the motion vector difference and indicate one of a set of predefined offsets representing a predefined motion vector difference from the starting point (reference motion vector). Then, an MV offset corresponding to the signaled index may be added to either the horizontal or vertical component of the starting (reference) motion vector. Whether the horizontal or vertical component of the reference motion vector should be offset may be determined by the direction information of the MVD. An exemplary predefined relationship between the distance index and the predefined offset is specified in Table 2.

[0155] [Table 2]

[0156] In some exemplary implementations of MMVD, a direction index is further signaled and can be used to represent the direction of the MVD relative to the reference motion vector. In some implementations, the direction may be limited to either the horizontal or vertical direction. An exemplary 2-bit direction index is shown in Table 3. In the example of Table 3, the interpretation of the MVD can vary according to the start / reference MV information. For example, if the start / reference MV corresponds to a single prediction block, or both reference frame lists correspond to a bi-prediction block that points to the same side of the current picture (i.e., the POCs of both reference pictures are both greater than the POC of the current picture or both are less than the POC of the current picture), the signs in Table 3 can specify the sign (direction) of the MV offset added to the start / reference MV. If the start / reference MV corresponds to a bi-prediction block having two reference pictures on different sides of the current picture (i.e., the POC of one reference picture is greater than the POC of the current picture and the POC of the other reference picture is less than the POC of the current picture), and the difference between the reference POC in picture reference list 0 and the current frame is greater than the difference between the reference POC in picture reference list 1 and the current frame, the signs in Table 3 can specify the sign of the MV offset added to the reference MV corresponding to the reference picture in picture reference list 0, and the sign of the offset of the MV corresponding to the reference picture in picture reference list 1 can have the opposite value (opposite sign of the offset). Otherwise, if the difference between the reference POC in picture reference list 1 and the current frame is greater than the difference between the reference POC in picture reference list 0 and the current frame, the signs in Table 3 can specify that the sign of the MV offset added to the reference MV associated with picture reference list 1 and the sign of the offset to the reference MV associated with picture reference list 0 have opposite values.

[0157]

Table 3

[0158] In some exemplary implementations, the MVD may be scaled according to the difference in POC in each direction. If the difference in POC in both lists is the same, no scaling is required. Otherwise, if the difference in POC in reference list 0 is greater than the difference in reference list 1, the MVD of reference list 1 is scaled. If the POC difference in reference list 1 is greater than that in list 0, the MVD of list 0 may similarly be scaled. When the starting MV is singly predicted, the MVD is added to the available or reference MV.

[0159] In some exemplary implementations of MVD coding and signaling for bidirectional composite prediction, in addition to or instead of coding and signaling two MVDs separately, symmetric MVD coding can be implemented such that only one MVD requires signaling and the other MVD can be derived from the signaled MVD. In such implementations, motion information including reference picture indices for both list 0 and list 1 is signaled. However, for example, only the MVD associated with reference list 0 is signaled and the MVD associated with reference list 1 is not signaled but derived. Specifically, at the slice level, a flag called "mvd_l1_0_flag" can be included in the bitstream to indicate whether reference list 1 is signaled in the bitstream. If this flag is 1, indicating that reference list 1 is equal to zero (and thus not signaled), the bidirectional prediction flag called "BiDirPredFlag" may be set to 0, which means there is no bidirectional prediction. Otherwise, if mvd_l1_0_flag is 0 and the closest reference picture in list 0 and the closest reference picture in list 1 form a pair of reference pictures in the forward and reverse directions or a pair of reference pictures in the reverse and forward directions, BiDirPredFlag may be set to 1 and the reference pictures in list 0 and list 1 are both short-term reference pictures. Otherwise, BiDirPredFlag is set to 0. A BiDirPredFlag of 1 may indicate that a symmetric mode flag is additionally signaled in the bitstream. The decoder may extract the symmetric mode flag from the bitstream if BiDirPredFlag is 1. The symmetric mode flag can be signaled, for example, at the CU level (if necessary) and can indicate whether the symmetric MVD coding mode is used for the corresponding CU.When the symmetric mode flag is 1, it indicates the use of the symmetric MVD coding mode, and only the reference picture indices for both list 0 and list 1 (referred to as "mvp_l0_flag" and "mvp_l1_flag") are signaled by the MVD associated with list 0 (referred to as "MVD0"), indicating that the other motion vector difference "MVD1" should be derived rather than signaled. For example, MVD1 can be derived as -MVD0. Thus, in an exemplary symmetric MVD mode, only one MVD is signaled. In some other exemplary implementations for MV prediction, a harmonic method may be used for both single-reference mode MV prediction and composite-reference mode MV prediction, for implementing the general merge mode MMVD, and several other types of MV prediction. Various syntax elements can be used to signal how the MV of the current block is predicted.

[0160] For example, in the case of the single-reference mode, the following MV prediction modes can be signaled.

[0161] Use one of the motion vector predictors (MVPs) within the list indicated by the DRL (dynamic reference list) index directly without NEARMV-MVD.

[0162] Use one of the motion vector predictors (MVPs) within the list signaled by the DRL index as a NEWMV-reference and apply a delta to the MVP (e.g., use MVD).

[0163] Use a motion vector based on global motion parameters at the frame level for GLOBALMV.

[0164] Similarly, in the case of the composite-reference inter-prediction mode that uses two reference frames corresponding to the two MVs to be predicted, the following MV prediction modes can be signaled.

[0165] For each of the two MVs to be predicted in NEAR_NEARMV, one of the motion vector predictors (MVPs) in the list signaled by the DRL index without MVD is used.

[0166] NEAR_NEWMV - To predict the first of the two motion vectors, one of the motion vector predictors (MVPs) in the list signaled by the DRL index without MVD is used as the reference MV, and to predict the second of the two motion vectors, one of the motion vector predictors (MVPs) in the list signaled by the DRL index is used as the reference MV in combination with the additionally signaled delta MV (MVD).

[0167] NEW_NEARMV - To predict the second of the two motion vectors, one of the motion vector predictors (MVPs) in the list signaled by the DRL index without MVD is used as the reference MV, and to predict the first of the two motion vectors, one of the motion vector predictors (MVPs) in the list signaled by the DRL index is used as the reference MV in combination with the additionally signaled delta MV (MVD).

[0168] NEW_NEWMV - One of the motion vector predictors (MVPs) in the list signaled by the DRL index is used as the reference MV, and it is used in combination with the additionally signaled delta MV to predict each of the two MVs.

[0169] GLOBAL_GLOBALMV - Based on the global motion parameters at the frame level, the MV from each reference is used.

[0170] Thus, the term "NEAR" refers to MV prediction that uses a reference MV without MVD as a general merge mode, while the term "NEW" refers to MV prediction that uses a reference MV and offsets it with an MVD signaled like in the MMVD mode. In the case of composite inter prediction, both the reference-based motion vector and the above-mentioned motion vector delta can be used even if they are correlated and such correlation can be utilized to reduce the amount of information required to signal the two motion vector deltas, and they may generally be different or independent between the two references. In such a situation, joint signaling of the two MVDs can be implemented and shown in the bitstream.

[0171] The above-mentioned dynamic reference list (DRL) can be used to be dynamically maintained and hold a set of indexed motion vectors regarded as candidate motion vector predictors.

[0172] In some exemplary implementations, a predefined resolution of the MVD may be allowed. For example, a motion vector accuracy (or precision) of 1 / 8 pixel may be allowed. The MVDs described above in various MV prediction modes can be constructed and signaled in various ways. In some implementations, various syntax elements can be used to signal the above-mentioned motion vector differences within reference frame list 0 or list 1.

[0173] For example, a syntax element called "mv_joint" can specify which components of the associated motion vector difference are non-zero. In the case of MVD, this is signaled together for all non-zero components. For example, mv_joint has the following values: 0 can indicate that there is no non-zero MVD along either the horizontal or vertical direction, 1 can indicate that there is a non-zero MVD only along the horizontal direction, 2 can indicate that there is a non-zero MVD only along the vertical direction, 3 can indicate that there are non-zero MVDs along both the horizontal and vertical directions.

[0174] When the "mv_joint" syntax element for MVD signals that there are no non-zero MVD components, no further MVD information can be signaled. However, when the "mv_joint" syntax signals that there is one or two non-zero components, additional syntax elements can be further signaled for each of the non-zero MVD components, as described below.

[0175] For example, a syntax element called "mv_sign" can be used to further specify whether the corresponding motion vector difference component is positive or negative.

[0176] In other examples, a syntax element called "mv_class" can be used to specify the class of the motion vector difference between a predefined set of classes of the corresponding non-zero MVD components. The predefined classes of motion vector differences can be used, for example, to separate the continuous magnitude space of the motion vector differences into non-overlapping ranges, each range corresponding to an MVD class. Thus, the signaled MVD class indicates the magnitude range of the corresponding MVD component. In the exemplary implementation shown in Table 4 below, higher classes correspond to motion vector differences with larger magnitude ranges. In Table 4, the symbol (n,m] is used to represent the range of motion vector differences greater than n pixels and less than or equal to m pixels.

[0177]

Table 4

[0178] In some other examples, a syntax element called "mv_bit" may be further used to specify the integer part of an offset between the size of the start of an MV class size range signaled corresponding to a non-zero motion vector difference component. Thus, mv_bit may indicate the size or amplitude of the MVD. The number of bits required for "my_bit" to signal the full range of each MVD class may vary depending on the MV class. For example, MV_CLASS 0 and MV_CLASS 1 in the implementation of Table 4 may require only a single bit to indicate an integer pixel offset of 1 or 2 from a starting MVD of 0, and each higher MV_CLASS in the exemplary implementation of Table 4 may require one more bit for "mv_bit" progressively than the previous MV_CLASS.

[0179] In some other examples, a syntax element called "mv_fr" may be further used to specify the first two fractional bits of the motion vector difference of the corresponding non-zero MVD component, and a syntax element called "mv_hp" may be used to specify the third fractional bit (high resolution bit) of the motion vector difference of the corresponding non-zero MVD component. The 2-bit "mv_fr" essentially provides an MVD resolution of 1 / 4 pixel, but the "mv_hp" bit may further provide a resolution of 1 / 8 pixel. In some other implementations, two or more "mv_hp" bits may be used to provide an MVD pixel resolution finer than 1 / 8 pixel. In some exemplary implementations, one or more additional flags may be signaled at one or more of various levels to indicate whether an MVD resolution of 1 / 8 pixel or higher is supported. If the MVD resolution is not applied to a particular coding unit, the above syntax elements for the corresponding unsupported MVD resolution may not be signaled.

[0180] In some of the above exemplary implementations, the fractional resolution may be independent of different classes of MVDs. In other words, regardless of the magnitude of the motion vector difference, similar options for the motion vector resolution may be provided using a predefined number of "mv_fr" and "mv_hp" bits to signal the fractional MVD of non-zero MVD components.

[0181] However, in some other exemplary implementations, the resolution of the motion vector difference in various MVD magnitude classes may be distinguished. Specifically, high-resolution MVDs for large MVD magnitudes in higher MVD classes may not result in a statistically significant improvement in compression efficiency. Thus, the MVD may be coded with a reduced resolution (integer pixel resolution or fractional pixel resolution) for larger MVD magnitude ranges corresponding to higher MVD magnitude classes. Similarly, the MVD may generally be coded with a reduced resolution (integer pixel resolution or fractional pixel resolution) for larger MVD values. Such MVD class-dependent or MVD magnitude-dependent MVD resolution is sometimes generally referred to as adaptive MVD resolution, amplitude-dependent adaptive MVD resolution, or magnitude-dependent MVD resolution. The term "resolution" may also be further referred to as "pixel resolution". Adaptive MVD resolution can be implemented in various ways as described by the following exemplary implementations to achieve generally better compression efficiency. In particular, reducing the number of signaling bits by aiming for less accurate MVDs may not significantly increase the inter-prediction residual coding efficiency of blocks with large or high-class MVDs, even if the MVD resolution of large or high-class MVDs is non-adaptively processed at the same level as that of small or low-class MVDs, due to the statistical observation that it can be larger than the additional bits required to code the inter-prediction residual as a result of such less accurate MVDs. In other words, using a higher MVD resolution for large or high-class MVDs may not result in more coding gain than using a lower MVD resolution.

[0182] In some common exemplary implementations, the pixel resolution or accuracy of the MVD may decrease or may not increase as the MVD class increases. Decreasing the pixel resolution of the MVD corresponds to a coarser MVD (or a larger step from one MVD level to the next). In some implementations, the correspondence between the MVD pixel resolution and the MVD class may be specified, predefined, or preconfigured, and thus need not be signaled in the encoded bitstream.

[0183] In some exemplary implementations, each of the MV classes in Table 3 may be associated with a different MVD pixel resolution.

[0184] In some exemplary implementations, each MVD class may be associated with a single allowed resolution. In some other implementations, one or more MVD classes may be associated with two or more optional MVD pixel resolutions. Thus, additional signaling can follow the signal within the bitstream of the current MVD component having such an MVD class to indicate which optional pixel resolution is selected for the current MVD component.

[0185] In some exemplary implementations, the adaptively allowed MVD pixel resolutions may include, but are not limited to, 1 / 64 pel (pixel), 1 / 32 pel, 1 / 16 pel, 1 / 8 pel, 1 / 4 pel, 1 / 2 pel, 1 pel, 2 pel, 4 pel... in descending order of resolution. Thus, each of the ascending MVD classes can be associated with one of these resolutions in non-ascending order. In some implementations, an MVD class may be associated with two or more of the above resolutions, and the higher resolution may be less than or equal to the lower resolution of the preceding MVD class. For example, if MV_CLASS_3 in Table 4 can be associated with optional 1 pel and 2 pel resolutions, the highest resolution that MV_CLASS_4 in Table 4 can be associated with is 2 pel. In some other implementations, the highest allowed resolution of an MV class may be higher than the lowest allowed resolution of the preceding (lower) MV class. However, the average of the allowed resolutions for ascending MV classes may be non-ascending only.

[0186] In some implementations, if fractional pixel resolutions higher than 1 / 8 pel are allowed, the "mv_fr" and "mv_hp" signaling can be extended to correspond to more than 3 fractional bits in total.

[0187] In some exemplary implementations, sub-pixel resolution may be allowed only for MVD classes below a threshold MVD class. For example, sub-pixel resolution may be allowed only for MVD-CLASS 0 and may not be allowed for all other MV classes in Table 4. Similarly, sub-pixel resolution may be allowed only for MVD classes below any one of the other MV classes in Table 4. For other MVD classes above the threshold MVD class, only the integer pixel resolution of the MVD is allowed. In this way, sub-resolution signaling, such as one or more of the "mv-fr" and / or "mv-hp" bits, need not be signaled for an MVD signaled in an MVD class above the threshold MVD class. For an MVD class having a resolution less than 1 pixel, the number of bits of "mv-bit" signaling may be further reduced. For example, for MV_CLASS_5 in Table 4, the range of the MVD pixel offset is (32,64], and thus 5 bits are required to signal the entire range at 1 pel resolution. However, if MV_CLASS_5 is associated with a 2 pel MVD resolution (a resolution lower than 1 pixel resolution), 4 bits instead of 5 bits may be required for "mv-bit", and neither "mv-fr" nor "mv-hp" need be signaled following the signaling of "mv_class" as MV-CLASS_5.

[0188] In some exemplary implementations, fractional pixel resolution may be allowed only for MVDs having an integer value less than a threshold integer pixel value. For example, fractional pixel resolution may be allowed only for MVDs smaller than 5 pixels. Corresponding to this example, fractional resolution may be permitted for MV_CLASS_0 and MV_CLASS_1 in Table 4 and may not be allowed for all other MV classes. In other examples, fractional pixel resolution may be allowed only for MVDs smaller than 7 pixels. Corresponding to this example, fractional resolution may be permitted for MV_CLASS_0 and MV_CLASS_1 (having a range less than 5 pixels) in Table 4 and may not be allowed for MV_CLASS_3 and above (having a range greater than 5 pixels). For MVDs belonging to MV_CLASS_2 whose pixel range includes 5 pixels, the fractional pixel resolution of the MVD may be allowed depending on the "mv-bit" value. When the "m-bit" value is signaled as 1 or 2 (such that the integer part of the signaled MVD is 5 or 6 as calculated as the start of the pixel range of MV_CLASS_2 having an offset 1 or 2 as indicated by the "m-bit"), fractional pixel resolution may be allowed. Otherwise, when the "mv-bit" value is signaled as 3 or 4 (such that the integer part of the signaled MVD is 7 or 8), fractional pixel resolution may not be allowed.

[0189] In some other implementations, for MV classes above a threshold MV class, only a single MVD value may be allowed. For example, such a threshold MV class may be MV_CLASS 2. Thus, above MV_CLASS_2, only having a single MVD value and not having sub-pixel resolution may be allowed. The single allowed MVD value for these MV classes may be pre-defined. In some examples, the single allowed value may be the upper limit value of the respective range of these MV classes in Table 4. For example, MV_CLASS_2 to MV_CLASS_10 may be above the threshold class of MV_CLASS 2, and the single allowed MVD value for these classes may be pre-defined as 8, 16, 32, 64, 128, 256, 512, 1024, and 2048 respectively. In some other examples, the single allowed value may be the median value of the respective range of these MV classes in Table 4. For example, MV_CLASS_2 to MV_CLASS_10 may exceed the class threshold, and the single allowed MVD value for these classes may be pre-defined as 3, 6, 12, 24, 48, 96, 192, 384, 768, and 1536 respectively. Any other value within the range may also be defined as the single allowed resolution for each MVD class.

[0190] In the above implementation, when the signaled "mv_class" is above a pre-defined MVD class threshold, only the "mv_class" signaling is sufficient to determine the MVD value. Next, the size and direction of the MVD are determined using "mv_class" and "mv_sign".

[0191] Therefore, when the MVD is signaled for only one reference frame (either from reference frame list 0 or list 1, but not both), or is signaled together for two reference frames, the accuracy (or resolution) of the MVD may depend on the relevant motion vector difference class in Table 3 and / or the size of the MVD.

[0192] In some other implementations, the pixel resolution or accuracy of the MVD may decrease or may not increase as the MVD size increases. For example, the pixel resolution may depend on the integer part of the MVD size. In some implementations, fractional pixel resolution may only be allowed for MVD sizes below the amplitude threshold. In the case of a decoder, the integer part of the MVD size may first be extracted from the bitstream. Then, the pixel resolution may be determined, and then a determination may be made as to whether any fractional MVD exists in the bitstream and needs to be analyzed (e.g., if sub-pixel resolution is not allowed for a particular extracted MVD integer size, the bitstream that requires extraction may not contain fractional MVD bits). The above exemplary implementations regarding MVD class-dependent adaptive MVD pixel resolution apply to MVD size-dependent adaptive MVD pixel resolution. In a specific example, an MVD class that exceeds or includes a size threshold may be allowed to have only one predefined value.

[0193] The various exemplary implementations described above apply to single-reference mode. These implementations also apply to the exemplary NEW_NEARMV, NEAR_NEWMV, and / or NEW_NEWMV modes in composite prediction under MMVD. These implementations generally apply to the adaptive resolution of any MVD.

[0194] In a particularly exemplary implementation for adaptive MVD pixel resolution, the MVD pixel resolution for MVD sizes less than 1 may be fractional, and for MV classes of MV_CLASS_1 or higher, only a single MVD size equal to the end value of the corresponding MVD size range in Table 4 may be allowed. In such an example, the allowed MVD values are shown in Table 4 for allowed fractional pixel resolutions of 1 / 8, 1 / 4, or 1 / 2 pixel.

[0195]

Table 5

[0196] In the case of a coding block, whether an adaptive MVD pixel resolution is used can be signaled (derived) explicitly or implicitly. If it is signaled that an adaptive MVD pixel resolution is not used, it indicates that different MVD classes can follow the MVD ranges shown in Table 4, and a non - adaptive MVD pixel resolution can be defined or signaled. Such a non - adaptive resolution can be fractional (e.g., 1 / 8, 1 / 4, or 1 / 2 pixel) or non - fractional (e.g., 1, 2, 4,... pixels) and is applicable to all MVD classes. The non - adaptive resolution basically determines the number of bits required to signal mv_bit, mv_fr, and mv_hp as described above. If the non - adaptive resolution is fractional, it can only determine the number of bits required to signal mv_fr and mv_hp for all MVD classes (regardless of the MVD class), and the number of bits for signaling Mv_bit can depend on the MVD class.

[0197] When it is signaled that an adaptive MVD pixel resolution is used, the allowed MVD levels or values may be pre - defined in an adaptive way such as those shown in Table 5 or may be signaled. For example, they can be signaled in the bitstream in various ways depending on the particular scheme for adaptive MVD resolution. In the example of Table 5, a set of signaling syntax can be used to indicate a fractional resolution (e.g., 1 / 8 pixel) and a size threshold (e.g., 1 - pixel MVD size) to which the signaled fractional resolution applies. Other sets of syntax (which can be more complex) can be used to signal other adaptive MVD resolution schemes. Such an indication of an adaptive MVD pixel resolution scheme can be signaled at one of various coding levels such as sequence level, picture level, frame level, slice level, super - block level, or coding - block level.

[0198] In some exemplary implementations, the overall adaptive MVD pixel resolution scheme, including but not limited to those shown in Table 5, may be defined or signaled at a particular coding level (e.g., sequence level, picture level, frame level, slice level, superblock level). Such an adaptive MVD pixel resolution scheme may be further modified at the same or other coding levels such that the allowable MVD pixel resolution values for various MVD classes may be adjusted or modified at the same or other coding levels. If no adjustment is made at a particular coding level, the signaled or pre - defined adaptive MVD pixel resolution scheme is applied without modification. For example, the overall adaptive MVD pixel resolution scheme may be defined or signaled at the frame level, while the adjustment may be made at one or more superblock levels or coding block levels, and vice versa.

[0199] Such an adjustment may be implemented as a limitation of MVD accuracy or an expansion of MVD accuracy. Information related to such an adjustment may be pre - defined or signaled. A pre - defined adjustment may be applied to all coding blocks. Alternatively, a pre - defined adjustment may be activated at various coding levels by signaling.

[0200] In some implementations, such an adjustment may be embodied as a maximum allowable MVD accuracy. For a particular coding block, when the adaptive MVD resolution is applied, such a maximum allowable MVD accuracy may be different from the MVD pixel accuracy of the adaptive MVD pixel resolution scheme specified / signaled / derived at the picture level, or superblock level, or coded block level as described above. In such a situation, the allowable MVD resolution values for various MVD classes may be determined by taking both the allowable values specified or derived by the adaptive MVD pixel resolution scheme and the maximum allowable MVD accuracy.

[0201] For example, assume that the adaptive MVD pixel resolution method in Table 5 is predefined / signaled / derived for a specific coding level. Further assume that the maximum allowable accuracy is 1 / 4 pixel. This means that for any MVD class, an accuracy of more than 1 / 8 pixel is not allowed, regardless of the adaptive MVD resolution related to Table 5. Next, by applying the maximum allowable pixel accuracy indiscriminately as a limitation in Table 5, the allowable MVD pixel levels or values for various MVD classes can be modified as follows:

[0202]

Table 6

[0203] Defining / signaling a maximum allowable MVD pixel accuracy of 1 / 4 pixel such that an accuracy of more than 1 / 8 pixel is not allowed for all MVD classes is just one example. In other examples, the maximum allowable pixel accuracy can be defined / signaled as 1 / 2 pixel. The allowable MVD values corresponding to the above MV_CLASS_0 can be (1 / 2, 1, 2) for an adaptive resolution method having fractional pixel resolutions of 1 / 8 pixel, 1 / 4 pixel, and 1 / 2 pixel, and can be (1, 2) for a pixel resolution of 1 pixel.

[0204] As illustrated in the above implementation form in relation to Table 6, when an adaptive resolution method defined / signaled / derived at a specific coding level is used and the adaptive MVD resolution is applied with an additionally defined / signaled maximum allowable MVD accuracy at the same or a different coding level, it may be required / limited that such a maximum allowable MVD accuracy is not greater than the MVD resolution of the adaptive resolution method. In other words, the actually applied MVD accuracy derived considering both the adaptive resolution method and the maximum allowable accuracy is clipped by the MVD resolution in the adaptive resolution method (i.e., the maximum allowable accuracy is not effective if it is greater than the resolution defined / signaled / derived from the adaptive resolution method).

[0205] However, in some other implementations, such clipping may not be required, and the defined / signaled maximum allowable MVD accuracy may control the actual MVD resolution for at least some MVD classes. In these implementations, when the adaptive MVD resolution is applied (as indicated by definition / signaling / derivation at various coding levels as described above), the adjustment of the MVD level for at least some MVD classes may involve increasing rather than limiting the adaptive MVD resolution defined / signaled / derived in an adaptive MVD pixel resolution scheme such as that associated with Table 5. For example, the adjustment may be made to allow higher accuracy than that specified / signaled / derived from the adaptive resolution scheme for MVD classes below the defined or signaled threshold MVD class level. Such higher accuracy may be defined / signaled as the maximum allowable MVD accuracy as described above. Such maximum allowable accuracy may be imposed below the threshold MVD class level regardless of the specified / signaled / derived MVD resolution in the adaptive resolution scheme. Specifically, such a threshold MVD class level may (but need not) be MV_CLASS_0 (or the lowest MVD class level of a set of MVD classes such as the set of MVD classes in Table 5). The maximum allowable pixel accuracy may be pre-defined / signaled. The maximum allowable pixel accuracy may be fractional. For a specific example, if the threshold MVD class is MV_CLASS_0 in the adaptive resolution scheme of Table 5, the MVD pixel resolution of MV_CLASS_0 is a non-fractional 1 pixel, and the maximum allowable fractional pixel accuracy for adjustment is 1 / 8, 1 / 4, or 1 / 2 pixel, the adjustable MVD values are as follows:

[0206]

Table 7

[0207] In some exemplary implementations that replace the implementation form of Table 7, a threshold MVD amplitude may be used instead of the threshold MVD class level. In these implementations, higher accuracy may be imposed by the maximum allowable MVD accuracy specified / signaled for MVDs having a magnitude below the threshold MVD amplitude rather than the threshold MVD class level. In such implementations, in addition to the mv_class information, the mv_bit information may be signaled early enough within the video stream so that the magnitude of the MVD can be determined over time to determine the allowable MVD value. For example, by replacing the threshold MVD class with a threshold MVD magnitude of 1 / 2 pixel and still assuming that the adaptive MVD resolution of MV_CLASS_0 is 1 pixel in the adaptive resolution scheme, Table 7 becomes Table 8 below:

[0208]

Table 8

[0209] In some other exemplary implementations, the above adjustment may include allowing only certain accuracies and lower accuracies (e.g., fractional accuracies of 1 / 8, 1 / 4, or 1 / 2 or less) when the magnitude of the MVD is below the threshold MVD magnitude. In such implementations, again, in addition to the mv_class information, the mv_bit information may be signaled early enough within the video stream so that the magnitude of the MVD can be determined over time to determine the allowable MVD value.

[0210] In such implementations, no additional resolution can be imposed on the MVD values derived from the adaptive resolution scheme (such as Table 5). Instead, when the amplitude of the MVD is greater than the threshold MVD magnitude, MVD values associated with a resolution above the defined / signaled accuracy level may not be allowed. Here too, in the example of Table 5, further assume that for MVD magnitudes greater than the threshold MVD magnitude of 1 / 2 pixel, MVD values associated with a resolution above the defined / signaled accuracy of 1 / 8 pixel accuracy are not allowed. Next, Table 5 is adjusted as follows:

[0211]

Table 9

[0212] In particular, as described above, the allowable MVD values for MV_CLASS_0 and fractional resolutions of 1 / 8 pixel (1 / 8, 2 / 8, 3 / 8, 1 / 2, 5 / 8, 6 / 8, 7 / 8, 1, 2) are adjusted to (1 / 8, 2 / 8, 3 / 8, 1 / 2, 6 / 8, 1, 2), and the MVD values associated with 1 / 8 precision are allowed and are only maintained below the threshold MVD magnitude of 1 / 2 pixel. When exceeding the size of 1 / 2 pixel, MVD values associated with 1 / 8 precision such as 5 / 8 pixel value and 7 / 8 pixel value are not allowed.

[0213] Similarly, in the example of Table 5, assume that when the MVD magnitude is greater than the threshold magnitude of 1 / 2 pixel, MVD values associated with resolutions above the defined / signaled precision of 1 / 4 pixel are not allowed. Next, adjust Table 5 as follows:

[0214]

Table 10

[0215] In some of the above implementation forms, the threshold MVD magnitude may be 2 pixels or less, such as the threshold of the size of 1 / 2 pixel given in the above example.

[0216] The above exemplary implementation forms are described for a specific MVD, regardless of whether the inter prediction mode is in a single reference mode or a composite reference mode. In some other exemplary implementation forms in the composite reference mode where the MV is predicted by multiple reference frames, a set of definitions / signaling may be used to indicate whether an adaptive MVD resolution is applied and to which of the multiple reference frames it is applied among the reference frames to which it is applied.

[0217] In some exemplary implementations, when the MVD is signaled for multiple reference frames, one (or more) flags / indexes may be signaled to indicate whether the adaptive MVD resolution is applied.

[0218] For example, when the MVD is signaled for multiple reference frames (e.g., in the NEW_NEWMV mode as described above, or in other composite reference inter-prediction modes), one flag / index may be signaled within the video stream to indicate whether the adaptive MVD resolution is applied to the MVD signaling for all of the multiple reference frames. If this flag / index is 1 (or 0), this indicates that the adaptive MVD resolution is applied to the MVD signaling for all of the multiple reference frames. Otherwise, if this flag / index is 0 (or 1), the adaptive MVD coding is not applied to the MVD signaling for any of the multiple reference frames. In such implementations, for multiple inter-prediction reference frames, the adaptive MVD resolution is applied in an all-or-nothing manner.

[0219] In some other examples, when the MVD is signaled for multiple reference frames (e.g., in the NEW_NEWMV mode as described above for the two-reference-frame composite inter-prediction mode or other composite inter-prediction modes), one flag / index may be signaled separately for each reference frame to indicate whether the adaptive MVD resolution is applied to that reference frame. In such implementations, whether the adaptive MVD resolution is applied may be determined individually for each of the reference frames. The determination of whether to apply the adaptive MVD resolution may be made independently by the encoder for each of the multiple reference frames and signaled separately within the video stream.

[0220] In some exemplary implementations, when the MVD is signaled for multiple reference frames, for each of the multiple reference frames, if the MVD of that reference frame is non-zero, one flag / index may be signaled to indicate whether the adaptive MVD resolution is applied to that reference frame. Otherwise, the flag / index need not be signaled. In other words, if the MVD of a particular reference frame is signaled / indicated as zero, there is no need to determine whether the adaptive MVD resolution is applied, and thus no corresponding signaling within the video stream is necessary. However, in such implementations, an indication that the MVD is zero needs to be signaled before the determination regarding whether the adaptive resolution is applied is made.

[0221] Turning further to the signaling of the MVD resolution, in some exemplary implementations, a flag / index may be signaled to explicitly indicate the MVD resolution of the current coding block, and the context used to entropy code such a flag / index may depend on the MVD class associated with the MVD. Such a flag / index may be either the MVD resolution used to derive an adaptive resolution scheme such as Table 5, or the maximum allowable MVD accuracy described above.

[0222] In some exemplary implementations regarding the signaling of the MVD resolution, the various components of the MVD may be signaled separately. The MVD may include, for example, a horizontal component and a vertical component. For each of the horizontal and vertical components of the MVD, a flag / index may be signaled to indicate the MVD resolution of the horizontal and vertical components, respectively.

[0223] In some exemplary implementations, the MVD resolution flag / index may be signaled after the MVD class information. Depending on the values of the signaled MVD class information such as MV_CLASS_0, MV_CLASS_1, MV_CLASS_2, etc., a context value may be derived and used to signal the MVD resolution flag / index for indicating the MVD resolution. In other words, the syntax(es) for signaling the MVD resolution may be entropy-coded using different contexts for different MVD classes or different MVD class groups.

[0224] FIG. 18 shows a flowchart 1800 of an exemplary method according to the principles underlying the above implementation for adaptive MVD resolution. The flow of the exemplary decoding method starts at S1801. At S1810, a video stream is received. At S1820, it is determined that a video block is inter-coded based on a predicted block and a motion vector (MV), and the MV should be derived from a reference motion vector (RMV) and a motion vector difference (MVD) of the video block. In response to determining that the MVD is coded at an adaptive MVD pixel resolution at S1830, the reference MVD pixel accuracy of the current video block is determined, the maximum allowable MVD pixel accuracy is identified, a set of allowable MVD levels for the current video block is determined based on the reference MVD pixel accuracy and the maximum allowable MVD pixel accuracy, and the MVD from the video stream is derived according to at least one MVD parameter signaled in the video stream for the current video block and the set of allowable MVD levels. The exemplary method stops at S1899.

[0225] FIG. 19 shows a flowchart 1900 of an exemplary method according to the principle underlying the above implementation for adaptive MVD resolution. The flow of the exemplary decoding method starts at S1901. At S1910, a video stream is received. At S1920, it is determined that the current video block is inter-coded and associated with a plurality of reference frames. At S1930, based on the signaling within the video stream, it is further determined whether an adaptive motion vector difference (MVD) pixel resolution is applied to at least one of the plurality of reference frames. The exemplary method stops at S1999.

[0226] FIG. 20 shows a flowchart 2000 of an exemplary method according to the principle underlying the above implementation for adaptive MVD resolution. The flow of the exemplary decoding method starts at S2001. At S2010, a video stream is received. At S2020, it is determined that the video block is inter-coded based on a prediction block and a motion vector (MV), and the MV should be derived from a reference motion vector (RMV) and a motion vector difference (MVD) of the video block. At S2030, the current MVD class of the MVD within a predefined set of MVD classes is determined. At S2040, at least one context for entropy decoding at least one explicit signaling within the video stream is derived based on the current MVD class, and the at least one explicit signaling is included in the video stream to specify the MVD pixel resolution of at least one component of the MVD. At S2050, at least one explicit signaling from the video stream is entropy decoded using at least one context to determine the MVD pixel resolution of at least one component of the MVD. The exemplary method stops at S2099.

[0227] In embodiments and implementations of the present disclosure, if desired, any steps and / or operations may be combined or arranged in any amount or order. Two or more of the steps and / or operations may be performed in parallel. The embodiments and implementations of the present disclosure may be used separately or combined in any order. Further, each of the methods (or embodiments), encoders, and decoders may be implemented by a processing circuit (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored on a non-transitory computer-readable medium. The embodiments of the present disclosure may be applied to luma blocks or chroma blocks. The term block may be interpreted as a prediction block, a coding block, or a coding unit, i.e., a CU. The term block here may also be used to refer to a transform block. When referring to block size in the following items, it may refer to either the width or height of the block, or the maximum value of the width and height, or the minimum value of the width and height, or the size of the area (width * height), or the aspect ratio of the block (width:height or height:width).

[0228] The techniques described above can be implemented as computer software using computer-readable instructions and physically stored on one or more computer-readable media. For example, FIG. 21 shows a computer system (2100) suitable for implementing certain embodiments of the disclosed subject matter.

[0229] The computer software can be coded using any suitable machine code or computer language that can undergo mechanisms such as assembly, compilation, linking, etc., and create code containing instructions that can be executed directly by one or more central processing units (CPUs), graphics processing units (GPUs), etc., or via interpretation, microcode execution, etc.

[0230] The command can be executed on various types of computers or computer components, including, for example, personal computers, tablet computers, servers, smartphones, game consoles, Internet of Things devices, and the like.

[0231] The components shown in FIG. 21 with respect to the computer system (2100) are essentially exemplary and do not imply any limitation regarding the use or function scope of the computer software implementing the embodiments of the present disclosure. The configuration of the components should not be construed as having any dependency or requirement regarding any one or combination of the components shown in the exemplary embodiment of the computer system (2100).

[0232] The computer system (2100) may include a specific human interface input device. Such a human interface input device can respond to input by one or more human users via, for example, tactile input (such as keystrokes, swipes, movements of a data glove), audio input (such as voice, clapping), visual input (such as gestures), and olfactory input (not described). The human interface device can also be used to capture specific media that is not necessarily directly related to conscious input by humans, such as audio (voice, music, ambient sound, etc.), images (scanned images, photographic images obtained from a still image camera, etc.), video (2D video, 3D video including stereoscopic video, etc.).

[0233] The input human interface device may include one or more of a keyboard (2101), a mouse (2102), a trackpad (2103), a touch screen (2110), a data glove (not shown), a joystick (2105), a microphone (2106), a scanner (2107), and a camera (2108) (only one of each is depicted).

[0234] The computer system (2100) may also include certain human interface output devices. Such human interface output devices may, for example, stimulate the senses of one or more human users via tactile output, sound, light, and smell / taste. Such human interface output devices include tactile output devices (e.g., tactile feedback by a touch screen (2110), a data glove (not shown), or a joystick (2105), although there may also be tactile feedback devices that do not function as input devices), audio output devices (such as speakers (2109), headphones (not depicted)), visual output devices (such as screens (2110) including CRT screens, LCD screens, plasma screens, OLED screens, some of which may output two-dimensional visual output or three-dimensional or higher output via means such as stereographic output, virtual reality glasses (not depicted), holographic displays, and smoke tanks (not depicted), regardless of the presence or absence of a touch screen input function and regardless of the presence or absence of a tactile feedback function), and may also include a printer (not depicted).

[0235] The computer system (2100) can also include human-accessible storage devices and associated media such as an optical medium including a CD / DVD ROM / RW (2120) having a CD / DVD or similar medium (2121), a thumb drive (2122), a removable hard drive or solid state drive (2123), legacy magnetic media such as tapes and floppy disks (not depicted), and special ROM / ASIC / PLD-based devices such as security dongles (not depicted).

[0236] One of ordinary skill in the art should also understand that the term "computer-readable medium" as used in connection with the subject matter of this disclosure does not include transmission media, carrier waves, or other transient signals.

[0237] The computer system (2100) can also include an interface (2154) to one or more communication networks (2155). The network can be, for example, wireless, wired, or optical. The network can further be local, wide area, metropolitan, vehicle and industrial, real-time, delay tolerant, etc. Examples of networks include local area networks such as Ethernet and wireless LAN, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., and television wired or wireless wide area digital networks including cable television, satellite television, and terrestrial broadcast television, and vehicle and industrial including CANbus. A particular network typically requires an external network interface adapter attached to a particular general-purpose data port or peripheral bus (2149) (such as a USB port of the computer system (2100)), and other networks are typically integrated into the core of the computer system (2100) by attaching to the system bus as described below (such as an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system (2100) can communicate with other entities. Such communication can be only unidirectional reception (such as broadcast TV), only unidirectional transmission (such as CANbus to a particular CANbus device), or bidirectional with other computer systems using, for example, local or wide area digital networks. Particular protocols and protocol stacks can be used for each of those networks and network interfaces as described above.

[0238] The aforementioned human interface device, human-accessible storage device, and network interface can be attached to the core (2140) of the computer system (2100).

[0239] The core (2140) can include special programmable processing devices in the form of one or more central processing units (CPUs) (2141), graphics processing units (GPUs) (2142), field programmable gate arrays (FPGAs) (2143), hardware accelerators for specific tasks (2144), graphics adapters (2150), etc. These devices may be connected via a system bus (2148) together with read-only memory (ROM) (2145), random access memory (2146), internal mass storage such as an internal hard drive or SSD that is not accessible to the user (2147). In some computer systems, the system bus (2148) may be accessible in the form of one or more physical plugs to allow for expansion by additional CPUs, GPUs, etc. Peripheral devices can be attached directly to the core's system bus (2148) or via a peripheral bus (2149). In one example, a screen (2110) can be connected to a graphics adapter (2150). Architectures for peripheral buses include PCI, USB, etc.

[0240] The CPU (2141), GPU (2142), FPGA (2143), and accelerator (2144) can together execute specific instructions that can make up the aforementioned computer code. That computer code can be stored in ROM (2145) or RAM (2146). Migration data can also be stored in RAM (2146), and persistent data can be stored, for example, in internal mass storage (2147). Fast storage and retrieval for any of the memory devices can be enabled using cache memory that can be closely associated with one or more CPUs (2141), GPUs (2142), mass storage (2147), ROM (2145), RAM (2146), etc.

[0241] A computer-readable medium can have computer code for performing various computer-implemented operations. The medium and the computer code may be specially designed and constructed for the purposes of this disclosure, or may be of the kind well-known and available to those having skill in the art of computer software technology.

[0242] As a non-limiting example, a computer system (2100) having an architecture, specifically a core (2140), can provide a function as a result of a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media can be the user-accessible mass storage introduced above, as well as media associated with specific storage of the core (2140) of a non-transitory nature such as core internal mass storage (2147) or ROM (2145). The software implementing various embodiments of the present disclosure can be stored in such devices and executed by the core (2140). The computer-readable media can include one or more memory devices or chips, depending on specific needs. The software can cause the core (2140), and specifically the processor (including a CPU, GPU, FPGA, etc.) therein, to define a data structure stored in the RAM (2146) and modify such data structure according to a process defined by the software, to execute a specific process described herein or a specific part of a specific process. Additionally, or alternatively, the computer system can provide a function as a result of logic hardwired or otherwise embodied in a circuit (e.g., an accelerator (2144)), which can operate in place of or in conjunction with software to execute a specific process described herein or a specific part of a specific process. References to software can, as necessary, include logic, and vice versa. References to computer-readable media can, as necessary, include circuits (such as integrated circuits (ICs)) storing software for execution, circuits embodying logic for execution, or both. The present disclosure encompasses any suitable combination of hardware and software.

[0243] Although this disclosure describes some exemplary embodiments, there are changes, substitutions, and various alternative equivalents that fall within the scope of this disclosure. Thus, it will be understood that those skilled in the art can devise numerous systems and methods that, while not explicitly illustrated or described herein, embody the principles of this disclosure and are thus within the spirit and scope of this disclosure. Appendix A: Acronyms JEM: Joint Exploration Model VVC: Versatile Video Coding BMS: Benchmark Set MV: Motion Vector HEVC: High Efficiency Video Coding SEI: Supplemental Enhancement Information VUI: Video User Interface Information GOPs: Groups of Pictures TUs: Transform Units PUs: Prediction Units CTUs: Coding Tree Units CTBs: Coding Tree Blocks PBs: Prediction Blocks HRD: Hypothetical Reference Decoder SNR: Signal-to-Noise Ratio CPUs: Central Processing Units GPUs: Graphics Processing Units CRT: Cathode Ray Tube LCD: Liquid Crystal Display OLED: Organic Light-Emitting Diode CD: Compact Disc DVD: Digital Versatile Disc ROM: Read-Only Memory RAM: Random Access Memory ASIC: Application-Specific Integrated Circuit PLD: Programmable Logic Device LAN: Local Area Network GSM: Global System for Mobile Communications LTE: Long Term Evolution CANBus: Controller Area Network Bus USB: Universal Serial Bus PCI: Peripheral Component Interconnect FPGA: Field Programmable Gate Array SSD: Solid State Drive IC: Integrated Circuit HDR: High Dynamic Range SDR: Standard Dynamic Range JVET: Joint Video Exploration Team MPM: Most Probable Mode WAIP: Wide Angle Intra Prediction CU: Coding Unit PU: Prediction Unit TU: Transform Unit CTU: Coding Tree Unit PDPC: Position Dependent Prediction Combination ISP: Intra Sub - partition SPS: Sequence Parameter Set PPS: Picture Parameter Set APS: Adaptive Parameter Set VPS: Video Parameter Set DPS: Decoding Parameter Set ALF: Adaptive Loop Filter SAO: Sample Adaptive Offset CC - ALF: Cross - Component Adaptive Loop Filter CDEF: Constrained Directional Enhancement Filter CCSO: Cross - Component Sample Offset LSO: Local Sample Offset LR: Loop Restoration Filter AV1: AOMedia Video 1 AV2: AOMedia Video 2 MVD: Motion Vector Difference CfL: Chroma from Luma SDT: Semi - detached Tree SDP: Semi - detached Partition SST: Semi - separate Tree SB: Super Block IBC (or IntraBC): Intra-block copy CDF: Cumulative density function SCC: Screen content coding GBI: Generalized bi-prediction BCW: Bi-prediction with CU-level weights CIIP: Combined intra-inter prediction POC: Picture order count RPS: Reference picture set DPB: Decoded picture buffer MMVD: Merge mode with motion vector difference

Explanation of symbols

[0244] 101 sample 102 arrow 103 arrow 104 block 201 current block 202 sample 203 sample 204 sample 205 sample 206 sample 300 communication system 310 terminal device 320 terminal device 330 terminal device 340 terminal device 350 network 400 communication system 401 video source 402 video picture stream 403 video encoder 404 encoded video data 405 streaming server 406 client subsystem 407 video data 408 client subsystem 409 video data 410 video decoder Output Stream of 411 Video Pictures 412 Display 413 Video Capture Subsystem 420 Electronic Device 430 Electronic Device 501 Channel 510 Video Decoder 512 Rendering Device 515 Buffer Memory 520 Parser 521 Symbol 530 Electronic Device 531 Receiver 551 Scaler / Inverse Transformation Unit 552 Intra Prediction Unit 553 Motion Compensation Prediction Unit 555 Aggregator 556 Loop Filter 557 Reference Picture Memory 558 Current Picture Buffer 601 Video Source 603 Video Encoder 620 Electronic Device 630 Source Coder 632 Coding Engine 633 Decoder 634 Reference Picture Memory 635 Predictor 640 Transmitter 645 Entropy Encoder 650 Controller 660 Communication Channel 703 Video Encoder 721 General-Purpose Controller 722 Intra Encoder 723 Residual Calculator 724 Residual Encoder 725 Entropy Encoder 726 Switch 728 Residual Decoder 730 Inter Encoder 810 Video Decoder 871 Entropy Decoder 872 Intra Decoder 873 Residual Decoder 874 Reconstruction Module 880 Inter Decoder 2100 Computer System 2101 Keyboard 2102 Mouse 2103 Track Pad 2105 Joystick 2106 Microphone 2107 Scanner 2108 Camera 2109 Speaker 2110 Touch Screen 2120 CD / DVD ROM / RW 2121 CD / DVD or Similar Media 2122 Flash Drive 2123 Removable Hard Drive or Solid State Drive 2140 Core 2141 Central Processing Unit (CPU) 2142 Graphics Processing Unit (GPU) 2143 Field Programmable Gate Array (FPGA) 2144 Accelerator 2145 Read Only Memory (ROM) 2146 Random Access Memory (RAM) 2147 Mass Storage 2148 System Bus 2149 Peripheral Bus 2150 Graphics Adapter 2154 Network Interface 2155 Communication Network

Claims

1. A video encoding method performed by a video encoder, comprising: receiving video data including a current video block; determining that the current video block should be intercoded based on a prediction block and a motion vector (MV), wherein the MV should be derived from a reference motion vector (RMV) and a motion vector difference (MVD) of the current video block; determining the reference MVD pixel precision of the current video block if the MVD is coded at an adaptive MVD pixel resolution; identifying a maximum allowable MVD pixel precision; determining a set of acceptable MVD levels for the current video block based on the reference MVD pixel precision and the maximum allowable MVD pixel precision; and deriving the MVD according to at least one MVD parameter for the current video block and the set of acceptable MVD levels.

2. The method according to claim 1, wherein the reference MVD pixel precision of the current video block is specified / signaled / derived at the sequence level, picture level, frame level, superblock level, or coding block level.

3. The method according to claim 2, wherein the reference MVD pixel precision of the current video block depends on the MVD class associated with the MVD of the current video block.

4. The method according to claim 2, wherein the reference MVD pixel precision of the current video block depends on the MVD size of the MVD of the current video block.

5. The method according to claim 2, wherein the maximum allowable MVD pixel accuracy is predefined.

6. The method according to any one of claims 1 to 5, further comprising the step of determining the current MVD class from a predefined set of MVD classes, wherein the step of determining the set of acceptable MVD levels for the MVD based on the reference MVD pixel precision and the maximum allowable MVD pixel precision includes the step of excluding MVD levels associated with MVD pixel precisions greater than or equal to the maximum allowable MVD pixel precision from the reference MVD level set determined based on the reference MVD pixel precision and the current MVD class in order to determine the set of acceptable MVD levels for the current video block.

7. The method according to claim 6, wherein the maximum allowable MVD pixel accuracy is 1 / 4 pixel.

8. The method according to any one of claims 1 to 5, wherein MVD levels associated with an accuracy of 1 / 8 pixel or greater are excluded from the set of acceptable MVD levels for the current video block.

9. The method according to any one of claims 1 to 5, further comprising the step of determining the current MVD class from a predefined set of MVD classes, wherein the MVD level associated with fractional MVD precision is included in the set of acceptable MVD levels, regardless of the reference MVD pixel precision, if the current MVD class is less than or equal to a threshold MVD class.

10. The method according to claim 9, wherein the threshold MVD class is the lowest MVD class in a predefined set of MVD classes.

11. The method according to any one of claims 1 to 5, further comprising the step of determining the size of the MVD, wherein an MVD level associated with an MVD accuracy higher than a threshold MVD accuracy is acceptable in the set of acceptable MVD levels only if the size of the MVD is less than or equal to the threshold MVD size.

12. The method according to claim 11, wherein the threshold MVD size is 2 pixels or less.

13. The method according to claim 12, wherein the threshold MVD accuracy is 1 pixel.

14. The method according to claim 11, wherein an MVD level associated with an MVD accuracy of 1 / 4 pixel or more is permitted only when the size of the MVD is 1 / 2 pixel or less.

15. The method according to any one of claims 1 to 5, wherein the maximum allowable MVD pixel accuracy is less than or equal to the reference MVD pixel accuracy.

16. A video encoding method performed by a video encoder, comprising: receiving video data including a current video block; determining that the current video block is to be intercoded and associated with a plurality of reference frames; determining whether an adaptive motion vector difference (MVD) pixel resolution is to be applied to at least one of the plurality of reference frames; and outputting a signaling including a single-bit flag to indicate whether the adaptive MVD pixel resolution is to be applied to all of the plurality of reference frames or not to any of them.

17. A method of video encoding performed by a video encoder, comprising: receiving video data comprising a current video block; determining that the current video block is to be intercoded and associated with a plurality of reference frames; determining whether adaptive motion vector difference (MVD) pixel resolution is to be applied to at least one of the plurality of reference frames; and outputting a signaling comprising a separate flag corresponding to each of the plurality of reference frames to indicate whether adaptive MVD pixel resolution is to be applied.

18. A video encoding method performed by a video encoder, comprising: receiving video data including a current video block; determining that the current video block is to be intercoded and associated with a plurality of reference frames; determining whether adaptive motion vector difference (MVD) pixel resolution is to be applied to at least one of the plurality of reference frames; and for each of the plurality of reference frames, outputting a signaling including an implicit indication that adaptive MVD pixel resolution is not to be applied when the MVD corresponding to each of the plurality of reference frames is zero; and a single-bit flag for indicating whether adaptive MVD pixel resolution is to be applied when the MVD corresponding to each of the plurality of reference frames is non-zero.

19. A video encoding method performed by a video encoder, comprising: receiving video data comprising a current video block; determining that the current video block is to be intercoded based on a prediction block and a motion vector (MV), wherein the MV should be derived from a reference motion vector (RMV) and a motion vector difference (MVD) for the current video block; determining the current MVD class of the MVD from a predefined set of MVD classes; deriving, based on the current MVD class, at least one context for entropy coding at least one explicit signaling for output, wherein the at least one explicit signaling specifies an MVD pixel resolution for at least one component of the MVD; and entropy coding the at least one explicit signaling using the at least one context to determine the MVD pixel resolution for the at least one component of the MVD.

20. The method according to claim 19, wherein the at least one component of the MVD includes a horizontal component and a vertical component of the MVD, and the at least one context includes two separate contexts associated with one of the horizontal component and the vertical component of the MVD, and the horizontal component and the vertical component are associated with separate MVD pixel resolutions.

21. The method according to claim 1, wherein the maximum allowable MVD pixel accuracy is lower than the reference MVD pixel accuracy.

22. The method according to claim 1, wherein the first accuracy is selected from the set of acceptable MVD levels.

23. A video processing device comprising memory for storing computer instructions and a processor, wherein the processor is configured to perform the method according to any one of claims 1 to 5 and 16 to 22 when executing the computer instructions.

24. A computer program for causing a computer to perform the method described in any one of claims 1 to 5 and 16 to 22.

25. A method for transmitting a video bitstream performed by a device having a video encoder, comprising: generating a bitstream comprising a plurality of encoded data blocks, each comprising a current video block, corresponding to a current frame; and transmitting the generated bitstream, wherein the step of generating the bitstream comprises: receiving video data comprising a current video block; determining that the current video block should be intercoded based on a prediction block and a motion vector (MV), wherein the MV should be derived from a reference motion vector (RMV) and a motion vector difference (MVD) of the current video block; determining the reference MVD pixel precision of the current video block if the MVD is coded at an adaptive MVD pixel resolution; identifying a maximum allowable MVD pixel precision; determining a set of acceptable MVD levels for the current video block based on the reference MVD pixel precision and the maximum allowable MVD pixel precision; and deriving the MVD according to at least one MVD parameter for the current video block and the set of acceptable MVD levels.

Images