Video decoding method and apparatus, and video encoding method

Abstract

The present application discloses signaling of motion vectors or motion vector differentials depending on whether a magnitude-dependent adaptive motion vector differential pixel resolution is used in video encoding and decoding. An exemplary method for processing video blocks of a video stream is disclosed. The method may include receiving a video stream, determining that a video block is inter-coded based on a predictive block and a motion vector (MV) to be derived from a motion vector differential (MVD) of the video block and a reference motion vector (RMV), extracting or deriving a data item related to at least one of the MVD or RMV from the video stream in a manner that depends at least on whether the MVD is coded with the magnitude-dependent adaptive MVD pixel resolution, extracting the MVD from the video stream, deriving the MV based on the extracted RMV and the MVD, and reconstructing the video block based at least on the MV and the predictive block.

Description

Technical Field

[0001] [Incorporation by Reference] This application claims the benefit of priority thereto under 35 U.S.C. § 119(e) to U.S. Provisional Patent Application No. 63 / 300,433, filed Jan. 18, 2022, entitled “Improvement for Adaptive MVD resolution,” and to U.S. Patent Application No. 17 / 824,168, filed May 25, 2022, entitled “Interdependence Between Adaptive Resolution of Motion Vector Difference and Signaling / Derivation of Motion Vector-Related Parameters.” These prior applications are hereby incorporated by reference in their entirety.

[0002] [Technical Field] The present disclosure generally relates to video coding, and more particularly, to methods and systems for providing an adaptive resolution of motion vector differences in inter prediction of video blocks.

Background Art

[0003] The description of the background provided herein is for the purpose of generally presenting the background of the disclosure. The study of the presently named inventors, insofar as it is described in this background section, and aspects of the description that might otherwise qualify as prior art at the time of filing, are not admitted to be prior art to the disclosure, either expressly or implicitly.

[0004] Video encoding and decoding can be performed using interpicture prediction with motion compensation. Uncompressed digital video can contain a sequence of pictures, each picture having spatial dimensions of, for example, 1920×1080 luminance samples and associated full or subsampled chrominance samples. A sequence of pictures can have a fixed or variable picture rate (alternatively referred to as frame rate), for example, 60 pictures per second or 60 frames per second. Uncompressed video has specific bitrate requirements for streaming or data processing. For example, video with a pixel resolution of 1920×1080, a frame rate of 60 frames / second, and 4:2:0 chroma subsampling at 8 bits / pixel per color channel requires a bandwidth close to 1.5 Gbit / s. One hour of such video requires more than 600 GB of storage space.

[0005] One purpose of video encoding and decoding can be to reduce the redundancy of uncompressed input video signals through compression. Compression can help reduce the bandwidth and / or storage space requirements mentioned above by more than two orders of magnitude in some cases. Both lossless and lossy compression, as well as combinations thereof, can be used. Lossless compression refers to a technique in which an exact copy of the original signal can be reconstructed from the compressed original signal by the decoding process. Lossy compression refers to an encoding / decoding process in which the original video information is not fully preserved during encoding and is not fully recoverable during decoding. When using lossy compression, the reconstructed signal may not be identical to the original signal, but the distortion between the original and reconstructed signals is made small enough that the reconstructed signal is useful for its intended use, despite some information loss. In the case of video, lossy compression is widely used in many applications. The amount of acceptable distortion depends on the application. For example, users of certain consumer video streaming applications may tolerate higher distortion than users of film or television broadcast applications. The compression ratio achievable by a particular coding algorithm can be selected or adjusted to reflect various strain tolerances, and generally, higher tolerable strains allow for coding algorithms that result in higher losses and higher compression ratios.

[0006] Video encoders and decoders can utilize techniques from several broad categories and steps, including, for example, motion compensation, Fourier transform, quantization, and entropic coding.

[0007] Video codec techniques may include a technique known as intra-coding. In intra-coding, sample values ​​are represented without referencing samples or other data from a previously reconstructed reference picture. In some video codecs, the picture is spatially subdivided into blocks of samples. A picture is sometimes called an intra-picture when all blocks of samples are coded in intra-mode. Intra-pictures and their derivatives, such as independent decoder refresh pictures, can be used to reset the decoder state and therefore can be used as the first picture in a coded bitstream and video session, or as a still image. Samples of a block after intra-prediction can then be converted to the frequency domain, and the resulting conversion coefficients can be quantized before entropycoding. Intra-prediction can represent a technique that minimizes sample values ​​in the pre-conversion domain. In some cases, the smaller the post-conversion DC value and the smaller the AC coefficient, the fewer bits are needed at a given quantization step size to represent the block after entropycoding.

[0008] For example, traditional intra-coding, such as that known from MPEG-2 generation coding techniques, does not use intra-prediction. However, some newer video compression techniques include, for example, techniques that attempt to encode / decode blocks of data based on surrounding sample data and / or metadata obtained during the encoding and / or decoding of blocks of data that are spatially adjacent to and preceding in the decoding order of the data being intra-encoded or decoded. Such techniques will hereafter be referred to as “intra-prediction” techniques. Note that in at least some cases, intra-prediction uses only reference data from the current picture being reconstructed, and not from other reference pictures.

[0009] A wide variety of intra-prediction forms can exist. When more than one of these techniques are available in a given video coding technique, the technique in use may be called an intra-prediction mode. One or more intra-prediction modes may be provided in a particular codec. In certain cases, a mode may have submodes and / or be associated with various parameters, and mode / submode information and intra-coding parameters of video blocks may be coded independently or collectively contained in a mode codeword. Since which codeword should be used for a given mode, submode, and / or parameter combination can affect the coding efficiency gain through intra-prediction, an entropy coding technique may be used to convert the codeword into a bitstream.

[0010] Intra-prediction in specific modes was introduced by H.264, refined in H.265, and further refined with newer coding techniques such as Joint Exploration Model (JEM), Versatile Video Coding (VVC), and Benchmark Set (BMS). Generally, in the case of intra-prediction, predictor blocks may be formed using available neighboring sample values. For example, available values ​​for a particular set of neighboring samples along a particular direction and / or line may be copied into the predictor block. The reference to the direction in use may be coded into the bitstream or may be predicted itself.

[0011] Referring to Figure 1A, the lower right shows a subset of nine predictor directions specified by the 33 possible intra-predictor directions in H.265 (corresponding to 33 of the 35 intra-modes defined in H.265, or angular modes). The point where the arrows converge (101) corresponds to the sample being predicted. The arrows indicate the direction in which adjacent samples are used to predict the sample at 101. For example, arrow (102) indicates that sample (101) is predicted from one or more adjacent samples located to the upper right at an angle of 45 degrees from the horizontal. Similarly, arrow (103) indicates that sample (101) is predicted from one or more adjacent samples located to the lower left of sample (101) at an angle of 22.5 degrees from the horizontal.

[0012] Still referring to Figure 1A, the upper left shows a square block (104) of 4x4 samples (indicated by a thick dashed line). The square block (104) contains 16 samples, each labeled with "S", its position in the Y dimension (e.g., row index), and its position in the X dimension (e.g., column index). For example, sample S21 is the second sample (from the top) in the Y dimension and the first sample (from the left) in the X dimension. Similarly, sample S44 is the fourth sample in block (104) in both the Y and X dimensions. Since the block is 4x4 samples in size, S44 is in the lower right. Furthermore, an exemplary reference sample following a similar numbering scheme is shown. The reference sample is labeled with "R" and its Y position (e.g., row index) and X position (column index) relative to block (104). In both H.264 and H.265, a predicted sample adjacent to the block being reconstructed is used.

[0013] Intra-picture prediction for block 104 may be initiated by copying a reference sample value from an adjacent sample according to the signaled prediction direction. For example, suppose the coded video bitstream includes signaling for this block 104 that indicates the prediction direction of arrow (102), i.e., that the sample is predicted from one or more prediction samples located to the upper right at a 45-degree angle from the horizontal. In such a case, samples S41, S32, S23, and S14 are predicted from the same reference sample R05. Then, sample S44 is predicted from reference sample R08.

[0014] In certain cases, the values ​​of multiple reference samples may be combined, for example, through interpolation, in order to calculate the reference sample, particularly when the directions are not equally divisible at 45 degrees.

[0015] The number of possible directions is increasing as video coding technology continues to develop. For example, in H.264 (2003), nine different directions are available for intra-prediction. This increased to 33 in H.265 (2013), and JEM / VVC / BMS can support up to 65 directions as of this disclosure. Empirical studies have been conducted to help identify the most appropriate intra-prediction direction, and certain techniques in entropy coding are used to encode those most appropriate directions with a small number of bits, while accepting a specific bit penalty for the direction. Furthermore, the direction itself can sometimes be predicted from the adjacent direction used in the intra-prediction of the decoded adjacent block.

[0016] Figure 1B shows a schematic diagram (180) representing 65 intra-prediction directions by JEM to illustrate the increase in the number of prediction directions in various coding techniques developed over time.

[0017] The method of mapping bits representing intra-prediction directions to prediction directions in a coded video bitstream can vary depending on the video coding technique, and may range from simple direct mapping of prediction directions to intra-prediction modes, codewords, complex adaptive schemes including most probable modes, and similar techniques. In all cases, however, there may be specific directions for intra-prediction that are statistically less likely to occur in video content than certain other directions. Since the goal of video compression is to reduce redundancy, those less likely directions may be represented by more bits than more likely directions in a well-designed video coding technique.

[0018] Interpicture prediction or interpretation may be based on motion compensation. In motion compensation, sample data from a previously reconstructed picture or a portion thereof (reference picture) may be used for predicting a newly reconstructed picture or a portion of a picture (e.g., a block) after being spatially shifted in the direction indicated by a motion vector (MV). In some cases, the reference picture may be the same as the picture currently being reconstructed. The MV may have two dimensions, X and Y, or it may have three dimensions, the third of which indicates the reference picture being used (homogeneous with the time dimension).

[0019] In some video compression techniques, the current motion vector (MV) applicable to a particular area of ​​sample data can be predicted from other MVs, for example, from other MVs related to other areas of sample data that are spatially adjacent to the area being reconstructed and precede the current MV in the decoding order. By doing so, the overall amount of data required to code the MV can be significantly reduced by relying on removing redundancy with correlated MVs, thereby increasing compression efficiency. For example, when coding an input video signal obtained from a camera (known as natural video), MV prediction can be effective because there is a statistical probability that an area larger than the area to which a single MV is applicable will move in a similar direction in the video sequence, and therefore, in some cases, it can be predicted using similar motion vectors derived from the MVs of adjacent areas. As a result, the actual MV of a given area is similar to or the same as the MV predicted from the surrounding MVs. Such an MV can then be represented with fewer bits than would be used if the MV were coded directly rather than predicted from adjacent MVs after entropy coding. In some cases, MV prediction can be an example of lossless compression of a signal (i.e., MV) derived from the original signal (i.e., sample stream). In other cases, the MV prediction itself may be irreversible, for example, due to rounding errors when calculating the predictor from some surrounding MVs.

[0020] Various MV prediction mechanisms are described in H.265 / HEVC (ITU-T Rec. H265, “High Efficiency Video Coding”, December 2016). Among the many MV prediction mechanisms specified in H.265, this specification will describe a technique hereafter referred to as “spatial merge”.

[0021] Specifically, referring to Figure 2, the current block (201) has samples that the encoder has determined during the motion search process to be predictable from the previous block of the same size, which is spatially shifted. Instead of directly coding its MV, the MV can be derived from metadata associated with one or more reference pictures, for example, from the most recent reference picture (in the decoding order), using MVs associated with any one of five surrounding samples represented as A0, A1 and B0, B1, B2 (202 through 206, respectively). In H.265, the MV prediction can use predictors from the same reference pictures used by the adjacent blocks. [Overview of the project]

[0022] This disclosure generally relates to video coding, and more particularly to methods and systems for signaling various motion vector syntax related to motion vector differences, based on whether or not magnitude-dependent adaptive resolution is used for motion vector differences in interpretation.

[0023] In an exemplary implementation, a method for processing video blocks of a video stream is disclosed. The method may include the steps of: receiving a video stream; determining that the video blocks are to be intercoded based on predicted blocks and motion vectors (MVs), the MVs being derived from motion vector difference (MVD) and reference motion vector (RMV) of the video blocks; extracting or deriving data items related to at least one of the MVD or RMV from the video stream in a manner that depends at least on whether the MVD is coded with magnitude-dependent adaptive MVD pixel resolution; extracting the MVD from the video stream; deriving the MV based on the extracted RMV and MVD; and reconstructing the video blocks based at least on the MV and predicted blocks.

[0024] In the above implementation, the data item may include a syntactic element related to at least one of the MVD or RMV.

[0025] In any one of the above implementations, the data item may include an RMV.

[0026] In any one of the above implementations, the data item may include an RMV index of a video block that maps to a Dynamic Reference List (DRL), and the DRL is configured to identify a plurality of ordered candidate RMVs.

[0027] In any one of the above implementations, the step of extracting a data item at least according to whether the MVD of a video block is coded with a size-dependent adaptive MVD pixel resolution includes a step of determining an RMV index range N at least according to whether the MVD of the video block is coded with a size-dependent adaptive MVD pixel resolution, where N is a positive integer, and a step of parsing a video stream based on the RMV index range to extract the RMV index of the video block.

[0028] In any one of the above implementations, the RMV indices from 1 to N may map to a predetermined set of positions within the DRL.

[0029] In any one of the above implementations, the RMV indices from 1 to N may map to the first N candidate RMVs among the plurality of ordered candidate RMVs identified by the DRL.

[0030] In any one of the above implementations, N may be 1 or 2.

[0031] In any one of the above implementations, N may be signaled in the video stream, and the method further comprises the step of extracting N from the video stream.

[0032] In any one of the above implementations, N may be signaled in the syntax element at the sequence level, frame level, slice level, title level, or superblock level.

[0033] In any one of the above implementations, N=1, and the RMV index may not be present in the video stream, and is derived in response to determining N=1.

[0034] In any one of the above implementations, the method for extracting or deriving the RMV index may further depend, in addition to whether the MVD is coded with size-dependent adaptive MVD pixel resolution, on whether the video blocks are predicted in single-reference mode.

[0035] In any one of the above implementations, the RMV index is extracted from the video stream, and the context for signaling the RMV index in the video stream may depend on whether the MVD is coded with a size-dependent adaptive MVD pixel resolution.

[0036] In any one of the above implementations, if the MVD is coded with size-dependent adaptive MVD pixel resolution, a first context may be used to signal the RMV in the video stream; on the other hand, if the MVD is not coded with size-dependent adaptive MVD pixel resolution, a second context, separate from the first context, may be used to signal the RMV in the video stream.

[0037] In any one of the above implementations, the method may further include the step of extracting information items from the video stream indicating whether Overlapped Block Motion Compensation (OBMC) or Warped Motion is used when the video block is predicted in single-reference mode, in response to the video block being coded with size-dependent adaptive MVD pixel resolution.

[0038] In any one of the above implementations, the method may further include the step of extracting information items from the video stream indicating whether a compound inter-intra prediction mode is used when a video block is predicted in single-reference mode, in response to the video block being coded with size-dependent adaptive MVD pixel resolution.

[0039] In any one of the above implementations, the context derivation for signaling at least one syntax element relating to MVD may depend on whether the video block is coded with size-dependent adaptive MVD pixel resolution.

[0040] In any one of the above implementations, at least one syntax element relating to the MVD includes at least one of the following: a first MVD syntax element indicating which components of the MVD are non-zero; a second MVD syntax element specifying the sign of the MVD; a third MVD syntax element specifying the size range of the MVD; a fourth MVD syntax element specifying an integer size offset within the size range of the MVD; or a fifth MVD syntax element specifying the pixel resolution of the MVD.

[0041] In any one of the above implementations, if the video block is coded with size-dependent adaptive MVD pixel resolution, a first context may be derived to decode at least one syntax element relating to the MVD; on the other hand, if the video block is not coded with size-dependent adaptive MVD pixel resolution, a second context, separate from the first context, may be derived to decode at least one syntax element relating to the MVD.

[0042] In other implementations, a method for decoding video blocks of a video stream is disclosed. The method includes the steps of: receiving a video stream; determining that the video blocks are to be intercoded based on predicted blocks and motion vectors (MVs), where the MVs are derived from the motion vector difference (MVD) and reference motion vector (RMV) of the video blocks; extracting the RMV index of the video blocks to map to a dynamic reference list (DRL) configured to identify a plurality of ordered candidate RMVs; and determining, based on the value of the RMV index, whether the MVD is coded with a size-dependent adaptive MVD pixel resolution.

[0043] In the implementation described above, the method further includes the steps of: extracting a flag from a video stream if the value of the RMV index indicates one of the first N RMV candidates among a plurality of ordered candidate RMVs identified by the DRL, where N is a positive integer; determining, based on the flag, whether the MVD is coded with size-dependent adaptive MVD pixel resolution; and determining that the MVD is not coded with size-dependent adaptive MVD pixel resolution if the value of the RMV index does not indicate any of the first N RMV candidates among a plurality of ordered candidate RMVs.

[0044] In any of the above implementations, N may be predefined as 1 or 2. In any of the above implementations, N is signaled separately in the video stream. In any of the above implementations, N may be signaled in the syntax element at the sequence level, frame level, slice level, title level, or superblock level.

[0045] The disclosure also provides a video coding or decoding device or apparatus including a circuit configuration configured to perform any of the methods described above.

[0046] The aspect of the disclosure also provides a non-temporary computer-readable medium that stores instructions causing a computer to perform a video decoding and / or encoding method, when performed by a computer for video decoding and / or encoding.

[0047] Further features, properties, and various advantages of the disclosed subject matter will become clearer from the following detailed description and accompanying drawings. [Brief explanation of the drawing]

[0048] [Figure 1A] A schematic diagram of an exemplary subset of intra-predictive direction modes is shown. [Figure 1B] An illustrative diagram of an example intra-prediction direction is shown. [Figure 2] A schematic diagram of the current block and its surrounding space merge candidates for motion vector prediction in one example is shown. [Figure 3] A schematic diagram of a simplified block diagram of a communication system (300) according to an example embodiment is shown. [Figure 4] A schematic diagram of a simplified block diagram of a communication system (400) according to an example embodiment is shown. [Figure 5] A schematic diagram of a schematic block diagram of a decoder according to an example embodiment is shown. [Figure 6] A schematic diagram of a simplified block diagram of an encoder according to an example embodiment is shown. [Figure 7] A block diagram of a video encoder according to another example embodiment is shown. [Figure 8] A block diagram of a video decoder according to another example embodiment is shown. [Figure 9] A coding block partitioning scheme according to an embodiment that serves as an example of the disclosure is shown. [Figure 10] Another scheme for coding block partitioning according to an embodiment that serves as an example of the disclosure is shown. [Figure 11] Another scheme for coding block partitioning according to an embodiment that serves as an example of the disclosure is shown. [Figure 12] This shows an exemplary partitioning of a base block into coding blocks that follows an example partitioning scheme. [Figure 13] An example of a 3-partitioning scheme is shown. [Figure 14] An example of a quad-tree binary coding block partitioning scheme is shown. [Figure 15] An embodiment illustrating an example of the disclosure shows a scheme for dividing a coding block into multiple transformation blocks and a coding order for the transformation blocks. [Figure 16] This document illustrates an example embodiment of the disclosure, showing another scheme for dividing a coding block into multiple transformation blocks and the coding order of the transformation blocks. [Figure 17] Another scheme for dividing a coding block into multiple transformation blocks is shown, according to an embodiment that serves as an example of the disclosure. [Figure 18] A flowchart of the method according to an embodiment that serves as an example of the disclosure is shown. [Figure 19] Another flowchart of the method according to the embodiment, which serves as an example of the disclosure, is shown. [Figure 20] A schematic diagram of a computer system according to an embodiment of the disclosure is shown. [Modes for carrying out the invention]

[0049] Throughout the specification and claims, terms may have nuances implied or suggested in context beyond their expressly stated meaning. The phrases “in one embodiment” or “in several embodiments” used herein do not necessarily refer to the same embodiment, and the phrases “in another embodiment” or “in other embodiments” used herein do not necessarily refer to different embodiments. Similarly, the phrases “in one implementation” or “in several implementations” used herein do not necessarily refer to the same implementation, and the phrases “in another implementation” or “in other implementations” used herein do not necessarily refer to different implementations. For example, the claimed subject matter is intended to include, in whole or in part, exemplary embodiments / implementations.

[0050] In general, terms can be understood at least partially from their use in context. For example, terms such as “and,” “or,” and “and / or” as used herein may have various meanings that depend at least partially on the context in which such terms are used. Typically, “or,” when used to relate a list such as A, B, or C, is intended to mean A, B, and C in an inclusive sense, and similarly, A or B or C in an exclusive sense. Furthermore, the terms “one or more” or “at least one” as used herein may, at least partially depending on the context, be used to describe any feature, structure, or characteristic in a singular sense, or to describe a combination of features, structures, or characteristics in a plural sense. Similarly, terms such as “one” (a or an) and “the” (the) may, as before, be understood, at least partially depending on the context, to convey singular use or plural use. Furthermore, the terms "based on" or "determined by" may be understood not necessarily as being intended to convey an exclusive set of factors, and instead, as before, depending at least partially on the context, the presence of additional factors that are not necessarily explicitly stated may be acknowledged.

[0051] Figure 3 shows a schematic block diagram of a communication system (300) according to an embodiment of the present disclosure. The communication system (300) includes a plurality of terminal devices that can communicate with each other, for example, via a network (350). For example, the communication system (300) includes a first pair of terminal devices (310) and (320) interconnected via the network (350). In the example of Figure 3, the first pair of terminal devices (310) and (320) can perform one-way transmission of data. For example, terminal device (310) may code video data (for example, a stream of video pictures captured by terminal device (310)) for transmission to other terminal devices (320) via the network (350). The coded video data can be transmitted in the form of one or more coded video bitstreams. Terminal device (320) may receive coded video data from the network (350), decode the coded video data to recover video pictures, and display video pictures according to the recovered video data. One-way data transmission is sometimes used in media serving applications, for example.

[0052] In another example, the communication system (300) includes a second pair of terminal devices (330) and (340) that perform bidirectional transmission of coded video data, which may be performed, for example, during video conferencing. For bidirectional transmission of data, in this example, each terminal device of terminal devices (330) and (340) may code video data (for example, a stream of video pictures captured by that terminal device) for transmission to the other terminal device of terminal devices (330) and (340) over the network (350). Each terminal device of terminal devices (330) and (340) may also receive coded video data transmitted by the other terminal device of terminal devices (330) and (340), decode the coded video data to recover video pictures, and display the video pictures on an accessible display device according to the recovered video data.

[0053] In the example in Figure 3, terminal devices (310), (320), (330), and (340) may be implemented as servers, personal computers, and smartphones, but the applicability of the fundamental principles of this disclosure is not limited thereto. Embodiments of this disclosure may be implemented in desktop computers, laptop computers, tablet computers, media players, wearable computers, dedicated video conferencing equipment, and / or similar devices. Network (350) corresponds to any number or type of network that transmits coded video data between terminal devices (310), (320), (330), and (340), including, for example, wireline and / or wireless communication networks. Communication network (350) may exchange data in circuit switching, packet switching, and / or other types of channels. Typical networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For the purposes of this discussion, the architecture and topology of network (350) may be irrelevant to the operation of this disclosure unless expressly described herein.

[0054] Figure 4 illustrates the arrangement of a video encoder and video decoder in a video streaming environment as an example of the disclosed subject matter. The disclosed subject matter may similarly be applicable to other video-enabled applications, including, for example, video conferencing, digital television broadcasting, games, virtual reality, and the storage of compressed video on digital media such as CDs, DVDs, and memory sticks.

[0055] The video streaming system may include a video capture subsystem (413) which may include a video source (401), such as a digital camera, that generates a stream (402) of uncompressed video pictures or images. In the example, the stream (402) of video pictures includes samples recorded by the digital camera of the video source (401). The stream (402) of video pictures is shown in bold to emphasize its high data volume compared to encoded video data (404) (or encoded video bitstream) and may be processed by electronic equipment (420) including a video encoder (403) coupled to the video source (401). The video encoder (403) may include hardware, software, or a combination thereof to enable or implement the aspects of the subject matter disclosed, as will be described in more detail below. The encoded video data (404) (or encoded video bitstream (404)) is represented by a thin line to highlight its lower data volume compared to the uncompressed video picture stream (402), and may be stored in a streaming server (405) for future use or for direct downstream to a video device (not shown). One or more streaming client subsystems, such as client subsystems (406) and (408) in Figure 4, can access the streaming server (405) to read copies (407) and (409) of the encoded video data (404). The client subsystem (406) may include, for example, a video decoder (410) in an electronic device (430). The video decoder (410) decodes the incoming copy (407) of the encoded video data and generates an outgoing stream (411) of video pictures that are uncompressed and can be rendered on a display (412) (e.g., a display screen) or other rendering device (not shown). The video decoder (410) may be configured to perform some or all of the various functions described herein.In some streaming systems, encoded video data (404), (407), and (409) (e.g., video bitstream) may be encoded according to a specific video coding / compression standard. An example of such a standard is ITU-T Recommendation H.265. In this example, the video coding standard under development is commonly known as Versatile Video Coding (VVC). The disclosed subject matter may be used in connection with VVC and other video coding standards.

[0056] The electronic devices (420) and (430) may include other components (not shown). For example, electronic device (420) may include a video decoder (not shown), and electronic device (430) may similarly include a video encoder (not shown).

[0057] Figure 5 shows a block diagram of a video decoder (510) according to any embodiment of the present disclosure described below. The video decoder (510) may be included in electronic equipment (530). Electronic equipment (530) may include a receiver (531) (e.g., a receiving circuit). The video decoder (510) may be used in place of the video decoder (410) in the example of Figure 4.

[0058] The receiver (531) may receive one or more coded video sequences to be decoded by the video decoder (510). In the same or other embodiments, one coded video sequence may be decoded at a time, in which case the decoding of each coded video sequence is independent of other coded video sequences. Each video sequence may be associated with multiple video frames or images. The coded video sequences may be received from a channel (501), which may be a storage device storing coded video data or a hardware / software link to a streaming source transmitting coded video data. The receiver (531) may receive the coded video data together with other data, such as coded audio data and / or auxiliary data streams, which may be transferred to their respective processing circuits (not shown). The receiver (531) may separate the coded video sequences from the other data. To counteract network jitter, a buffer memory (515) may be located between the receiver (531) and the entropy decoder / parser (520) (hereinafter referred to as "parser (520)"). In certain applications, the buffer memory (515) may be implemented as part of the video decoder (510). In other applications, it can be separated outside the video decoder (510) (not shown). In yet other applications, for example, to counteract network jitter, there may be a buffer memory (not shown) outside the video decoder (510), and another additional buffer memory (515) may exist inside the video decoder (510), for example, to manipulate playback timing. When the receiver (531) is receiving data from a sufficiently bandwidth and controllable storage / transfer device or from an isosynchronous network, the buffer memory (515) may not be necessary or can be small.For use in best-effort packet networks such as the Internet, a sufficiently large buffer memory (515) may be required, and that size may be relatively large. Such buffer memory may be implemented with adaptive sizing and may be at least partially implemented in an operating system or similar element (not shown) outside the video decoder (510).

[0059] The video decoder (510) may include a parser (520) for reconstructing symbols (521) from the coded video sequence. The categories of these symbols include information used to manage the operation of the video decoder (510) and information for controlling rendering devices such as a display (512) (e.g., a display screen), which may or may not be an essential part of the electronic device (530) but can be coupled to the electronic device (530) as shown in Figure 5. The control information for rendering devices may take the form of Supplemental Enhancement Information (SEI) messages or Video Usability Information (VUI) parameter set fragments (not shown). The parser (520) may parse / entropy decode the coded video sequence received by the parser (520). The coding of the coded video sequence may follow video coding techniques or standards and may follow various principles, including variable-length coding, Huffman coding, context-dependent or independent arithmetic coding, etc. The parser(520) may extract from the coded video sequence a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder, based on at least one parameter corresponding to the subgroup. Subgroups may include Group of Picture (GOP), picture, tile, slice, macroblock, coding unit (CU), block, transform unit (TU), prediction unit (PU), and so on. The parser(520) may also extract from the coded video sequence information such as transform coefficients (e.g., Fourier transform coefficients), quantizer parameter values, and motion vectors.

[0060] The parser (520) may perform an entropy decoding / parsing operation on the video sequence received from the buffer memory (515) in order to generate a symbol (521).

[0061] The reconstruction of the symbol (521) can have a number of different processing or function units depending on the type of coded video picture or part thereof (e.g., inter and intra pictures, inter and intra blocks) and other factors. The units involved and the way they are involved may be controlled by subgroup control information parsed from the coded video sequence by the parser (520). The flow of such subgroup control information between the parser (520) and the following multiple processing or function units is not shown for simplicity' sake.

[0062] Beyond the functional blocks already described, the video decoder (510) can be conceptually subdivided into numerous functional units, as described below. In actual implementations operating under commercial constraints, many of these functional units may interact closely with each other and be at least partially integrated with one another. However, for the purpose of clearly describing the various functions of the disclosed subject, a conceptual subdivision into functional units is adopted hereafter in this disclosure.

[0063] The first unit may include a scaler / inverse unit (551). The scaler / inverse unit (551) may receive control information from the parser (520) as symbols (521) along with the quantized transformation coefficients, including information indicating which inverse transform should be used, such as block size, quantization coefficients / parameters, and quantization scaling matrix. The scaler / inverse unit (551) can output a block containing sample values ​​that can be input to the aggregator (555).

[0064] In some cases, the output samples of the scaler / inverse converter (551) may relate to intracoded blocks, i.e., blocks that do not use prediction information from previously reconstructed pictures but can use prediction information from portions reconstructed before the current picture. Such prediction information may be supplied by an intrapicture prediction unit (552). In some cases, the intrapicture prediction unit (552) may generate a block of the same size and shape as the block being reconstructed, using surrounding block information that has already been reconstructed and is stored in the current picture buffer (558). The current picture buffer (558) buffers, for example, partially reconstructed current pictures and / or fully reconstructed current pictures. In some implementations, the aggregator (555) may add, sample by sample, the prediction information generated by the intraprediction unit (552) to the output sample information supplied by the scaler / inverse converter (551).

[0065] In other cases, the output samples of the scaler / inverse unit (551) may relate to an intercoded and potentially motion-compensated block. In such cases, the motion-compensated prediction unit (553) may access the reference picture memory (557) to fetch samples to be used for interpicture prediction. After motion-compensating the fetched samples according to the symbols (521) relating to the block, those samples may be added by the aggregator (555) to the output of the scaler / inverse unit (551) (the output of unit 551 may be called residual samples or residual signals) to generate output sample information. The addresses in the reference picture memory (557) from which the motion-compensated prediction unit (553) fetches prediction samples may be controlled by motion vectors that are available to the motion-compensated prediction unit (553) in the form of symbols (521) which may have, for example, X and Y components (shift) and a reference picture component (time). Motion compensation may also include interpolation of sample values ​​fetched from reference picture memory (557) when the exact motion vectors of subsamples are used, and may also be associated with motion vector prediction mechanisms, etc.

[0066] The output samples of the aggregator (555) can undergo various loop filtering techniques in the loop filter unit (556). The video compression technique may include an in-loop filtering technique. This technique is contained in the coded video sequence (also called the coded video bitstream) and is controlled by parameters made available to the loop filter unit (556) as symbols (521) from the parser (520), but can also respond to metadata obtained during the decoding of the coded picture or the previous part (in the decoding order) of the coded video sequence, and further, it can respond to previously configured loop-filtered sample values. Several types of loop filters may be included as parts of the loop filter unit 556 in various orders, as will be described in more detail below.

[0067] The output of the loop filter unit (556) can be a sample stream that is output to the rendering device (512) and can also be stored in a reference picture memory (557) for use in future interpicture prediction.

[0068] A particular coded picture, once fully reconfigured, can be used as a reference picture for future interpicture prediction. For example, once the coded picture corresponding to the current picture is fully reconfigured and the coded picture is identified as a reference picture (e.g., by the parser (520)), the current picture buffer (558) can become part of the reference picture memory (557), and any unused current picture buffer can be reallocated before the reconfiguration of subsequent coded pictures begins.

[0069] The video decoder (510) may perform the decoding operation according to a specified video compression technique adopted in a standard such as the ITU-T recommended H.265. The coded video sequence may conform to the syntax defined by the video compression technique or standard in use, in the sense that the coded video sequence conforms to both the syntax of the video compression technique or standard and the profile documented in the video compression technique or standard. Specifically, a profile may select a particular tool from all the tools available in the video compression technique or standard as the only tool available for use under that profile. For compliance with the standard, the complexity of the coded video sequence may be within the boundaries defined by the level of the video compression technique or standard. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples / second), maximum reference picture size, etc. The limitations set by the level may, in some cases, be further restricted through the Hypothetical Reference Decoder (HRD) specification and metadata for HRD buffer management that is communicated in the coded video sequence.

[0070] In some exemplary embodiments, the receiver (531) may receive additional (redundant) data along with the encoded video. The additional data may be included as a portion of the coded video sequence. The additional data may be used by the video decoder (510) to properly decode the data and / or to more accurately reconstruct the original video data. The additional data may take the form of, for example, a time, space, or signal-to-noise ratio (SNR) enhancement layer, redundant slices, redundant pictures, forward error correction codes, etc.

[0071] Figure 6 shows a block diagram of a video encoder (603) according to an exemplary embodiment of the present disclosure. The video encoder (603) may be included in electronic equipment (620). The electronic equipment (620) may further include a transmitter (640) (e.g., a transmitting circuit). The video encoder (603) may be used in place of the video encoder (403) in the example of Figure 4.

[0072] The video encoder (603) may receive video samples from a video source (601) (not part of the electronic device (620) in the example in Figure 6) that can capture video images to be coded by the video encoder (603). In other examples, the video source (601) may be implemented as part of the electronic device (620).

[0073] The video source (601) may supply a source video sequence to be coded by the video encoder (603) in the form of a digital video sample stream, which can have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, etc.), any color space (e.g., BT.601 YCrCb, RGB, XYZ, etc.), and any suitable sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media serving system, the video source (601) may be a storage device capable of storing pre-prepared video. In a video conferencing system, the video source (601) may be a camera that captures local image information as a video sequence. The video data may be supplied as a series of individual pictures or images that give motion when viewed sequentially. The picture itself may be organized as a spatial array of pixels, each pixel may have one or more samples depending on the sampling structure, color space, etc., in use. Those skilled in the art will readily understand the relationship between pixels and samples. This specification will focus on samples below.

[0074] According to several exemplary embodiments, the video encoder (603) may encode and compress the pictures of a source video sequence into a coded video sequence (643) in real time or under any other time constraints required by the application. Enforcing an appropriate coding speed constitutes a function of the controller (650). In some embodiments, the controller (650) may be functionally coupled to and control other functional units, such as those described below. The couplings are not shown for simplicity. Parameters set by the controller (650) may include parameters related to rate control (e.g., picture skip, quantizer, lambda value of rate distortion optimization technique), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (650) may be configured to have other appropriate functions related to the video encoder (603) optimized for a particular system design.

[0075] In some exemplary embodiments, the video encoder (603) may be configured to operate in a coding loop. In an overly simplified description, in the example, the coding loop may include a source coder (630) (which is involved in generating symbols, such as a symbol stream, based, for example, an input picture to be coded and a reference picture) and a (local) decoder (633) embedded in the video encoder (603). The decoder (633) reconstructs the symbols to generate sample data in a similar manner to what a (remote) decoder would generate, even if the embedded decoder (633) processes the video stream coded by the source coder (630) without entropic coding (meaning that any compression between the symbols and the coded video stream can be reversible in the video compression techniques considered in the disclosed subject). The reconstructed sample stream (sample data) is input to a reference picture memory (634). Symbol stream decoding yields bit-exact results that are independent of the decoder's location (local or remote), so the content in reference picture memory (634) is also bit-exact between the local and remote encoders. That is, the predictive portion of the encoder "sees" the exact same sample values ​​as the reference picture samples that the decoder "sees" when using predictions during decoding. This fundamental principle of reference picture synchronicity (and the resulting drift when synchronicity cannot be maintained, for example, due to channel errors) is used to improve coding quality.

[0076] The operation of the “local” decoder (633) can be the same as that of “remote” decoders such as the video decoder (510), which have already been described in detail above with reference to Figure 5. Referring temporarily to Figure 5, however, the entropy decoding portion of the video decoder (510), including the buffer memory (515) and parser (520), does not have to be fully implemented in the local decoder (633) within the encoder, provided that symbols are available and the encoding / decoding of symbols to the coded video sequence by the entropycoder (645) and parser (520) is reversible.

[0077] What can be observed at this point is that any decoder technique, with the exception of parsing / entropy decoding which may exist only in the decoder, must necessarily exist in substantially the same functional form in the corresponding encoder. For this reason, the disclosed subject matter may, from time to time, focus on the decoder operation related to the decoding portion of the encoder. Thus, the description of encoder techniques may be omitted, as they are the inverse of the more comprehensively described decoder techniques. A more detailed description of encoders is given below only in specific areas or aspects.

[0078] During operation, in some embodiments, the source coder (630) may perform motion-compensated predictive coding. This predictively codes the input picture by referencing one or more previously coded pictures from a video sequence, designated as “reference pictures.” In this way, the coding engine (632) codes the difference (or residual) of the color channels between the pixel blocks of the reference picture, which may be selected as a predictive reference for the input picture, and the pixel blocks of the input picture. The terms “residue” or its adjective “residual” may be used interchangeably.

[0079] A local video decoder (633) may decode the coded video data of a picture that may be designated as a reference picture, based on symbols generated by the source coder (630). The operation of the coding engine (632) may, advantageously, be an irreversible process. When the coded video data can be decoded by a video decoder (not shown in Figure 6), the reconstructed video sequence may typically be a copy of the source video sequence with some errors. The local video decoder (633) may replicate the decoding process that may be performed by the video decoder on the reference picture, so that the reconstructed reference picture is stored in the reference picture cache (634). In this way, the video encoder (603) can locally store a copy of the reconstructed reference picture that has the same content as the reconstructed reference picture that will be obtained by a far-end (remote) video decoder (without transmission errors).

[0080] The predictor (635) may perform a predictive search for the coding engine (632). That is, in order for a new picture to be coded, the predictor (635) may search the reference picture memory (634) for specific metadata or sample data (as candidate reference pixel blocks), such as reference picture motion vectors, block shapes, etc., which may serve as appropriate predictive criteria for the new picture. The predictor (635) may operate on a sample block-by-pixel block basis to find appropriate predictive criteria. In some cases, the input picture may have predictive criteria drawn from multiple reference pictures stored in the reference picture memory (634), as determined by the search results obtained by the predictor (635).

[0081] The controller (650) may manage the coding operation of the source coder (630), including, for example, setting parameters and subgroup parameters used to encode video data.

[0082] The outputs of all the above functional units may undergo entropic coding in the entropicorder (645). The entropicorder (645) converts the symbols generated by the various functional units into coded video sequences by lossless compression of symbols according to techniques such as Huffman coding, variable-length coding, and arithmetic coding.

[0083] The transmitter (640) may buffer the coded video sequence generated by the entropicorder (645) to prepare it for transmission over the communication channel (660). The communication channel (660) may be a hardware / software link to a storage device that stores the coded video data. The transmitter (640) may merge the coded video data from the videocoder (603) with other data to be transmitted, such as coded audio data and / or auxiliary data streams (sources not shown).

[0084] The controller (650) may manage the operation of the video encoder (603). During coding, the controller (650) may assign each coded picture to a specific coded picture type that may affect the coding techniques that may be applied to each picture. For example, a picture may often be assigned to one of the following picture types:

[0085] An intra-picture (I-picture) may be a picture that can be encoded and decoded without using any other pictures in the sequence as a source of prediction. Some video codecs allow various types of intra-pictures, including, for example, independent decoder refresh (IDR) pictures. Those skilled in the art will know such variations of I-pictures and their respective applications and characteristics.

[0086] A predictive picture (P-picture) may be a picture that can be encoded and decoded by intra-prediction or inter-prediction using at most one motion vector and reference index to predict the sample values ​​of each block.

[0087] A bidirectionally predictive picture (B-picture) may be a picture that can be encoded and decoded by intra-prediction or inter-prediction using at most two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, multiple-predictive picture(s) may use more than two reference pictures and associated metadata for the reconstruction of a single block.

[0088] A source picture may generally be spatially subdivided into multiple sample coding blocks (e.g., 4x4, 8x8, 4x8, or 16x16 sample blocks, respectively), and each block may be coded. Blocks may be predictively coded by referring to other (already coded) blocks determined by the coding assignment applied to each picture in the 'block'. For example, blocks of picture I may be coded non-predictively, or they may be coded predictively by referring to already coded blocks of the same picture (spatial prediction or intra-prediction). Pixel blocks of picture P may be coded predictively by spatial prediction or temporal prediction by referring to one previously coded reference picture. Blocks of picture B may be coded predictively by spatial prediction or temporal prediction by referring to one or two previously coded reference pictures. The source picture or intermediate processed picture may be subdivided into other types of blocks for other purposes. The subdivision of coding blocks and other types of blocks may follow or not follow the same methods, as will be described in more detail below.

[0089] The video encoder (603) may perform coding operations in accordance with a specified video coding technique or standard, such as ITU-T Recommended H.265. During such operations, the video encoder (603) may perform various compression operations, including predictive coding operations that utilize temporal and spatial redundancy in the input video sequence. Accordingly, the coded video data may conform to the syntax defined by the video coding technique or standard being used.

[0090] In some exemplary embodiments, the transmitter (640) may transmit additional data along with the encoded video. The source coder (630) may include such data as part of the coded video sequence. The additional data may include time / space / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, SEI messages, VUI parameter set fragments, and the like.

[0091] Video may be captured as multiple source pictures (video pictures) in a time sequence. Intra-picture prediction (often abbreviated as intra-prediction) utilizes spatial correlations in a given picture, while inter-picture prediction utilizes temporal or other correlations between pictures. For example, a particular picture being encoded / decoded, called the current picture, may be partitioned into blocks. A block within the current picture may be coded by a vector called a motion vector if it is similar to a reference block in a reference picture that was coded before the video and is still buffering. The motion vector points to a reference block in the reference picture and may have a third dimension that identifies the reference picture if multiple reference pictures are used.

[0092] In some exemplary embodiments, a biprediction technique may be used for interpicture prediction. According to such a biprediction technique, two reference pictures are used, for example, a first reference picture and a second reference picture, both preceding the current picture in the video in decoding order (but in display order, they may be past or future, respectively). A block in the current picture may be coded by a first motion vector pointing to a first reference block in the first reference picture and a second motion vector pointing to a second reference block in the second reference picture. That block is predictable together by a combination of the first and second reference blocks.

[0093] Furthermore, merge mode techniques may be used in interpicture prediction to improve coding efficiency.

[0094] According to some exemplary embodiments of this disclosure, predictions such as interpicture prediction and intrapicture prediction are performed in units of blocks. For example, pictures in a sequence of video pictures are partitioned into coding tree units (CTUs) for compression, and the CTUs in a picture may have the same size, such as 64x64 pixels, 32x32 pixels, or 16x16 pixels. Generally, a CTU may contain three parallel coding tree blocks (CTBs), i.e., one lumar CTB and two chroma CTBs. Each CTU may be recursively quadtree-partitioned into one or more coding units (CUs). For example, a 64x64 pixel CTU can be divided into one 64x64 pixel CU or four 32x32 pixel CUs. One or more of the 32x32 blocks may be further divided into four 16x16 pixel CUs. In some exemplary embodiments, each CU may be analyzed during encoding to determine the prediction type for that CU from among various prediction types, such as interprediction type or intraprediction type. A CU may be divided into one or more prediction units (PUs) depending on its temporal and / or spatial predictability. Generally, each PU includes one lumar prediction block (PB) and two chroma PBs. In embodiments, the prediction operation in coding (encoding / decoding) is performed in units of prediction blocks. The division of CUs into PUs (or PBs for different color channels) may be performed in various spatial patterns. A lumar or chroma PB may include a matrix of sample values ​​(e.g., lumar values), such as 8x8 pixels, 16x16 pixels, 8x16 pixels, 16x8 pixels, etc.

[0095] Figure 7 shows a diagram of a video encoder (703) according to another exemplary embodiment of the disclosure. The video encoder (703) is configured to receive a processing block (e.g., a prediction block) of sample values ​​in the current video picture that are included in a sequence of video pictures, and to encode the processing block into a coded picture which is a part of the coded video sequence. The example video encoder (703) may be used instead of the example video encoder (403) in Figure 4.

[0096] For example, the video encoder (703) receives a matrix of sample values ​​for a processing block, such as an 8x8 sample prediction block. The video encoder (703) then determines, for example, using rate-distortion optimization (RDO), whether the processing block is best coded in intra-mode, inter-mode, or bi-prediction mode. If it is determined that the processing block is coded in intra-mode, the video encoder (703) may use the intra-prediction technique to encode the processing block into a coded picture; if it is determined that the processing block is coded in inter-mode or bi-prediction mode, the video encoder (703) may use the inter-prediction or bi-prediction technique, respectively, to encode the processing block into a coded picture. In some exemplary embodiments, merge mode may be used as a submode of interpicture prediction in which motion vectors are derived from one or more motion vector predictors without benefiting from coded motion vector components outside the predictors. In some other exemplary embodiments, there may be motion vector components applicable to the target block. Accordingly, the video encoder (703) may include components not explicitly shown in Figure 7, such as a mode determination module, to determine the prediction mode of the processing block.

[0097] In the example shown in Figure 7, the video encoder (703) includes an interencoder (730), an intraencoder (722), a residual calculation unit (723), a switch (726), a residual encoder (724), a general-purpose controller (721), and an entropy encoder (725), all coupled together as shown in the exemplary arrangement in Figure 7.

[0098] The interencoder (730) is configured to receive a sample of the current block (e.g., a processing block), compare that block to one or more reference blocks in the reference picture (e.g., blocks in the previous and subsequent pictures in the standard), generate interprediction information (e.g., a description of redundant information, motion vectors, and merge mode information according to the intercoding technique), and compute an interprediction result (e.g., a predicted block) based on the interprediction information using some appropriate technique. In some examples, the reference picture is a decoded reference picture that has been decoded based on video information encoded using a decoding unit (633) embedded in the exemplary encoder (620) in Figure 6 (shown as a residual decoder 728 in Figure 7, as will be described in more detail below).

[0099] The intra encoder (722) is configured to receive a sample of the current block (e.g., a processing block), compare it with a block already coded in the same picture, and generate transformed quantized coefficients, and in some cases, further generate intra prediction information (e.g., intra prediction direction information following one or more intra coding techniques). The intra encoder (722) may compute an intra prediction result (e.g., a prediction block) based on the intra prediction information and a reference block in the same picture.

[0100] The general-purpose controller (721) may be configured to determine general-purpose control data and to control other components of the video encoder (703) based on the general-purpose control data. In an example, the general-purpose controller (721) determines the prediction mode of a block and supplies control signals to the switch (726) based on the prediction mode. For example, if the prediction mode is intra-mode, the general-purpose controller (721) controls the switch (726) to select the intra-mode result for use by the residual calculation unit (723), and then controls the entropy encoder (725) to select the intra-prediction information and include the intra-prediction information in the bitstream. If the prediction mode of a block is inter-mode, the general-purpose controller (721) controls the switch (726) to select the inter-prediction result for use by the residual calculation unit (723), and then controls the entropy encoder (725) to select the inter-prediction information and include the inter-prediction information in the bitstream.

[0101] The residual calculation unit (723) is configured to calculate the difference (residual data) between the received block and the prediction result of a block selected from the intra-encoder (722) or inter-encoder (730). The residual encoder (724) may be configured to encode the residual data to generate conversion coefficients. For example, the residual encoder (724) may be configured to convert the residual data from the spatial domain to the frequency domain to generate conversion coefficients. The conversion coefficients are then quantized to obtain quantized conversion coefficients. In various exemplary embodiments, the video encoder (703) also includes a residual decoder (728). The residual decoder (728) is configured to perform an inverse transform to generate decoded residual data. The decoded residual data can be appropriately used by the intra-encoder (722) and inter-encoder (730). For example, an interencoder (730) can generate a decoded block based on the decoded residual data and interprediction information, and an intraencoder (722) can generate a decoded block based on the decoded residual data and intraprediction information. The decoded block is appropriately processed to generate a decoded picture, which is buffered in a memory circuit (not shown) and can be used as a reference picture.

[0102] The entropy encoder (725) may be configured to format the bitstream to include the encoded blocks and to perform entropy coding. The entropy encoder (725) may be configured to include various types of information in the bitstream. For example, the entropy encoder (725) may be configured to include general-purpose control data, selected prediction information (e.g., intra-prediction information or inter-prediction information), residual information, and other appropriate information in the bitstream. Residual information may not be present when coding blocks in either inter-mode or bi-prediction mode merge submode.

[0103] Figure 8 shows a diagram of an exemplary video decoder (810) according to another embodiment of the disclosure. The video decoder (810) is configured to receive a coded picture, which is a portion of a coded video sequence, and to decode the coded picture to produce a reconstructed picture. In the example, the video decoder (810) may be used instead of the video decoder (410) in the example of Figure 4.

[0104] In the example shown in Figure 8, the video decoder (810) includes an entropy decoder (871), an interdecoder (880), a residual decoder (873), a reconfiguration module (874), and an intradecoder (872), which are coupled together as shown in the exemplary arrangement in Figure 8.

[0105] An entropy decoder (871) may be configured to reconstruct specific symbols representing syntax elements from a coded picture, from which the coded picture is composed. Such symbols may include, for example, the mode in which a block is coded (e.g., intra-mode, inter-mode, bi-prediction mode, merge sub-mode, or other sub-mode), prediction information (e.g., intra-prediction information or inter-prediction information) that can identify specific samples or metadata used for prediction by an intra-decoder (872) or inter-decoder (880), residual information (e.g., in the form of quantized transformation coefficients), and so on. In the example, if the prediction mode is inter- or bi-prediction mode, inter-prediction information is supplied to the inter-decoder (880), and if the prediction type is intra-prediction type, intra-prediction information is supplied to the intra-decoder (872). The residual information may undergo inverse quantization and be supplied to the residual decoder (873).

[0106] The interdecoder (880) may be configured to receive interprediction information and generate interprediction results based on the interprediction information.

[0107] The intra decoder (872) may be configured to receive intra prediction information and generate prediction results based on the intra prediction information.

[0108] The residual decoder (873) may be configured to perform inverse quantization to extract inversely quantized conversion coefficients, process the inversely quantized conversion coefficients, and convert the residual from the frequency domain to the spatial domain. The residual decoder (873) may also utilize specific control information (to include quantization parameters (QP)), which may be supplied by the entropy decoder (871) (this is only low-capacity control information, and the data path is not shown).

[0109] The reconstruction module (874) may be configured to combine the residuals output by the residual decoder (873) and the prediction results (optionally output by the inter or intra prediction module) in the spatial domain to form reconstructed blocks that form parts of the reconstructed picture as parts of the reconstructed video. Other appropriate operations, such as deblocking, may be performed to improve visual quality.

[0110] The video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) can be implemented by any suitable technology. In some exemplary embodiments, the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) can be implemented using one or more integrated circuits. In other embodiments, the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) can be implemented using one or more processors that execute software instructions.

[0111] Referring to block partitioning for encoding and decoding, general partitioning may begin with a base block and follow a predefined set of rules, a specific pattern, a partition tree, or any partition structure or scheme. Partitioning can be hierarchical and recursive. After dividing or partitioning the base block according to one or a combination thereof of the exemplary partitioning procedures described below, a final set of partitions or coding blocks may be obtained. Each of these partitions may be at one of the various partitioning levels in the partitioning hierarchy and may be of various shapes. Each partition may be called a coding block (CB). For the various exemplary partitioning implementations further described below, each resulting CB may be of any of the allowed size and partitioning level. Such partitions are called coding blocks in that they may form units in which some basic encoding / decoding decisions can be made and encoding / decoding parameters can be optimized, determined, and signaled in the encoded video bitstream. The highest or deepest level of the final partitions represents the depth of the coding block partitioning structure of the tree. The coding blocks may be ruma coding blocks or chroma coding blocks. The CB tree structure for each color may be called a coding block tree (CBT).

[0112] The coding blocks for all color channels can be collectively referred to as coding units (CUs). The hierarchical structure of all color channels can be collectively referred to as coding tree units (CUs). The partitioning patterns or structures of the various color channels in a CTU may or may not be the same.

[0113] In some implementations, the partition tree schemes or structures used for lumern and chroma channels do not need to be the same. In other words, lumern and chroma channels may have separate coding tree structures or patterns. Furthermore, whether lumern and chroma channels use the same or different coding partition tree structures, and the actual coding partition tree structures to be used, may depend on whether the slice being coded is a P, B, or I slice. In the case of an I slice, chroma and lumern channels may have separate coding partition tree structures or coding partition tree structure modes, while in the case of a P or B slice, lumern and chroma channels may share the same coding partition tree scheme. When separate coding partition tree structures or modes are applied, a lumern channel may be partitioned into CBs by one coding partition tree structure, and a chroma channel may be partitioned into chroma CBs by another coding partition tree structure.

[0114] In some embodiments, a predetermined partitioning pattern may be applied to the base block. As shown in Figure 9, four example partition trees may begin at a first predefined level (64x64 block level or other size as the base block size), and the base block may be hierarchically partitioned down to a lower predefined level (e.g., 4x4 level). For example, the base block may follow one of four predefined partitioning options or patterns shown in 902, 904, 906, and 908, where partitions denoted as R are recursively partitioned, in that the same partitioning options shown in Figure 9 may be repeated at a lower scale down to the lowest level (e.g., 4x4 level). In some implementations, additional restrictions may apply to the partitioning scheme in Figure 9. In the implementation of Figure 9, rectangular partitions (e.g., 1:2 / 2:1 rectangular partitions) may be allowed, but they may not be recursive, while square partitions may be recursive. The recursive partitioning according to Figure 9 generates the final set of coding blocks as needed. The coding tree depth may be further defined to indicate the partitioning depth from the root node or root block. For example, the coding tree depth of the root node or root block, e.g., a 64x64 block, may be set to 0, and after the root block is further partitioned once according to Figure 9, the coding tree depth increases by 1. The maximum or deepest level from the 64x64 base block to the smallest 4x4 partition is 4 in the above scheme (starting from level 0). Such a partitioning scheme may be applied to one or more color channels. Each color channel may be partitioned independently according to the scheme in Figure 9 (for example, a partitioning pattern or option from a predefined pattern may be determined independently for each color channel at each hierarchical level).Alternatively, two or more color channels may share the same hierarchical pattern tree in Figure 9 (for example, the same partitioning pattern or option in a predefined pattern may be selected for two or more color channels at each hierarchical level).

[0115] Figure 10 shows other exemplary predefined partitioning patterns that allow recursive partitioning to form a partitioning tree. Ten exemplary partitioning structures or patterns may be predefined, as shown in Figure 10. The root block may start at a predefined level (e.g., from a base block at a 128x128 level or a 64x64 level). The exemplary partitioning structures in Figure 10 include various 2:1 / 1:2 and 4:1 / 1:4 rectangular partitions. Partition types containing three subpartitions, shown in the second row of Figure 10 as 1002, 1004, 1006, and 1008, are sometimes called "T-type" partitions. The "T-type" partitions 1002, 1004, 1006, and 1008 may be called Left T-Type, Top T-Type, Right T-Type, and Bottom T-Type. In some embodiments, none of the rectangular partitions in Figure 10 can be further subdivided. The coding tree depth may be further defined to indicate the partitioning depth from the root node or root block. For example, the coding tree depth of the root node or root block, e.g., a 128x128 block, may be set to 0, and after the root block is further partitioned once according to Figure 10, the coding tree depth increases by 1. In some implementations, only all 1010 square partitions may be allowed to undergo recursive partitioning to the next level of the partitioning tree following the pattern in Figure 10. In other words, recursive partitioning is not allowed for square partitions in T-type patterns 1002, 1004, 1006, and 1008. The recursive partitioning procedure following Figure 10 generates a final set of coding blocks as needed. Such a scheme may be applied to one or more color channels. In some implementations, further flexibility may be added to the use of partitions below the 8x8 level. For example, 2x2 chroma interpretation may be used in particular cases.

[0116] In some other embodiments for coding block partitioning, a quadtree structure may be used to divide a base block or intermediate block into quadtree partitions. Such quadtree partitioning may be applied hierarchically and recursively to any square partition. Whether the base block or intermediate block or partition is further quadtree partitioned may be adapted to various local properties of the base block or intermediate block / partition. Quadtree partitioning at picture boundaries may be further adapted. For example, implicit quadtree partitioning may be performed at picture boundaries so that the block continues to quadtree partition until its size fits the picture boundary.

[0117] In some other embodiments, hierarchical bipartitioning from a base block may be used. In such a scheme, the base block or intermediate level block may be divided into two partitions. The bipartitioning may be horizontal or vertical. For example, horizontal bipartitioning may divide the base block or intermediate block into equal left and right partitions. Similarly, vertical bipartitioning may divide the base block or intermediate block into equal upper and lower partitions. Such bipartitioning may be hierarchical and recursive. Whether the bipartitioning scheme should continue may be determined for each base block or intermediate block, and if the scheme continues, whether horizontal or vertical bipartitioning should be used may be determined. In some implementations, further partitioning may stop at a predefined minimum partition size (in one or both dimensions). Alternatively, further partitioning may stop when a predefined partitioning level or depth from the base block is reached. In some implementations, the aspect ratio of the partitions may be restricted. For example, the aspect ratio of a partition cannot be smaller than 1:4 (or larger than 4:1). Thus, a vertical strip partition with a 4:1 vertical-to-horizontal aspect ratio can only be further divided vertically into two partitions, one above the other, each having a 2:1 vertical-to-horizontal aspect ratio.

[0118] In further examples, a 3-partitioning scheme may be used to partition a base block or any intermediate block, as shown in Figure 13. The 3-partitioning pattern may be implemented vertically, as shown in 1302 of Figure 13, or horizontally, as shown in 1304 of Figure 13. The exemplary partition ratios in Figure 13, vertically or horizontally, are shown as 1:2:1, but other ratios may be predefined. In some implementations, two or more different ratios may be predefined. Such a 3-partitioning scheme may be used to complement quadtree or binary tree partitioning structures, in that such 3-tree partitioning can capture objects located at the center of a block within a single contiguous partition, while quadtrees and binary trees always partition along the center of a block, thus dividing objects into separate partitions. In some implementations, the width and height of the partitions in the exemplary 3-tree are always powers of 2 to avoid additional transformations.

[0119] The above partitioning schemes may be combined in any way at different partitioning levels. For example, the quadtree and binary partitioning schemes described above may be combined to partition a base block into a quadtree-binary-tree (QTBT) structure. In such a scheme, the base block or intermediate block / partition may be quadtree-partitioned or binary-partitioned according to a set of predefined conditions, if specified. A concrete example is shown in Figure 14. In the example in Figure 14, the base block is first quadtree-partitioned into four partitions as shown by 1402, 1404, 1406, and 1408. Subsequently, each of the resulting partitions may be quad-tree partitioned into four further partitions at the next level (e.g., 1408), or binary into two further partitions (e.g., either horizontally or vertically) (e.g., 1402 or 1406, both symmetrical), or not partitioned at all (e.g., 1404). Binary or quad-tree partitioning may be recursively allowed for square partitions, as shown by the overall exemplary partitioning pattern in 1410 and the corresponding tree structure / representation in 1420, where solid lines represent quad-tree partitioning and dashed lines represent binary partitioning. A flag may be used for each binary partition node (non-leaf binary partition) to indicate whether the binary partitioning is horizontal or vertical. For example, as shown in 1420, consistent with the partitioning structure in 1410, a flag "0" may represent horizontal binary partitioning and a flag "1" may represent vertical binary partitioning. In the case of a quadtree partition, the quadtree partition always divides the block or partition both horizontally and vertically, producing four subblocks / partitions of equal size, so it is not necessary to indicate the partition type. In some implementations, the flag "1" may represent a horizontal binary partition, and the flag "0" may represent a vertical binary partition.

[0120] In some implementations of QTBT, the quadtree and binary partitioning rule set can be represented by the following predefined parameters and their associated corresponding functions: • CTU size: Root node size of a quad tree (size of the base block) • MinQTSize: Minimum allowable quad tree leaf node size • MaxBTSize: Maximum allowable 2-minute tree root node size • MaxBTDepth: Maximum allowable binary tree depth • MinBTSize: Minimum allowable size of a binary tree leaf node

[0121] In some embodiments of the QTBT partitioning structure, the CTU size may be set as a 128x128 chroma sample having two corresponding 64x64 blocks of chroma sample (when example chroma subsampling is considered and used), MinQTSize may be set to 16x16, MaxBTSize may be set to 64x64, MinBTSize (for both width and height) may be set to 4x4, and MaxBTDepth may be set to 4. Quadratic partitioning may be applied to the CTU first to generate a quadratic leaf node. The quadratic leaf node can have a size from its minimum allowable size (i.e., MinQTSize) of 16x16 to 128x128 (i.e., the CTU size). When the node is 128x128, the size exceeds MaxBTSize (i.e., 64x64), so the node is not initially partitioned by a binary tree. Otherwise, nodes that do not exceed MaxBTSize may be partitioned by a binary tree. In the example in Figure 14, the base block is 128x128. The base block can only be quadrubed according to a predefined set of rules. The base block has a partitioning depth of 0. Each of the four resulting partitions is 64x64 and does not exceed MaxBTSize, and may be further quadrubed or binary at level 1. The process continues. When the binary tree depth reaches MaxBTDepth (i.e., 4), no further partitioning is possible. When a binary tree node has a width equal to MinBTSize (i.e., 4), no further horizontal partitioning is possible. Similarly, when a binary tree node has a height equal to MinBTSize, no further vertical partitioning is possible.

[0122] In some embodiments, the above QTBT scheme may be configured to support the flexibility for the lumera and chroma to have the same QTBT structure or separate QTBT structures. For example, in the case of P and B slices, the lumera CTB and chroma CTB within a single CTU may share the same QTBT structure. However, in the case of I slices, the lumera CTB may be partitioned into CBs by a QTBT structure, and the chroma CTB may be partitioned into chroma CBs by another QTBT structure. This means that CUs may be used to refer to different color channels in I slices; for example, an I slice may consist of a coding block for the lumera component and coding blocks for the two chroma components, while a CU in a P or B slice may consist of coding blocks for all three color components.

[0123] In some other implementations, the QTBT scheme may be supplemented by the ternary scheme described above. Such implementations may be called multi-type-tree (MTT) structures. For example, in addition to binary partitioning of nodes, one of the ternary partitioning patterns in Figure 13 may be selected. In some implementations, only square nodes can undergo ternary partitioning. Additional flags may be used to indicate whether the ternary partitioning is horizontal or vertical.

[0124] The design of two-level or multi-level trees, such as QTBT implementation and QTBT implementation supplemented by 3-part division, can be motivated primarily by complexity reduction. Theoretically, the complexity of traversing the tree is T D Here, T represents the number of partition types and D is the depth of the tree. A trade-off may be made by using multiple types (T) while reducing the depth (D).

[0125] In some implementations, the CB may be further partitioned. For example, the CB may be further partitioned into multiple prediction blocks (PBs) for intra or interframe predictions during the coding and decoding processes. In other words, the CB may be further divided into different subpartitions, where individual prediction decisions / configurations may be made. In parallel, the CB may be further partitioned into multiple transformation blocks (TBs) to delineate the levels at which transformations or inverse transformations of the video data are performed. The partitioning schemes of the CB into PBs and TBs may be the same or different. For example, each partitioning scheme may be executed using its own procedure, for example, based on various characteristics of the video data. The PB and TB partitioning schemes may be independent in some implementations. The PB and TB partitioning schemes and boundaries may be correlated in some implementations. In some implementations, for example, the TBs may be partitioned after the PB partitioning, and in particular, each PB may be determined following the partitioning of the coding blocks and then further partitioned into one or more TBs. For example, in some implementations, the PB may be divided into one, two, four, or any other number of TBs.

[0126] In some implementations, lumar channels and chroma channels may be treated differently for partitioning base blocks into coding blocks, and further into prediction and / or transformation blocks. For example, in some implementations, partitioning coding blocks into prediction and / or transformation blocks may be permitted for lumar channels, while such partitioning of coding blocks into prediction and / or transformation blocks may not be permitted for chroma channels. In such implementations, transformation and / or prediction of lumar blocks may therefore only be performed at the coding block level. As another example, the minimum transformation block size for lumar channels and chroma channels may differ; for example, coding blocks in lumar channels may be permitted to be partitioned into smaller transformation and / or prediction blocks than those in chroma channels. As yet another example, the maximum depth of partitioning coding blocks into transformation and / or prediction blocks may differ between lumar channels and chroma channels; for example, coding blocks in lumar channels may be permitted to be partitioned into deeper transformation and / or prediction blocks than those in chroma channels. For example, a ruma coding block may be partitioned into multiple sized transformation blocks that can be repeated by recursive partitions down to a maximum of two levels, and transformation block shapes such as square, 2:1 / 1:2, and 4:1 / 1:4, as well as transformation block sizes from 4x4 to 64x64, may be permitted. In the case of a chroma block, however, the maximum number of transformation blocks specified for the ruma block may be permitted.

[0127] In some embodiments for partitioning coding blocks into PB, the depth, shape, and / or other characteristics of the PB partitioning may depend on whether the PB is intra or interconnected.

[0128] Partitioning of coding blocks (or prediction blocks) into transformation blocks may be performed recursively or non-recursively, further considering the transformation blocks at the boundaries of the coding or prediction blocks, in a variety of exemplary schemes including, but not limited to, quadtree partitioning and predefined pattern partitioning. In general, the resulting transformation blocks may be at different partitioning levels, may not be the same size, and may not be square in shape (for example, they may be rectangular with some permitted size and aspect ratio). Further examples are described in more detail below in relation to Figures 15, 16, and 17.

[0129] In some other implementations, however, the CB obtained by any of the above partitioning schemes may be used as the basic or minimal coding block for prediction and / or transformation. In other words, no further partitioning is performed for inter-prediction / intra-prediction and / or transformation. For example, the CB obtained by the above QTBT scheme may be used directly as the unit for performing prediction. Specifically, such a QTBT structure eliminates the concept of multiple partition types; for example, it eliminates the separation of CU, PU, ​​and TU, supporting greater flexibility in the CU / CB partition shapes described above. In such a QTBT block structure, the CU / CB can have either a square or rectangular shape. The leaf nodes of such a QTBT are used as the unit for prediction and transformation processing without further partitioning. This means that the CU, PU, ​​and TU have the same block size in such an exemplary QTBT coding block structure.

[0130] The various CB partitioning schemes described above, as well as further partitioning of CBs into PBs and / or TBs (including no PB / TB partitioning), may be combined in any way. The following specific implementations are given as non-limiting examples.

[0131] Specific examples of partitioning coding blocks and transformation blocks are described below. In such examples, the base block may be partitioned into coding blocks using recursive quadtree partitioning or the predefined partitioning patterns described above (e.g., the partitioning patterns in Figures 9 and 10). At each level, whether further quadtree partitioning of a particular partition should follow may be determined by local video data characteristics. The resulting CBs may have various quadtree partitioning levels and sizes. The decision on whether to code a picture area by interpicture (time) or intrapicture (spatial) prediction may be made at the CB level (or, for all three color channels, at the CU level). Each CB may be further partitioned into one, two, four, or other number of PBs according to a predefined PB partitioning type. The same prediction process may be applied within a single PB, and relevant information may be transmitted to the decoder at the PB level. After obtaining residual blocks by applying the prediction process based on the PB partitioning type, the CB may be partitioned into TBs according to other quadtree structures similar to the coding tree of the CB. In this particular implementation, the CB or TB does not need to be limited to a square shape. Furthermore, in this particular example, the PB may be square or rectangular for interpretation and square only for intrapretation. A coding block may be divided, for example, into four square-shaped TBs. Each TB may be further recursively (by quadtree partitioning) into smaller TBs called Residual Quadtrees (RQTs).

[0132] Further embodiments for partitioning base blocks into CBs, PBs, and / or TBs are described below. For example, instead of using multiple partition unit types as shown in Figure 9 or Figure 10, a quadtree segmentation structure including nested multi-type trees with binary and ternary partitioning (e.g., the QTBT described above or a QTBT with ternary partitioning) may be used. The separation of CBs, PBs, and TBs (i.e., partitioning CBs into PBs and / or TBs, and partitioning PBs into TBs) may be abandoned unless required for CBs that are too large for the maximum transformation length. However, such CBs may need to be further partitioned. This example partitioning scheme may be designed to support greater flexibility in the CB partition shape so that both prediction and transformation can be performed at the CB level without further partitioning. In such a coding tree structure, the CB may have either a square or rectangular shape. Specifically, a coding tree block (CTB) may first be partitioned by a quadtree structure. Next, the quadtree leaf nodes may be further partitioned by nested multitype trees. An example of a nested multitype tree structure using bipartite or tripartite partitioning is shown in Figure 11. Specifically, the example multitype tree structure in Figure 11 includes four partitioning types called vertical bipartite (SPLIT_BT_VER)(1102), horizontal bipartite (SPLIT_BT_HOR)(1104), vertical tripartite (SPLIT_TT_VER)(1106), and horizontal tripartite (SPLIT_TT_VER)(1108). In this case, CB corresponds to the leaf of the multitype tree. In this embodiment, this segmentation is used for both prediction and transformation processing without further partitioning, as long as CT is not too large relative to the maximum transformation length. This means that in most cases, CB, PB, and TB have the same block size in a quadtree structure containing nested multitype tree coding blocks. An exception occurs when the maximum supported transformation length is smaller than the width or height of the color component of CB.In some implementations, in addition to binary or trifoliation, the nested pattern in Figure 11 may further include quadtree partitioning.

[0133] Figure 12 shows one specific example of a quadtree structure containing nested multi-type tree coding blocks of block partitions (including quadtree, binary, and ternary partitioning options) for a single block base. More specifically, Figure 12 shows that the base block 1200 is quadtree partitioned into four square partitions 1202, 1204, 1206, and 1208. The decision to use further multi-tree structures and quadtrees as shown in Figure 11 for further partitioning is made for each of the quadtree-partitioned partitions. In the example in Figure 12, partition 1204 is not further partitioned. Partitions 1202 and 1208 each employ another quadtree partition. For partition 1202, the upper left, upper right, lower left, and lower right partitions, which are quadruced at the second level, are each divided at the third level into quadruced partitioning, horizontal bipartitioning 1104 in Figure 11, no division, and horizontal tripartitioning 1108 in Figure 11, respectively. Partition 1208 employs another quadruced partitioning, with the upper left, upper right, lower left, and lower right partitions, which are quadruced at the second level, each being divided at the third level into vertical tripartitioning 1106 in Figure 11, no division, no division, and horizontal bipartitioning 1104 in Figure 11, respectively. Two of the third-level subpartitions of the upper left partition of 1208 at the second level are further divided according to horizontal bipartitioning 1104 and horizontal tripartitioning 1108 in Figure 11, respectively. Partition 1206 is divided into two partitions by adopting a second-level division pattern according to the vertical bipartite division 1102 in Figure 11, and these two partitions are further divided at a third level according to the horizontal tripartite division 1108 and vertical bipartite division 1102 in Figure 11. A fourth-level division is further applied to one of these two partitions according to the horizontal bipartite division 1104 in Figure 11.

[0134] In the specific example above, the maximum rumor conversion size may be 64x64, and the maximum supported chroma conversion size may differ from the rumor, for example, 32x32. Even if the CB illustrated in Figure 12 is not generally further divided into smaller PB and / or TB, if the width or height of a rumor coding block or chroma coding block is greater than the maximum conversion width or height, the rumor coding block or chroma coding block may be automatically divided in that direction to satisfy the conversion size limitations in the horizontal and / or vertical directions.

[0135] In the specific examples of base block partitioning to the CB described above, the coding tree scheme can support the ability of lumens and chromens to have separate block tree structures, as stated above. For example, in the case of P and B slices, the lumens CTB and chromens CTB within a single CTU may share the same coding tree structure. In the case of I slices, for example, lumens and chromens may have separate coding block tree structures. When separate block tree structures are applied, the lumens CTB may be partitioned into lumens CBs by one coding tree structure, and the chromens CTB may be partitioned into chromens CBs by another coding tree structure. This means that in I slices, a CU can consist of coding blocks for the lumens component or coding blocks for the two chromens component, and in P or B slices, a CU always consists of coding blocks for all three color components unless the video is monochrome.

[0136] When a coding block is further partitioned into multiple transformation blocks, these transformation blocks may be ordered in the bitstream according to various orders or scanning schemes. Examples of partitioning coding blocks or prediction blocks into transformation blocks, and the coding order of the transformation blocks, are described further below. In some embodiments, as described above, transformation partitioning may support multiple shapes, e.g., 1:1 (square), 1:2 / 2:1, and 1:4 / 4:1, with transformation block sizes ranging, for example, from 4x4 to 64x64. In some implementations, when the coding block is 64x64 or less, transformation block partitioning may be applied only to the lumen component, so that for the chroma block, the transformation block size is the same as the coding block size. Otherwise, if the width or height of a coding block is greater than 64, both the rumor coding block and the chroma coding block may be implicitly divided into multiple transformation blocks of min(W,64)×min(H,64) and min(W,32)×min(H,32), respectively.

[0137] In some embodiments of transformation block partitioning, for both intra-coded and interconnected blocks, the coding block may be further partitioned into multiple transformation blocks with a partitioning depth of up to a predefined number of levels (e.g., 2 levels). The depth and size of the transformation block partitioning can be associated. For some embodiments, the mapping from the transformation size at the current depth to the transformation size at the next depth is shown in Table 1 below. [Table 1]

[0138] Based on the example mappings in Table 1, for a 1:1 square block, the next level of transformation partitioning may generate four 1:1 square sub-transformation blocks. The transformation partitioning may stop at, for example, 4x4. Thus, the transformation size of the current depth of 4x4 corresponds to the same size of 4x4 at the next depth. In the example in Table 1, for a 1:2 / 2:1 non-square block, the next level of transformation partitioning may generate two 1:1 square sub-transformation blocks, while for a 1:4 / 4:1 non-square block, the next level of transformation partitioning may generate two 1:2 / 2:1 sub-transformation blocks.

[0139] In some embodiments, additional restrictions may be applied to the transformation block partitioning in the case of the lumen component of an intra-coded block. For example, for each level of transformation partitioning, all sub-transformation blocks may be restricted to having equal sizes. For example, in the case of a 32×16 coding block, a level 1 transformation partition produces two 16×16 sub-transformation blocks, and a level 2 transformation partition produces eight 8×8 sub-transformation blocks. In other words, the second level partitioning should be applied to all first-level sub-blocks so that the transformation units are kept equal in size. An example of transformation block partitioning for an intra-coded square block according to Table 1 is shown in Figure 15, along with the coding order indicated by arrows. Specifically, 1502 shows a square coding block. A first-level partition into four equally sized transformation blocks according to Table 1 is shown in 1504, along with the coding order indicated by arrows. A second-level partition of all first-level equally sized blocks into 16 equally sized transformation blocks according to Table 1 is shown in 1506, along with the coding order indicated by arrows.

[0140] In some embodiments, the above restrictions on intracoding may not apply to the rumor component of an interconnected block. For example, after the first level of transform partitioning, one of the sub-transformation blocks may be further partitioned independently at one further level. Thus, the resulting transformation blocks may or may not be the same size. An example of partitioning an interconnected block into transformation blocks is shown in Figure 16, along with their coding order. In the example in Figure 16, the interconnected block 1602 is partitioned into transformation blocks at two levels according to Table 1. At the first level, the interconnected block is partitioned into four transformation blocks of equal size. Then, only one of the four transformation blocks (but not all of them) is further partitioned into four sub-transformation blocks, resulting in a total of seven transformation blocks of two different sizes, as shown by 1604. An example of the coding order of these seven transformation blocks is shown by the arrow in 1604 of Figure 16.

[0141] In some embodiments, additional restrictions may be applied to the chroma component on the transformation block. For example, the transformation block size for the chroma component may be the same as the coding block size, but it may not be smaller than a predefined size, e.g., 8x8.

[0142] In some other embodiments, for coding blocks where either the width (W) or height (H) is greater than 64, both the rumor coding block and the chroma coding block may be implicitly divided into multiple transformation units of min(W,64)×min(H,64) and min(W,32)×min(H,32), respectively. Hereinafter, "min(a,b)" may return the smaller of a and b.

[0143] Figure 17 further illustrates another alternative example of a scheme for partitioning coding blocks or prediction blocks into transformation blocks. As shown in Figure 17, instead of using recursive transformation partitioning, a predefined set of partitioning types may be applied to the coding block depending on the transformation type of the coding block. In the specific example shown in Figure 17, one of the six example partitioning types may be applied to divide the coding block into a varying number of transformation blocks. Such a scheme for generating transformation block partitioning may be applied to either coding blocks or prediction blocks.

[0144] More specifically, the partitioning scheme in Figure 17 provides up to six example partition types for any given transformation type (where the transformation type refers to, for example, the type of primary transformation such as ADST and others). In this scheme, any coding block or prediction block may be assigned a transformation partition type, for example, based on rate strain cost. For example, the transformation partition type assigned to a coding block or prediction block may be determined based on the transformation type of the coding block or prediction block. A particular transformation partition type may correspond to the transformation block partition size and pattern, as shown by the six transformation partition types represented in Figure 17. The correspondence between various transformation types and various transformation partitions may be predefined. An example is shown below using capital letters to indicate the transformation partition types that can be assigned to a coding block or prediction block based on rate strain cost: • PARTITION_NONE: Allocates a conversion size equal to the block size. • PARTITION_SPLIT: Allocates a conversion size that is half the width of the block size and half the height of the block size. • PARTITION_HORZ: Allocates a conversion size that has the same width as the block size and half the height of the block size. • PARTITION_VERT: Allocates a conversion size that is half the width of the block size and the same height as the block size. • PARTITION_HORZ4: Allocates a conversion size that has the same width as the block size and 1 / 4 of the block size's height. • PATRITION_VERT4: Allocates a conversion size that is 1 / 4 the width of the block size and the same height as the block size.

[0145] In the example above, all the transformation partition types shown in Figure 17 include a uniform transformation size for the partitioned transformation block. This is merely an example, not an limitation. In some other implementations, a mixed transformation block size may be used for the partitioned transformation block with a specific partition type (or pattern).

[0146] A PB (or CB, also called a PB if not further partitioned into prediction blocks) obtained from any of the above partitioning schemes can then become individual blocks for coding by either intra-prediction or inter-prediction. In the case of inter-prediction on the current PB, a residual is generated between the current block and the prediction block, coded, and may be included in the coded bitstream.

[0147] Interpretation may be performed, for example, in single-reference mode or compound-reference mode. In some implementations, a skip flag may be initially included in the bitstream of the current block (or at a higher level) to indicate whether the current block should be intercoded and not skipped. If the current block is intercoded, other flags may be further included in the bitstream as signals to indicate whether single-reference mode or compound-reference mode is used for predicting the current block. In single-reference mode, one reference block may be used to generate a predictive block for the current block. In compound-reference mode, two or more reference blocks may be used to generate a predictive block, for example, by a weighted average. Compound-reference mode may be called more-than-one reference mode, 2-reference mode, or multiple-reference mode. One or more reference blocks may be identified using a reference frame index, and further, using corresponding motion vectors that indicate the positional shift between the reference block and the current block, for example, in horizontal and vertical pixels. For example, in single-reference mode, the interpretation block of the current block may be generated from a single reference block identified as the prediction block by a single motion vector in the reference frame, whereas in composite-reference mode, the prediction block may be generated by a weighted average of two reference blocks in two reference frames indicated by two reference frame indices and two corresponding motion vectors. The motion vectors may be coded in various ways and included in the bitstream.

[0148] In some implementations, the encoding or decoding system may maintain a decoded picture buffer (DPB). Some images / pictures may be held in the DPB awaiting display (in decoding order), and some images / pictures in the DPB may be used as reference frames to enable interpretation (in the decoding or encoding system). In some implementations, reference frames in the DPB may be tagged as either short-term or long-term references for the current image being encoded or decoded. For example, short-term reference frames may include frames used for interpretation of blocks in the current frame or in a predefined number (e.g., 2) of subsequent video frames closest to the current frame in decoding order. Long-term reference frames are used to predict image blocks in frames that are a predefined number of frames away from the current frame in decoding order. Information about such tags for short-term and long-term reference frames may be called a Reference Picture Set (RPS) and may be added to the header of each frame in the encoded bitstream. Each frame in the encoded video stream may be identified by a Picture Order Counter (POC), which is numbered in an absolute manner according to the playback order, or in relation to a group of pictures starting, for example, from frame I.

[0149] In some embodiments, one or more reference picture lists, including the identification of short-term and long-term reference frames for interpretation, may be formed based on RPS information. For example, a single picture reference list may be formed for one-way interpretation and represented as L0 reference (or reference list 0), while two picture reference lists may be formed for bidirectional interpretation and represented as L0 (or reference list 0) and L1 (or reference list 1) for each of the two prediction directions. The reference frames included in the L0 and L1 lists may be ordered in various predetermined ways. The lengths of the L0 and L1 lists may be signaled in the video bitstream. One-way interpretation may be either single-reference mode or composite-reference mode if the multiple references for generating prediction blocks by weighted averaging in composite prediction mode are on the same side of the block to be predicted. Bidirectional interpretation is only composite mode in that bidirectional interpretation requires at least two reference blocks.

[0150] In some implementations, a merge mode (MM) may be used for interpretation. Generally, for a merge mode, one or more motion vectors in a single reference prediction or a composite reference prediction for the current PB may be derived from other motion vectors rather than being computed and signaled independently. For example, in an encoding system, the current motion vector of the current PB may be represented by the difference between the current motion vector and one or more other already encoded motion vectors (called reference motion vectors). Rather than the entire current motion vector, such a difference of motion vectors may be encoded and included in the bitstream and associated with the reference motion vectors. Correspondingly, in a decoding system, the motion vector corresponding to the current PB may be derived based on the decoded motion vector difference and the associated reference motion vectors. As a specific form of general merge mode (MM) interpretation, such interpretation based on motion vector differences is sometimes called a Merge Mode with Motion Vector Difference (MMVD). Generally, MM, or in particular MMVD, may be implemented to utilize correlations between motion vectors associated with different PPBs in this way to improve coding efficiency. For example, adjacent PBs may have similar motion vectors, so the MVD may be small and can be coded efficiently. As another example, motion vectors may be temporally correlated (between frames) for similarly positioned / located blocks in space.

[0151] In some embodiments, the MM flag may be included in the bitstream during encoding to indicate whether the current PB is in merge mode. Additionally, or alternatively, the MMVD flag may be included in the bitstream during encoding and signaled to indicate whether the current PB is in MMVD mode. The MM and / or MMVD flags or indicators may be supplied at the PB level, CB level, CU level, CTB level, CTU level, slice level, picture level, etc. For certain examples, both the MM and MMVD flags may be included for the current CU, and the MMVD flag may be signaled immediately after the skip flag and the MM flag to specify whether MMVD mode is used for the current CU.

[0152] In some embodiments of MMVD, a list of reference motion vectors (RMVs) or MV predictor candidates for motion vector prediction may be formed for the block being predicted. The list of RMV candidates may include a predetermined number (e.g., 2) of MV predictor candidate blocks whose motion vectors could be used to predict the current motion vector. The RMV candidate blocks may include blocks selected from adjacent blocks in the same frame and / or time blocks (blocks located identically in preceding or succeeding frames of the current frame). These options represent blocks that are spatially or temporally located relative to the current block and are likely to have a similar or identical motion vector. The size of the list of MV predictor candidates may be predetermined. For example, the list may include two or more candidates. In order to be included in the RMV candidate list, candidate blocks may be required to have, for example, the same reference frame (or multiple frames) as the current block, and must exist (e.g., boundary checking must be performed if the current block is near the edge of the frame), must have been encoded during the encoding process, and / or must have been decoded during the decoding process. In some implementations, the list of merge candidates may be populated first in spatially adjacent blocks (scanned in a specific predefined order) if available and satisfying the above conditions, and then in temporal blocks if space is still available in the list. Adjacent RMV candidate blocks may be selected, for example, from the blocks to the left and above the current block. The list of RMV predictor candidates may be dynamically formed as a Dynamic Reference List (DRL) at various levels (sequence, picture, frame, slice, superblock, etc.). The DRL may be signaled in the bitstream.

[0153] In some implementations, the actual MV predictor candidates used as reference motion vectors to predict the motion vector of the current block may be signaled. If the RMV candidate list contains two candidates, a one-bit flag called a merge candidate flag may be used to indicate the selection of the reference merge candidate. When the current block is predicted in composite mode, each of the multiple motion vectors predicted by the MV predictor may be associated with a reference motion vector from the merge candidate list. The encoder may determine which of the RMV candidates more closely predicts the current coding block and signal the selection as an index in the DRL.

[0154] In some embodiments of MMVD, after an RMV candidate is selected and used as a base motion vector prediction for predicting the motion vector, a motion vector difference (MVD or delta-MV representing the difference between the motion vector to be predicted and the reference candidate motion vector) may be calculated in the encoding system. Such an MVD may include information representing the magnitude and direction of the MV difference, both of which may be signaled in the bitstream. The magnitude and direction of the motion difference may be signaled in various ways.

[0155] In some embodiments of the MMVD, the distance index may be used to specify the magnitude information of the motion vector difference and to indicate one of a set of predefined offsets that represent a predefined motion vector difference from a starting point (reference motion vector). The MV offset, according to the signaled index, may then be added to either the horizontal or vertical component of the starting (reference) motion vector. Whether the horizontal or vertical component of the reference motion vector should be offset may be determined by the directional information of the MVD. Examples of predefined relationships between the distance index and predefined offsets are specified in Table 2. [Table 2]

[0156] In some embodiments of the MMVD, the distance index may be further signaled and used to represent the direction of the MVD relative to the reference motion vector. In some implementations, the direction may be restricted to either the horizontal or vertical direction. As an example, a 2-bit direction index is shown in Table 3. In the example in Table 3, the interpretation of the MVD may vary depending on the information of the start / reference MV. For example, if the start / reference MV corresponds to a single predictive block or a double predictive block, and both reference frame lists point to the same side of the current picture (i.e., the POCs of both reference pictures are either greater than or less than the POC of the current picture), the sign in Table 3 may specify the sign (direction) of the MV offset applied to the start / reference MV. If the start / reference MV corresponds to a biprediction block having two reference pictures on different sides of the current picture (i.e., the POC of one reference picture is greater than the POC of the current picture, and the POC of the other reference picture is smaller than the POC of the current picture), and the difference between the reference POC in picture reference list 0 and the current frame is greater than the difference between the reference POC in picture reference list 1 and the current frame, then the sign in Table 3 can specify the sign of the MV offset applied to the reference MV corresponding to the reference picture in picture reference list 0, and the sign of the offset for the MV corresponding to the reference picture in picture reference list 1 may have the opposite value (inverse sign of the offset). Otherwise, if the difference between the reference POC in picture reference list 1 and the current frame is greater than the difference between the reference POC in picture reference list 0 and the current frame, then the sign in Table 3 can specify the sign of the offset applied to the reference MV associated with picture reference list 1, and the sign of the offset for the reference MV associated with picture reference list 0 has the opposite value. [Table 3]

[0157] In some embodiments, the MVD may be scaled according to the difference in POC in each direction. If the difference from POC in both lists is the same, scaling is not necessary. If, instead, the difference in POC in reference list 0 is greater than that in reference list 1, the MVD of reference list 1 is scaled. If the difference in POC in reference list 1 is greater than that of list 0, the MVD of list 0 may be scaled similarly. When the starting MV is single-predicted, the MVD is added to the available MV or reference MV.

[0158] In some embodiments of MVD coding and signaling for bidirectional composite prediction, in addition to or instead of coding and signaling two MVDs separately, symmetric MVD coding may be performed so that only one MVD requires signaling, and the other MVD can be derived from the signaled MVD. In such an implementation, motion information including reference picture indices in both List 0 and List 1 is signaled. However, for example, only the MVD associated with reference list 0 is signaled, and the MVD associated with reference list 1 is derived without signaling. Specifically, to indicate whether reference list 1 is not signaled in the bitstream, a flag called "mvd_l1_zero_flag" may be included in the bitstream at the slice level. If this flag is 1, indicating that reference list 1 is equal to zero (and therefore not signaled), then a bidirectional prediction flag called "BiDirPredFlag" may be set to 0, meaning there is no bidirectional prediction. If mvd_l1_zero_flag is zero, BiDirPredFlag may be set to 1 if the nearest reference picture in List 0 and the nearest reference picture in List 1 form a forward-backward or backward-forward pair of reference pictures, and both reference pictures in List 0 and List 1 are short-term reference pictures. Otherwise, BiDirPredFlag is set to 0. A BiDirPredFlag of 1 may indicate that the symmetric mode flag is further signaled in the bitstream. The decoder may extract the symmetric mode flag from the bitstream when BiDirPredFlag is 1. The symmetric mode flag may be signaled (optionally) at the CU level, for example, to indicate whether a symmetric MVD coding mode is used for the corresponding CU. When the symmetric mode flag is 1, it means using the symmetric MVD coding mode and the reference picture indices in both List 0 and List 1 (called "mvp_l0_flag" and "mvp_l1_flag").This indicates that only the MVD associated with List 0 (referred to as "MVD0") is signaled, while the other motion vector difference "MVD1" should be derived rather than signaled. For example, MVD1 may be derived as -MVD0. Thus, in the example symmetric MVD mode, only one MVD is signaled. In some other embodiments of MV prediction, harmonized schemes may be used to implement common merged mode, MMVD, and other types of MV prediction for both single-reference mode and compound-reference mode MV predictions. Various syntactic elements may be used to indicate how the MV of the current block is predicted.

[0159] For example, in single-reference mode, the following MV prediction modes may be signaled: NEARMV: Directly uses one of the motion vector predictors (MVPs) in a list pointed to by a DRL (Dynamic Reference List) index without any MVD. • NEWMV: Uses one of the motion vector predictors (MVPs) in the list notified by the DRL index as a reference, and applies the delta to the MVP (e.g., using MVD). • GLOBALMV: Uses motion vectors based on global motion parameters at the frame level.

[0160] Similarly, in the case of a composite reference interpretation mode using two reference frames corresponding to the two MVs to be predicted, the following MV prediction modes may be signaled: • NEAR_NEARMV: For each of the two MVs to be predicted, use one of the motion vector predictors (MVPs) in the list notified by the DRL index without MVD. • NEAR_NEWMV: To predict the first of two motion vectors, use one of the motion vector predictors (MVPs) in the list notified by the DRL index as the reference MV without the MVD, and to predict the second of the two motion vectors, use one of the motion vector predictors (MVPs) in the list notified by the DRL index as the reference MV in conjunction with the further notified delta MV (MVD). • NEW_NEARMV: To predict the second of two motion vectors, use one of the motion vector predictors (MVPs) in the list notified by the DRL index as the reference MV without the MVD, and to predict the first of the two motion vectors, use one of the motion vector predictors (MVPs) in the list notified by the DRL index as the reference MV in conjunction with the further notified delta MV (MVD). • NEW_NEWMV: Uses one of the motion vector predictors (MVPs) in the list notified by the DRL index as the reference MV, and uses it in conjunction with the further notified delta MV to make predictions for each of the two MVs. • GLOBAL_GLOBALMV: Uses MV from each reference based on their frame-level global motion parameters.

[0161] The above terms “NEAR” refer to MV prediction that uses a reference MV without relying on MVD as a general merge mode, while “NEW” refer to MV prediction that uses a reference MV and involves offsetting it with a signaled MVD, as seen in MMVD mode. In the case of composite interpretation, both the above reference-based motion vector and motion vector delta may be generally different or independent between the two references, even if they are correlated and such correlation is used to reduce the amount of information needed to signal the two motion vector deltas. In such a situation, joint signaling of the two MVDs may be performed and indicated in the bitstream.

[0162] The above dynamic reference list (DRL) may be used to dynamically maintain and hold a set of indexed motion vectors that are considered candidate motion vector predictors.

[0163] In some embodiments, a predefined resolution for the MVD may be permitted. For example, a motion vector precision (or accuracy) of 1 / 8 of a pixel may be permitted. The above MVDs in various MV prediction modes may be configured and signaled in various ways. In some implementations, various syntax elements may be used to signal the above motion vector difference in reference frame list 0 or list 1.

[0164] For example, a syntax element called "mv_joint" can specify which components of the associated motion vector difference are non-zero. For MVD, this signals all non-zero components together. For example, mv_joint can have the following values: A value of 0 may indicate that there are no non-zero MVDs in either the horizontal or vertical direction. The value 1 may indicate that non-zero MVDs exist only along the horizontal direction. 2 could show that non-zero MVDs exist only along the vertical direction. 3 may indicate that non-zero MVDs exist in both the horizontal and vertical directions.

[0165] When the "mv_joint" syntax element of an MVD indicates that no non-zero MVDs exist, no further MVD information needs to be signaled. However, if the "mv_joint" syntax element indicates the presence of one or two non-zero components, additional syntax elements may be signaled for each of the non-zero MVD components, as described below.

[0166] For example, a syntax element called "mv_sign" may be used to further specify whether the corresponding motion vector difference component is positive or negative.

[0167] As another example, a syntax element called "mv_class" may be used to specify a class for a motion vector difference from a set of predefined classes for the corresponding non-zero MVD components. The predefined classes for motion vector differences may be used, for example, to divide a continuous size space of motion vector differences into non-overlapping ranges, each range corresponding to an MVD class. Thus, the notified MVD class indicates the size range of the corresponding MVD components. In the embodiments shown in Table 4 below, higher classes correspond to motion vector differences with larger size ranges. In Table 4, the symbol (n,m) is used to represent a range of motion vector differences that is greater than n pixels and less than or equal to m pixels. [Table 4]

[0168] In some other examples, a syntax element called “mv_bit” may be further used to specify the integer part of the offset between the start size of the MV class size range and the non-zero motion vector difference component, which is signaled in correspondence with the MV class size range. The number of bits required in “mv_bit” to indicate the full range of each MVD class may vary depending on the MV class. For example, in the implementations in Table 4, MV_CLASS0 and MV_CLASS1 require only a single bit to indicate an integer pixel offset of 1 or 2 from a start MVD of 0, and in the examples in Table 4, as the MV_CLASS increases, the number of bits required in “mv_bit” increases by one for each higher MV_CLASS than for the previous MV_CLASS.

[0169] In some other implementations, a syntax element called “mv_fr” may be further used to specify the first two fractional bits of the motion vector difference for the corresponding non-zero MVD component, while a syntax element called “mv_hp” may be used to specify the third fractional bit (high-resolution bit) of the motion vector difference for the corresponding non-zero MVD component. One bit of “mv_fr” essentially provides a 1 / 4 pixel MVD resolution, while the “mv_hp” bit may further provide an 1 / 8 pixel resolution. In some other implementations, more than one “mv_hp” may be used to provide an MVD pixel resolution finer than 1 / 8 pixel. In some embodiments, additional flags may be signaled at one or more different levels to indicate whether 1 / 8 pixel or higher MVD resolution is supported. The above syntax elements may not be signaled for the corresponding unsupported MVD resolution when the MVD resolution does not apply to a particular coding unit.

[0170] In some of the above embodiments, the fractional resolution may be independent of different classes of MVD. In other words, regardless of the magnitude of the motion vector difference, similar options for motion vector resolution may be provided using a predefined number of "mv_fr" and "mv_hp" bits to signal fractional MVD for non-zero MVD components.

[0171] However, in some other embodiments, the resolution of motion vector differences across different MVD size classes may be distinguished. Specifically, high-resolution MVDs for larger MVD sizes in higher MVD classes may not result in a statistically significant improvement in comparative efficiency. Therefore, MVDs may be coded with reduced resolution (integer pixel resolution or fractional pixel resolution) for larger MVD size ranges corresponding to higher MVD size classes. Similarly, MVDs may generally be coded with reduced resolution (integer pixel resolution or fractional pixel resolution) for larger MVD values. Such MVD class-dependent or MVD size-dependent MVD resolutions may generally be called adaptive MVD resolution, amplitude-dependent adaptive MVD resolution, or size-dependent MVD resolution. The term “resolution” may be further referred to as “pixel resolution.” Adaptive MVD resolution may be implemented in various ways, as described by the embodiments below, to achieve better overall compression efficiency. In particular, statistical observations suggest that treating the MVD resolution of large-scale or high-class MVDs at the same level as that of low-scale or low-class MVDs in an unadapted manner does not significantly improve the inter-predictive residual coding efficiency of blocks by large-scale or high-class MVDs. Therefore, the reduction in the number of signaling bits by aiming for a lower-precision MVD may outweigh the additional bits required to code the inter-predictive residuals as a result of such a lower-precision MVD. In other words, using a higher MVD resolution for large-scale or high-class MVDs may not yield as much coding gain as using a lower MVD resolution.

[0172] In some common implementations, the pixel resolution or precision of the MVD may or may not decrease as the MVD class increases. Lowering the pixel resolution of the MVD results in a coarser MVD (i.e., the step from one MVD level to the next becomes larger). In some implementations, the correspondence between MVD pixel resolution and MVD class may be specified, predefined, or pre-configured, and therefore may not need to be signaled in the encoded bitstream.

[0173] In some embodiments, the MV classes in Table 3 may be associated with different MVD pixel resolutions.

[0174] In some embodiments, each MVD class may be associated with a single permitted resolution. In some other embodiments, one or more MVD classes may be associated with two or more arbitrary MVD pixel resolutions. Following the signal in the bitstream for the current MVD component having such MVD classes may be additional signaling indicating which arbitrary pixel resolution is selected for the current MVD component.

[0175] In some embodiments, adaptively permitted MVD pixel resolutions may include, but are not limited to, 1 / 64pel (pixel), 1 / 32pel, 1 / 16pel, 1 / 8pel, 1 / 4pel, 1 / 2pel, 1pel, 2pel, 4pel, etc. (in descending order of resolution). Thus, each MVD class in ascending order may be associated with one of these resolutions in non-ascending order. In some implementations, an MVD class may be associated with two or more of the above resolution classes, and a higher resolution may be lower than or equal to a lower resolution of the previous MVD class. For example, if MV_CLASS_3 in Table 4 can be associated with any 1pel and 2pel resolutions, the highest resolution that MV_CLASS_4 in Table 4 can be associated with is 2pel. In some other implementations, the highest acceptable resolution of an MV class may be higher than the lowest acceptable resolution of the previous (lower) MV class. However, the average of the permitted resolutions of ascending MV classes may only be in non-ascending order.

[0176] In some implementations, where fractional pixel resolutions higher than 1 / 8pel are permitted, the "mv_fr" and "mv_hp" signaling can be extended accordingly to more than three fractional bits in total.

[0177] In some embodiments, fractional pixel resolution may only be permitted for MVD classes below the threshold MVD class. For example, fractional pixel resolution may only be permitted for MVD_CLASS_0 and not for all other classes in Table 4. Similarly, fractional pixel resolution may only be permitted for MVD classes below one of the other MV classes in Table 4. For the remaining MVD classes above the threshold MVD class, only integer pixel resolution of the MVD is permitted. In this way, fractional resolution signaling, such as one or more "mv_fr" and / or "mv_hp" bits, may not need to be signaled for MVDs that are signaled in MVD classes above the threshold MVD class. For MVD classes with resolutions lower than 1 pixel, the number of bits in the "mv_bit" signaling can be further reduced. For example, for MV_CLASS_5 in Table 4, the range of the MVD pixel offset is (32,649), so 5 bits are required to indicate the entire range with 1pel resolution. However, if MV_CLASS_5 is associated with 2pel MVD resolution (a resolution lower than 1 pixel resolution), then 4 bits are required for "mv_bit" instead of 5 bits, and neither "mv_fr" nor "mv_hp-" needs to be signaled after the signaling of "mv_class" as MV_CLASS_5.

[0178] In some embodiments, fractional pixel resolution may only be permitted for MVDs that have integer values ​​below a threshold integer pixel value. For example, fractional pixel resolution may only be permitted for MVDs smaller than 5 pixels. Corresponding to this example, fractional resolution may be permitted for MV_CLASS_0 and MV_CLASS_1 in Table 4, but not for all other MV classes. As another example, fractional pixel resolution may only be permitted for MVDs smaller than 7 pixels. Corresponding to this example, fractional resolution may be permitted for MV_CLASS_0 and MV_CLASS_1 (with a range of less than 5 pixels) in Table 4, but not for MV_CLASS_3 and higher classes (with a range of more than 5 pixels). For MVDs belonging to MV_CLASS_2 whose pixel range encompasses 5 pixels, fractional pixel resolution of the MVD may or may not be permitted depending on the value of "mv_bit". Fractional resolution may be permitted if the value of "mv_bit" is notified as 1 or 2 (so that the integer part of the notified MVD is 5 or 6, which is calculated as the start of the pixel range of MV_CLASS_2 with offset 1 or 2 indicated by "mv_bit"). Otherwise, fractional pixel resolution may not be permitted if the value of "mv_bit" is notified as 3 or 4 (so that the integer part of the notified MVD is 7 or 8).

[0179] In some other implementations, only a single MVD value may be allowed for MV classes above the threshold MV class. For example, such a threshold MV class may be MV_CLASS_2. Thus, MV_CLASS_2 and higher classes may only be allowed to have a single MVD value and have no fractional pixel resolution. The single allowed MVD values ​​for these MV classes may be predefined. In some examples, the allowed single value may be the upper limit of each range of these MV classes in Table 4. For example, MV_CLASS_2 through MV_CLASS_10 may be above the threshold class of MV_CLASS_2, and the single allowed MVD values ​​for these classes may be predefined as 8, 16, 32, 64, 128, 256, 512, 1024, and 2048, respectively. In some other examples, the allowed single value may be the median of each range of these MV classes in Table 4. For example, MV_CLASS_2 through MV_CLASS_10 may exceed the class threshold, and the single allowed MVD values ​​for these classes may be predefined as 6, 12, 24, 48, 96, 192, 384, 768, and 1536, respectively. Any other value within the range may also be defined as the single allowed resolution for each MVD class.

[0180] In the above implementation, if the signaled "mv_class" is greater than or equal to a predefined MVD class threshold, then "mv_class" alone is sufficient to determine the MVD value. In that case, the size and method of MVD are determined using "mv_class" and "mv_sign".

[0181] Therefore, when the MVD is notified for only one reference frame (either from reference frame list 0 or list 1, but not both), or when it is notified for two reference frames together, the precision (or resolution) of the MVD may depend on the relevant class of motion vector difference in Table 3 and / or the magnitude of the MVD.

[0182] In some other implementations, the pixel resolution or precision of an MVD may or may not decrease with increasing MVD class. For example, the pixel resolution may depend on the integer part of the MVD size. In some embodiments, fractional pixel resolution may only be permitted for MVD sizes below an amplitude threshold. In the case of a decoder, the integer part of the MVD size may first be extracted from the bitstream. Then the pixel resolution may be determined, and then it may be determined whether any fractional MVDs are present in the bitstream and need to be parsed (for example, if fractional pixel resolution is not permitted for the integer size of a particular extracted MVD, fractional MVD bits cannot be included in the bitstream that requires extraction). The above embodiments regarding adaptive MVD pixel resolution dependent on MVD class apply to adaptive MVD pixel resolution dependent on MVD size. For a particular example, MVD classes that exceed or contain a size threshold may be permitted to have only one predefined value.

[0183] The various embodiments described above apply to single-reference modes. These implementations also apply to NE_NEARMV, NEAR_NEWMV, and / or NEW_NEWMV modes, which are examples of composite prediction under MMVD. These implementations generally apply to the adaptive resolution of any MVD.

[0184] When adaptive (more specifically, size-adaptive) pixel resolution is used for MVD, various parameters relating to the MV and MVD of a coding block can be interdependent. Parameters considered to be related to MV or MVD broadly refer to information items that may affect how RMV is selected and detected, and how RMV and MVD are signaled, computed, or derived. These MV or MVD-related parameters include, but are not limited to, the following:

[0185] The above Dynamic Reference List (DRL) identifies an ordered list of candidate reference motion vectors (RMVs) for predicting the MV of the current coding block. For example, the DRL may identify a set of spatial or temporal neighbor block locations that are likely to have a motion vector similar to that of the current block. These locations correspond to RMVs that can be used as candidates for predicting the current motion vector. The encoder may select the RMV that most closely matches that of the current coding block from among these candidate RMVs, and then use that RMV to derive the corresponding MVD. The selected RMV may be represented or identified, for example, by its corresponding location or index in the DRL.

[0186] • The DRL index corresponding to the selected candidate RMV for the current coding block.

[0187] • Instructions for adopting adaptive MVD pixel resolution.

[0188] MVD information including, but not limited to, the aforementioned mv_joint, mv_sign, mv_class, mv_bit, mv_fr, and mv_hp.

[0189] Instructions for using motion compensation modes, such as overlap block motion compensation mode.

[0190] Instructions for using advanced motion compensation modes such as warp motion mode.

[0191] These information items or parameters can be interdependent, especially when adaptive MVD pixel resolution is used, so whether they are signaled or derived, the order in which they are signaled, the number of syntaxes used for signaling, and the context derivation for encoding / signaling those syntax elements can all be taken into consideration when designing a more efficient encoding / decoding scheme.

[0192] In some embodiments, whether one or more parameters related to MV or MVD are signaled in the video stream or derived from other signaled information may depend on whether adaptive MVD resolution is applied. Alternatively or additionally, if such particular parameters are signaled in the video stream, the way they are signaled may depend on whether adaptive MVD resolution is applied.

[0193] For example, when adaptive MVD pixel resolution is applied, it may be specified that only the first N RMV candidates in the DRL are allowed to be used by the encoder. In other words, the encoder may be required to select one of the first N entries from the DRL of the current coding block intercoded with adaptive MVD pixel resolution. In such an implementation, the encoder is free to select from the entire DRL list for the prediction of the current MV when adaptive MVD pixel resolution is not applied. The rationale for such an implementation may be based on the observation that RMVs with lower indices in the DRL are statistically more likely to predict the motion vector of the current block more precisely. In this way, the DRL index signaled in the bitstream for blocks using adaptive MVD pixel resolution requires fewer bits, and separate syntax elements may be used for signaling, for example.

[0194] Here, N is a positive number. N can be smaller than the entire indexing space of the DRL. In some specific implementations, N may be 1 or 2, meaning that when adaptive MVD pixel resolution is used in the current block, the predictor MV or RMV is always the first RMV or the first or second RMV in the DRL.

[0195] The value of N may be predefined or signaled in the bitstream. For example, N may be signaled at various coding levels, including but not limited to sequence level, frame level, slice level, title level, or superblock level. The value N is applied to various blocks within the signaled level that use adaptive MVD pixel resolution.

[0196] In some implementations, when the value of N is specified or signaled as 1, the DRL index does not need to be signaled in the bitstream for blocks using adaptive MVD pixel resolution. The index for extracting RMVs based on the DRL is automatically derived by the decoder as 1, referring to the first RMV candidate in the DRL. When the value of N is specified or signaled as 2, as another example, it may be used to signal the DRL index of a single-bit coding block.

[0197] In some other implementations, the above methods for signaling or deriving the DRL index may apply only when interpretation is in single-reference mode. In other words, when adaptive MVD pixel resolution is applied and interpretation is in single-reference mode rather than composite-reference mode, it may be specified or signaled that only the first N MVP candidates in the DRL are permitted to be used by the encoder. The derivation or signaling of the range of the DRL index under such conditions may be the same as described above, for example, N may be restricted, signaled, or predefined. When the interpretation mode is in composite-reference mode, or when the interpretation mode is in single-reference mode but adaptive MVD pixel resolution is not used, the RMV of the current coding block may be selected by the encoder from candidate RMVs corresponding to the entire indexing range of the DRL.

[0198] Therefore, the DRL indices signaled in the bitstream for various coding blocks can have different ranges (or number of bits). For coding blocks using adaptive MVD pixel resolution, their DRL indices may range from 1 to N when signaled, while for coding blocks not using adaptive MVD pixel resolution, their DRL indices may range from 1 to the entire DRL indexing range. Thus, the DRL indices under these different circumstances may follow different probabilistic models, and therefore, the entropic coding context of the signaled DRL indices may be used / derived in different ways depending on whether adaptive MVD pixel resolution is used or not. In other words, when adaptive MVD pixel resolution is used, a set of contexts may be used to signal the DRL indices. Otherwise, a set of contexts may be used to signal the DRL indices. The derivation of the context in the decoder for compounding the DRL indices will accordingly depend on whether adaptive MVD pixel resolution is used or not.

[0199] In some embodiments, signaling for motion compensation modes such as OBMC mode and / or warp motion may depend on whether adaptive MVD pixel resolution is used. For example, a flag indicating whether OBMC mode and / or warp motion is used may be signaled only if adaptive MVD pixel resolution is used for the current coding block. Otherwise, such a flag may not be signaled (for example, the decoder may infer that OBMC mode and / or warp motion are not used).

[0200] In some embodiments, the above signaling for OBMC mode and / or warp motion may also be conditional on the current coding block's interprediction mode being single-reference mode rather than composite-reference mode. In other words, the OBMC mode and / or warp motion flags may only be signaled if the current coding block is associated with single-reference mode and adaptive MVD pixel resolution is used.

[0201] In some other embodiments, adaptive MVD pixel resolution may be used only in single-reference interprediction mode. Therefore, when adaptive MVD pixel resolution is used in the current coding block determined by signaling or derivation, it indicates that interprediction is not in composite-reference mode. In that situation, a flag does not need to be signaled in the bitstream to indicate whether single-reference mode or composite-reference mode is being used in the current block.

[0202] In some other embodiments, a flag indicating whether the inter-intra composite mode is used may depend on whether adaptive MVD pixel resolution is applied and whether the current block is in single-reference mode. For example, the inter-intra composite mode for inter and intra-composite prediction of the current block can potentially only be used in single-reference mode and only when the inter-prediction is based on the use of adaptive MVD pixel resolution. Therefore, a flag signaling whether the inter-intra composite mode is used is unnecessary, and it is inferred that the inter-intra composite mode is not used when the current block is not predicted by single-reference mode and does not depend on adaptive MVD pixel resolution.

[0203] In some other embodiments, the context derivation for signaling other MVD-related syntax may depend on whether adaptive MVD pixel resolution is applied. These MVD-related syntax may include, as before, the aforementioned mv_joint, mv_class, mv_bit, mv_fr, mv_hp, etc. For example, mv_joint and / or mv_class can be statistically correlated with adaptive MVD pixel resolution and may follow different probabilistic models depending on whether adaptive MVD pixel resolution is applied. Specifically, when adaptive MVD resolution is applied, one context may be used / derived to signal mv_joint (or mv_class). Otherwise, one or more different contexts may be used / derived to signal mv_joint (or mv_class). The context dependency of mv_joint and / or mv_class on the use of adaptive MVD pixel resolution is merely an example. Other MVD-related syntax elements may also be associated with context derivations that depend on whether or not adaptive MVD pixel resolution is applied.

[0204] In the various embodiments described above, it is assumed that whether or not adaptive MVD pixel resolution is used in the current coding block can be first extracted (if signaled) or derived from the bitstream before determining any other MV or MVD-related information items or parameters, such as the DRL index. In some other embodiments, such other information items may instead be signaled or derived before determining whether or not adaptive MVD pixel resolution is used in the current coding block. Due to a correlation or interrelationship between such other information items and whether or not adaptive MVD pixel resolution should be used, the signaling or derivation of whether or not adaptive MVD pixel resolution is used may depend on such other information already extracted or derived from the bitstream.

[0205] For example, in some specific implementations, whether or not adaptive MVD resolution is applied may depend on the value of the DRL index. In such implementations, the DRL index used to determine the RMV of the current coding block may first be signaled in the bitstream (or derived from the bitstream in some other way). If it is notified that the DRL index of the current coding block is a value within the range of 1 to N, it is indicated that the selection of RMV candidates in the DRL can be performed within a finite range of DRL indexes. In that situation, the bitstream may further include a flag by the encoder to indicate whether or not adaptive MVD pixel resolution is used for the current coding block. However, if the notified DRL index is outside the range of 1 to n, it may be indicated that adaptive MVD pixel resolution is not used for the current coding block, so the flag to indicate whether or not adaptive MVD pixel resolution is used does not need to be included in the bitstream. In that situation, the decoder simply determines by derivation that adaptive MVD pixel resolution is not used, where N is a positive number smaller than the DRL's dex range. For example, N may be 1 or 2. In some implementations, N may be predefined. In some other implementations, N may be signaled. For example, N may be signaled at various signaling levels, such as sequence level, frame level, slice level, title level, or superblock level.

[0206] Figure 18 shows a flowchart 1800 of an exemplary method following the principles underlying the above implementation for adaptive MVD resolution and its signaling. The exemplary decoding method begins at S1801. At S1810, the video stream is received. At S1820, it is determined that video blocks will be intercoded based on predicted blocks and motion vectors (MV), where the MV is derived from the motion vector difference (MVD) and reference motion vector (RMV) of the video blocks. At S1830, data items related to at least one of the MVD or RMV are extracted or derived from the video stream in a manner that at least depends on whether the MVD is coded with a size-dependent adaptive MVD pixel resolution. At S1840, the MVD is extracted from the video stream, the MV is derived based on the extracted RMV and MVD, and the video blocks are reconstructed based at least on the MV and predicted blocks. The exemplary method ends at S1899.

[0207] Figure 19 shows a flowchart 1900 of an exemplary method following the principles underlying the above implementation for adaptive MVD resolution and its signaling. The exemplary decoding method begins at S1901. At S1910, the video stream is received. At S1920, it is determined that the video blocks will be intercoded based on predicted blocks and motion vectors (MV), where the MVs are derived from the motion vector difference (MVD) and reference motion vector (RMV) of the video blocks. At S1930, the RMV indices of the video blocks are extracted to map to a dynamic reference list (DRL), which is configured to identify multiple ordered candidate RMVs. At S1940, it is determined, based on the RMV index value, whether the MVDs will be coded with a size-dependent adaptive MVD pixel resolution. The exemplary method ends at S1999.

[0208] In embodiments and implementations of the present disclosure, any steps and / or operations may be combined or arranged in any quantity or order as desired. Two or more steps and / or operations may be performed in parallel. Embodiments and implementations of the present disclosure may be used separately or combined in any order. Furthermore, each of the methods (or embodiments), encoders, and decoders may be implemented by processing circuits (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored on a non-temporary computer-readable medium. Embodiments of the present disclosure may be applied to rumor blocks or chroma blocks. The term block may be interpreted as a prediction block, coding block, or coding unit, i.e., CU. The term block may also be used to refer to a transformation block. In the following sections, when we refer to block size, we may mean the width or height of the block, or the maximum width and height, or the minimum width and height, or the area size (width × height), or the aspect ratio of the block (width:height or height:width).

[0209] The above technology is implemented as computer software using computer-readable instructions and can be physically stored on one or more computer-readable media. For example, Figure 20 shows a computer system (2000) suitable for implementing a particular embodiment of the subject matter of the disclosure.

[0210] Computer software can be coded in any suitable machine code or computer language that can follow mechanisms such as assembly, compilation, and linking to generate code containing instructions that can be executed directly or through interpretation, microcode execution, etc., by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc.

[0211] The instructions can be executed on various types of computers or their components, including, for example, personal computers, tablet computers, servers, smartphones, game consoles, and Internet of Things devices.

[0212] The components shown in Figure 20 with respect to the computer system (2000) are illustrative in nature and are not intended to imply any limitation on the scope of use or functionality of computer software implementing embodiments of this disclosure. The configuration of the components should not be construed as having any dependency or requirement on any one or combination of the components described in the exemplary embodiments of the computer system (2000).

[0213] The computer system (2000) may include certain human interface input devices. Such human interface input devices may respond to input from one or more users, for example, through tactile input (e.g., keystrokes, swipes, dataglobe movements), voice input (e.g., voice, clapping), visual input (e.g., gestures), and olfactory input (not shown). The human interface devices may also be used to capture certain media that are not necessarily directly related to conscious human input, such as voice (e.g., speech, music, ambient sounds), images (e.g., scanned images, photographic images acquired from a still camera), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0214] An input human interface device may include one or more of the following: keyboard (2001), mouse (2002), trackpad (2003), touchscreen (2010), data glove (not shown), joystick (2005), microphone (2006), scanner (2007), and camera (2008) (only one of each is shown).

[0215] The computer system (2000) may also include certain human interface output devices. Such human interface output devices may stimulate the senses of one or more users, for example, through tactile output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touchscreen (2010), data glove (not shown), or joystick (2005), although haptic feedback devices that do not function as input devices may also exist), audio output devices (e.g., speakers (2009), headphones (not shown)), visual output devices (e.g., CRT screens, LCD screens, plasma screens, OLED screens, each with or without touchscreen input functionality and each with or without haptic feedback functionality, some of which are screens (2010) capable of outputting two-dimensional visual output or output in more than three dimensions by means such as stereoscopic output, virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).

[0216] A computer system (2000) may also include human-accessible storage devices and their associated media, such as CD / DVD ROM / RW (2020), thumb drives (2022), removable hard disks or solid-state drives (2023), legacy magnetic media, such as tapes and floppy disks (not shown), dedicated ROM / ASIC / PLD-based devices, such as security dongles (not shown), and the like.

[0217] Those skilled in the art will understand that the term “computer-readable medium” as used in relation to the subject matter currently disclosed does not include transmission media, carrier waves, or other transient signals.

[0218] A computer system (2000) may also include an interface (2054) to one or more communication networks (2055). These networks may be, for example, wireless, wireline, or optical. Networks may further be local, wide-area, metropolitan, vehicle and industrial, real-time, latency-tolerant, etc. Examples of networks include local area networks such as Ethernet®, cellular networks including wireless LAN, GSM, 3G, 4G, 5G, LTE, etc., TV wireline or wireless wide-area digital networks including cable TV, satellite TV, and terrestrial TV, and vehicle and factory networks including CANBus. Certain networks generally require an external network interface adapter attached to a specific general-purpose data port or peripheral bus (2049) (e.g., a USB port on a computer system (2000)). Others are generally integrated into the core of the computer system (2000) by attachment to a system bus as described below (e.g., an Ethernet network to a PC computer system, or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system (2000) can communicate with other entities. Such communication can be unidirectional and receive-only (e.g., broadcast TV) or unidirectional and transmit-only (e.g., CANBus to a specific CANBus device), or it can be bidirectional to other computer systems using, for example, a local or wide-area digital network. Specific protocols or protocol stacks are available for use with each of the networks and network interfaces described above.

[0219] The above-mentioned human interface devices, human-accessible memory devices, and network interfaces may be attached to the core (2040) of the computer system (2000).

[0220] The core (2040) may include one or more central processing units (CPUs) (2041), graphics processing units (GPUs) (2042), dedicated programmable processing units in the form of field-programmable gate areas (FPGAs) (2043), hardware accelerators for specific tasks (2044), graphics adapters (2050), etc. These devices may be connected via a system bus (2048) along with read-only memory (ROM) (2045), random access memory (RAM) (2046), internal mass storage devices such as internal user-inaccessible hard drives, SSDs, etc. (2047). In some computer systems, the system bus (2048) may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripherals may be attached directly to the core's system bus (2048) or via a peripheral bus (2049). For example, a screen (2010) could be connected to a graphics adapter (2050). Architectures for peripheral buses include PCI and USB.

[0221] The CPU (2041), GPU (2042), FPGA (2043), and accelerator (2044) are capable of executing specific instructions that can be combined to form the computer code described above. This computer code can be stored in ROM (2045) or RAM (2046). Temporary data can also be stored in RAM (2046), while persistent data can be stored, for example, in a built-in mass storage device (2047). High-speed storage and retrieval to any of the memory devices is made possible by the use of cache memory. Cache memory may be closely associated with one or more CPUs (2041), GPUs (2042), mass storage devices (2047), ROMs (2045), RAM (2046), etc.

[0222] Computer-readable media may contain computer code for performing various computer implementation operations. The media and computer code may be specifically designed and configured for the purposes of this disclosure, or they may be of a type commonly known and available to those with ordinary skill in computer software technology.

[0223] As a non-limiting example, a computer system having architecture (2000), specifically core (2040), can provide functionality as a result of a processor (including CPUs, GPUs, FPGAs, accelerators, etc.) that runs software embodied in one or more tangible computer-readable media. Such computer-readable media may be media related to user-accessible mass storage devices described above, in addition to specific storage devices of core (2040) that are non-transient in nature, such as core-integrated mass storage (2047) or ROM (2045). Software implementing various embodiments of this disclosure is stored in such devices and is executable by core (2040). Depending on the specific needs, the computer-readable media may include one or more memory devices or chips. The software may cause the core (2040) and, specifically, the processor within it (including CPUs, GPUs, FPGAs, etc.) to execute a particular process or a particular part of a particular process as described herein, including defining data structures stored in RAM (2046) and modifying such data structures according to a process defined by the software. Additionally, or alternatively, a computer system may provide functionality as a result of logic (e.g., accelerators (2044)) hardwired or otherwise embodied in the circuitry, which can operate in place of or with the software to execute a particular process or a particular part of a particular process as described herein. References to software may, as necessary, include logic, and vice versa. References to computer-readable media may, as necessary, include circuitry storing software for execution (e.g., integrated circuits (ICs)), circuitry embodying logic for execution, or both. This disclosure also encompasses any suitable combination of hardware and software.

[0224] While this disclosure has described several exemplary embodiments, there are alternatives, substitutions, and various alternative equivalents within the scope of the disclosure. Therefore, those skilled in the art should understand that numerous systems and methods embodying the principles of this disclosure, and thus falling within its spirit and scope, can be conceived, even if not expressly illustrated or described herein.

[0225] Appendix A: Acronyms JEM: Joint Exploration Model VVC: Versatile Video Coding BMS: Benchmark Set MV: Motion Vector HEVC:High Efficiency Video Coding SEI:Supplementary Enhancement Information VUI:Video Usability Information GOP: Group of Picture(s) TU: Transform Unit(s) PU: Prediction Unit(s) CTU: Coding Tree Unit(s) CTB: Coding Tree Block(s) PB: Prediction Block(s) HRD:Hypothetical Reference Decoder SNR: Signal Noise Ratio CPU:Central Processing Unit(s) GPU:Graphics Processing Unit(s) CRT: Cathode Ray Tube LCD: Liquid-Crystal Display OLED:Organic Light-Emitting Diode CD:Compact Disc DVD:Digital Video Disc ROM:Read-Only Memory RAM:Random Access Memory ASIC:Application-Specific Integrated Circuit PLD:Programmable Logic Device LAN:Local Area Network GSM:Global System for Mobile communications LTE:Long-Term Evolution CANBus:Controller Area Network Bus USB:Universal Serial Bus PCI:Peripheral Component Interconnect FPGA:Field Programmable Gate Area(s) SSD:Solid-State Drive IC:Integrated Circuit HDR:High Dynamic Range SDR:Standard Dynamic Range JVET:Joint Video Exploration Team MPM:Most Probable Mode WAIP:Wide-Angle Intra Prediction CU:Coding Unit PU:Prediction Unit TU:Transform Unit CTU:Coding Tree Unit PDPC:Position Dependent Prediction Combination ISP: Intra Sub-Partitions SPS: Sequence Parameter Setting PPS: Picture Parameter Set APS: Adaptation Parameter Set VPS: Video Parameter Set DSP: Decoding Parameter Set ALF: Adaptive Loop Filter SAO: Sample Adaptive Offset CC-ALF: Cross-Component Adaptive Loop Filter CDEF: Constrained Directional Enhancement Filter CCSO: Cross-Component Sample Offset LSO: Local Sample Offset LR: Loop Restoration Filter AV1: AOMedia Video 1 AV2: AOMedia Video 2 MVD: Motion Vector difference CfL: Chroma from Luma SDT: Semi Decoupled Tree SDP: Semi Decoupled Partitioning SST: Semi Separate Tree SB: Super Block IBC(or Intra BC): Intra Block Copy CDF: Cumulative Density Function SCC: Screen Content Coding GBI: Generalized Bi-prediction BCW:Bi-prediction with CU-level Weights CIIP:Combined intra-inter prediction POC:Picture Order Count RPS:Reference Picture Set DPB:Decoded Picture Buffer MMVD:Merge Mode with Motion Vector Difference

Claims

1. A method of video decoding performed by a video decoder, The steps include receiving coded information for the current block in the current picture from the coded video bitstream, The steps include partitioning the current block into multiple subblocks, The step of obtaining a first flag contained in the coded information, the first flag indicating whether a cross-component linear model prediction (CCLM) is applied to the current block, in which the chroma sample of the current block is predicted based on the reconfigured lumen sample of the current block. In response to the first flag indicating that the CCLM is applied to the current block, the steps include determining the predicted sample value of each of the chroma samples in each of the plurality of subblocks of the current block based on the CCLM, The steps include: reconstructing the current block based on the respective predicted sample values ​​of the chroma samples in each of the plurality of subblocks of the current block; It has, The step of partitioning the current block is: The method further comprises the step of partitioning the current block into a plurality of subblocks along the width direction, based on the fact that the width of the current block is equal to or greater than the height of the current block. method.

2. A method for video decoding performed by a video decoder, The steps include receiving coded information for the current block in the current picture from the coded video bitstream, The steps include partitioning the current block into multiple subblocks, The step of obtaining a first flag contained in the coded information, the first flag indicating whether a cross-component linear model prediction (CCLM) is applied to the current block, in which the chroma sample of the current block is predicted based on the reconfigured lumen sample of the current block. In response to the first flag indicating that the CCLM is applied to the current block, the steps include determining the predicted sample value of each of the chroma samples in each of the plurality of subblocks of the current block based on the CCLM, The steps include: reconstructing the current block based on the respective predicted sample values ​​of the chroma samples in each of the plurality of subblocks of the current block; It has, The step of partitioning the current block is: The method further comprises the step of partitioning the current block into a plurality of subblocks along the height direction, based on the fact that the width of the current block is less than the height of the current block. method.

3. A method for video decoding performed by a video decoder, The steps include receiving coded information for the current block in the current picture from the coded video bitstream, The steps include partitioning the current block into multiple subblocks, The step of obtaining a first flag contained in the coded information, the first flag indicating whether a cross-component linear model prediction (CCLM) is applied to the current block, in which the chroma sample of the current block is predicted based on the reconfigured lumen sample of the current block. In response to the first flag indicating that the CCLM is applied to the current block, the steps include determining the predicted sample value of each of the chroma samples in each of the plurality of subblocks of the current block based on the CCLM, The steps include: reconstructing the current block based on the respective predicted sample values ​​of the chroma samples in each of the plurality of subblocks of the current block; It has, The step of partitioning the current block is: The method further comprises the step of partitioning the current block into a plurality of subblocks of the minimum subblock size along both the height and width directions. method.

4. The steps include obtaining syntax elements within the coded information, The steps include determining the minimum subblock size based on the syntax elements, and It further possesses, The syntax element is located in one of the following: sequence parameter set (SPS), picture parameter set (PPS), slice, and tile. The method according to claim 3.

5. The aforementioned decision-making step is: The method further includes the step of determining the predicted sample value of the chroma sample in the second subblock of the plurality of subblocks based on the reconfigured sample of the first subblock of the plurality of subblocks, The second subblock is adjacent to the first subblock, The method according to any one of claims 1 to 3.

6. The aforementioned decision-making step is: In response to the first flag indicating that the CCLM is applied to the current block, The steps include obtaining a second flag contained in the coded information, the second flag indicating whether the CCLM is applied to each of the plurality of subblocks, In response to the second flag indicating that the CCLM is applied to each of the plurality of subblocks, the steps include determining the respective predicted sample values ​​of the chroma sample in each of the plurality of subblocks of the current block based on the CCLM. It further has, The method according to any one of claims 1 to 3.

7. The step of determining each of the aforementioned predicted sample values ​​is: A step of determining a predicted sample value of the chroma sample in the first subblock of the plurality of subblocks based on a first mode of the CCLM in response that the reconstructed neighboring sample is adjacent to the left side of the first subblock of the plurality of subblocks, wherein the first mode of the CCLM indicates that the predicted sample value of the chroma sample in the first subblock is determined based on the reconstructed neighboring sample adjacent to the left side of the first subblock. A step of determining the predicted sample value of the chroma sample in the second subblock of the plurality of subblocks based on a second mode of the CCLM in response that the reconstructed neighboring sample is adjacent to the left and above the second subblock of the plurality of subblocks, wherein the second mode of the CCLM indicates that the predicted sample value of the chroma sample in the second subblock is determined based on the reconstructed neighboring sample that is adjacent to the left and above the second subblock. It further has, The method according to claim 6.

8. The aforementioned decision-making step is: In response to the first flag indicating that the CCLM is applied to the current block, The steps include obtaining a second flag contained in the coded information, the second flag indicating whether the CCLM is applied to each of the plurality of subblocks, A step of obtaining an index contained in the coded information, wherein the index indicates the CCLM mode of the CCLM, and the CCLM mode indicates which reconstructed neighboring samples are applied by the CCLM to generate the respective predicted sample values ​​of the chroma samples in each of the plurality of subblocks. The second flag indicates that the CCLM is applied to each of the plurality of subblocks, and the index indicates the CCLM mode. In response to this, the steps include determining the respective predicted sample values ​​of the chroma samples in each of the plurality of subblocks of the current block based on the CCLM using the CCLM mode. It further has, The method according to any one of claims 1 to 3.

9. The aforementioned decision-making step is: In response to the index indicating a first CCLM mode, the steps include determining the respective predicted sample values ​​of the chroma sample in each of the plurality of subblocks based on reconstructed neighbor samples adjacent to the left and above one of the plurality of subblocks, In response to the index indicating a second CCLM mode, the steps include determining the respective predicted sample values ​​of the chroma sample in each of the multiple subblocks based on the reconstructed neighboring sample adjacent to the left of each of the multiple subblocks, In response to the index indicating a third CCLM mode, the steps include determining the respective predicted sample values ​​of the chroma sample in each of the multiple subblocks based on the reconstructed neighboring sample adjacent to the upper side of each of the multiple subblocks, and It further has, The method according to claim 8.

10. At least one memory configured to store program code, A processing circuit configured to read the program code and operate as instructed by the program code It has, An apparatus that, when the program code is executed by the processing circuit, causes the processing circuit to perform the method according to any one of claims 1 to 3.

11. A video encoding method performed by a video encoder, The steps involve partitioning the current block within the current picture of a video into multiple subblocks, The steps include determining the predicted sample value of each chroma sample in each of the plurality of subblocks of the current block based on a cross-component linear model prediction (CCLM) in which the chroma sample of the current block is predicted based on the reconstructed lumen sample of the current block, The steps include performing an intra-prediction on the current block based on the predicted sample values ​​of the chroma samples in each of the plurality of sub-blocks of the current block, The steps include generating a first flag indicating that the CCLM is applied to the plurality of subblocks of the current block, and encoding the information including the first flag into a coded video bitstream. It has, The step of partitioning the current block is: The method further comprises the step of partitioning the current block into a plurality of subblocks along the width direction, based on the fact that the width of the current block is equal to or greater than the height of the current block. method.

Images