Improved context derivation for motion vector difference coding

Abstract

This disclosure relates to encoding and decoding motion vector differences for interpreting video blocks. [Solution] An example of decoding interpredicted video blocks of a video stream is disclosed. The method includes the steps of: deriving a motion vector prediction mode for a video block; deriving a context from the video stream for signaling a set of syntax elements associated with a motion vector difference (MVD) associated with the video block, based on the interpretation mode and / or the motion vector prediction mode; and decoding the video block based on the set of syntax elements and the derived context.

Description

[[Technical Field]]

[0001] Cross-Reference to Related Applications This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 272,648 filed on October 27, 2021, U.S. Provisional Patent Application No. 63 / 289,124 filed on December 13, 2021, and U.S. Non-Provisional Patent Application No. 17 / 700,887 filed on March 22, 2022, all of which are hereby incorporated by reference in their entireties into this specification.

[0002] The present disclosure generally relates to video coding and / or decoding, and more particularly to an improved context derivation method for motion vector difference coding. [[Background Art]]

[0003] The description of the background art provided herein is for the purpose of generally presenting the context of the present disclosure. The work of the inventors, to the extent that the work is described in this section of background art, together with any aspects of the description that may not otherwise be recognized as prior art at the time of filing of the present application, is not expressly or implicitly admitted as prior art to the present disclosure.

[0004] Video coding and decoding can be performed using interpicture prediction with motion compensation. Uncompressed digital video can contain a series of pictures, each picture having spatial dimensions of, for example, 1920×1080 luminance samples and associated fully sampled or subsampled color difference samples. The series of pictures can have a fixed or variable picture rate (also called frame rate), for example, 60 pictures per second or 60 frames per second. Uncompressed video has specific bitrate requirements for streaming or data processing. For example, video with a pixel resolution of 1920×1080, a frame rate of 60 frames / second, and 4:2:0 chroma subsampling with 8 bits per pixel per color channel requires a bandwidth of nearly 1.5 Gbit / s. One hour of such video requires more than 600 GByte of storage space.

[0005] One purpose of video coding and decoding may be to reduce the redundancy of uncompressed input video signals through compression. Compression can help reduce the aforementioned bandwidth and / or storage space requirements by more than two orders of magnitude, in some cases. Both lossless and lossy compression, as well as combinations thereof, can be employed. Lossless compression refers to a technique in which an exact copy of the original signal can be reconstructed from the compressed original signal through the decoding process. Lossy compression refers to a coding / decoding process in which the original video information is not fully preserved during coding and cannot be fully restored during decoding. When using lossy compression, the reconstructed signal may not be identical to the original signal, but the distortion between the original and reconstructed signals is small enough, with some information loss, to make the reconstructed signal useful for its intended purpose. For video, lossy compression is widely adopted in many applications. The amount of distortion that can be tolerated depends on the application. For example, users of certain consumer video streaming applications may tolerate higher distortion than users of film or television broadcast applications. The compression ratio achievable by a particular coding algorithm may be selected or adjusted to reflect varying distortion tolerances. In other words, generally speaking, the higher the distortion tolerance, the more possible coding algorithms are that result in higher loss and higher compression ratios.

[0006] Video encoders and decoders can utilize techniques from several broad categories and steps, including, for example, motion compensation, Fourier transform, quantization, and entropy coding.

[0007] Video codec techniques may include a technique known as intra-coding. In intra-coding, sample values ​​are represented without referencing samples or other data from a previously reconstructed reference picture. In some video codecs, the picture is spatially subdivided into blocks of samples. If all blocks of samples are coded in intra-mode, the picture can be called an intra-picture. Intra-pictures and their derived pictures, such as independent decoder refresh pictures, may be used to reset the decoder state and therefore may be used as the first picture in the coded video bitstream and video session, or as a still image. The samples in the intra-predicted blocks can then be transformed into the frequency domain, and the resulting transformation coefficients can be quantized before entropy coding. Intra-prediction represents a technique that minimizes the sample values ​​in the pre-transformation domain. In some cases, the smaller the post-transformation DC value and the smaller the AC coefficient, the fewer bits are required at a given quantization step size to represent the block after entropy coding.

[0008] For example, traditional intra-coding, as known from MPEG-2 generation coding techniques, does not use intra-prediction. However, some newer video compression techniques include methods that attempt to code / decode blocks based on surrounding sample data and / or metadata that precede the block of data being intra-coded or intra-decoded in decoding order, obtained, for example, during spatially adjacent encoding and / or decoding. Such techniques are hereafter referred to as “intra-prediction” techniques. It should be noted that, at least in some cases, intra-prediction uses reference data only from the current picture being reconstructed and not from reference data from other reference pictures.

[0009] Intra-prediction can take many different forms. If two or more of these techniques are available in a given video coding technique, the techniques used can be called an intra-prediction mode. One or more intra-prediction modes may be provided in a particular codec. In certain cases, a mode may have submodes and / or be associated with various parameters, and the mode / submode information and intra-coding parameters of blocks of video can be coded individually or collectively included in the mode's codeword. The choice of codeword for a given combination of mode, submode, and / or parameters can affect the improvement of coding efficiency via intra-prediction, and therefore the entropy coding technique used to convert the codeword into a bitstream can also have an effect.

[0010] Certain modes of intra-prediction were introduced in H.264, improved in H.265, and further refined in newer coding techniques such as Joint Search Models (JEM), Versatile Video Coding (VVC), and Benchmark Sets (BMS). Generally, intra-prediction allows predictor blocks to be formed using available neighboring sample values. For example, available values ​​for a particular set of neighboring samples along a specific direction and / or line may be copied into a predictor block. References to the direction in use may be coded in the bitstream or may be predicted themselves.

[0011] Referring to Figure 1A, the lower right shows a subset of the nine predictor directions specified by the 33 possible intra-predictor directions in H.265 (corresponding to 33 of the 35 intra-modes specified in H.265, i.e., 33 angular modes). The point where the arrows converge (101) represents the predicted sample. The arrows indicate the direction from which neighboring samples are used to predict the sample at 101. For example, arrow (102) indicates that sample (101) is predicted from one or more neighboring samples to the upper right at an angle of 45 degrees from the horizontal. Similarly, arrow (103) indicates that sample (101) is predicted from one or more neighboring samples to the lower left of sample (101) at an angle of 22.5 degrees from the horizontal.

[0012] Referring further to Figure 1A, the upper left shows a 4x4 sample square block (104) (indicated by a thick dashed line). The square block (104) contains 16 samples, each labeled with "S", its position in the Y dimension (e.g., row index), and its position in the X dimension (e.g., column index). For example, sample S21 is the second sample (from the top) in the Y dimension and the first sample (from the left) in the X dimension. Similarly, sample S44 is the fourth sample in both the Y and X dimensions within block (104). Since the block is 4x4 samples in size, S44 is in the lower right. Further exemplary reference samples following a similar numbering scheme are shown. The reference samples are labeled with R relative to block (104), its Y position (e.g., row index), and its X position (column index). In both H.264 and H.265, predicted samples adjacent to the block being reconstructed are used.

[0013] Intra-picture prediction in block 104 may begin by copying a reference sample value from an adjacent sample according to the signaled prediction direction. For example, suppose the coded video bitstream includes signaling for this block 104 indicating the prediction direction of arrow (102), i.e., a sample is predicted from one or more prediction samples to the upper right at an angle of 45 degrees from the horizontal. In such a case, samples S41, S32, S23, and S14 are predicted from the same reference sample R05. Then, sample S44 is predicted from reference sample R08.

[0014] In certain cases, to calculate the reference sample, especially when the direction is not evenly divisible by 45 degrees, the values ​​of multiple reference samples may be combined, for example, by interpolation.

[0015] The number of possible directions has increased as video coding techniques continue to develop. For example, in H.264 (2003), nine different directions are available for intra-prediction. This increased to 33 in H.265 (2013), and JEM / VVC / BMS can support up to 65 directions as of the present disclosure. Experimental studies have been conducted to help identify the most appropriate intra-prediction directions, and using certain techniques of entropy coding, those most appropriate directions may be encoded with a small number of bits, accepting a specific bit penalty for the direction. Furthermore, the direction itself can sometimes be predicted from the adjacent directions used in the intra-prediction of the decoded adjacent block.

[0016] Figure 1B shows a schematic diagram (180) illustrating the 65 intra predictive directions by JEM to illustrate the increasing number of predictive directions in various encoding technologies that have developed over time.

[0017] The method for mapping bits representing intra-prediction directions in a coded video bitstream to prediction directions can vary depending on the video coding technique, ranging from simple direct mapping of prediction direction versus intra-prediction mode to complex adaptive schemes involving codewords, most probable modes, and similar techniques. However, in all cases, there may be a particular direction of intro prediction that is statistically less likely to occur in video content than a given other direction. Since the purpose of video compression is to reduce redundancy, in a well-designed video coding technique, those less likely directions may be represented by more bits than the more likely directions.

[0018] Interpicture prediction, or interpretation, may be based on motion compensation. In motion compensation, sample data from a previously reconstructed picture or a portion of it (a reference picture) may be spatially shifted in the direction indicated by a motion vector (hereinafter referred to as MV) and then used to predict the newly reconstructed picture or portion of the picture (e.g., a block). In some cases, the reference picture may be the same as the picture currently being reconstructed. The MV may have two dimensions, X and Y, or three dimensions, the third of which is an indication of the reference picture in use (similar to the time dimension).

[0019] In some video compression techniques, the current motion vector (MV) applicable to a particular area of ​​sample data can be predicted from other MVs, for example, from other MVs related to other areas of the sample data that are spatially adjacent to the area being reconstructed and precede the current MV in the decoding order. Doing so significantly reduces the overall amount of data required to code the MV by relying on the removal of redundancy in correlated MVs, thereby increasing compression efficiency. MV prediction can work effectively, for example, when coding an input video signal derived from a camera (known as natural video), areas larger than the area to which a single MV is applicable have a statistical likelihood of moving in a similar direction in the video sequence, and therefore, in some cases, can be predicted using similar motion vectors derived from the MVs of adjacent areas. As a result, the actual MV of a given area is similar to or identical to the MV predicted from the surrounding MVs. Such an MV may further be represented with fewer bits after entropy coding than would be used if the MV were coded directly rather than predicted from adjacent MVs. In some cases, MV prediction can be an example of lossless compression of a signal (i.e., MV) derived from the original signal (i.e., sample stream). In other cases, the MV prediction itself may be irreversible, for example, due to rounding errors when calculating the predictor from some surrounding MVs.

[0020] Various MV prediction mechanisms are described in H.265 / HEVC (ITU-T Recommendation H.265, "High Efficiency Video Coding," December 2016). Of the many MV prediction mechanisms specified in H.265, the technique described below will be hereafter referred to as "spatial merging."

[0021] Specifically, referring to Figure 2, the current block (201) contains samples that the encoder detected during the motion search process as predictable from a previous block of the same size, but spatially shifted. Instead of directly coding its MV, the MV can be derived from metadata associated with one or more reference pictures, for example, from the most recent reference picture (in decoding order), using the MV associated with one of the five surrounding samples, denoted as A0, A1, and B0, B1, B2 (202-206, respectively). In H.265, the MV prediction can use predictors from the same reference pictures used by the adjacent blocks. [Overview of the project] [Means for solving the problem]

[0022] This disclosure describes various embodiments of methods, apparatus, and computer-readable storage media for video encoding and / or decoding.

[0023] According to one aspect, one embodiment of the present disclosure provides a method for decoding an interpredicted video block. The method includes the steps of: deriving a motion vector prediction mode for a video block; deriving a context from a video stream for signaling a set of syntax elements associated with a motion vector difference (MVD) associated with the video block, based on the interpretation mode and / or the motion vector prediction mode; and decoding the video block based on the set of syntax elements and the derived context.

[0024] According to another aspect, an embodiment of the present disclosure provides an apparatus for video encoding and / or decoding. The apparatus includes a memory for storing instructions, and a processor in communication with the memory. When the processor executes the instructions, the processor is configured to cause the apparatus to implement the above method for video decoding and / or encoding.

[0025] According to still another aspect, an embodiment of the present disclosure provides a non-transitory computer-readable medium that stores instructions which, when executed by a computer for video decoding and / or encoding, cause the computer to implement the above method for video decoding and / or encoding.

[0026] The above and other aspects and their implementations are described in further detail in the drawings, the description, and the claims.

[0027] Further features, properties, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings. Brief Description of the Drawings

[0028] [Figure 1A] FIG. 1 is a schematic diagram of an exemplary subset of intra prediction directional modes. [Figure 1B] FIG. 2 is a diagram illustrating exemplary intra prediction directions. [Figure 2] FIG. 3 is a schematic diagram illustrating a current block and its surrounding spatial merge candidates for motion vector prediction in an example. [Figure 3] FIG. 4 is a schematic diagram of a simplified block diagram of a communication system (300) according to an exemplary embodiment. [Figure 4] FIG. 5 is a schematic diagram of a simplified block diagram of a communication system (400) according to an exemplary embodiment. [Figure 5] FIG. 6 is a schematic diagram of a simplified block diagram of a video decoder according to an exemplary embodiment. [Figure 6] This is a schematic diagram of a simplified block diagram of a video encoder according to an exemplary embodiment. [Figure 7] This is a block diagram of a video encoder according to another exemplary embodiment. [Figure 8] This is a block diagram of a video decoder according to another exemplary embodiment. [Figure 9] This figure shows a coding block partitioning method according to an exemplary embodiment of the present disclosure. [Figure 10] This figure shows another method of coding block partitioning according to an exemplary embodiment of the present disclosure. [Figure 11] This figure shows another method of coding block partitioning according to an exemplary embodiment of the present disclosure. [Figure 12] This figure shows an exemplary division of a base block into coding blocks using an exemplary division scheme. [Figure 13] This diagram illustrates an example of a three-part division of law. [Figure 14] This diagram illustrates an exemplary quadtree-binary-coding block partitioning scheme. [Figure 15] This figure shows a method for dividing a coding block into a plurality of transformation blocks and a coding order for the transformation blocks according to an exemplary embodiment of the present disclosure. [Figure 16] This figure shows another method for dividing a coding block into multiple transformation blocks and the coding order of the transformation blocks, according to an exemplary embodiment of the present disclosure. [Figure 17] This figure shows another method for dividing a coding block into multiple transformation blocks, according to an exemplary embodiment of the present disclosure. [Figure 18] This figure shows a flowchart of the method according to an exemplary embodiment of the present disclosure. [Figure 19] This is a schematic diagram of a computer system according to an exemplary embodiment of the present disclosure. [Modes for carrying out the invention]

[0029] Next, the present invention will be described in detail below with reference to the accompanying drawings, which form part of the present invention and illustrate specific examples of embodiments. However, it should be noted that the present invention may be embodied in various different forms, and therefore the subject matter included or claimed is not intended to be limited to any of the embodiments described below. It should also be noted that the present invention may be embodied as a method, device, component, or system. Thus, embodiments of the present invention may take the form of, for example, hardware, software, firmware, or any combination thereof.

[0030] Throughout this specification and the claims, terms may have nuances implied or suggested in context beyond their expressly stated meanings. As used herein, the phrases “in one embodiment” or “in some embodiments” do not necessarily refer to the same embodiment, and the phrases “in another embodiment” or “in other embodiments” do not necessarily refer to different embodiments. Similarly, the phrases “in one implementation” or “in some implementations” do not necessarily refer to the same implementation, and the phrases “in another implementation” or “in other implementations” do not necessarily refer to different implementations. For example, the claimed subject matter is intended to include all or some combinations of exemplary embodiments / implementations.

[0031] In general, technical terms may be understood at least partially from their usage in context. For example, terms such as “and,” “or,” or “and / or” as used herein may have various meanings that may at least partially depend on the context in which such terms are used. Typically, when “or” is used to relate a list such as A, B, or C, it is intended to mean A, B, and C, used here in an inclusive sense, as well as A, B, or C, used here in an exclusive sense. In addition, the terms “one or more” or “at least one” as used herein may, at least partially depending on the context, be used to describe any feature, structure, or characteristic in a singular sense, or to describe a combination of features, structures, or characteristics in a plural sense. Similarly, terms such as “a,” “an,” or “the” may also be understood, at least partially depending on the context, to convey either a singular usage or a plural usage. Furthermore, the terms “based on” or “determined by” may be understood not necessarily as conveying an exclusive set of factors, but rather, depending at least partially on the context, may allow for the existence of further factors that are not necessarily explicitly described.

[0032] Figure 3 shows a simplified block diagram of a communication system (300) according to one embodiment of the present disclosure. The communication system (300) includes, for example, a plurality of terminal devices that can communicate with each other over a network (350). For example, the communication system (300) includes a first pair of terminal devices (310) and (320) interconnected over the network (350). In the example of Figure 3, the first pair of terminal devices (310) and (320) may perform unidirectional transmission of data. For example, terminal device (310) may code video data (for example, a stream of video pictures captured by terminal device (310)) for transmission to the other terminal device (320) over the network (350). The encoded video data may be transmitted in the form of one or more coded video bitstreams. Terminal device (320) may receive coded video data from the network (350), decode the coded video data to restore video pictures, and display video pictures according to the restored video data. One-way data transmission can be implemented for applications such as media serving.

[0033] In another example, the communication system (300) includes a second pair of terminal devices (330) and (340) that perform bidirectional transmission of coded video data, which may be performed, for example, during video conferencing. For bidirectional transmission of data, in one example, each terminal device of terminal devices (330) and (340) may code video data (e.g., a stream of video pictures captured by that terminal device) for transmission to the other terminal device of terminal devices (330) and (340) via the network (350). Each terminal device of terminal devices (330) and (340) may also receive coded video data transmitted by the other terminal device of terminal devices (330) and (340), decode the coded video data to restore video pictures, and display video pictures on an accessible display device according to the restored video data.

[0034] In the example in Figure 3, terminal devices (310), (320), (330), and (340) may be implemented as servers, personal computers, and smartphones, but the applicability of the underlying principles of this disclosure is not limited in this way. Embodiments of this disclosure may be implemented in desktop computers, laptop computers, tablet computers, media players, wearable computers, dedicated video conferencing equipment, etc. Network (350) represents any number or type of network that transmits coded video data between terminal devices (310), (320), (330), and (340), including, for example, wired and / or wireless communication networks. Communication network (350)9 may exchange data over circuit-switched channels, packet-switched channels, and / or other types of channels. Typical networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For the purposes of this discussion, the architecture and topology of network (350) may not be important to the operation of this disclosure unless expressly described herein.

[0035] Figure 4 shows an example of the application of the disclosed subject matter, illustrating the arrangement of a video encoder and video decoder in a video streaming environment. The disclosed subject matter may equally apply to other video-enabled applications, including, for example, video conferencing, digital television broadcasting, games, virtual reality, and storage of compressed video on digital media such as CDs, DVDs, and memory sticks.

[0036] A video streaming system may include a video acquisition subsystem (413) which may include a video source (401), such as a digital camera, for creating an uncompressed stream (402) of video pictures or images. In one example, the stream (402) of video pictures includes samples recorded by the digital camera of the video source 401. The stream (402) of video pictures, shown as a thick line to emphasize the large amount of data compared to encoded video data (404) (or encoded video bitstream), may be processed by an electronic device (420) which includes a video encoder (403) coupled to the video source (401). The video encoder (403) may include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject, as will be described in more detail below. The encoded video data (404) (or encoded video bitstream (404)) is shown in thin lines to highlight its low data size compared to the stream of uncompressed video pictures (402), and may be stored in a streaming server (405) or directly in a downstream video device (not shown) for future use. One or more streaming client subsystems, such as client subsystems (406) and (408) in Figure 4, can access the streaming server (405) to retrieve copies (407) and (409) of the encoded video data (404). The client subsystem (406) may include, for example, a video decoder (410) in an electronic device (430). The video decoder (410) decodes the input copy (407) of the encoded video data to create an output stream (411) of a video picture that can be rendered on a display (412) (e.g., a display screen) or other rendering device (not shown). The video decoder 410 may be configured to perform some or all of the various functions described herein.In some streaming systems, encoded video data (404), (407), and (409) (e.g., video bitstream) may be encoded according to specific video coding / compression standards. Examples of such standards include ITU-T Recommendation H.265. For example, a video coding standard under development is informally known as Multipurpose Video Coding (VVC). The disclosed subject matter may be used in the context of VVC and other video coding standards.

[0037] It should be noted that electronic devices (420) and (430) may include other components (not shown). For example, electronic device (420) may include a video decoder (not shown), and electronic device (430) may also include a video encoder (not shown).

[0038] Figure 5 shows a block diagram of a video decoder (510) according to any embodiment of the present disclosure described below. The video decoder (510) may be included in an electronic device (530). The electronic device (530) may include a receiver (531) (e.g., a receiving circuit). The video decoder (510) can be used in place of the video decoder (410) in the example of Figure 4.

[0039] The receiver (531) may receive one or more coded video sequences to be decoded by the video decoder (510). In the same or different embodiments, one coded video sequence may be decoded at a time, and the decoding of each coded video sequence is independent of other coded video sequences. Each video sequence may be associated with multiple video frames or video images. Coded video sequences may be received from a channel (501), which may be a hardware / software link to a storage device that stores encoded video data, or a streaming source that transmits encoded video data. The receiver (531) may receive the encoded video data together with other data, such as coded audio data and / or auxiliary data streams, which may be transferred to their respective processing circuits (not shown). The receiver (531) may isolate the coded video sequences from other data. To counteract network jitter, a buffer memory (515) may be placed between the receiver (531) and the entropy decoder / parser (520) (hereinafter, "Parser (520)"). For certain applications, the buffer memory (515) may be implemented as part of the video decoder (510). For other applications, the buffer memory (515) may be separate from the video decoder (510) and located externally (not shown). For yet other applications, for example, there may be a buffer memory (not shown) outside the video decoder (510) to counteract network jitter, or there may be another additional buffer memory (515) inside the video decoder (510) to handle playback timing, for example. When the receiver (531) is receiving data from a storage / transfer device with sufficient bandwidth and controllability, or from an isosynchronous network, the buffer memory (515) may not be necessary or can be made small. For use in best-effort packet networks such as the Internet, a sufficiently large buffer memory (515) may be required, and its size may be relatively large.Such buffer memory may be implemented with an adaptive size and may be at least partially implemented in an operating system or similar element (not shown) outside the video decoder (510).

[0040] The video decoder (510) may include a parser (520) to reconstruct symbols (521) from the coded video sequence. The categories of these symbols include information used to manage the operation of the video decoder (510) and information for controlling a rendering device, such as a display (512) (e.g., a display screen), which may or may not be an integral part of the electronic device (530) as shown in Figure 5, but which can be coupled to the electronic device (530). The control information for the rendering device may be in the form of supplemental extension information (SEI messages) or parameter set fragments (not shown) of video usability information (VUI). The parser (520) may parse / entropy decode the coded video sequence received by the parser (520). The entropy coding of the coded video sequence may conform to video coding techniques or standards and may follow various principles, including variable-length coding, Huffman coding, context-dependent or non-context-dependent arithmetic coding, etc. The parser(520) may extract from the coded video sequence a set of at least one subgroup parameters for a subgroup of pixels in the video decoder, based on at least one parameter corresponding to the subgroup. Subgroups may include picture groups (GOPs), pictures, tiles, slices, macroblocks, coding units (CUs), blocks, transform units (TUs), and predictive units (PUs). The parser(520) may also extract information from the coded video sequence such as transform coefficients (e.g., Fourier transform coefficients), quantization parameter values, and motion vectors.

[0041] The parser (520) can perform entropy decoding / parsing operations on the video sequence received from buffer memory (515) in order to create symbols (521).

[0042] The reconstruction of the symbol (521) may include multiple different processing units or functional units, depending on the type of the coded video picture or part thereof (interpicture and intrapicture, interblock and intrablock, etc.) and other factors. The units to be included and how they are included may be controlled by subgroup control information parsed from the video sequence coded by the parser (520). The flow of such subgroup control information between the parser (520) and the following multiple processing units or functional units is not illustrated for brevity.

[0043] In addition to the functional blocks already described, the video decoder (510) can be conceptually subdivided into several functional units, as described below. In actual implementations operating under commercial constraints, many of these functional units may interact closely with each other and, at least partially, be integrated with one another. However, in order to clearly illustrate the various functions of the disclosed subject, the following disclosure employs a conceptual subdivision into functional units.

[0044] The first unit is the scaler / inverse unit (551). The scaler / inverse unit (551) may receive control information from the parser (520) as symbols (521), including quantization transformation coefficients, information indicating which type of inverse transformation to use, block size, quantization coefficients / parameters, and quantization scaling matrix. The scaler / inverse unit (551) may output a block containing sample values ​​that can be input to the aggregator (555).

[0045] In some cases, the output samples of the scaler / inverse transform (551) may relate to intracoded blocks, i.e., blocks that do not use prediction information from previously reconstructed pictures but can use prediction information from previously reconstructed portions of the current picture. Such prediction information may be provided by an intrapicture prediction unit (552). In some cases, the intrapicture prediction unit (552) may generate a block of the same size and shape as the block being reconstructed, using information from surrounding blocks that have already been reconstructed and are stored in the current picture buffer (558). The current picture buffer (558) buffers, for example, partially reconstructed current pictures and / or fully reconstructed current pictures. In some implementations, the aggregator (555) may, sample by sample, add the prediction information generated by the intraprediction unit (552) to the output sample information, such as that provided by the scaler / inverse transform unit (551).

[0046] In other cases, the output samples of the scaler / inverse unit (551) may be associated with an intercoded and possibly motion-compensated block. In such cases, the motion-compensated prediction unit (553) can access the reference picture memory (557) to fetch samples to be used for interpicture prediction. After motion-compensating the fetched samples according to the symbols (521) associated with the block, these samples can be added by the aggregator (555) to the output of the scaler / inverse unit (551) to generate output sample information (the output of unit 551 may be called residual samples or residual signals). The address in the reference picture memory (557) from which the motion-compensated prediction unit (553) fetches prediction samples may be controlled by a motion vector available to the motion-compensated prediction unit (553) in the form of a symbol (521) which may have, for example, an X component, a Y component (shift), and a reference picture component (time). Motion compensation may also include interpolation of sample values ​​fetched from reference picture memory (557) when the exact motion vectors of subsamples are used, and may be associated with a motion vector prediction mechanism or the like.

[0047] The output samples of the aggregator (555) may be subjected to various loop filtering techniques in the loop filter unit (556). The video compression technique may include in-loop filtering techniques that are controlled by parameters contained in the coded video sequence (also called the coded video bitstream) and are available to the loop filter unit (556) as symbols (521) from the parser (520), but may also respond to metadata obtained during decoding of earlier parts (in decoding order) of the coded picture or coded video sequence, and may also respond to previously reconstructed and loop-filtered sample values. Several types of loop filters may be included as part of the loop filter unit 556 in various orders, as will be described in more detail below.

[0048] The output of the loop filter unit (556) can be a sample stream that can be output to the rendering device (512) and can also be stored in reference picture memory (557) for use in future interpicture prediction.

[0049] A particular coded picture, once fully reconfigured, can be used as a reference picture for future interpicture prediction. For example, once the coded picture corresponding to the current picture is fully reconfigured and the coded picture is identified as a reference picture (e.g., by the parser (520)), the current picture buffer (558) can become part of the reference picture memory (557), and any unused current picture buffer can be reallocated before starting the reconfiguration of the next coded picture.

[0050] The video decoder (510) may perform decoding operations according to a predetermined video compression technique adopted in a standard such as ITU-T Recommendation H.265. The coded video sequence may conform to the syntax specified by the video compression technique or standard used, in the sense that the coded video sequence is faithful to both the syntax of the video compression technique or standard and the profile documented in the video compression technique or standard. Specifically, a profile may select a particular tool from all the tools available in the video compression technique or standard as the only tool that can be used under that profile. In order to conform to the standard, the complexity of the coded video sequence may be within the range defined by the level of the video compression technique or standard. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples per second), maximum reference image size, etc. The limitations set by the level may, in some cases, be further limited by the specifications of the virtual reference decoder (HRD) and metadata for HRD buffer management signaled within the coded video sequence.

[0051] In some exemplary embodiments, the receiver (531) may receive additional (redundant) data along with the encoded video. The additional data may be included as part of the encoded video sequence. The additional data may be used by the video decoder (510) to properly decode the data and / or to more accurately reconstruct the original video data. The additional data may take the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) extension layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0052] Figure 6 shows a block diagram of a video encoder (603) according to an example embodiment of the present disclosure. The video encoder (603) may be included in an electronic device (620). The electronic device (620) may further include a transmitter (640) (e.g., a transmitting circuit). The video encoder (603) can be used in place of the video encoder (403) in the example of Figure 4.

[0053] The video encoder (603) may receive video samples from a video source (601) (not part of the electronic device (620) in the example in Figure 6) from which video images to be coded by the video encoder (603) can be captured. In another example, the video source (601) may be implemented as part of the electronic device (620).

[0054] The video source (601) may provide a source video sequence to be coded by the video encoder (603) in the form of a digital video sample stream, which can have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 YCrCb, RGB, XYZ, ...), and any suitable sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media serving system, the video source (601) may be a storage device capable of storing previously prepared video. In a video conferencing system, the video source (601) may be a camera that captures local image information as a video sequence. The video data may be provided as a series of individual pictures or images that give motion when viewed sequentially. The pictures themselves may be organized as a spatial array of pixels, each pixel may contain one or more samples depending on the sampling structure, color space, etc., used. Those skilled in the art will readily understand the relationship between pixels and samples. The following description focuses on samples.

[0055] According to some exemplary embodiments, the video encoder (603) can encode and compress pictures of a source video sequence into an encoded video sequence (643) in real time or under any other time constraints required by the application. Enforcing an appropriate coding speed constitutes one function of the controller (650). In some embodiments, the controller (650) can be functionally coupled to and control other functional units, as described below. For brevity, the couplings are not illustrated. Parameters set by the controller (650) may include rate control-related parameters (such as picture skip, quantizer, lambda value of rate distortion optimization technique), picture size, picture group (GOP) layout, maximum motion vector search range, etc. The controller (650) can be configured to have other appropriate functions related to the video encoder (603) optimized for a particular system design.

[0056] In some exemplary embodiments, the video encoder (603) may be configured to operate in a coding loop. In an overly simplified explanation, in one example, the coding loop may include a source coder (630) (which is involved in generating symbols, such as a symbol stream, based, for example, on an input picture and a reference picture to be coded) and a (local) decoder (633) built into the video encoder (603). The decoder (633) reconstructs the symbols to create sample data in a similar manner to what a (remote) decoder would create, even though the built-in decoder 633 processes the video stream coded by the source coder 630 without entropy coding (because in the video compression techniques considered in the disclosed subject, any compression between the symbols and the coded video bitstream may be reversible). The reconstructed sample stream (sample data) is input to the reference picture memory (634). Symbol stream decoding yields bit-exact results regardless of the decoder's location (local or remote), so the contents of the reference picture memory (634) are also bit-exact between the local and remote encoders. In other words, the predictive portion of the encoder "sees" the exact same sample values ​​as the reference picture samples that the decoder "sees" when using predictions during decoding. This fundamental principle of reference picture synchronization (and the resulting drift if synchronization cannot be maintained due to, for example, channel errors) is used to improve coding quality.

[0057] The operation of the “local” decoder (633) may be the same as that of a “remote” decoder, such as the video decoder (510), which has already been described in detail above with reference to Figure 5. However, as also briefly referring to Figure 5, the entropy decoding portion of the video decoder (510), including the buffer memory (515) and parser (520), may not be fully implemented in the local decoder (633) within the encoder, since symbols are available and the encoding / decoding of symbols to the coded video sequence by the entropy coder (645) and parser (520) may be reversible.

[0058] At this point, it can be said that any decoder technique other than parse / entropy decoding, which can only exist within the decoder, must also necessarily exist in the corresponding encoder in substantially the same functional form. For this reason, the disclosed subject matter may focus on decoder operation, which is similar to the decoding portion of the encoder. Thus, the description of encoder technique may be omitted, as it is the reverse of the comprehensively described decoder technique. A more detailed description of the encoder is given below only in specific areas or embodiments.

[0059] In operation, in some example implementations, the source coder (630) may perform motion-compensated predictive coding, predictively coding the input picture by referencing one or more previously coded pictures from a video sequence designated as a “reference picture”. In this way, the coding engine (632) codes the difference (or residual) of the color channels between the pixel blocks of the input picture and the pixel blocks of the reference picture that may be selected as a predictive reference to the input picture. The terms “residual” and its adjective form “residual” may be used interchangeably.

[0060] The local video decoder (633) can decode coded video data of a picture that may be designated as a reference picture based on symbols created by the source coder (630). The operation of the coding engine (632) may, advantageously, be a lossy process. When coded video data can be decoded by a video decoder (not shown in Figure 6), the reconstructed video sequence may typically be a replica of the source video sequence with some errors. The local video decoder (633) may replicate the decoding process that may be performed by the video decoder on the reference picture so that the reconstructed reference picture is stored in the reference picture cache (634). In this way, the video encoder (603) may locally store a copy of the reconstructed reference picture that has content common to the reconstructed reference picture acquired (without transmission errors) by the far-end (remote) video decoder.

[0061] The predictor (635) can perform a predictive search for the coding engine (632). That is, for a new picture to be coded, the predictor (635) can search the reference picture memory (634) for sample data (as candidate reference pixel blocks) or reference picture motion vectors, block shapes, and other specific metadata that can serve as appropriate predictive references for the new pixels. The predictor (635) can work on sample blocks for each pixel block to find appropriate predictive references. In some cases, the input picture may have predictive references drawn from multiple reference pictures stored in the reference picture memory (634), as determined by the search results obtained by the predictor (635).

[0062] The controller (650) can manage the coding operations of the source coder (630), including, for example, setting parameters and subgroup parameters used to encode video data.

[0063] The output of all the aforementioned functional units can undergo entropy coding within the entropy coder (645). The entropy coder (645) converts the symbols generated by the various functional units into coded video sequences by lossless compression of symbols according to techniques such as Huffman coding, variable-length coding, and arithmetic coding.

[0064] The transmitter (640) can buffer the coded video sequence created by the entropy coder (645) and prepare it for transmission over the communication channel (660), which may be a hardware / software link to a storage device that stores the encoded video data. The transmitter (640) can merge the coded video data from the video coder (603) with other data to be transmitted, such as coded audio data and / or auxiliary data streams (sources not shown).

[0065] The controller (650) can manage the operation of the video encoder (603). During coding, the controller (650) can assign a specific coded picture type to each coded picture, which may affect the coding technique that can be applied to each picture. For example, a picture may often be assigned as one of the following picture types:

[0066] An intra-picture (I-picture) can be a picture that can be coded and decoded without using any other pictures in the sequence as a source of prediction. Some video codecs enable different types of intra-pictures, including, for example, Independent Decoder Refresh ("IDR") pictures. Those skilled in the art will recognize these variations of I-pictures and their respective uses and characteristics.

[0067] A predictive picture (P-picture) can be a picture that can be coded and decoded using intra-prediction or inter-prediction, which uses at most one motion vector and reference index to predict the sample values ​​of each block.

[0068] A bidirectional predictive picture (B-picture) can be a picture that can be coded and decoded using intra-prediction or inter-prediction, which uses at most two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, multiple predictive pictures can use three or more reference pictures and associated metadata to reconstruct a single block.

[0069] A source picture can generally be spatially subdivided into multiple sample coding blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples each), and each block can be coded. Blocks may be coded predictively by referencing other (already coded) blocks, as determined by the coding assignment applied to each picture in the block. For example, blocks of picture I may be coded non-predictively, or they may be coded predictively by referencing already coded blocks of the same picture (spatial prediction or intra-prediction). Pixel blocks of picture P may be coded predictively by referencing one previously coded reference picture, either via spatial prediction or temporal prediction. Blocks of picture B may be coded predictively by referencing one or two previously coded reference pictures, either via spatial prediction or temporal prediction. A source picture or an intermediate picture may be subdivided into other types of blocks for other purposes. The subdivision of coding blocks and other types of blocks may or may not follow the same method, as will be described in more detail below.

[0070] The video encoder (603) can perform coding operations in accordance with a given video coding technique or standard, such as ITU-T Rec.H.265. In these operations, the video encoder (603) can perform various compression operations, including predictive coding operations that utilize temporal and spatial redundancy in the input video sequence. Thus, the coded video data may conform to the syntax specified by the video coding technique or standard being used.

[0071] In some exemplary embodiments, the transmitter (640) may transmit additional data along with the encoded video. The source coder (630) may include such data as part of the coded video sequence. The additional data may include time / space / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, SEI messages, VUI parameter set fragments, and the like.

[0072] Video may be captured as multiple source pictures (video pictures) in a time series. Intra-picture prediction (often abbreviated as intra-prediction) utilizes spatial correlations within a given picture, while inter-picture prediction utilizes temporal or other correlations between pictures. For example, a particular picture being encoded / decoded, called the current picture, may be divided into blocks. Blocks within the current picture may be coded by vectors called motion vectors if they are similar to reference blocks in previously coded, still-buffering reference pictures within the video. Motion vectors point to reference blocks within reference pictures and may have a third dimension to identify the reference pictures if multiple reference pictures are used.

[0073] In some exemplary embodiments, a bi-prediction technique can be used for interpicture prediction. According to such a bi-prediction technique, two reference pictures are used, such as a first reference picture and a second reference picture, both of which advance the current picture in the video in decoding order (however, in display order, they can be past or future, respectively). Blocks in the current picture can be coded by a first motion vector pointing to a first reference block in the first reference picture, and a second motion vector pointing to a second reference block in the second reference picture. Blocks can be predicted in conjunction by a combination of the first and second reference blocks.

[0074] Furthermore, merge mode techniques may be used to improve coding efficiency in interpicture prediction.

[0075] According to some exemplary embodiments of this disclosure, predictions such as interpicture prediction and intrapicture prediction are performed in block units. For example, pictures in a sequence of video pictures are divided into coding tree units (CTUs) for compression, and the CTUs in a picture may have the same size, such as 64x64 pixels, 32x32 pixels, or 16x16 pixels. Generally, a CTU may contain three parallel coding tree blocks (CTBs), i.e., one luma CTB and two chroma CTBs. Each CTU can be recursively quadtree-partitioned into one or more coding units (CUs). For example, a 64x64 pixel CTU can be divided into one 64x64 pixel CU or four 32x32 pixel CUs. Each of one or more of the 32x32 blocks may be further divided into four 16x16 pixel CUs. In some exemplary embodiments, each CU may be analyzed during encoding to determine the prediction type of that CU from among various prediction types, such as inter-prediction type and intra-prediction type. A CU can be divided into one or more prediction units (PUs) depending on its temporal and / or spatial predictability. Generally, each PU contains one luma prediction block (PB) and two chroma PBs. In one embodiment, the prediction operation in coding (encoding / decoding) is performed in units of prediction blocks. The division of the CU into PUs (or PBs for different color channels) can be carried out in various spatial patterns. A luma PB or chroma PB may contain a matrix of sample values ​​(e.g., luma values), such as 8x8 pixels, 16x16 pixels, 8x16 pixels, 16x8 pixels, etc.

[0076] Figure 7 shows a diagram of a video encoder (703) according to another example embodiment of the present disclosure. The video encoder (703) is configured to receive a processing block (e.g., a prediction block) of sample values ​​in the current video picture within a sequence of video pictures, and to encode the processing block into a coded picture which is part of a coded video sequence. The example video encoder (703) may be used instead of the example video encoder (403) in Figure 4.

[0077] For example, the video encoder (703) receives a matrix of sample values ​​for a processing block, such as an 8x8 sample prediction block. The video encoder (703) then determines, for example, using rate-distortion optimization (RDO), whether the processing block is best coded using intra-mode, inter-mode, or bi-prediction mode. If it is determined that the processing block is coded in intra-mode, the video encoder (703) encodes the processing block into a coded picture using the intra-prediction technique; if it is determined that the processing block is coded in inter-mode or bi-prediction mode, the video encoder (703) may encode the processing block into a coded picture using the inter-prediction technique or the bi-prediction technique, respectively. In some exemplary embodiments, a merge mode may be used as a submode of inter-picture prediction, in which the motion vector is derived from one or more motion vector predictors without benefiting from the coded motion vector components outside the predictor. In some other exemplary embodiments, there may be motion vector components applicable to the target block. Therefore, the video encoder (703) may include components not explicitly shown in Figure 7, such as a mode determination module, to determine the prediction mode of the processing block.

[0078] In the example shown in Figure 7, the video encoder (703) includes an interencoder (730), an intraencoder (722), a residual calculator (723), a switch (726), a residual encoder (724), a general-purpose controller (721), and an entropy encoder (725), all coupled together as shown in the example configuration of Figure 7.

[0079] The interencoder (730) is configured to receive a sample of the current block (e.g., a processing block), compare that block to one or more reference blocks in the reference picture (e.g., blocks in the previous and subsequent pictures in display order), generate interprediction information (e.g., a description of redundant information, motion vectors, and merge mode information by the interencoding technique), and compute an interprediction result (e.g., a predicted block) based on the interprediction information using any appropriate technique. In some examples, the reference picture is a reference picture decoded based on video information encoded using a decoding unit 633 incorporated into the example encoder 620 in Figure 6 (shown as a residual decoder 728 in Figure 7, as will be described in more detail below).

[0080] The intra encoder (722) is configured to receive a sample of the current block (e.g., a processing block), compare the block to an already coded block in the same picture, generate a converted quantization coefficient, and optionally also generate intra prediction information (e.g., intra prediction direction information by one or more intra encoding techniques). Based on the intra prediction information and a reference block in the same picture, the intra prediction result (e.g., a predicted block) may be calculated.

[0081] The general-purpose controller (721) may be configured to determine general-purpose control data and control other components of the video encoder (703) based on the general-purpose control data. For example, the general-purpose controller (721) determines the prediction mode of a block and provides a control signal to the switch (726) based on the prediction mode. For example, if the prediction mode is intra-mode, the general-purpose controller (721) controls the switch (726) to select an intra-mode result for use by the residual calculator (723), and controls the entropy encoder (725) to select intra-prediction information and include that intra-prediction information in the bitstream. If the description mode of a block is inter-mode, the general-purpose controller (721) controls the switch (726) to select an inter-prediction result for use by the residual calculator (723), and controls the entropy encoder (725) to select inter-prediction information and include that inter-prediction information in the bitstream.

[0082] A residual calculator (723) may be configured to calculate the difference (residual data) between the received block and the prediction result for a block selected from an intra-encoder (722) or an inter-encoder (730). A residual encoder (724) may be configured to encode the residual data to generate conversion coefficients. For example, a residual encoder (724) may be configured to convert the residual data from the spatial domain to the frequency domain to generate conversion coefficients. The conversion coefficients are then quantized to obtain quantized conversion coefficients. In various exemplary embodiments, the video encoder (703) also includes a residual decoder (728). The residual decoder (728) is configured to perform an inverse transform to generate decoded residual data. The decoded residual data can be appropriately used by the intra-encoder (722) and the inter-encoder (730). For example, an interencoder (730) can generate a decoded block based on decoded residual data and interprediction information, and an intraencoder (722) can generate a decoded block based on decoded residual data and intraprediction information. The decoded block is appropriately processed to generate a decoded picture, which is buffered in a memory circuit (not shown) and can be used as a reference picture.

[0083] The entropy encoder (725) is configured to format the bitstream to include encoded blocks and to perform entropy coding. The entropy encoder (725) is configured to include various types of information in the bitstream. For example, the entropy encoder (725) may be configured to include general-purpose control data, selected prediction information (e.g., intra-prediction information or inter-prediction information), residual information, and other appropriate information in the bitstream. Residual information may not be present when coding blocks in either inter-mode or bi-prediction mode merge submodes.

[0084] Figure 8 shows a diagram of an example video decoder (810) according to another embodiment of the present disclosure. The video decoder (810) is configured to receive a coded picture which is part of a coded video sequence and to decode the coded picture to produce a reconstructed picture. In one example, the video decoder (810) may be used instead of the video decoder (410) in the example of Figure 4.

[0085] In the example shown in Figure 8, the video decoder (810) includes an entropy decoder (871), an interdecoder (880), a residual decoder (873), a reconfiguration module (874), and an intradecoder (872) coupled together as shown in the example configuration of Figure 8.

[0086] The entropy decoder (871) can be configured to reconstruct specific symbols from the coded picture that represent the syntax elements that make up the coded picture. Such symbols may include, for example, the mode in which the block is coded (e.g., intra-mode, inter-mode, bi-prediction mode, merge sub-mode, or another sub-mode), prediction information (e.g., intra-prediction information or inter-prediction information) that can identify specific samples or metadata used for prediction by the intra-decoder (872) or inter-decoder (880), and residual information in the form of quantization transformation coefficients. For example, if the prediction mode is inter-prediction mode or bi-prediction mode, inter-prediction information is provided to the inter-decoder (880), and if the prediction type is intra-prediction type, intra-prediction information is provided to the intra-decoder (872). The residual information can undergo inverse quantization and be provided to the residual decoder (873).

[0087] The interdecoder (880) may be configured to receive interprediction information and generate interprediction results based on the interprediction information.

[0088] The intra decoder (872) may be configured to receive intra prediction information and generate prediction results based on the intra prediction information.

[0089] The residual decoder (873) may be configured to perform inverse quantization to extract inverse quantization conversion coefficients, and to process these inverse quantization conversion coefficients to convert the residual from the frequency domain to the spatial domain. The residual decoder (873) may also utilize certain control information (to include quantization parameters (QP)), which may be provided by the entropy decoder (871) (the data path is not illustrated as this may only be a small amount of control information).

[0090] The reconstruction module (874) may be configured to combine the residuals as output from the residual decoder (873) and the prediction results (optionally as output from the inter-prediction module or intra-prediction module) in the spatial domain to form reconstructed blocks that form part of the reconstructed picture as part of the reconstructed video. Note that other appropriate operations, such as deblocking operations, may be performed to improve visual quality.

[0091] It should be noted that the video encoders (403), (603), and (703), as well as the video decoders (410), (510), and (810), can be implemented using any suitable technique. In some exemplary embodiments, the video encoders (403), (603), and (703), as well as the video decoders (410), (510), and (810), can be implemented using one or more integrated circuits. In another embodiment, the video encoders (403), (603), and (603), as well as the video decoders (410), (510), and (810), can be implemented using one or more processors that execute software instructions.

[0092] Turning to block partitioning for coding and decoding, a typical partition can begin with a base block and follow a predefined set of rules, a specific pattern, a partition tree, or any partition structure or scheme. The partition may be hierarchical and recursive. A final set of partitions or coding blocks may be obtained after separating or partitioning the base block according to one of the exemplary partitioning procedures or other procedures, or a combination thereof, described below. Each of these partitions may be at one of various partitioning levels in the partitioning hierarchy and may be of various shapes. Each of the partitions may be called a coding block (CB). In the various exemplary partitioning implementations described further below, each resulting CB may be any CB of an acceptable size and partitioning level. Such partitions are called coding blocks because they may form units for which several basic coding / decoding decisions can be made, for which coding / decoding parameters can be optimized, determined, and signaled in the encoded video bitstream. The highest or deepest level in the final partition represents the depth of the coding block partitioning tree structure. Coding blocks may be luma coding blocks or chroma coding blocks. The CB tree structure for each color is sometimes called a coding block tree (CBT).

[0093] The coding blocks for all color channels are sometimes collectively called a coding unit (CU). The hierarchical structure of all color channels is sometimes collectively called a coding tree unit (CTU). The division patterns or structures of the various color channels within a CTU may or may not be the same.

[0094] In some implementations, the partitioning tree scheme or structure used for lumar channels and chroma channels does not have to be the same. In other words, lumar channels and chroma channels may have separate coding tree structures or patterns. Furthermore, whether lumar channels and chroma channels use the same coding partitioning tree structure or different coding partitioning tree structures, and the actual coding partitioning tree structure to be used, may depend on whether the slice being coded is a P slice, a B slice, or an I slice. For example, in the case of an I slice, chroma channels and lumar channels may have separate coding partitioning tree structures or coding partitioning tree structure modes, but in the case of a P slice or a B slice, lumar channels and chroma channels may share the same coding partitioning tree scheme. When separate coding partitioning tree structures or modes are applied, a lumar channel may be partitioned into CBs by one coding partitioning tree structure, and a chroma channel may be partitioned into chroma CBs by another coding partitioning tree structure.

[0095] In some exemplary implementations, a predetermined partitioning pattern may be applied to the base block. As shown in Figure 9, the exemplary four-way partitioning tree may start at a first predefined level (e.g., a 64x64 block level or other size as the base block size), and the base block may be hierarchically partitioned down to a predefined lowest level (e.g., a 4x4 level). For example, the base block can follow one of four predefined partitioning options or patterns shown by 902, 904, 906, and 908, and the partition designated as R can be recursively partitioned in such a way that the same partitioning option shown in Figure 9 can be repeated at a lower scale down to the lowest level (e.g., a 4x4 level). In some implementations, additional restrictions may be placed on the partitioning scheme in Figure 9. In the implementation of Figure 9, rectangular partitions (e.g., 1:2 / 2:1 rectangular partitions) may be allowed but not recursive, while square partitions may be recursive. If necessary, the recursive partitioning following Figure 9 generates the final set of coding blocks. A coding tree depth may be further defined to indicate the division depth from the root node or root block. For example, the coding tree depth for the root node or root block of a 64x64 block may be set to 0, and after the root block is further divided one more time according to Figure 9, the coding tree depth increases by 1. The maximum or deepest level from the 64x64 base block to the smallest 4x4 partition is 4 in the above scheme (starting from level 0). Such a division scheme may be applied to one or more of the color channels. Each color channel may be divided independently according to the scheme in Figure 9 (for example, for each color channel at each hierarchy level, the division pattern or option in the predefined pattern may be determined independently). Alternatively, two or more color channels may share the same hierarchy pattern tree in Figure 9 (for example, the same division pattern or option in the predefined pattern may be selected for two or more color channels at each hierarchy level).

[0096] Figure 10 shows another exemplary predefined partition pattern that allows recursive partitioning to form a partition tree. As shown in Figure 10, ten exemplary partition structures or patterns may be predefined. The root block can start from a predefined level (e.g., from a base block at a 128x128 level or 64x64 level). The exemplary partition structures in Figure 10 include various 2:1 / 1:2 and 4:1 / 1:4 rectangular partitions. Partition types with three subpartitions, shown in 1002, 1004, 1006, and 1008 in the second column of Figure 10, may be called "T-shaped" partitions. The "T-shaped" partitions 1002, 1004, 1006, and 1008 may be called left T-shaped, top T-shaped, right T-shaped, and bottom T-shaped. In some exemplary implementations, none of the rectangular partitions in Figure 10 can be further subdivided. A coding tree depth may be further defined to indicate the partition depth from the root node or root block. For example, the coding tree depth for the root node or root block of a 128x128 block may be set to 0, and after the root block is further divided one more time according to Figure 10, the coding tree depth increases by 1. In some implementations, only all square partitions of 1010 can allow recursive partitioning to the next level of the partition tree following the pattern in Figure 10. In other words, recursive partitioning may not be possible for square partitions in T-shaped patterns 1002, 1004, 1006, and 1008. If necessary, the recursive partitioning procedure following Figure 10 generates the final set of coding blocks. Such a scheme may be applied to one or more of the color channels. In some implementations, more flexibility may be added to the use of partitions with fewer than 8x8 levels. For example, in some cases, 2x2 chromainter prediction may be used.

[0097] In some other exemplary implementations of coding block partitioning, a quadtree structure may be used to partition a base block or intermediate block into quadtree partitions. Such quadtree partitioning may be applied hierarchically and recursively to any square partition. Whether the base block or intermediate block or partition is further quadtree partitioned may be adapted to various local characteristics of the base block or intermediate block / partition. Quadtree partitioning at picture boundaries may be further adapted. For example, implicit quadtree partitioning may be performed at picture boundaries so that the block continues to quadtree partition until its size fits within the picture boundary.

[0098] In some other exemplary implementations, hierarchical bifurcation from a base block may be used. In such a scheme, the base block or intermediate level block may be divided into two partitions. The bifurcation may be either horizontal or vertical. For example, horizontal bifurcation can divide the base block or intermediate block into equal left and right partitions. Similarly, vertical bifurcation can divide the base block or intermediate block into equal upper and lower partitions. Such bifurcation may be hierarchical and recursive. Whether the bifurcation scheme should be continued, and if so, whether horizontal bifurcation or vertical bifurcation should be used, may be determined at each of the base block or intermediate block. In some implementations, further bifurcation may be stopped at a predefined minimum partition size (of one or both dimensions). Alternatively, further bifurcation may be stopped when a predefined bifurcation level or depth is reached from the base block. In some implementations, the aspect ratio of the partitions may be restricted. For example, the aspect ratio of the partitions may not be less than 1:4 (or greater than 4:1). Therefore, a vertical strip partition with a 4:1 vertical-to-horizontal aspect ratio can only be further divided vertically into upper and lower partitions, each having a 2:1 vertical-to-horizontal aspect ratio.

[0099] In several other examples, a ternary scheme may be used to partition a base block or any intermediate block, as shown in Figure 13. The ternary pattern may be implemented vertically, as shown in 1302 of Figure 13, or horizontally, as shown in 1304 of Figure 13. The exemplary partition ratio in Figure 13 is shown as 1:2:1, either vertically or horizontally, but other ratios may be predefined. In some implementations, two or more different ratios may be predefined. Such a ternary scheme may be used to complement a quadtree or binary structure, in that while such a ternary partition can take in objects located at the center of a block within a single contiguous partition, quadtrees and binary trees always partition along the center of a block and thus divide objects into separate partitions. In some implementations, the width and height of the partitions in the exemplary ternary are always powers of 2 to avoid further transformations.

[0100] The above partitioning schemes can be combined in any way at different partitioning levels. For example, the quadtree and binary partitioning schemes described above may be combined to partition a base block into a quadtree-binary (QTBT) structure. In such a scheme, the base block or intermediate block / partition may be either quadtree partitioned or binary partitioned, if specified, according to a set of predefined conditions. A specific example is shown in Figure 14. In the example in Figure 14, the base block is first quadtree partitioned into four partitions, as shown by 1402, 1404, 1406, and 1408. Each of the resulting partitions is then quadtree partitioned into four further partitions (such as 1408), or binary partitioned into two further partitions at the next level (e.g., horizontal or vertical, such as 1402 or 1406, both of which are symmetric), or not partitioned at all (such as 1404). Binary or quadtree partitioning may be recursively possible for square partitions, as shown by the overall exemplary partitioning patterns in 1410 and the corresponding tree structures / representations in 1420, where solid lines represent quadtree partitioning and dashed lines represent binary partitioning. A flag may be used for each binary node (non-leaf binary partition) to indicate whether the binary is horizontal or vertical. For example, as shown in 1420, which matches the partitioning structure in 1410, a flag "0" may represent horizontal binary and a flag "1" may represent vertical binary. In the case of quadtree partitioning, there is no need to indicate the partition type, as quadtree partitioning always divides a block or partition both horizontally and vertically to produce four subblocks / partitions of equal size. In some implementations, a flag "1" may represent horizontal binary and a flag "0" may represent vertical binary.

[0101] In some exemplary implementations of QTBT, the quadtree and binary rule set may be represented by the following predefined parameters and their associated corresponding functions. -CTU size: The size of the root node of the quadtree (the size of the base block). -MinQTSize: Minimum allowable quadtree leaf node size -MaxBTSize: Maximum allowable binary tree root node size -MaxBTDepth: Maximum allowable binary tree depth -MinBTSize: Minimum allowable binary tree leaf node size In some exemplary implementations of the QTBT partition structure, the CTU size may be set as a 128x128 chromasample with two corresponding 64x64 blocks of chromasamples (when exemplary chroma subsampling is considered and used), MinQTSize may be set as 16x16, MaxBTSize may be set as 64x64, MinBTSize may be set as 4x4 (for both width and height), and MaxBTDepth may be set as 4. The quadtree partition may be applied to the CTU first to generate quadtree leaf nodes. A quadtree leaf node can have a size from its minimum allowable size (i.e., MinQTSize) of 16x16 to 128x128 (i.e., CTU size). If a node is 128x128, it will not be partitioned by the binary tree first because its size exceeds MaxBTSize (i.e., 64x64). Otherwise, nodes that do not exceed MaxBTSize may be partitioned by the binary tree. In the example in Figure 14, the base block is 128x128. The base block can only be quadtree-partitioned according to a predefined set of rules. The base block has a partitioning depth of 0. Each of the four resulting partitions is 64x64, not exceeding MaxBTSize, and may be further quadtree-partitioned or binary-partitioned at level 1. The process continues. When the binary tree depth reaches MaxBTDepth (i.e., 4), further partitioning is not considered. When the width of a binary tree node is equal to MinBTSize (i.e., 4), further horizontal partitioning is not considered. Similarly, when the height of a binary tree node is equal to MinBTSize, further vertical partitioning is not considered.

[0102] In some exemplary implementations, the above QTBT scheme may be configured to support the flexibility for lumens and chromians to have the same QTBT structure or separate QTBT structures. For example, in the case of P-slice and B-slice, the lumens CTB and chromens CTB within one CTU may share the same QTBT structure. However, in the case of I-slice, the lumens CTB may be divided into CBs by a QTBT structure, and the chromens CTB may be divided into chromens CBs by another QTBT structure. This means that CUs may be used to refer to different color channels within an I-slice, for example, an I-slice may consist of a coding block for the lumens component or a coding block for two chromens components, and a CU in a P-slice or B-slice may consist of a coding block for all three color components.

[0103] In some other implementations, the QTBT scheme may be complemented by the ternary scheme described above. Such implementations are sometimes called multi-type tree (MTT) structures. For example, in addition to the bipartiteization of nodes, one of the three-partiteization patterns in Figure 13 may be selected. In some implementations, only square nodes can undergo three-partiteization. Additional flags may be used to indicate whether the three-partiteization is horizontal or vertical.

[0104] The design of two-level or multi-level trees, such as QTBT implementations and QTBT implementations complemented by three-part division, may be primarily motivated by complexity reduction. Theoretically, the complexity of traversing the tree is T D Here, T represents the number of partition types and D is the depth of the tree. A trade-off may be made by using multiple types (T) while reducing the depth (D).

[0105] In some implementations, the CB may be further subdivided. For example, the CB may be further subdivided into multiple prediction blocks (PBs) for the purpose of intra-frame prediction or inter-frame prediction during the coding and decoding processes. In other words, the CB may be further subdivided into different subpartitions where individual prediction decisions / constructions may be made. In parallel, the CB may be further subdivided into multiple transformation blocks (TBs) for the purpose of describing the level at which transformation or inverse transformation of the video data is performed. The subdivision schemes of the CB into PBs and TBs may be the same or different. For example, each subdivision scheme may be implemented using its own procedure based on various characteristics of the video data, for example. The subdivision schemes of the PBs and TBs may be independent in some exemplary implementations. The subdivision schemes and boundaries of the PBs and TBs may be correlated in some other exemplary implementations. In some implementations, for example, the TBs may be subdivided after the PB subdivision, and in particular, each PB may be determined after the subdivision of the coding blocks and then further subdivided into one or more TBs. For example, in some implementations, the PBs may be subdivided into one, two, four, or other numbers of TBs.

[0106] In some implementations, luma channels and chroma channels may be treated differently in order to divide the base block into coding blocks, and further into prediction and / or transformation blocks. For example, in some implementations, the division of coding blocks into prediction and / or transformation blocks may be permitted for luma channels, but such division of coding blocks into prediction and / or transformation blocks may not be permitted for chroma channels. In such implementations, transformation and / or prediction of luma blocks can therefore only be performed at the coding block level. In another example, the minimum transformation block size of luma channels and chroma channels may differ; for example, coding blocks of luma channels may be allowed to be divided into smaller transformation and / or prediction blocks than those of chroma channels. In yet another example, the maximum depth of the division of coding blocks into transformation and / or prediction blocks may differ between luma channels and chroma channels; for example, coding blocks of luma channels may be allowed to be divided into deeper transformation and / or prediction blocks than those of chroma channels. For example, a luma coding block may be divided into transformation blocks of multiple sizes that can be represented by recursive partitioning down by up to two levels, and transformation block shapes such as square, 2:1 / 1:2, and 4:1 / 1:4, as well as transformation block sizes from 4x4 to 64x64, may be allowed. However, in the case of a chroma block, only the largest possible transformation block specified for the luma block may be allowed.

[0107] In some exemplary implementations for dividing a coding block into PBs, the depth, shape, and / or other properties of the PB division may depend on whether the PB is intracoded or intercoded.

[0108] The partitioning of coding blocks (or prediction blocks) into transformation blocks may be carried out recursively or non-recursively, with further consideration of the transformation blocks at the boundaries of the coding or prediction blocks, in a variety of exemplary schemes including, but not limited to, quadtree partitioning and predefined pattern partitioning. In general, the resulting transformation blocks may be at different partitioning levels, not necessarily the same size, and not necessarily square in shape (for example, they may be rectangles with some allowed sizes and aspect ratios). Further examples are described in more detail below in relation to Figures 15, 16, and 17.

[0109] However, in some other implementations, the CB obtained through any of the above partitioning schemes may be used as the basic or smallest coding block for prediction and / or transformation. In other words, no further partitioning is performed for the purpose of performing inter-prediction / intra-prediction and / or transformation. For example, the CB obtained from the above QTBT scheme may be used as is as the unit for performing prediction. Specifically, such a QTBT structure eliminates the concept of multiple partitioning types, i.e., removes the separation of CU, PU, ​​and TU, and supports further flexibility regarding the CU / CB partition shape as described above. In such a QTBT block structure, the CU / CB can have either a square or rectangular shape. The leaf nodes of such a QTBT are used as units for prediction and transformation processing without further partitioning. This means that the CU, PU, ​​and TU have the same block size in such an exemplary QTBT coding block structure.

[0110] The various CB partitioning schemes described above, as well as further partitioning of the CB into PB and / or TB (including no PB / TB partitioning), can be combined in any way. The following specific implementations are provided as non-limiting examples.

[0111] Specific and exemplary implementations of the partitioning of coding blocks and transform blocks are described below. In such exemplary implementations, the base block may be partitioned into coding blocks using recursive quadtree partitioning or the predefined partitioning patterns described above (such as the partitioning patterns in Figures 9 and 10). At each level, whether further quadtree partitioning of a particular partition should be continued may be determined by local video data characteristics. The resulting CBs may be at various quadtree partitioning levels and may be of various sizes. The decision of whether to code the picture area using interpicture (time) prediction or intrapicture (spatial) prediction may be made at the CB level (or at the CU level for all three color channels). Each CB may be further partitioned into one, two, four, or other number of PBs according to a predefined PB partitioning type. Within one PB, the same prediction process may be applied, and the relevant information may be sent to the decoder at the PB level. After obtaining residual blocks by applying the prediction process based on the PB partitioning type, the CB can be partitioned into TBs according to another quadtree structure similar to the coding tree for the CB. In this particular implementation, the CB or TB is not limited to squares. Furthermore, in this particular example, the PB may be square or rectangular for interpretations, and square only for intrapredictions. A coding block may be divided, for example, into four square TBs. Each TB may be further divided recursively (using quadtree partitioning) into smaller TBs called residual quadtrees (RQTs).

[0112] Another exemplary implementation for partitioning a base block into CBs, PBs, and / or TBs is described further below. For example, a quadtree with nested multitype trees using dichotomous and trichotomous segmentation structures (e.g., QTBT or the trichotomous QTBT described above) may be used instead of using multiple partition unit types such as those shown in Figure 9 or Figure 10. The separation of CBs, PBs, and TBs (i.e., partitioning a CB into PBs and / or TBs, and partitioning a PB into TBs) may be abandoned except when necessary for a CB that is too large for the maximum transformation length, if such a CB requires further partitioning. This exemplary partitioning scheme may be designed to support further flexibility regarding the CB partition shape so that both prediction and transformation can be performed at the CB level without further partitioning. In such a coding tree structure, the CB can have either a square or rectangular shape. Specifically, a coding tree block (CTB) may first be partitioned by a quadtree structure. The quadtree leaf nodes may then be further partitioned by nested multitype tree structures. An example of a nested multitype tree structure using bipartite or tripartite is shown in Figure 11. Specifically, the exemplary multitype tree structure in Figure 11 includes four segmentation types called vertical bipartite (SPLIT_BT_VER)(1102), horizontal bipartite (SPLIT_BT_HOR)(1104), vertical tripartite (SPLIT_TT_VER)(1106), and horizontal tripartite (SPLIT_TT_HOR)(1108). The CB then corresponds to the leaf of the multitype tree. In this exemplary implementation, this segmentation is used for both prediction and transformation processing without further segmentation, as long as the CB is not too large relative to the maximum transformation length. This means that in most cases, the CB, PB, and TB have the same block size in a quadtree with a nested multitype tree coding block structure. The exception occurs when the supported maximum transformation length is smaller than the width or height of the color component of the CB.In some implementations, in addition to two- or three-partitioning, the nested pattern in Figure 11 may further include quadtree partitioning.

[0113] Figure 12 shows one specific example of a quadtree having a nested multi-type tree coding block structure of block partitions (including options for quadtree partitioning, bipartite, and tripartite partitioning) for a single base block. More specifically, Figure 12 shows that the base block 1200 is quadtree partitioned into four square partitions 1202, 1204, 1206, and 1208. The decision to further use the multi-type tree structure and quadtrees of Figure 11 for further partitioning is made for each of the quadtree-partitioned partitions. In the example of Figure 12, partition 1204 is not further partitioned. Partitions 1202 and 1208 each adopt a different quadtree partitioning. In partition 1202, the second-level quadtree-partitioned upper-left, upper-right, lower-left, and lower-right partitions adopt a third-level partitioning of a quadtree, horizontal bipartite 1104 in Figure 11, unpartitioned, and horizontal tripartite 1108 in Figure 11, respectively. Partition 1208 employs a different quadtree partitioning pattern, with the second-level quadtree-partitioned upper-left, upper-right, lower-left, and lower-right partitions employing third-level partitioning patterns: vertical tripartitioning 1106, unpartitioned, unpartitioned, and horizontal bipartitioning 1104 in Figure 11, respectively. Two of the subpartitions of the upper-left partition at the third level of 1208 are further partitioned according to horizontal bipartitioning 1104 and horizontal tripartitioning 1108 in Figure 11, respectively. Partition 1206 employs a second-level partitioning pattern into two partitions, following vertical bipartitioning 1102 in Figure 11, and the two partitions are further partitioned at the third level according to horizontal tripartitioning 1108 and vertical bipartitioning 1102 in Figure 11. A fourth-level partitioning pattern is further applied to one of them according to horizontal bipartitioning 1104 in Figure 11.

[0114] In the example above, the maximum luma conversion size may be 64x64, and the maximum supported chroma conversion size may be different from luma, for example, 32x32. Even if the exemplary CB in Figure 12 is not generally divided further into smaller PB and / or TB, if the width or height of a luma coding block or chroma coding block is greater than the maximum conversion width or maximum conversion height, the luma coding block or chroma coding block may be automatically divided in that direction to satisfy the horizontal and / or vertical conversion size limitations.

[0115] In the specific example of the above division of the base block into CBs, as stated above, the coding tree scheme can support the ability for lumens and chromens to have separate block tree structures. For example, in the case of P-slice and B-slice, the lumens CTB and chromens CTB within a single CTU can share the same coding tree structure. In the case of I-slice, for example, lumens and chromens may have separate coding block tree structures. When separate block tree structures are applied, a lumens CTB may be divided into lumens CBs by one coding tree structure, and a chromens CTB may be divided into chromens CBs by another coding tree structure. This means that a CU in an I-slice may consist of coding blocks for the lumens component or coding blocks for the two chromens component, and a CU in a P-slice or B-slice will always consist of coding blocks for all three color components unless the video is monochrome.

[0116] When a coding block is further divided into multiple transformation blocks, these transformation blocks may be ordered within the bitstream according to various orders or scanning schemes. Exemplary implementations for dividing a coding block or prediction block into transformation blocks, and the coding order of the transformation blocks, are described in more detail below. In some exemplary implementations, as described above, the transformation division can support multiple shapes of transformation blocks, e.g., 1:1 (square), 1:2 / 2:1, and 1:4 / 4:1, with transformation block sizes ranging from, for example, 4x4 to 64x64. In some implementations, if the coding block is smaller than or equal to 64x64, the transformation block division may be applied only to the luma component, in the case of a chroma block, such that the transformation block size is the same as the coding block size. Alternatively, if the width or height of a coding block is greater than 64, both the luma coding block and the chroma coding block may be implicitly divided into transformation blocks that are multiples of min(W,64) × min(H,64) and min(W,32) × min(H,32), respectively.

[0117] In some exemplary implementations of transform block partitioning, for both intracoded and interconnected blocks, a coding block may be further subdivided into multiple transform blocks having a predefined number of levels of partitioning depth (e.g., 2 levels). The partitioning depth and size of the transform blocks may be related. For some exemplary implementations, the mapping from the transform size at the current depth to the transform size at the next depth is shown in Table 1 below.

[0118] [Table 1]

[0119] Based on the exemplary mapping in Table 1, for a 1:1 square block, the next level of transformation partitioning can create four 1:1 square subtransformation blocks. The transformation partitioning may stop at, for example, 4x4. Thus, the transformation size of the current depth of 4x4 corresponds to the same size of 4x4 at the next depth. In the example in Table 1, for a 1:2 / 2:1 non-square block, the next level of transformation partitioning can create two 1:1 square subtransformation blocks, while for a 1:4 / 4:1 non-square block, the next level of transformation partitioning can create two 1:2 / 2:1 subtransformation blocks.

[0120] In some exemplary implementations, further restrictions may be applied to the Luma components of an intra-coded block with respect to the transformation block partitioning. For example, at each level of transformation partitioning, all sub-transformation blocks may be restricted to having equal sizes. For instance, for a 32x16 coding block, a level 1 transformation partitioning would create two 16x16 sub-transformation blocks, and a level 2 transformation partitioning would create eight 8x8 sub-transformation blocks. In other words, to keep the transformation units of equal size, a second-level partitioning must be applied to all first-level sub-blocks. An example of transformation block partitioning for an intra-coded square block according to Table 1 is shown in Figure 15, along with the coding order indicated by arrows. Specifically, 1502 shows a square coding block. A first-level partitioning into four equally sized transformation blocks according to Table 1 is shown in 1504, along with the coding order indicated by arrows. A second-level partitioning of all first-level equally sized blocks into 16 equally sized transformation blocks according to Table 1 is shown in 1506, along with the coding order indicated by arrows.

[0121] In some exemplary implementations, the above restrictions on intracoding may not apply to the luma components of an interconnected block. For example, after a first level of transform partitioning, one of the sub-transformation blocks may be further independently partitioned at another level. Thus, the resulting transformation blocks may or may not be of the same size. An exemplary partitioning of an interconnected block into transformation blocks according to their coding order is shown in Figure 16. In the example in Figure 16, the interconnected block 1602 is partitioned into transformation blocks at two levels according to Table 1. At the first level, the interconnected block is partitioned into four transformation blocks of equal size. Then, only one of the four transformation blocks (but not all of them) is further partitioned into four sub-transformation blocks, resulting in a total of seven transformation blocks of two different sizes, as shown by 1604. The exemplary coding order of these seven transformation blocks is indicated by the arrow in 1604 of Figure 16.

[0122] In some exemplary implementations, for chroma components, some further restrictions may be applied to the transformation block. For example, for chroma components, the transformation block size may be the same as the coding block size, but it cannot be smaller than a predefined size, e.g., 8x8.

[0123] In some other exemplary implementations, for coding blocks where either the width (W) or height (H) is greater than 64, both the luma coding block and the chroma coding block may be implicitly divided into transformation units of multiples of min(W,64) × min(H,64) and min(W,32) × min(H,32), respectively. Hereinafter, "min(a,b)" may return the smaller value between a and b.

[0124] Figure 17 further illustrates another alternative exemplary scheme for splitting a coding block or prediction block into transformation blocks. As shown in Figure 17, instead of using recursive transformation partitioning, a predefined set of partitioning types may be applied to the coding block according to the transformation type of the coding block. In the particular example shown in Figure 17, one of the six exemplary partitioning types may be applied to split the coding block into a varying number of transformation blocks. Such a scheme for generating transformation block partitioning may be applied to either the coding block or the prediction block.

[0125] More specifically, the partitioning scheme in Figure 17 provides up to six exemplary partitioning types for any given transformation type (where the transformation type refers to, for example, the type of primary transformation such as ADST). In this scheme, all coding blocks or prediction blocks may be assigned a transformation partitioning type based, for example, on rate distortion cost. In one example, the transformation partitioning type assigned to a coding block or prediction block may be determined based on the transformation type of the coding block or prediction block. As illustrated by the six transformation partitioning types illustrated in Figure 17, a particular transformation partitioning type can correspond to the partitioning size and pattern of the transformation block. The correspondence between various transformation types and various transformation partitioning types may be predefined. An example of a transformation partitioning type that can be assigned to a coding block or prediction block based on rate distortion cost, indicated by capital letter labels, is shown below.

[0126] • PARTITION_NONE: Allocates a conversion size equal to the block size.

[0127] • PARTITION_SPLIT: Assigns a conversion size that is half the width and half the height of the block size.

[0128] • PARTITION_HORZ: Assigns a conversion size that is the same width as the block size and half the height of the block size.

[0129] • PARTITION_VERT: Assigns a conversion size that is half the width of the block size and the same height as the block size.

[0130] • PARTITION_HORZ4: Assigns a conversion size that is the same width as the block size and 1 / 4 the height of the block size.

[0131] • PARTITION_VERT4: Allocates a conversion size that is 1 / 4 the width of the block size and the same height as the block size.

[0132] In the example above, the transformation partitioning types shown in Figure 17 all involve a uniform transformation size for the partitioned transformation blocks. This is merely an example, not an limitation. In some other implementations, a mixed transformation block size may be used for the partitioned transformation blocks in a particular partitioning type (or pattern).

[0133] A PB (or CB, also called a PB if not further divided into prediction blocks) obtained from any of the above partitioning methods can become individual blocks for coding via either intra-prediction or inter-prediction. For inter-prediction in the current PB, a residual is generated between the current block and the prediction block, which can be coded and included in the coded bitstream.

[0134] Interpretation can be performed, for example, in single-reference mode or composite-reference mode. In some implementations, a skip flag may be initially included in the bitstream of the current block (or at a higher level) to indicate whether the current block is being intercoded and will not be skipped. If the current block is being intercoded, other flags may be further included in the bitstream as signals indicating whether single-reference mode or composite-reference mode is being used for the current block. In single-reference mode, one reference block may be used to generate the predicted block for the current block. In composite-reference mode, two or more reference blocks may be used, for example, by weighted averaging to generate the predicted block. Composite-reference mode may be called multiple-reference mode, two-reference mode, or multi-reference mode. One or more reference blocks may be identified using one or more reference frame indices, and further using one or more corresponding motion vectors indicating the shift between one or more reference blocks and the current block, for example, in position within horizontal and vertical pixels. For example, the interpretation block of the current block may be generated from a single reference block identified by a single motion vector in the reference frame as a prediction block in single reference mode, but in composite reference mode, the prediction block may be generated by a weighted average of two reference blocks in two reference frames indicated by two motion vectors. Motion vectors can be coded in various ways and included in the bitstream.

[0135] In some implementations, an encoding or decoding system may maintain a decoded picture buffer (DPB). Some images / pictures may be maintained in the DPB waiting to be displayed (in the decoding system), and some images / pictures in the DPB may be used as reference frames to enable interpretation. In some implementations, reference frames in the DPB may be tagged as either short-term or long-term references to the current image being encoded or decoded. For example, short-term reference frames may include, in decoding order, frames used for interpretation of blocks in the current frame or a predetermined number (e.g., 2) of subsequent video frames closest to the current frame. Long-term reference frames may include frames in the DPB that can be used to predict image blocks in frames beyond a predetermined number of frames from the current frame in decoding order. Information regarding such tags for short-term and long-term reference frames is referred to as a reference picture set (RPS) and may be appended to the header of each frame in the encoded bitstream. Each frame in an encoded video stream can be identified by a picture sequence counter (POC), which is numbered according to the playback sequence, either in an absolute manner or in relation to a group of pictures starting with, for example, frame I.

[0136] In some exemplary implementations, one or more reference picture lists, including the identification of short-term and long-term reference frames for inter-prediction, may be formed based on information in the RPS. For example, a single picture reference list may be formed for unidirectional inter-prediction, denoted as L0 reference (or reference list 0), and two picture reference lists may be formed for bidirectional inter-prediction, denoted as L0 (or reference list 0) and L1 (or reference list 1) for each of the two prediction directions. The reference frames included in the L0 and L1 lists may be ordered in various predetermined ways. The lengths of the L0 and L1 lists may be signaled in the video bitstream. Unidirectional inter-prediction can be either single-reference mode or composite-reference mode, provided that multiple references for generating prediction blocks by weighted averaging in composite-prediction mode are on the same side of the block to be predicted. Bidirectional inter-prediction can only be composite mode, in that bidirectional inter-prediction includes at least two reference blocks.

[0137] In some implementations, a merge mode (MM) for interpretation may be implemented. Generally, in merge mode, one or more motion vectors in a single reference prediction or a composite reference prediction of the current PB may be derived from other motion vectors rather than being computed and signaled independently. For example, in an encoding system, the current motion vector of the current PB may be reduced to the difference between the current motion vector and one or more other already encoded motion vectors (called reference motion vectors). Such a difference of motion vectors, rather than the entire current motion vector, may be encoded and included in the bitstream and linked to the reference motion vectors. Correspondingly, in a decoding system, the motion vector corresponding to the current PB may be derived based on the difference of the decoded motion vectors and the decoded reference motion vector linked to it. As a specific form of general merge mode (MM) interpretation, such interpretation based on the difference of motion vectors is sometimes called merge mode by difference of motion vectors (MMVD). Thus, general MM, or MMVD in particular, may be implemented to improve coding efficiency by leveraging the correlation between motion vectors associated with different PBs. For example, adjacent PBs may have similar motion vectors. In another example, motion vectors can be correlated temporally (between frames) for blocks that are similarly positioned / placed in space.

[0138] In some exemplary implementations, the MM flag may be included in the bitstream during the encoding process to indicate whether the current PB is in merge mode. Additionally or alternatively, the MMVD flag may be included and signaled in the bitstream during the encoding process to indicate whether the current PB is in MMVD mode. The MM and / or MMVD flags or indicators may be provided at the PB level, CB level, CU level, CTB level, CTU level, slice level, picture level, etc. In certain examples, both the MM and MMVD flags may be included for the current CU, and the MMVD flag may be signaled immediately after the skip flag and the MM flag to specify whether MMVD mode is used for the current CU.

[0139] In some exemplary implementations of MMVD, a list of merge candidates for motion vector prediction may be formed for a predicted block. The list of merge candidates can contain a predetermined number (e.g., 2) of MV predictor candidate blocks whose motion vectors can be used to predict the current motion vector. MVD candidate blocks can include blocks selected from adjacent blocks within the same frame and / or time blocks (e.g., blocks that are identically located in frames in progress or subsequent to the current frame). These options represent blocks that are spatially or temporally located relative to the current block and are likely to have similar or identical motion vectors to the current block. The size of the list of MV predictor candidates may be predetermined. For example, the list can contain 2 candidates. In order to be on the list of merge candidates, candidate blocks must, for example, have the same reference frame (or multiple frames) as the current block, must exist (e.g., boundary checking must be performed if the current block is near a frame edge), must have already been encoded during the encoding process, and / or must have already been decoded during the decoding process. In some implementations, the list of merge candidates may be filled first with spatially adjacent blocks (scanned in a specific predetermined order) if available and satisfying the above conditions, and then with time blocks if space is still available in the list. Adjacent candidate blocks can be selected, for example, from the blocks to the left and above the current block. The list of merge MV predictor candidates may be signaled within the bitstream.

[0140] In some implementations, the actual merge candidate used as a reference motion vector for predicting the motion vector of the current block may be signaled. If the merge candidate list contains two candidates, a one-bit flag called the merge candidate flag may be used to indicate the selection of the reference merge candidate. For the current block being predicted in composite mode, each of the multiple motion vectors predicted using the MV predictor may be associated with a reference motion vector from the merge candidate list.

[0141] In some exemplary implementations of MMVD, a merge candidate is selected and used as the base motion vector predictor for the predicted motion vector, after which the motion vector difference (MVD or deltaMV representing the difference between the motion vector to be predicted and the reference candidate motion vector) can be calculated in the encoding system. Such an MVD may contain information representing the magnitude and direction of the MV difference, which can be signaled within the bitstream. The magnitude and direction of the motion difference can be signaled in various ways.

[0142] In some exemplary implementations of MMVD, a distance index can be used to specify the magnitude of the motion vector difference and indicate one of a set of predetermined offsets that represent a given motion vector difference from a starting point (reference motion vector). The MV offset corresponding to the signaled index can then be added to either the horizontal or vertical component of the starting (reference) motion vector. Whether either the horizontal or vertical component of the reference motion vector should be offset is determined by the exemplary directional information of the MVD. Exemplary predetermined relationships between distance indexes and predetermined offsets are specified in Table 2.

[0143] [Table 2]

[0144] In some exemplary implementations of MMVD, a direction index may be further signaled and used to represent the direction of the MVD relative to the reference motion vector. In some implementations, the direction may be restricted to either the horizontal or vertical direction. Exemplary 2-bit direction indices are shown in Table 3. In the examples in Table 3, the interpretation of the MVD may vary depending on the information of the start / reference MV. For example, if the start / reference MV corresponds to a single predictive block, or if both reference framelists correspond to two predictive blocks pointing to the same side of the current picture (i.e., the POCs of both reference pictures are either greater than the POC of the current picture, or both are less than the POC of the current picture), the sign in Table 3 may specify the sign (direction) of the MV offset applied to the start / reference MV. If the start / reference MV corresponds to a biprediction block having two reference pictures on different sides of the current picture (i.e., the POC of one reference picture is greater than the POC of the current picture, and the POC of the other reference picture is less than the POC of the current picture), and the difference between the reference POC in picture reference list 0 and the current frame is greater than the difference between the reference POC in picture reference list 1 and the current frame, then the sign in Table 3 can specify the sign of the MV offset applied to the reference MV corresponding to the reference picture in picture reference list 0, and the sign of the offset of the MV corresponding to the reference picture in picture reference list 1 can have the opposite value (opposite sign of the offset). Otherwise, if the difference between the reference POC in picture reference list 1 and the current frame is greater than the difference between the reference POC in picture reference list 0 and the current frame, then the sign in Table 3 can specify that the sign of the MV offset applied to the reference MV associated with picture reference list 1 and the sign of the offset to the reference MV associated with picture reference list 0 have opposite values.

[0145] [Table 3]

[0146] In some exemplary implementations, the MVD may be scaled according to the difference in POCs in each direction. If the difference in POCs in both lists is the same, scaling is not necessary. If, instead, the difference in POCs in reference list 0 is greater than the difference in reference list 1, the MVD of reference list 1 is scaled. If the difference in POCs in reference list 1 is greater than that of list 0, the MVD of list 0 may be scaled similarly. If the initial MV is single predicted, the MVD is added to the available or reference MVs.

[0147] In some exemplary implementations of MVD coding and signaling for bidirectional composite prediction, in addition to coding and signaling two MVDs separately, or instead, symmetric MVD coding may be implemented such that only one MVD requires signaling, and the other MVD can be derived from the signaled MVD. In such implementations, motion information containing reference picture indices in both List 0 and List 1 is signaled. However, for example, only the MVD associated with reference list 0 is signaled, and the MVD associated with reference list 1 is not signaled and is derived. Specifically, at the slice level, a flag called "mvd_l1_0_flag" may be included in the bitstream to indicate whether reference list 1 is not signaled in the bitstream. If this flag is 1, indicating that reference list 1 is equal to 0 (and therefore not signaled), then a bidirectional prediction flag called "BiDirPredFlag" may be set to 0, which means there is no bidirectional prediction. If mvd_l1_0_flag is 0, BiDirPredFlag may be set to 1 if the nearest reference picture in List 0 and the nearest reference picture in List 1 form a forward-reverse pair of reference pictures or a reverse-forward pair of reference pictures, and both reference pictures in List 0 and List 1 are short-term reference pictures. Otherwise, BiDirPredFlag is set to 0. A BiDirPredFlag of 1 indicates that a symmetric mode flag is additionally signaled in the bitstream. The decoder can extract the symmetric mode flag from the bitstream if BiDirPredFlag is 1. The symmetric mode flag is signaled, for example, at the CU level (if necessary) to indicate whether a symmetric MVD coding mode is being used for the corresponding CU.When the symmetric mode flag is 1, it indicates the use of the symmetric MVD coding mode, where only the reference picture indices in both List 0 and List 1 (called "mvp_l0_flag" and "mvp_l1_flag") are signaled by the MVD associated with List 0 (called "MVD0"), and the other motion vector difference "MVD1" should be derived rather than signaled. For example, MVD1 may be derived as -MVD0. Thus, in the exemplary symmetric MVD mode, only one MVD is signaled. In some other exemplary implementation forms for MV prediction, harmonic schemes may be used to implement the general merge mode MMVD, and several other types of MV prediction, for both single-reference mode MV prediction and compound-reference mode MV prediction. Various syntactic elements may be used to signal how the MV of the current block is predicted.

[0148] For example, in single-reference mode, the following MV prediction modes may be signaled.

[0149] NEARMV uses one of the motion vector predictors (MVPs) in a list directly indicated by a DRL (Dynamic Reference List) index without MVD.

[0150] NEWMV uses one of the motion vector predictors (MVPs) in the list signaled by the DRL index as a reference, and applies the delta to the MVP (e.g., use MVD).

[0151] GLOBALMV uses motion vectors based on global motion parameters at the frame level.

[0152] Similarly, in the case of a composite reference interpretation mode that uses two reference frames corresponding to the two MVs to be predicted, the following MV prediction modes may be signaled:

[0153] NEAR_NEARMV - For each of the two MVs to be predicted, use one of the motion vector predictors (MVPs) in a list signaled by a DRL index without MVD.

[0154] NEAR_NEWMV - To predict the first of two motion vectors, use MVD and one of the motion vector predictors (MVPs) in the DRL index-signaled list as the reference MV, and to predict the second of the two motion vectors, use one of the motion vector predictors (MVPs) in the DRL index-signaled list as the reference MV, in conjunction with an additionally signaled delta MV (MVD).

[0155] NEW_NEARMV - To predict the second of two motion vectors, it uses MVD and one of the motion vector predictors (MVPs) in a list signaled by the DRL index as the reference MV, and to predict the first of two motion vectors, it uses one of the motion vector predictors (MVPs) in a list signaled by the DRL index as the reference MV in conjunction with an additionally signaled delta MV (MVD).

[0156] NEW_NEWMV uses one of the motion vector predictors (MVPs) in a list signaled by the DRL index as the reference MV, and uses it in conjunction with an additionally signaled delta MV to make predictions for each of the two MVs.

[0157] GLOBAL_GLOBALMV - Uses MV from each reference based on frame-level global motion parameters.

[0158] Therefore, the term "NEAR" above refers to MV prediction using a reference MV without an MVD as a general merge mode, while the term "NEW" refers to MV prediction that uses a reference MV and involves offsetting it with a signaled MVD, similar to the MMVD mode. In the case of composite interpretation, both the reference-based motion vector and the motion vector delta described above may be generally different or independent between the two references, even though they may be correlated, and such correlation may be used to reduce the amount of information required to signal the two motion vector deltas. In such situations, joint signaling of the two MVDs may be implemented and shown in the bitstream.

[0159] The above dynamic reference list (DRL) can be used to maintain a set of indexed motion vectors that are dynamically maintained and considered candidate motion vector predictors.

[0160] In some exemplary implementations, a predefined resolution for the MVD may be acceptable. For example, a motion vector accuracy (or precision) of 1 / 8 of a pixel may be acceptable. The MVDs described above in various MV prediction modes can be constructed and signaled in various ways. In some implementations, various syntax elements can be used to signal the above motion vector difference in reference frame list 0 or list 1.

[0161] For example, a syntax element called "mv_joint" can specify which components of the associated motion vector difference are non-zero. In the case of MVD, this signals all non-zero components together. For example, mv_joint can have the following values: A value of 0 can indicate that there are no non-zero MVDs along either the horizontal or vertical axis. 1 can be shown to indicate that there is a non-zero MVD only along the horizontal direction. 2 can be shown that there is a non-zero MVD only along the vertical direction. 3 can be shown to have a non-zero MVD along both the horizontal and vertical directions.

[0162] If the “mv_joint” syntax element for MVD signals that there are no non-zero MVD components, no further MVD information can be signaled. However, if the “mv_joint” syntax signals that there are one or two non-zero components, additional syntax elements can further signal each of the non-zero MVD components, as described below.

[0163] For example, a syntax element called "mv_sign" may be used to further specify whether the corresponding motion vector difference component is positive or negative.

[0164] In another example, a syntax element called "mv_class" can be used to specify the class of motion vector differences between a predefined set of classes for corresponding non-zero MVD components. These predefined classes of motion vector differences can be used, for example, to separate a contiguous size space of motion vector differences into non-overlapping ranges, each range corresponding to an MVD class. Thus, the signaled MVD classes indicate the size ranges of the corresponding MVD components. In the exemplary implementation shown in Table 4, higher classes correspond to motion vector differences with larger size ranges. In Table 4, the symbol (n,m) is used to represent a range of motion vector differences greater than n pixels and less than or equal to m pixels.

[0165] [Table 4]

[0166] In some other examples, a syntax element called "mv_bit" can be used to specify the integer part of the offset between the non-zero motion vector difference component and the magnitude of the starting MV class that is signaled in correspondence. The number of bits required for "my_bit" to signal the entire range of each MVD class may vary depending on the MV class. For example, in the implementations in Table 4, MV_CLASS 0 and MV_CLASS 1 may require only a single bit to indicate an integer pixel offset of 1 or 2 from the starting MVD of 0, and each higher MV_CLASS may progressively require one more bit for "mv_bit" than the previous MV_CLASS.

[0167] In some other implementations, a syntax element called “mv_fr” may be further used to specify the first two fractional bits of the motion vector difference for the corresponding non-zero MVD component, and a syntax element called “mv_hp” may be used to specify a third fractional bit (high-resolution bit) of the motion vector difference for the corresponding non-zero MVD component. Two bits of “mv_fr” essentially provide a quarter-pixel MVD resolution, but the “mv_hp” bits can further provide a resolution of eighth pixels. In some other implementations, two or more “mv_hp” bits may be used to provide MVD pixel resolutions finer than eighth pixels. In some exemplary implementations, additional flags may be signaled at one or more of various levels to indicate whether MVD resolutions of eighth pixels or higher are supported. If an MVD resolution does not apply to a particular coding unit, the above syntax elements for the corresponding unsupported MVD resolution may not be signaled.

[0168] In this disclosure, in the case of a composite reference mode, if the point of control (POC) of both reference frames for a given motion vector pair is greater than or less than the point of control (POC) of the current frame, then the directions of the two reference frames are the same. Otherwise, if the POC of one reference frame is greater than the POC of the current frame, and the POC of the other reference frame is less than the POC of the current frame, then the directions of the two reference frames are different.

[0169] As described above, various interprediction modes, such as single-reference mode and combined-reference mode (or single-reference interprediction mode and combined-reference interprediction mode), can be applied to blocks under interprediction. Single-reference mode can include NEARMV, NEWMV, and GLOBALMV regarding the motion vector prediction mode, or how the motion vectors are predicted (e.g., whether to use some MVD). Combined-reference mode can include NEAR_NEARMV, NEAR_NEWMV, NEW_NEARMV, NEW_NEWMV, and GLOBAL_GLOBALMV regarding how each of the motion vectors should be predicted. In some situations, the combinations, distributions, and characteristics of MVD-related signaling parameters, such as the signaling parameters or syntax associated with motion vector prediction parameters and the various exemplary parameters mentioned above (e.g., "mv_class", "mv_bit", "mv_joint", "mv_sign", "mv_fr", "mv_hp", etc.), may differ among these interconnected modes, but may be more similar among certain interconnection modes (single-reference modes or composite modes) and / or motion vector prediction modes (e.g., NEARMV, NEWMV, or GLOBALMV, etc.). Therefore, when performing entropy encoding of one or more MVD-related syntaxes, a single context derivation method that is optimal for one of these modes may not be optimal for another mode if the single context derivation method does not take into account different interconnection modes and / or motion vector prediction modes. Sharing the same context derivation method among all of these modes may be suboptimal and inefficient. Various exemplary implementations for entropy encoding MVD-related syntax are disclosed below, utilizing context models derived from various context derivation methods and leveraging the correlation between motion vector predictions and MVD-related syntax in each of the inter-prediction modes and / or prediction modes of motion vectors.

[0170] In some exemplary implementations, MVD-related syntax can be entropy encoded using a context model that can be derived using a context derivation method. MVD-related syntax is signaled and applied to the MVD associated with the block. As mentioned above, MVD-related syntax can include mv_joint, mv_sign, mv_class, mv_bit, mv_fr, and mv_hp. The context derivation method may depend on the block's reference mode. The context derivation method may also depend on whether the MVD is signaled to only one reference framelist or to two reference lists. The context derivation method may also depend on the block's motion information or the motion information of the MVD associated with the block.

[0171] In some exemplary implementations, the interpretation mode may include single-reference interpretation mode or compound-reference interpretation mode. The context derivation method may depend on whether the current block is in single-reference mode or compound-reference mode. The context may differ between single-reference mode and compound-reference mode for one or more sets of MVD syntax described above. The different contexts may be predefined in the encoder or determined dynamically.

[0172] In some exemplary implementations, the interpretation mode can indicate whether a single reference list or multiple (e.g., 2) reference lists are used for interpretation in the current block. Context derivation for signaling MVD-related syntax may depend on whether MVD is signaled for only one reference framelist or for both. The context may differ for one or more sets of MVD syntax described above, based on whether MVD is signaled for only one reference framelist or for both. These different contexts may be predefined in the encoder or determined dynamically.

[0173] In some exemplary implementations, the context may depend on the motion vector prediction mode. The context may differ for one or more sets of MVD syntaxes among the various NEWMV, NEARMV, GLOBALMV, NEW_NEARMV, NEAR_NEWMV, NEW_NEWMV, and GLOBAL_GLOBALMV motion vector prediction modes.

[0174] In some exemplary implementations, the interpretation mode may be a composite reference mode. The context derivation method may depend on whether the motion vector prediction mode of the current block is NEW_NEARMV mode or NEAR_NEWMV mode. If the motion vector prediction mode of the current block is NEW_NEARMV or NEAR_NEWMV mode, a first context may be used for the set of one or more MVD syntax elements described above. If the motion vector prediction mode of the current block is not NEW_NEARMV or NEAR_NEWMV mode, a second context may be used for the set of one or more MVD syntax elements described above. The first and second contexts are different. The first and second contexts may be predefined in the encoder or may be determined dynamically.

[0175] For example, if the current block is coded as NEW_NEARMV and / or NEAR_NEWMV mode, one context may be selected to signal mv_joint. Otherwise, one or more different contexts may be selected to signal mv_joint. The one context and one or more different contexts may be predetermined or may be selected dynamically, for example, during a video data encoding session.

[0176] In another example, if the current block is coded as NEW_NEARMV and / or NEAR_NEWMV mode, one context may be selected to signal the mv_class. Otherwise, one or more different contexts may be selected to signal the mv_class. The one context and one or more different contexts may be predetermined or selected dynamically, for example, during a video data encoding session.

[0177] In some exemplary implementations, the context derivation method may depend on whether the current block's reference mode is NEWMV mode or NEW_NEWMV mode. If the current block's motion vector prediction mode is NEW mode or NEW_NEWMV mode, a first context may be used for the set of one or more MVD syntax elements described above. If the current block's motion vector prediction mode is NEWMV mode or NEW_NEWMV mode, a second context may be used for the set of one or more MVD syntax elements described above. The first and second contexts are distinct. The first and second contexts may be predefined in the encoder or determined dynamically.

[0178] For example, if the current block is coded as NEWMV mode and / or NEW_NEWMV mode, one context may be selected to signal mv_joint. Otherwise, a different context may be selected to signal mv_joint. These different contexts may be predefined in the encoder or determined dynamically.

[0179] In embodiments of this disclosure, any steps and / or operations may be combined or arranged in any quantity or order as needed. Two or more of the steps and / or operations may be performed in parallel.

[0180] The embodiments of this disclosure may be used separately or combined in any order. Furthermore, each of the methods (or embodiments), encoder, and decoder may be implemented by a processing circuit (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored on a non-temporary computer-readable medium. The embodiments of this disclosure may be applied to luma blocks or chroma blocks.

[0181] Figure 18 shows an exemplary method 1800 for decoding a video block in a video stream. Method 1800 may include some or all of the following steps: receiving a video stream 1810; determining the inter-prediction mode of a video block from the video stream 1820; determining the motion vector prediction mode of a video block 1830; deriving a context from the video stream for signaling a set of syntax elements associated with a motion vector difference (MVD) associated with the video block, based on at least one of the inter-prediction mode and the motion vector prediction mode 1840; and decoding the video block based on the set of syntax elements and the derived context 1850.

[0182] The techniques described above can be implemented as computer software using computer-readable instructions and can be physically stored on one or more computer-readable media. For example, Figure 19 shows a computer system (2000) suitable for carrying out a particular embodiment of the disclosed subject matter.

[0183] Computer software can be coded using any suitable machine code or computer language that can undergo mechanisms such as assembly, compilation, and linking, and can create code containing instructions that can be executed directly or via interpretation, microcode execution, etc., by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc.

[0184] Instructions can be executed on various types of computers or computer components, including, for example, personal computers, tablet computers, servers, smartphones, game consoles, and Internet of Things devices.

[0185] The components shown in Figure 19 with respect to the computer system (2000) are illustrative in nature and are not intended to imply any limitation on the scope of use or functionality of the computer software implementing embodiments of the present disclosure. The configuration of the components should not be construed as having any dependencies or requirements on any one or combination of components shown in the exemplary embodiment of the computer system (2000).

[0186] Computer systems (2000) may include certain human interface input devices. Such human interface input devices may respond to input from one or more human users, for example, through tactile input (such as keystrokes, swipes, or data glove movements), audio input (such as voices or clapping), visual input (such as gestures), or olfactory input (not depicted). Human interface devices may also be used to capture certain media that are not necessarily directly related to conscious human input, such as audio (such as voices, music, or ambient sounds), images (such as scanned images or photographic images taken from a still camera), or video (such as two-dimensional video or three-dimensional video, including stereoscopic video).

[0187] Input human interface devices may include one or more of the following (only one of each is shown): keyboard (2001), mouse (2002), trackpad (2003), touchscreen (2010), data glove (not shown), joystick (2005), microphone (2006), scanner (2007), and camera (2008).

[0188] The computer system (2000) may also include certain human interface output devices. Such human interface output devices may stimulate the senses of one or more human users, for example, through tactile output, sound, light, and smell / taste. Such human interface output devices may include tactile output devices (e.g., tactile feedback via touchscreens (2010), data gloves (not shown), or joysticks (2005), but there may also be tactile feedback devices that do not function as input devices), audio output devices (e.g., speakers (2009), headphones (not shown)), visual output devices (screens (2010), including CRT screens, LCD screens, plasma screens, OLED screens, etc., each with or without touchscreen input functionality, each with or without tactile feedback functionality, some of which are capable of outputting two-dimensional visual output or three-dimensional or more output by means such as stereographic output, virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).

[0189] The computer system (2000) may also include human-accessible storage devices and their associated media, such as optical media including CD / DVD ROM / RW (2020) with media such as CD / DVD (2021), thumb drives (2022), removable hard drives or solid-state drives (2023), legacy magnetic media such as tapes and floppy disks (not shown), and dedicated ROM / ASIC / PLD-based devices such as security dongles (not shown).

[0190] Those skilled in the art should also understand that the term “computer-readable medium” as used in relation to the subject matter of this disclosure does not include transmission media, carrier waves, or other transient signals.

[0191] The computer system (2000) may also include interfaces (2054) to one or more communication networks (2055). These networks may be, for example, wireless, wired, or optical. Networks may further be local, wide-area, metropolitan, automotive, and industrial, real-time, or latency-tolerant. Examples of networks include local area networks such as Ethernet, cellular networks including wireless LAN, GSM, 3G, 4G, 5G, and LTE, wired or wireless wide-area digital networks for television including cable television, satellite television, and terrestrial television, and automotive and industrial networks including CAN bus. Certain networks typically require an external network interface adapter connected to a specific general-purpose data port or peripheral bus (2049) (e.g., a USB port on the computer system (2000)), while others are generally integrated into the core of the computer system (2000) by connecting to a system bus, as described below (e.g., an Ethernet interface to a PC computer system, or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system (2000) may communicate with other entities. Such communications can be, for example, one-way receive only (e.g., terrestrial television broadcasting), one-way transmit only (e.g., CANbus to a specific CANbus device), or bidirectional, to other computer systems using local or wide-area digital networks. Specific protocols and protocol stacks may be used on each of those networks and network interfaces, as described above.

[0192] The aforementioned human interface device, human-accessible memory device, and network interface can be attached to the core (2040) of the computer system (2000).

[0193] The core (2040) may include one or more central processing units (CPUs) (2041), graphics processing units (GPUs) (2042), dedicated programmable processing units in the form of field-programmable gate areas (FPGAs) (2043), hardware accelerators for specific tasks (2044), graphics adapters (2050), and the like. These devices may be connected via a system bus (2048) along with read-only memory (ROM) (2045), random access memory (2046), internal non-user-accessible hard drives, SSDs, and other internal mass storage (2047). In some computer systems, the system bus (2048) may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripherals can be connected directly to the core's system bus (2048) or via a peripheral bus (2049). For example, a screen (2010) may be connected to a graphics adapter (2050). Peripheral bus architectures include PCI, USB, and others.

[0194] The CPU (2041), GPU (2042), FPGA (2043), and accelerator (2044) can execute specific instructions that, in combination, can constitute the aforementioned computer code. This computer code can be stored in ROM (2045) or RAM (2046). Temporary data can also be stored in RAM (2046), and persistent data can be stored, for example, in internal mass storage (2047). High-speed storage and retrieval to any of the memory devices can be enabled by using cache memory, which can be closely associated with one or more CPUs (2041), GPUs (2042), mass storage (2047), ROM (2045), RAM (2046), etc.

[0195] A computer-readable medium may contain computer code for performing various computer implementation operations. The medium and computer code may be specifically designed and constructed for the purposes of this disclosure, or they may be of a type that is well known and available to those skilled in the computer software technology.

[0196] As a non-limiting example, a computer system having an architecture (2000), in particular a core (2040), can provide functionality as a result of (one or more) processors (including CPUs, GPUs, FPGAs, accelerators, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media can be user-accessible mass storage as described above, as well as media associated with specific storage of the core (2040) that are non-transient in nature, such as core internal mass storage (2047) or ROM (2045). Software implementing various embodiments of the present disclosure can be stored in such devices and executed by the core (2040). The computer-readable media can include one or more memory devices or chips, depending on the specific needs. The software can cause the core (2040), specifically the processors (including CPUs, GPUs, FPGAs, etc.) therein, to execute specific processes or specific parts of specific processes described herein, including defining data structures stored in RAM (2046) and modifying such data structures according to processes defined by the software. In addition, or as an alternative, a computer system may provide functionality as a result of logic hardwired or otherwise embodied in a circuit (e.g., an accelerator (2044)), which may operate in place of or with software to perform a particular process or a particular part of a particular process as described herein. References to software may, as necessary, include logic and vice versa. References to computer-readable media may, as necessary, include circuitry that stores software for execution (such as an integrated circuit (IC)), circuitry that embodies logic for execution, or both. This disclosure encompasses any suitable combination of hardware and software.

[0197] While a particular invention has been described with reference to exemplary embodiments, this description is not intended to be limiting. A reader of this description will see various modifications of the exemplary and additional embodiments of the invention. A reader of the art will readily understand that various other modifications can be made to the exemplary embodiments illustrated and described herein without departing from the spirit and scope of the invention. Accordingly, the appended claims are intended to encompass any such modifications and alternative embodiments. Certain parts in the drawings may be exaggerated, and other parts may be minimized. Therefore, this disclosure and the drawings should be considered illustrative, not limiting.

[0198] The following is a list of acronyms that may appear in this disclosure. JEM: Collaborative Search Model VVC: Versatile Video Coding BMS: Benchmark Set MV: Motion Vector HEVC: High-Efficiency Video Coding SEI: Supplementary and Extended Information VUI: Video Usability Information GOP: Picture Group TU: Conversion Unit PU: Prediction Unit CTU: Coding Tree Unit CTB: Coding Tree Block PB: Prediction Block HRD: Virtual Reference Decoder SNR: Signal-to-noise ratio CPU: Central Processing Unit GPU: Graphics Processing Unit CRT: cathode ray tube LCD: Liquid crystal display OLED: Organic Light-Emitting Diode CD: Compact Disc DVD: Digital Video Disc ROM: Read-only memory RAM: Random Access Memory ASIC: Application-Specific Integrated Circuit PLD: Programmable Logical Device LAN: Local Area Network GSM: Global System for Mobile Communications LTE: Long-Term Evolution CANBus: Controller Area Network Bus USB: Universal Serial Bus PCI: Peripheral component interconnection FPGA: Field-Programmable Gate Area SSD: Solid State Drive IC: Integrated Circuit HDR: High Dynamic Range SDR: Standard Dynamic Range JVET: Joint Video Exploration Team MPM: Highest probability mode WAIP: Wide-Angle Intra Prediction CU: Coding Unit PU: Prediction Unit TU: Conversion Unit CTU: Coding Tree Unit PDPC: Location-dependent predictive coupling ISP: Intra Subpartition SPS: Sequence Parameter Settings PPS: Picture Parameter Set APS: Adaptive Parameter Set VPS: Video Parameter Set DPS: Decoding parameter set ALF: Adaptive Loop Filter SAO: Sample Adaptive Offset CC-ALF: Cross-Component Adaptive Loop Filter CDEF: Constraint Directional Enhancement Filter CCSO: Cross-component sample offset LSO: Local Sample Offset LR: Loop Restoration Filter AV1: AOMedia Video 1 AV2:AOMedia Video 2 MVD: Motion Vector Difference CfL: Chroma from Luma SDT: Semi-Separated Tree SDP: Semi-separate split SST: Semi-separated tree SB: Superblock IBC (or IntraBC): Intrablock Copy CDF: Cumulative Density Function SCC: Screen Content Coding GBI: Generalized Bidirectional Prediction BCW: Biprediction using CU level weights CIIP: Combined Intra-Interface Prediction POC: Picture Order Count RPS: Reference Picture Set DPB: Decoded Picture Buffer MMVD: Merge mode with motion vector difference [Explanation of Symbols]

[0199] 101 samples 102 Arrow 103 Arrow 104 square 201 Current Block 202 samples 203 samples 204 samples 205 samples 206 samples 300 Communication Systems 310 Terminal devices 320 terminal devices 330 terminal devices 340 terminal devices 350 Networks 400 Communication Systems 401 Video Source 402 Video Picture Stream 403 Video Encoder 404 Encoded video data 405 Streaming Server 406 Client Subsystem 407 Input copy of encoded video data 408 Client Subsystem 409 Input copy of encoded video data 410 Video Decoder 411 Output Stream 412 displays 413 Video Acquisition Subsystem 420 Electronic Devices 430 Electronic Devices 501 Channel 510 Video Decoder 512 displays 515 buffer memory 520 Parser 521 Symbols 530 Electronic Devices 531 Receiver 551 Scaler / Inverse Unit 552 Intrapicture Prediction Units 553 Motion Compensation Prediction Unit 555 Aggregator 556 Loop Filter Unit 557 Reference Picture Memory 558 Picture buffer 601 Video Sources 603 Video Encoder 620 Electronic Devices 630 Source Coder 632 Coding Engine 633 Local Decoder 634 Reference Picture Memory 635 Predictors 640 Transmitter 643 coded video sequences 645 Entropy Coder 650 Controller 660 communication channels 703 Video Encoder 721 General-purpose controller 722 Intra Encoders 723 Residual Calculator 724 Residual Encoder 725 Entropy Encoder 726 switches 728 Residual Decoder 730 Interencoder 810 Video Decoder 871 Entropy Decoder 872 Intra Decoder 873 Residual Decoder 874 Reconfiguration Module 880 Interdecoder 902 Base Block 904 Base Block 906 Base Block 908 Base Block 1002 subpartition 1004 subpartition 1006 subpartition 1008 subpartitions 1010 Square Partition 1102 Split Type 1104 Split Type 1106 Split Type 1108 Split Type 1200 base block 1202 Square Partition 1204 Square Partition 1206 Square Partition 1208 Square Partition 1302 Three-Pattern Pattern 1304 Three-Pattern Pattern 1410 Base Block 1402 partitions 1404 partitions 1406 partitions 1408 partitions 1420 Tree structure 1502 Square Coding Blocks 1504 Division of the First Level 1506 Second level division 1602 Intercoded Block 1604 Conversion Block 1800 methods 2000 Computer System 2001 Keyboard 2002 Mouse 2003 Trackpad 2005 Joystick 2006 Microphone 2007 Scanner 2008 Camera 2009 Speakers 2010 Touchscreen 2020 CD / DVD ROM / RW 2021 Media 2022 Sam Drive 2023 Removable hard drive or solid state drive 2040 cores 2041 One or more central processing units (CPUs) 2042 Graphics Processing Unit (GPU) 2043 Field-Programmable Gate Area (FPGA) 2044 Hardware Accelerator 2045 Read-only memory (ROM) 2046 Random Access Memory (RAM) 2047 Internal large-capacity storage 2048 System Bus 2049 Specific general-purpose data port or peripheral bus 2050 Graphics Adapter 2054 Interface 2055 One or more communication networks

Claims

1. A method for decoding a video block in a video stream, wherein the method is The steps include receiving the aforementioned video stream, The steps include determining the interprediction mode of the video block from the video stream, The steps include determining the motion vector prediction mode of the video block, The steps include: deriving a context from the video stream for signaling a set of syntax elements associated with a motion vector difference (MVD) associated with the video block, based on at least one of the interpretation mode and the motion vector prediction mode; A step of decoding the video block based on the set of syntax elements and the derived context. Methods that include...

2. The method according to claim 1, wherein the video block includes one of a coded block, a prediction block, or a coding unit.

3. The set of syntax elements associated with the aforementioned MVD is: The mv_joint syntax element represents the non-zero components of the aforementioned MVD, The mv_sign syntax element indicates whether the aforementioned MVD is positive, The MVD class of the aforementioned MVD is represented by the mv_class syntax element, The mv_bit syntax element represents the corresponding integer part of the offset between the MVD and the starting size of the MVD class, The mv_fr syntax element represents the first two fractional bits of the aforementioned MVD, The mv_hp syntax element represents the third fractional bit of the aforementioned MVD and The method according to claim 1, comprising at least one of the following.

4. The method according to claim 1, wherein the interpretation mode indicates whether one motion vector difference or multiple motion vector differences are signaled to a single interpretation reference frame list or to both of two interpretation reference frame lists.

5. The step of deriving the context for signaling the set of syntax elements associated with the MVD associated with the video block is: The steps include: deriving the context as a first predefined context in response to the signaling of the MVD to the single interpredictive reference framelist; The step of deriving the context as a second predefined context in response to the MVD being signaled to both of the two interpredictive reference framelists, The first predefined context and the second predefined context are different, The method according to claim 4.

6. The method according to claim 1, wherein the inter-prediction mode includes one of a single reference inter-prediction mode or a composite reference inter-prediction mode.

7. The method according to claim 6, wherein the step of deriving the context includes the step of deriving the context for signaling the set of syntax elements associated with the MVD associated with the video block, based at least on the interprediction mode.

8. The step of deriving the context for signaling the set of syntax elements associated with the MVD associated with the video block is: In response to the inter prediction mode being the single reference inter prediction mode, the step of deriving the context as a first predefined context, The step of deriving the context as a second predefined context in response to the interpretation mode being the composite reference interpretation mode, The first predefined context and the second predefined context are different, The method according to claim 7.

9. In the case of the single reference interpretation mode, the motion vector prediction mode includes one of the following: direct merge motion vector prediction mode (NEARMV), merge mode motion vector difference prediction mode (NEWMV), and global motion vector prediction mode (GLOBALMV), associated with a single interpretation reference frame of the video block. In the case of the composite reference interpretation mode having two reference interpretation prediction frames, the motion vector prediction mode is: The direct merge motion vector prediction mode for the first reference interpretation frame among the two reference interpretation frames and the merge mode motion vector difference prediction mode (NEAR_NEWMV) for the second reference interpretation frame among the two reference interpretation frames, The direct merge motion vector prediction mode for the second reference interpretation frame of the two reference interpretation prediction frames and the merge mode motion vector difference prediction mode (NEW_NEARMV) for the first reference interpretation prediction frame of the two reference interpretation prediction frames, The merge mode motion vector difference prediction mode (NEW_NEWMV) for both the first and second reference inter prediction frames of the two reference inter prediction frames, The global motion vector prediction mode (GLOBAL_GLOBALMV) for both the first and second reference interpretation frames of the two reference interpretation prediction frames and Including one of the following, The method according to claim 6.

10. The step of deriving the context for signaling the set of syntax elements associated with the MVD associated with the video block is: The steps include: deriving the context as a first predefined context in response to the motion vector prediction mode being NEWMV or NEW_NEWMV; The step of deriving the context as a second predefined context in response to the fact that the motion vector prediction mode is neither NEWMV nor NEW_NEWMV, The first predefined context and the second predefined context are different, The method according to claim 9.

11. The method according to claim 10, wherein the first predefined context and the second predefined context are configured to signal one of the mv_joint syntax for indicating the non-zero components of the MVD.

12. The method according to claim 9, wherein the interpretation mode includes the composite reference interpretation mode, and the step of deriving the context includes the step of deriving the context for signaling the set of syntax elements associated with the MVD associated with the video block, at least based on whether the motion vector prediction mode is one of NEW_NEARMV or NEAR_NEWMV.

13. The step of deriving the context for signaling the set of syntax elements associated with the MVD associated with the video block is: The steps include: deriving the context as a first predefined context in response to the motion vector prediction mode being NEW_NEARMV or NEAR_NEWMV; The step of deriving the context as a second predefined context in response to the fact that the motion vector prediction mode is neither NEW_NEARMV nor NEAR_NEWMV, The first predefined context and the second predefined context are different, The method according to claim 12.

14. The method according to claim 13, wherein the first predefined context and the second predefined context are configured to signal the mv_joint syntax for indicating non-zero MVD components.

15. The method according to claim 13, wherein the first predefined context and the second predefined context are configured to signal the mv_class syntax indicating the MVD class of the MVD.

16. A device for decoding video blocks in a video stream, the device comprising a memory for storing computer instructions and a processor for communicating with the memory, wherein when the processor executes the computer instruction, the processor sends to the device, Receive the video stream, The interprediction mode of the video block is determined from the video stream. Determine the motion vector prediction mode of the aforementioned video block. From the video stream, a context is derived for signaling a set of syntax elements associated with the motion vector difference (MVD) associated with the video block, based on at least one of the interpretation mode and the motion vector prediction mode. The video block is decoded based on the set of syntax elements and the derived context. A device configured in such a way.

17. A non-temporary storage medium for storing computer-readable instructions, wherein, when the computer-readable instructions are executed by a processor, the processor receives the instructions. Receive the coded video stream, The interprediction mode of the video block is determined from the coded video stream. Determine the motion vector prediction mode of the aforementioned video block. From the video stream, a context is derived for signaling a set of syntax elements associated with the motion vector difference (MVD) associated with the video block, based on at least one of the interpretation mode and the motion vector prediction mode. The video block is decoded based on the set of syntax elements and the derived context. Non-transitory storage medium.

18. A device comprising a circuit configured to carry out the method described in any one of claims 2 to 15.

19. A computer program product comprising a non-temporary computer-readable program medium storing computer code, wherein when the computer code is executed by one or more processors, the computer program product causes one or more processors to carry out the method described in any one of claims 2 to 15.

Images