Method and apparatus for adaptive reordering of reference frames

Abstract

The present disclosure relates to adaptively reordering reference frames for video coding techniques. Template matching (TM) may be used to reorder reference frames or pairs of reference frames for each block by comparing the difference between the template of the current block and the template of the reference block by referencing motion information of spatial reference motion information (or spatial motion vectors) and / or temporal reference motion information (or temporal motion vectors). The difference between the template of the current block and the template of the reference block may be calculated for each spatial reference motion information and / or temporal reference motion information and may be marked as a score value for the associated reference frames or pairs of reference frames. Available reference frames or pairs of reference frames are ranked based on the score value.

Description

Technology Field

[0001] Integration by citation

[0002] The present application is based on and claims the benefit of priority to U.S. Regular Application No. 17 / 932,333, filed September 15, 2022, titled "METHOD AND APPARATUS FOR ADAPTIVE REORDERING FOR REFERENCE FRAMES," which claims priority to U.S. Provisional Application No. 63 / 247,088, filed September 22, 2021, titled "METHOD AND APPARATUS FOR ADAPTIVE REORDERING FOR REFERENCE FRAMES," each of which is incorporated herein by reference in its entirety.

[0003] Technology field

[0004] The present disclosure describes a series of advanced video coding techniques. More specifically, the disclosed technique involves adaptively reordering reference frames. Background Technology

[0005] The background description provided in this specification is intended to provide a general context for the present disclosure. The research of the currently registered inventors described in this background section and the modes of description that may not otherwise be considered prior art at the time of filing this application are not recognized as prior art to the present disclosure, either explicitly or implicitly.

[0006] Video coding and decoding can be performed using inter-picture prediction accompanied by motion compensation. Uncompressed digital video may contain a series of pictures, each having spatial dimensions of, for example, 1920x1080 luminance samples and associated whole or subsampled chrominance samples. The series of pictures may have a fixed or variable picture rate (alternatively referred to as the frame rate), for example, 60 pictures per second or 60 frames per second. Uncompressed video has specific bitrate requirements for streaming or data processing. For example, video with a pixel resolution of 1920 x 1080, a frame rate of 60 frames / second, and 4:2:0 chroma subsampling at 8 bits per pixel per color channel requires bandwidth approaching 1.5 Gbit / s. One hour of such video requires more than 600 GBytes of storage space.

[0007] One objective of video coding and decoding may be to reduce redundancy in the uncompressed input video signal through compression. Compression can help reduce the previously described bandwidth and / or storage requirements by more than two orders of magnitude in some cases. Both lossless and lossy compression, as well as combinations thereof, can be utilized. Lossless compression refers to techniques where an exact copy of the original signal can be reconstructed from the compressed original signal through the decoding process. Lossy compression refers to coding / decoding processes where the original video information is not fully preserved during coding and is not fully recoverable during decoding. When using lossy compression, the reconstructed signal may not be identical to the original signal, but the distortion between the original and reconstructed signals is made small enough to render the reconstructed signal useful for the intended application, despite some loss of information. For video, lossy compression is widely used in many applications. The amount of acceptable distortion depends on the application. For example, users of certain consumer video streaming applications may tolerate higher distortion than users of cinematic or television broadcast applications. The compression ratios achievable by specific coding algorithms can be selected or adjusted to reflect varying distortion tolerances: higher acceptable distortion generally allows for coding algorithms that yield higher losses and higher compression ratios.

[0008] Video encoders and decoders can utilize techniques from several broad categories and steps, including, for example, motion compensation, Fourier transform, quantization, and entropy coding.

[0009] Video codec technologies may include techniques known as intra coding. In intra coding, sample values ​​are represented without referencing samples from previously reconstructed reference pictures or other data. In some video codecs, a picture is spatially subdivided into blocks of samples. When all blocks of samples are coded in intra mode, the picture may be referred to as an intra picture. Intra pictures and their derivatives, such as independent decoder refresh pictures, can be used to reset the decoder state and thus can be used as the first picture or a still image in a coded video bitstream and video session. Samples in the block following intra prediction can then be transformed into the frequency domain, and the resulting transform coefficients can be quantized before entropy coding. Intra prediction refers to a technique that minimizes sample values ​​in the pre-transform domain. In some cases, the smaller the DC value after conversion and the smaller the AC coefficients, the fewer bits are required at a given quantization step size to represent the block after entropy coding.

[0010] For example, traditional intra-coding, such as that known from MPEG-2 generation coding techniques, does not use intra-prediction. However, some newer video compression techniques include methods that attempt to code / decode blocks based on surrounding sample data and / or metadata that are acquired, for example, during the encoding and / or decoding of spatial neighbors and that precede the data blocks being intra-coded or decoded in the decoding order. These methods are hereinafter referred to as "intra-prediction" techniques. Note that, in at least some cases, intra-prediction uses only reference data from the current picture being reconstructed, rather than from other reference pictures.

[0011] There may be many different forms of intra-prediction. When more than one of these techniques is available in a given video coding technique, the technique in use may be referred to as an intra-prediction mode. One or more intra-prediction modes may be provided in a specific codec. In certain cases, modes may have submodes and / or may be associated with various parameters, and mode / submode information and intra-coding parameters for blocks of video may be coded individually or collectively included in mode codewords. The codewords used for a given combination of mode, submode, and / or parameters can influence the coding efficiency gain through intra-prediction, and the entropy coding technique used to convert the codewords into a bitstream can do so as well.

[0012] Specific modes of intra prediction were introduced with H.264, improved in H.265, and further enhanced in newer coding techniques such as JEM (joint exploration model), VVC (versatile video coding), and BMS (benchmark set). Generally, for intra prediction, a predictor block can be formed using available neighbor sample values. For example, available values ​​from a specific set of neighbor samples following a particular direction and / or lines can be copied into the predictor block. References to the direction in use can be encoded in the bitstream or predicted themselves.

[0013] Referring to FIG. 1a, a subset of 9 predictor directions specified from the 33 possible intra predictor directions of H.265 (corresponding to 33 angle modes out of 35 intra modes specified in H.265) is depicted in the lower right. The point (101) where the arrows converge indicates the sample being predicted. The arrows indicate the direction in which neighboring samples are used to predict the sample from 101. For example, arrow (102) indicates that sample (101) is predicted from neighboring samples or samples to the upper right at a 45-degree angle from the horizontal direction. Similarly, arrow (103) indicates that sample (101) is predicted from neighboring samples or samples to the lower left of sample (101) at a 22.5-degree angle from the horizontal direction.

[0014] Referring again to FIG. 1a, a square block (104) of 4x4 samples (indicated by a bold dashed line) is depicted in the upper left. The square block (104) contains 16 samples, each labeled "S", a position in the Y dimension (e.g., row index), and a position in the X dimension (e.g., column index). For example, sample S21 is the second sample in the Y dimension (from the top) and the first sample in the X dimension (from the left). Similarly, sample S44 is the fourth sample in both the Y and X dimensions in the block (104). Since the block is 4x4 samples in size, S44 is located in the lower right. Exemplary reference samples following a similar numbering scheme are additionally illustrated. For the block (104), the reference sample is labeled R, its Y position (e.g., row index) and X position (column index). In both H.264 and H.265, neighboring prediction samples adjacent to the block being reconstructed are used.

[0015] Intra-picture prediction of block (104) can be initiated by copying reference sample values ​​from neighboring samples according to the signaled prediction direction. For example, assume that the coded video bitstream includes signaling indicating the prediction direction of the arrow (102) for this block (104)—that is, samples are predicted from the prediction sample or samples at an angle of 45 degrees from the horizontal direction to the upper right. In this case, samples S41, S32, S23, and S14 are predicted from the same reference sample R05. Subsequently, sample S44 is predicted from reference sample R08.

[0016] In certain cases, the values ​​of multiple reference samples can be combined, for example, through interpolation to calculate a reference sample, particularly when the directions are not evenly divided by 45 degrees.

[0017] As video coding technology continues to advance, the number of possible directions has increased. In H.264 (2003), for example, nine different directions are available for intra-prediction. This increased to 33 in H.265 (2013), and JEM / VVC / BMS can support up to 65 directions at the time of this publication. Experimental studies have been conducted to help identify the most suitable intra-prediction directions, and specific bit penalties for directions can be tolerated by encoding these most suitable directions with a small number of bits using specific techniques in entropy coding. Additionally, the directions themselves can sometimes be predicted from the neighbor directions used in the intra-prediction of decoded neighbor blocks.

[0018] FIG. 1b illustrates a schematic diagram (180) depicting 65 intra-predicted directions according to JEM to illustrate an increasing number of predicted directions in various encoding technologies developed over time.

[0019] The method of mapping bits representing intra-prediction directions to prediction directions in a coded video bitstream can vary depending on the video coding technique; for example, it can range from simple direct mappings of prediction directions to intra-prediction modes to complex adaptive methods involving codewords, most probable modes, and similar techniques. However, in all cases, there may be specific directions for intra-prediction that are statistically less likely to occur in the video content than certain other directions. Since the goal of video compression is to reduce redundancy, in a well-designed video coding technique, these less likely directions can be represented by a larger number of bits than the more likely directions.

[0020] Inter-picture prediction, or inter-prediction, may be based on motion compensation. In motion compensation, sample data from a previously reconstructed picture or part thereof (reference picture) may be used to predict a newly reconstructed picture or part of a picture (e.g., a block) after being spatially shifted in the direction indicated by a motion vector (hereinafter MV). In some cases, the reference picture may be the same as the picture currently being reconstructed. MVs may have two dimensions X and Y, or three dimensions, and the third dimension is an indication of the reference picture in use (similar to a time dimension).

[0021] In some video compression techniques, the current MV applicable to a specific region of sample data can be predicted from other MVs, for example, from other regions of sample data spatially adjacent to the region being reconstructed, and from those other MVs that precede the current MV in the decoding order. By doing so, the total amount of data required to code the MVs can be substantially reduced by relying on the elimination of redundancy in correlated MVs, thereby increasing compression efficiency. MV prediction can work effectively, for example when coding an input video signal derived from a camera (known as natural video), because there is a statistical probability that regions larger than the area where a single MV is applicable move in similar directions within the video sequence; thus, in some cases, prediction can be made using similar motion vectors derived from the MVs of neighboring regions. As a result, the actual MV for a given region becomes similar or identical to the MV predicted from the surrounding MVs. These MVs can eventually be represented with fewer bits than when the MV is coded directly rather than predicted from neighboring MV(s) after entropy coding. In some cases, MV prediction can be an example of lossless compression of signals (i.e., MVs) derived from the original signal (i.e., sample stream). In other cases, MV prediction itself can be lossy, for example, due to rounding errors when calculating the predictor from several surrounding MVs.

[0022] Various MV prediction mechanisms are described in H.265 / HEVC (ITU-T Rec. H.265, "High Efficiency Video Coding", December 2016). Among the many MV prediction mechanisms specified by H.265, a technique referred to as "spatial merge" is described below.

[0023] Specifically, referring to FIG. 2, the current block (201) contains samples discovered by the encoder during the motion search process that are predictable from a previous block of the same size that is spatially shifted. Instead of directly coding the MV, the MV may be derived from metadata associated with one or more reference pictures, for example, from the most recent (in decoding order) reference picture, using an MV associated with any one of five surrounding samples represented as A0, A1, and B0, B1, B2 (202 to 206, respectively). In H.265, the MV prediction may use predictors from the same reference picture used by neighboring blocks.

[0024] AV1 (AOMedia Video 1) is an open video coding format designed for video transmission over the Internet. It was developed as a successor to VP9 by incorporating additional techniques based on the VP9 codebase. The AV1 bitstream specification includes reference video codecs such as H.265, the HEVC (High Efficiency Video Coding) standard, or VVC (Versatile Video Coding).

[0025] Embodiments of the present disclosure provide methods and apparatuses for adaptively reordering reference frames for video coding techniques. Template matching (TM) may be used to reorder reference frames or pairs of reference frames for each block by comparing the difference between the template of the current block and the template of the reference block by referencing motion information of spatial reference motion information (or spatial motion vectors) and / or temporal reference motion information (or temporal motion vectors). The difference between the template of the current block and the template of the reference block may be calculated for each spatial reference motion information and / or temporal reference motion information and may be marked as a score value for the associated reference frames or pairs of reference frames. Available reference frames or pairs of reference frames are ranked based on the score value.

[0026] In one embodiment, a method for reordering reference frames for each block by template matching (TM) comprises: comparing a template of the current block with a template of the reference block for motion information—the motion information includes spatial reference motion information or temporal reference motion information—; calculating the difference between the template of the current block and the template of the reference block; determining a score value for the associated reference frame based on the calculated difference; and reordering the reference frames based on the determined score. The reference frames further include reference frame pairs. The TM includes decoder-side motion vector derivation for refining the motion information of the current block. The reordering step further comprises ranking available reference frames based on the score value for each of the reference frames. When the score values ​​of a plurality of reference frames are equal, the ranking corresponds to the scan order of the spatial reference motion information or temporal reference motion information. When the score values ​​of multiple reference frames are equal, ranking is based on the frequency of occurrence of these reference frames used in spatial reference motion information or temporal reference motion information. The calculation includes at least one of SAD (sum of absolute differences), SSD (sum of squared differences), MSE (mean squared error), or SATD (sum of absolute transform differences). The template includes the top or left neighbor block. Spatial reference motion information includes one or more spatial motion vectors. Temporal reference motion information includes one or more temporal motion vectors.When multiple motion vectors point to one of the reference frames, the single motion vector with the smallest difference among the motion vectors is used to determine the score value. When no motion vector points to one of the reference frames, the score value is determined to be the maximum allowed value. Allowed unidirectional and bidirectional composite reference frames are ranked together using TM, and the index of the reference frames for the current block is signaled in the bitstream. Allowed single reference frames are ranked together using TM, and the index of the reference frame for the current block is signaled in the bitstream.

[0027] In some other embodiments, a device for processing video information is disclosed. The device may include a circuit configured to perform any one of the above method implementations.

[0028] Embodiments of the present disclosure also provide non-transient computer-readable media storing instructions that, when executed by a computer for video decoding and / or encoding, cause the computer to perform methods for video decoding and / or encoding. Brief explanation of the drawing

[0029] Additional features, nature, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings. Figure 1a illustrates a schematic example of an exemplary subset of intra-predicted directional modes. Figure 1b illustrates examples of exemplary intra-predicted directions. Figure 2 illustrates a schematic example of the current block and spatial merge candidates around it for motion vector prediction in one example. FIG. 3 illustrates a schematic example of a simplified block diagram of a communication system (300) according to an exemplary embodiment. FIG. 4 illustrates a schematic example of a simplified block diagram of a communication system (400) according to an exemplary embodiment. FIG. 5 illustrates a schematic example of a simplified block diagram of a video decoder according to an exemplary embodiment. FIG. 6 illustrates a schematic example of a simplified block diagram of a video encoder according to an exemplary embodiment. FIG. 7 illustrates a block diagram of a video encoder according to another exemplary embodiment. FIG. 8 illustrates a block diagram of a video decoder according to another exemplary embodiment. FIG. 9 illustrates a method of coding block partitioning according to exemplary embodiments of the present disclosure. FIG. 10 illustrates another method of coding block partitioning according to exemplary embodiments of the present disclosure. FIG. 11 illustrates another method of coding block partitioning according to exemplary embodiments of the present disclosure. FIG. 12 illustrates an example of partitioning a base block into coding blocks according to an exemplary partitioning method. Figure 13 illustrates an exemplary ternary partitioning method. FIG. 14 illustrates an exemplary quadtree binary tree coding block partitioning method. FIG. 15 illustrates a method for partitioning a coding block into a plurality of conversion blocks and a coding order of conversion blocks according to exemplary embodiments of the present disclosure. FIG. 16 illustrates another method for partitioning a coding block into a plurality of conversion blocks and the coding order of the conversion blocks according to exemplary embodiments of the present disclosure. FIG. 17 illustrates another method for partitioning a coding block into a plurality of conversion blocks according to exemplary embodiments of the present disclosure. FIG. 18 illustrates an exemplary partition tree for block partitioning. Figure 19 illustrates an exemplary partition and tree for a QTBT (quad-tree plus binary tree) structure. FIG. 20 illustrates exemplary triple-tree partitioning. FIG. 21 illustrates an exemplary template matching (TM) on a search area around an initial motion vector (MV). FIG. 22 illustrates an example of decoder-side motion vector refinement. FIG. 23 illustrates an exemplary spatial motion vector search pattern. FIG. 24 illustrates a flowchart of a method according to an exemplary embodiment of the present disclosure. FIG. 25 illustrates a schematic diagram of a computer system according to exemplary embodiments of the present disclosure. Specific details for implementing the invention

[0030] Throughout the specification and claims, terms may have nuanced meanings implied or suggested by the context beyond their explicitly stated meanings. Phrases such as "in one embodiment" or "in some embodiments" as used herein do not necessarily refer to the same embodiment, nor do phrases such as "in another embodiment" or "in other embodiments" as used herein refer to different embodiments. Likewise, phrases such as "in one implementation" or "in some implementations" as used herein do not necessarily refer to the same implementation, nor do phrases such as "in another implementation" or "in other implementations" as used herein refer to different implementations. For example, the claimed subject matter is intended to include combinations of exemplary embodiments / implements, wholly or in part.

[0031] Generally, terms may be understood at least partially from their use in context. For example, terms such as “and,” “or,” or “and / or” as used herein may include various meanings that may depend at least partially on the context in which these terms are used. Typically, when used to associate a list such as A, B, or C, “or” is intended to mean A, B, and C as used herein in an inclusive sense, as well as A, B, or C as used herein in an exclusive sense. Additionally, terms such as “one or more” or “at least one” as used herein may, depending at least partially on the context, be used to describe any feature, structure, or characteristic in a singular sense, or to describe combinations of features, structures, or characteristics in a plural sense. Similarly, terms such as “one,” “one,” or “that” may again, depending at least partially on the context, be understood to convey a singular usage or a plural usage. Additionally, the terms "based on" or "determined by" may be understood as not necessarily intended to convey an exclusive set of arguments, but instead, again, depending at least partially on the context, may allow for the existence of additional arguments that are not necessarily explicitly described.

[0032] FIG. 3 illustrates a simplified block diagram of a communication system (300) according to an embodiment of the present disclosure. The communication system (300) includes a plurality of terminal devices capable of communicating with each other, for example, through a network (350). For example, the communication system (300) includes a first pair of terminal devices (310 and 320) interconnected through the network (350). In the example of FIG. 3, the first pair of terminal devices (310 and 320) can perform unidirectional transmission of data. For example, the terminal device (310) can encode video data (for example, a stream of video pictures captured by the terminal device (310)) for transmission to another terminal device (320) through the network (350). The encoded video data may be transmitted in the form of one or more coded video bitstreams. The terminal device (320) receives coded video data from the network (350), decodes the coded video data to recover video pictures, and can display the video pictures according to the recovered video data. Unidirectional data transmission can be implemented in media serving applications, etc.

[0033] In another example, the communication system (300) includes a second pair of terminal devices (330 and 340) that perform bidirectional transmission of coded video data, which may be implemented, for example, during a video conferencing application. For bidirectional transmission of data, in one example, each terminal device among the terminal devices (330 and 340) may code video data (e.g., a stream of video pictures captured by the terminal device) for transmission to another terminal device among the terminal devices (330 and 340) via a network (350). Each terminal device among the terminal devices (330 and 340) may also receive coded video data transmitted by the other terminal device among the terminal devices (330 and 340), decode the coded video data to recover video pictures, and display the video pictures on an accessible display device according to the recovered video data.

[0034] In the example of FIG. 3, the terminal devices (310, 320, 330, and 340) may be implemented as servers, personal computers, and smartphones, but the applicability of the basic principles of the present disclosure may not be so limited. Embodiments of the present disclosure may be implemented in desktop computers, laptop computers, tablet computers, media players, wearable computers, dedicated video conferencing equipment, and / or similar devices. Network (350) represents any number or type of networks that transmit coded video data between the terminal devices (310, 320, 330, and 340), including, for example, wired and / or wireless communication networks. The communication network (350) may exchange data on circuit-switched, packet-switched, and / or other types of channels. Representative networks include communication networks, local area networks, wide area networks, and / or the Internet. For the purposes of this discussion, the architecture and topology of the network (350) may not be important to the operation of the present disclosure unless explicitly described in this specification.

[0035] FIG. 4 illustrates the arrangement of a video encoder and a video decoder in a video streaming environment as an example of an application for the disclosed subject. The disclosed subject may be equally applicable to other video applications, such as, for example, video conferencing, digital TV broadcasting, gaming, virtual reality, and storage of compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0036] A video streaming system may include a video capture subsystem (413) which may include a video source (401), e.g., a digital camera, for generating a stream (402) of uncompressed video pictures or images. In one example, the stream (402) of video pictures includes samples recorded by the digital camera of the video source (401). The stream (402) of video pictures, depicted in bold lines to emphasize the high data capacity compared to encoded video data (404) (or coded video bitstream), may be processed by an electronic device (420) comprising a video encoder (403) coupled to the video source (401). The video encoder (403) may include hardware, software, or a combination thereof to enable or implement the embodiments of the disclosed subject matter as described in more detail below. Encoded video data (404) (or encoded video bitstream (404)), depicted as a thin line to emphasize the lower data capacity compared to a stream (402) of uncompressed video pictures, may be stored on a streaming server (405) or directly on downstream video devices (not shown) for future use. One or more streaming client subsystems, such as the client subsystems (406 and 408) in FIG. 4, may access the streaming server (405) to retrieve copies (407 and 409) of the encoded video data (404). The client subsystem (406) may include a video decoder (410) within an electronic device (430), for example. A video decoder (410) decodes an incoming copy (407) of encoded video data and generates an outgoing stream (411) of video pictures that are uncompressed and can be rendered on a display (412) (e.g., a display screen) or other rendering devices (not described).The video decoder (410) may be configured to perform some or all of the various functions described in this disclosure. In some streaming systems, encoded video data (404, 407, and 409) (e.g., video bitstreams) may be encoded according to specific video coding / compression standards. Examples of such standards include ITU-T Recommendation H.265. In one example, a video coding standard under development is informally known as VVC (Versatile Video Coding). The disclosed subject matter may be used in the context of VVC and other video coding standards.

[0037] Note that the electronic devices (420 and 430) may include other components (not shown). For example, the electronic device (420) may also include a video decoder (not shown) and the electronic device (430) may also include a video encoder (not shown).

[0038] FIG. 5 illustrates a block diagram of a video decoder (510) according to any embodiment of the present disclosure below. The video decoder (510) may be included in an electronic device (530). The electronic device (530) may include a receiver (531) (e.g., a receiving circuit). The video decoder (510) may be used instead of the video decoder (410) in the example of FIG. 4.

[0039] The receiver (531) may receive one or more coded video sequences to be decoded by the video decoder (510). In the same or different embodiments, one coded video sequence may be decoded at a time, wherein the decoding of each coded video sequence is independent of other coded video sequences. Each video sequence may be associated with multiple video frames or images. The coded video sequence may be received from a channel (501), which may be a hardware / software link to a storage device storing the encoded video data or to a streaming source transmitting the encoded video data. The receiver (531) may receive the encoded video data along with other data, such as encoded audio data and / or auxiliary data streams, which may be forwarded to their respective processing circuits (not described). The receiver (531) may separate the coded video sequence from the other data. To prevent network jitter, a buffer memory (515) may be placed between the receiver (531) and the entropy decoder / parser (520) (hereinafter referred to as "parser (520)"). In certain applications, the buffer memory (515) may be implemented as part of the video decoder (510). In other applications, it may be outside the video decoder (510) (not described) and separated from it. In other applications, for example, for the purpose of preventing network jitter, a buffer memory (not described) outside the video decoder (510) may exist, and for example, to handle playback timing, another additional buffer memory (515) inside the video decoder (510) may exist. When the receiver (531) is receiving data from a storage / forward device of sufficient bandwidth and controllability, or from an isosynchronous network, the buffer memory (515) may not be needed or may be small.For use on best-effort packet networks such as the Internet, a buffer memory (515) of sufficient size may be required, and the size may be relatively large. This buffer memory may be implemented with an adaptive size and may be implemented at least partially in an operating system or similar elements (not described) outside the video decoder (510).

[0040] The video decoder (510) may include a parser (520) to reconstruct symbols (521) from a coded video sequence. The categories of the symbols include information used to manage the operation of the video decoder (510), and information for controlling a rendering device, such as a display (512) (e.g., a display screen), which may or may not be an integral part of the electronic device (530) as illustrated in FIG. 5, but may be coupled to the electronic device (530). The control information for the rendering device(s) may be in the form of Supplemental Enhancement Information (SEI) messages or Video Usability Information (VUI) parameter set fragments (not depicted). The parser (520) may parse / entropy decode the coded video sequence received by the parser (520). Entropy coding of a coded video sequence may follow video coding techniques or standards and may follow various principles including variable-length coding, Huffman coding, and arithmetic coding with or without context sensitivity. The parser (520) may extract a set of subgroup parameters for at least one of a subgroup of pixels in a video decoder based on at least one parameter corresponding to the subgroups from the coded video sequence. The subgroups may include Group of Pictures (GOP), pictures, tiles, slices, macroblocks, Coding Units (CU), blocks, Transform Units (TU), Prediction Units (PU), etc.The parser (520) can also extract information such as transform coefficients (e.g., Fourier transform coefficients), quantizer parameter values, motion vectors, etc. from the coded video sequence.

[0041] The parser (520) can generate symbols (521) by performing an entropy decoding / parsing operation on a video sequence received from a buffer memory (515).

[0042] The reconstruction of the symbols (521) may involve a number of different processing or functional units depending on the type of the coded video picture or its parts (e.g., inter- and intra-picture, inter- and intra-block), and other factors. The units involved and how they are involved may be controlled by subgroup control information parsed from the video sequence coded by the parser (520). The flow of this subgroup control information between the parser (520) and the number of processing or functional units below is not depicted for the sake of simplification.

[0043] In addition to the functional blocks already mentioned, the video decoder (510) may be conceptually subdivided into a number of functional units as described below. In actual implementations operating under commercial constraints, many of these functional units may interact closely with one another and be at least partially integrated with one another. However, to clearly explain the various functions of the disclosed subject matter, the conceptual subdivision into functional units is adopted in the following disclosure.

[0044] The first unit may include a scaler / inverse transform unit (551). The scaler / inverse transform unit (551) may receive quantized transform coefficients as well as control information, including information indicating which type of inverse transform to use, block size, quantization factor / parameters, quantization scaling matrices, etc., as symbol(s) (521) from the parser (520). The scaler / inverse transform unit (551) may output blocks containing sample values ​​that can be input to an aggregator (555).

[0045] In some cases, the output samples of the scaler / inverse transform (551) may relate to an intra-coded block, that is, a block that does not use prediction information from previously reconstructed pictures but can use prediction information from previously reconstructed parts of the current picture. Such prediction information may be provided by an intra-picture prediction unit (552). In some cases, the intra-picture prediction unit (552) may generate a block of the same size and shape as the block being reconstructed by using surrounding block information that has already been reconstructed and stored in the current picture buffer (558). The current picture buffer (558) buffers, for example, a partially reconstructed current picture and / or a fully reconstructed current picture. In some implementations, the aggregator (555) may add the prediction information generated by the intra-prediction unit (552) to the output sample information provided by the scaler / inverse transform unit (551) on a per-sample basis.

[0046] In other cases, the output samples of the scaler / inverse conversion unit (551) may be intercoded and potentially associated with a motion-compensated block. In such cases, the motion compensation prediction unit (553) may access the reference picture memory (557) to fetch samples used for inter-picture prediction. After motion-compensating the fetched samples according to the symbols (521) associated with the block, these samples may be added by the aggregator (555) to the output of the scaler / inverse conversion unit (551) (the output of the unit (551) may be referred to as residual samples or residual signals) to generate output sample information. The addresses in the reference picture memory (557) where the motion compensation prediction unit (553) fetches prediction samples may be controlled by motion vectors available to the motion compensation prediction unit (553) in the form of symbols (521) that may have, for example, X, Y components (shift), and reference picture components (time). Motion compensation may also include interpolation of sample values ​​fetched from reference picture memory (557) when subsample accurate motion vectors are in use, and may also be associated with motion vector prediction mechanisms, etc.

[0047] Various loop filtering techniques within the loop filter unit (556) may be performed on the output samples of the aggregator (555). Video compression techniques may include in-loop filter techniques that are made available to the loop filter unit (556) as symbols (521) from the parser (520) and are controlled by parameters included in the coded video sequence (also referred to as the coded video bitstream), but which respond not only to meta-information acquired during the decoding of the previous (in decoding order) parts of the coded picture or coded video sequence, but also to previously reconstructed and loop-filtered sample values. Several types of loop filters may be included in various order as part of the loop filter unit (556), which will be described in more detail below.

[0048] The output of the loop filter unit (556) may be a sample stream that is not only output to the rendering device (512) but may also be stored in the reference picture memory (557) for use in future inter-picture predictions.

[0049] Certain coded pictures, when fully reconstructed, can be used as reference pictures for future inter-picture prediction. For example, when a coded picture corresponding to a current picture is fully reconstructed and the coded picture is identified as a reference picture (e.g., by the parser (520)), the current picture buffer (558) can become part of the reference picture memory (557), and a new current picture buffer can be reallocated before starting the reconstruction of the next coded picture.

[0050] A video decoder (510) can perform decoding operations according to a predetermined video compression technique adopted in a standard such as ITU-T Rec. H.265. In that the coded video sequence adheres to both the syntax of the video compression technique or standard and the profiles documented in the video compression technique or standard, the coded video sequence may comply with the syntax specified by the video compression technique or standard in use. Specifically, the profile may select specific tools from all tools available in the video compression technique or standard as the only tools available to use under that profile. To comply with the standard, the complexity of the coded video sequence may be within the boundaries defined by the levels of the video compression technique or standard. In some cases, the levels limit the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. The limits set by the levels may, in some cases, be further restricted through HRD (Hypothetical Reference Decoder) specifications and metadata for managing HRD buffers signaled in the coded video sequence.

[0051] In some exemplary embodiments, the receiver (531) may receive additional (redundant) data along with the encoded video. This additional data may be included as part of the encoded video sequence(s). This additional data may be used by the video decoder (510) to properly decode the data and / or more accurately reconstruct the original video data. The additional data may be in the form, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0052] FIG. 6 illustrates a block diagram of a video encoder (603) according to an exemplary embodiment of the present disclosure. The video encoder (603) may be included in an electronic device (620). The electronic device (620) may further include a transmitter (640) (e.g., a transmitting circuit). The video encoder (603) may be used instead of the video encoder (403) in the example of FIG. 4.

[0053] The video encoder (603) can receive video samples from a video source (601) (not part of the electronic device (620) in the example of FIG. 6) capable of capturing video image(s) to be encoded by the video encoder (603). In another example, the video source (601) may be implemented as part of the electronic device (620).

[0054] A video source (601) may provide a source video sequence to be coded by a video encoder (603) in the form of a digital video sample stream, which may have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 YCrCb, RGB, XYZ...), and any suitable sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media serving system, the video source (601) may be a storage device capable of storing previously prepared video. In a video conferencing system, the video source (601) may be a camera that captures local image information as a video sequence. The video data may be provided as a plurality of individual pictures or images that impart motion when viewed sequentially. The pictures themselves may be organized as a spatial array of pixels, wherein each pixel may contain one or more samples depending on the sampling structure, color space, etc. being used. A person skilled in the art can easily understand the relationship between pixels and samples. The following description focuses on samples.

[0055] According to some exemplary embodiments, the video encoder (603) can code and compress pictures of a source video sequence into a coded video sequence (643) in real time or under any other time constraints required by the application. Implementing an appropriate coding rate constitutes one function of the controller (650). In some embodiments, the controller (650) may be functionally coupled to other functional units to control them as described below. The coupling is not depicted for the sake of simplification. Parameters set by the controller (650) may include rate control related parameters (picture skip, quantizer, lambda values ​​of rate-distortion optimization techniques, ...), picture size, picture group (GOP) layout, maximum motion vector search range, etc. The controller (650) may be configured to have other suitable functions related to the video encoder (603) optimized for a specific system design.

[0056] In some exemplary embodiments, the video encoder (603) may be configured to operate in a coding loop. For the sake of oversimplification, in one example, the coding loop may include a source coder (630) (responsible for generating symbols, such as a symbol stream, based on, for example, the input picture to be coded and reference picture(s)), and a (local) decoder (633) embedded in the video encoder (603). The decoder (633) reconstructs the symbols to generate sample data in a manner similar to that generated by the (remote) decoder, even though the embedded decoder (633) processes the video stream coded by the source coder (630) without entropy coding (since any compression between the video bitstream and symbols coded in entropy coding may be lossless in the video compression techniques considered in the subject matter disclosed). The reconstructed sample stream (sample data) is input into the reference picture memory (634). Because the decoding of the symbol stream produces bit-exact results independently of the decoder location (local or remote), the content within the reference picture memory (634) is also bit exact between the local encoder and the remote encoder. In other words, the prediction part of the encoder "sees" the exact same sample values ​​as the reference picture samples that the decoder "sees" when using prediction during decoding. This fundamental principle of reference picture synchronicity (and the resulting drift when synchronicity cannot be maintained, for example, due to channel errors) is used to improve coding quality.

[0057] The operation of the “local” decoder (633) may be the same as the operation of the “remote” decoder, such as the video decoder (510) already described in detail above in relation to FIG. 5. However, with brief reference to FIG. 5, since symbols are available and the encoding / decoding of symbols into a coded video sequence by the entropy coder (645) and the parser (520) may be lossless, the entropy decoding parts of the video decoder (510), including the buffer memory (515) and the parser (520), may not be fully implemented in the local decoder (633) in the encoder.

[0058] An observation that can be made at this point is that any decoder technique, excluding parsing / entropy decoding which may exist only in the decoder, may also inevitably need to exist in the corresponding encoder in substantially the same functional form. For this reason, the disclosed subject matter may sometimes focus on decoder operations associated with the decoding part of the encoder. Since encoder techniques are the inverse of the comprehensively described decoder techniques, their description may therefore be condensed. A more detailed description of the encoder is provided below only in specific domains or modalities.

[0059] During operation, in some exemplary implementations, the source coder (630) may perform motion-compensated predictive coding, which predictively codes the input picture by referencing one or more previously coded pictures from a video sequence designated as a “reference picture.” In this way, the coding engine (632) codes the differences (or residuals) in color channels between the pixel blocks of the input picture and the pixel blocks of the reference picture(s) that can be selected as predictive reference(s) for the input picture. The term “residue” and its adjective form “residual” may be used interchangeably.

[0060] The local video decoder (633) can decode the coded video data of pictures that may be designated as reference pictures based on the symbols generated by the source coder (630). The operations of the coding engine (632) may advantageously be lossy processes. If the coded video data can be decoded in a video decoder (not shown in FIG. 6), the reconstructed video sequence may be a replica of the source video sequence, typically having some errors. The local video decoder (633) can replicate the decoding processes that can be performed by the video decoder on the reference pictures and allow the reconstructed reference pictures to be stored in the reference picture cache (634). In this way, the video encoder (603) can locally store copies of reconstructed reference pictures having common content as reconstructed reference pictures to be acquired by the original (remote) video decoder (without transmission errors).

[0061] The predictor (635) can perform prediction searches for the coding engine (632). That is, for a new picture to be coded, the predictor (635) can search the reference picture memory (634) for specific metadata or sample data (as candidate reference pixel blocks), such as reference picture motion vectors, block shapes, etc., which can serve as appropriate prediction references for the new pictures. The predictor (635) can operate on a sample block-by-pixel block basis to find appropriate prediction references. In some cases, as determined by the search results obtained by the predictor (635), the input picture may have prediction references drawn from a number of reference pictures stored in the reference picture memory (634).

[0062] The controller (650) can manage the coding operation of the source coder (630), including, for example, the settings of parameters and subgroup parameters used to encode video data.

[0063] The outputs of all the aforementioned function units may undergo entropy coding in an entropy coder (645). The entropy coder (645) converts the symbols generated by the various function units into a coded video sequence by lossless compression of the symbols according to techniques such as Huffman coding, variable-length coding, and arithmetic coding.

[0064] The transmitter (640) may buffer the coded video sequence(s) generated by the entropy coder (645) to prepare for transmission over a communication channel (660), which may be a hardware / software link to a storage device for storing the encoded video data. The transmitter (640) may merge the coded video data from the video coder (603) with other data to be transmitted, e.g., coded audio data and / or auxiliary data streams (sources not shown).

[0065] The controller (650) can manage the operation of the video encoder (603). During coding, the controller (650) can assign a specific coded picture type to each coded picture, which can influence the coding techniques that can be applied to each picture. For example, pictures can often be assigned as one of the following picture types:

[0066] An Intra Picture (I Picture) may be one that can be coded and decoded without using any other picture in the sequence as a source of prediction. Some video codecs allow different types of Intra Pictures, including, for example, Independent Decoder Refresh ("IDR") pictures. A person skilled in the art recognizes such variations of I Pictures and their respective applications and features.

[0067] The predictive picture (P picture) may be coded and decoded using intra prediction or inter prediction, using at most one motion vector and reference index to predict the sample values ​​of each block.

[0068] A bi-directionally predictive picture (Picture B) may be coded and decoded using intra-prediction or inter-prediction, using at most two motion vectors and reference indices to predict sample values ​​of each block. Similarly, multi-prediction pictures may use more than two reference pictures and associated metadata for the reconstruction of a single block.

[0069] Source pictures are generally spatially subdivided into multiple sample coding blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples, respectively) and can be coded on a block-by-block basis. Blocks can be predictively coded by referencing other (already coded) blocks determined by the coding assignment applied to each of the blocks' pictures. For example, blocks of pictures I can be non-predictively coded, or they can be predictively coded by referencing already coded blocks of the same picture (spatial prediction or intra prediction). Pixel blocks of picture P can be predictively coded through spatial prediction or temporal prediction by referencing one previously coded reference picture. Blocks of pictures B can be predictively coded through spatial prediction or temporal prediction by referencing one or two previously coded reference pictures. Source pictures or intermediately processed pictures may be subdivided into different types of blocks for different purposes. The division of coding blocks and other types of blocks may or may not follow the same method, as described in more detail below.

[0070] The video encoder (603) can perform coding operations according to a predetermined video coding technology or standard, such as ITU-T Rec. H.265. In the operation, the video encoder (603) can perform various compression operations, including predictive coding operations that utilize temporal and spatial redundancies in the input video sequence. Thus, the coded video data can comply with the syntax specified by the video coding technology or standard in use.

[0071] In some exemplary embodiments, the transmitter (640) may transmit additional data along with the encoded video. The source coder (630) may include such data as part of the encoded video sequence. The additional data may include other forms of redundant data such as temporal / spatial / SNR enhancement layers, redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

[0072] Video can be captured as multiple source pictures (video pictures) in a temporal sequence. Intra-picture prediction (often abbreviated as intra prediction) utilizes spatial correlations within a given picture, while inter-picture prediction utilizes temporal or other correlations between pictures. For example, a specific picture currently being encoded / decoded, referred to as the current picture, can be partitioned into blocks. A block within the current picture can be coded by a vector referred to as a motion vector when it is similar to a reference block within a previously coded and still buffered reference picture in the video. The motion vector points to the reference block within the reference picture and, if multiple reference pictures are in use, may have a third dimension identifying the reference picture.

[0073] In some exemplary embodiments, a bi-prediction technique may be used for inter-picture prediction. According to this bi-prediction technique, two reference pictures, such as a first reference picture and a second reference picture, are used, both of which advance the current picture in the decoding order in the video (but may be in the past or future, respectively, in the display order). A block in the current picture may be coded by a first motion vector pointing to a first reference block in the first reference picture, and a second motion vector pointing to a second reference block in the second reference picture. A block may be jointly predicted by a combination of the first reference block and the second reference block.

[0074] In addition, merge mode techniques can be used in inter-picture prediction to improve coding efficiency.

[0075] According to some exemplary embodiments of the present disclosure, predictions such as inter-picture predictions and intra-picture predictions are performed on a block basis. For example, a picture within a sequence of video pictures is partitioned into coding tree units (CTUs) for compression, and the CTUs within the picture may have the same size, such as 64x64 pixels, 32x32 pixels, or 16x16 pixels. Generally, a CTU may include three parallel coding tree blocks (CTBs): one luma CTB and two chroma CTBs. Each CTU may be recursively quadtree split into one or more coding units (CUs). For example, a CTU of 64x64 pixels may be split into one CU of 64x64 pixels or four CUs of 32x32 pixels. One or more of the 32x32 blocks may be further subdivided into four CUs of 16x16 pixels each. In some exemplary embodiments, each CU may be analyzed during encoding to determine a prediction type for the CU among various prediction types, such as inter-prediction type or intra-prediction type. The CU may be subdivided into one or more prediction units (PUs) based on temporal and / or spatial predictability. Generally, each PU includes a luminance prediction block (PB) and two chroma PBs. In one embodiment, the prediction operation during coding (encoding / decoding) is performed on a unit of prediction blocks. The subdivision of the CU into PUs (or PBs of different color channels) may be performed in various spatial patterns. The luminance or chroma PB may include a matrix of values ​​(e.g., luminance values) for samples, such as, for example, 8x8 pixels, 16x16 pixels, 8x16 pixels, 16x8 samples, etc.

[0076] FIG. 7 illustrates a video encoder (703) according to another exemplary embodiment of the present disclosure. The video encoder (703) is configured to receive a processing block (e.g., a prediction block) of sample values ​​within a current video picture in a sequence of video pictures and to encode the processing block into a coded picture that is part of a coded video sequence. The exemplary video encoder (703) may be used instead of the video encoder (403) in the example of FIG. 4.

[0077] For example, the video encoder (703) receives a matrix of sample values ​​for a processing block, such as a prediction block of 8x8 samples. Then, the video encoder (703) determines which of the intra mode, inter mode, or two prediction mode the processing block is best coded, for example, using rate-distortion optimization (RDO). When it is determined that the processing block is coded in intra mode, the video encoder (703) may encode the processing block into the coded picture using an intra prediction technique; when it is determined that the processing block is coded in inter mode or two prediction mode, the video encoder (703) may encode the processing block into the coded picture using an inter prediction technique or a two prediction technique, respectively. In some exemplary embodiments, a merge mode may be used as a submode of inter-picture prediction in which motion vectors are derived from one or more motion vector predictors without the benefit of coded motion vector components outside the predictors. In some other exemplary embodiments, there may be motion vector components applicable to the target block. Accordingly, the video encoder (703) may include components not explicitly shown in FIG. 7, such as a mode determination module, to determine the prediction mode of the processing blocks.

[0078] In the example of FIG. 7, the video encoder (703) includes an inter encoder (730), an intra encoder (722), a residue calculator (723), a switch (726), a residue encoder (724), a general controller (721), and an entropy encoder (725) combined together as shown in the exemplary arrangement of FIG. 7.

[0079] The inter-encoder (730) is configured to receive samples of a current block (e.g., a processing block), compare the block with one or more reference blocks within reference pictures (e.g., blocks within previous and subsequent pictures in the display order), generate inter-prediction information (e.g., description of redundant information according to the inter-encoding technique, motion vectors, merge mode information), and calculate inter-prediction results (e.g., predicted blocks) based on the inter-prediction information using any suitable technique. In some examples, the reference pictures are decoded reference pictures that are decoded based on encoded video information using a decoding unit (633) built into the exemplary encoder (620) of FIG. 6 (shown as the residual decoder (728) of FIG. 7, as described in more detail below).

[0080] The intra-encoder (722) is configured to receive samples of the current block (e.g., processing block), compare the block with already coded blocks within the same picture, generate quantized coefficients after transformation, and in some cases also generate intra-prediction information (e.g., intra-prediction direction information according to one or more intra-encoding techniques). The intra-encoder (722) can calculate intra-prediction results (e.g., prediction blocks) based on reference blocks within the same picture and intra-prediction information.

[0081] A general controller (721) may be configured to determine general control data and to control other components of the video encoder (703) based on the general control data. In one example, the general controller (721) determines the prediction mode of a block and provides a control signal to a switch (726) based on the prediction mode. For example, when the prediction mode is intra mode, the general controller (721) controls the switch (726) to select an intra mode result for use by the residual calculator (723), and controls the entropy encoder (725) to select intra prediction information and include the intra prediction information in the bitstream; when the prediction mode for a block is inter mode, the general controller (721) controls the switch (726) to select an inter prediction result for use by the residual calculator (723), and controls the entropy encoder (725) to select inter prediction information and include the inter prediction information in the bitstream.

[0082] The residual calculator (723) may be configured to calculate the difference (residual data) between the received block and the prediction results for the block selected from the intra-encoder (722) or the inter-encoder (730). The residual encoder (724) may be configured to encode the residual data to generate transformation coefficients. For example, the residual encoder (724) may be configured to generate transformation coefficients by converting the residual data from the spatial domain to the frequency domain. Then, quantization processing is performed on the transformation coefficients to obtain quantized transformation coefficients. In various exemplary embodiments, the video encoder (703) also includes a residual decoder (728). The residual decoder (728) is configured to perform an inverse transformation and generate decoded residual data. The decoded residual data may be suitably used by the intra-encoder (722) and the inter-encoder (730). For example, the inter encoder (730) can generate decoded blocks based on decoded residual data and inter prediction information, and the intra encoder (722) can generate decoded blocks based on decoded residual data and intra prediction information. The decoded blocks are appropriately processed to generate decoded pictures, and the decoded pictures are buffered in a memory circuit (not shown) and can be used as reference pictures.

[0083] The entropy encoder (725) may be configured to format the bitstream to include the encoded block and to perform entropy coding. The entropy encoder (725) is configured to include various information in the bitstream. For example, the entropy encoder (725) may be configured to include general control data, selected prediction information (e.g., intra prediction information or inter prediction information), residual information, and other suitable information within the bitstream. When coding the block in the inter mode or the merged submode of the two prediction modes, residual information may not be present.

[0084] FIG. 8 illustrates an exemplary video decoder (810) according to another embodiment of the present disclosure. The video decoder (810) is configured to receive coded pictures that are part of a coded video sequence and to decode the coded pictures to generate reconstructed pictures. In one example, the video decoder (810) may be used instead of the video decoder (410) in the example of FIG. 4.

[0085] In the example of FIG. 8, the video decoder (810) includes an entropy decoder (871), an inter decoder (880), a residual decoder (873), a reconstruction module (874), and an intra decoder (872) combined together as shown in the exemplary arrangement of FIG. 8.

[0086] The entropy decoder (871) may be configured to reconstruct specific symbols representing the syntax elements that constitute the coded picture from the coded picture. Such symbols may include, for example, the mode in which the block is coded (e.g., intra mode, inter mode, bi-predicted mode, merged submode, or other submode), prediction information (e.g., intra prediction information or inter prediction information) capable of identifying specific samples or metadata used for prediction by the intra decoder (872) or inter decoder (880), for example, residual information in the form of quantized transformation coefficients. In one example, when the prediction mode is an inter or bi-predicted mode, the inter prediction information is provided to the inter decoder (880); and when the prediction type is an intra prediction type, the intra prediction information is provided to the intra decoder (872). Inverse quantization may be performed on the residual information and is provided to the residual decoder (873).

[0087] The inter decoder (880) can be configured to receive inter prediction information and generate inter prediction results based on the inter prediction information.

[0088] The intra decoder (872) can be configured to receive intra prediction information and generate prediction results based on the intra prediction information.

[0089] The residual decoder (873) may be configured to perform inverse quantization to extract dequantized transform coefficients and process the dequantized transform coefficients to transform the residual from the frequency domain to the spatial domain. The residual decoder (873) may also utilize specific control information (to include quantizer parameters (QP)) that may be provided by the entropy decoder (871) (this may be merely low data capacity control information so that the data path is not described).

[0090] The reconstruction module (874) may be configured to form a reconstructed block that forms part of a reconstructed picture as part of a reconstructed video by combining the residuals output by the residual decoder (873) and the prediction results (output by inter or intra prediction modules, depending on the case) in the spatial domain. Note that other suitable operations, such as deblocking operations, may also be performed to improve visual quality.

[0091] It should be noted that the video encoders (403, 603, and 703), and video decoders (410, 510, and 810) can be implemented using any suitable technique. In some exemplary embodiments, the video encoders (403, 603, and 703), and video decoders (410, 510, and 810) can be implemented using one or more integrated circuits. In other embodiments, the video encoders (403, 603, and 703), and video decoders (410, 510, and 810) can be implemented using one or more processors that execute software instructions.

[0092] Referring to block partitioning for coding and decoding, general partitioning may start from a base block and may follow a predefined set of rules, specific patterns, partition trees, or any partition structure or method. Partitioning may be hierarchical and recursive. After partitioning the base block according to exemplary partitioning procedures or any of the other procedures described below, or a combination thereof, a final set of partitions or coding blocks may be obtained. Each of these partitions may be at one of various partitioning levels within the partitioning hierarchy and may have various shapes. Each of the partitions may be referred to as a coding block (CB). For various exemplary partitioning implementations further described below, each resulting CB may be of any of the allowed sizes and partitioning levels. These partitions are referred to as coding blocks because they can form units where some basic coding / decoding decisions can be made, coding / decoding parameters can be optimized, determined, and signaled in the encoded video bitstream. The highest or deepest level in the final partitions indicates the depth of the tree's coding block partitioning structure. A coding block can be a luminance coding block or a chroma coding block. Each color's CB tree structure can be referred to as a coding block tree (CBT).

[0093] Coding blocks of all color channels can be collectively referred to as Coding Units (CUs). The hierarchical structure of all color channels can be collectively referred to as Coding Tree Units (CTUs). Partitioning patterns or structures for various color channels within a CTU may or may not be identical.

[0094] In some implementations, the partition tree schemes or structures used for the Luma and Chroma channels may not need to be identical. In other words, the Luma and Chroma channels may have distinct coding tree structures or patterns. Additionally, whether the Luma and Chroma channels use the same or different coding partition tree structures, and the actual coding partition tree structures to be used, may depend on whether the slice being coded is a P, B, or I slice. For example, in the case of an I slice, the Chroma channels and Luma channels may have distinct coding partition tree structures or modes of coding partition tree structures, whereas in the case of a P or B slice, the Luma and Chroma channels may share the same coding partition tree scheme. When distinct coding partition tree structures or modes are applied, the Luma channel may be partitioned into CBs by one coding partition tree structure, and the Chroma channel may be partitioned into Chroma CBs by another coding partition tree structure.

[0095] In some exemplary implementations, a predetermined partitioning pattern may be applied to the base block. As illustrated in FIG. 9, an exemplary 4-way partition tree may start from a first predefined level (e.g., a 64x64 block level as the base block size or other sizes), and the base block may be hierarchically partitioned down to a predefined lowest level (e.g., a 4x4 level). For example, the base block may be subject to four predefined partitioning options or patterns indicated by 902, 904, 906, and 908, and partitions designated as R are permitted for recursive partitioning in that the same partition options shown in FIG. 9 may be repeated at a lower scale down to the lowest level (e.g., a 4x4 level). In some implementations, additional restrictions may be applied to the partitioning method of FIG. 9. In the implementation of FIG. 9, rectangular partitions (e.g., 1:2 / 2:1 rectangular partitions) may be allowed, but they may not be recursive, whereas square partitions are allowed to be recursive. Partitioning according to FIG. 9 using recursion generates a final set of coding blocks if necessary. The coding tree depth may be further defined to indicate the splitting depth from the root node or root block. For example, the coding tree depth for the root node or root block, e.g., a 64x64 block, may be set to 0, and after the root block is split once more according to FIG. 9, the coding tree depth is increased by 1. The maximum or deepest level from the 64x64 base block to the minimum 4x4 partition will be 4 (starting from level 0) in the case of the above method. This partitioning method may be applied to one or more of the color channels.Each color channel can be partitioned independently according to the method of FIG. 9 (for example, a partitioning pattern or option between predefined patterns can be determined independently for each color channel at each hierarchical level). Alternatively, two or more color channels may share the same hierarchical pattern tree of FIG. 9 (for example, the same partitioning pattern or option can be selected from predefined patterns for two or more color channels at each hierarchical level).

[0096] FIG. 10 illustrates another exemplary predefined partitioning pattern that allows recursive partitioning to form a partitioning tree. As illustrated in FIG. 10, an exemplary 10-way partitioning structure or pattern may be predefined. The root block may start from a predefined level (e.g., from a base block at a 128x128 level, or at a 64x64 level). The exemplary partitioning structure of FIG. 10 includes various 2:1 / 1:2 and 4:1 / 1:4 rectangular partitions. Partition types having three sub-partitions labeled 1002, 1004, 1006, and 1008 in the second row of FIG. 10 may be referred to as "T-type" partitions. The "T-type" partitions (1002, 1004, 1006, and 1008) may be referred to as the left T-type, top T-type, right T-type, and bottom T-type. In some exemplary implementations, none of the rectangular partitions of FIG. 10 are allowed to be further subdivided. The coding tree depth may be further defined to indicate the splitting depth from the root node or root block. For example, the coding tree depth for the root node or root block, e.g., a 128x128 block, may be set to 0, and after the root block is split once more according to FIG. 10, the coding tree depth is increased by 1. In some implementations, only 1010 all-square partitions may be allowed for recursive partitioning to the next level of the partitioning tree following the pattern of FIG. 10. In other words, recursive partitioning may not be allowed for square partitions within the T-type patterns (1002, 1004, 1006, and 1008). The partitioning procedure according to FIG. 10 using recursion generates a final set of coding blocks if necessary. This method can be applied to one or more of the color channels.In some implementations, more flexibility can be added to the use of partitions below the 8x8 level. For example, 2x2 chroma inter-prediction can be used in certain cases.

[0097] In some other exemplary implementations for coding block partitioning, a quadtree structure may be used to partition a base block or intermediate block into quadtree partitions. Such quadtree partitioning can be applied hierarchically and recursively to partitions of any square shape. Whether a base block, intermediate block, or partition is an additional quadtree partition may be adapted to various local characteristics of the base block or intermediate block / partition. Quadtree partitioning at picture boundaries may be further adapted. For example, an implicit quadtree split may be performed at picture boundaries so that the block maintains the quadtree split until its size fits within the picture boundary.

[0098] In some other exemplary implementations, hierarchical binary partitioning from a base block may be used. In this manner, a base block or an intermediate level block may be partitioned into two partitions. Binary partitioning can be horizontal or vertical. For example, horizontal binary partitioning may divide a base block or an intermediate block into equal right and left partitions. Similarly, vertical binary partitioning may divide a base block or an intermediate block into equal top and bottom partitions. Such binary partitioning can be hierarchical and recursive. A decision regarding whether the binary partitioning method should continue, and if so, whether horizontal binary partitioning or vertical binary partitioning should be used, can be made at the base block or the intermediate block, respectively. In some implementations, additional partitioning may stop at a predefined minimum partition size (in one or both dimensions). Alternatively, additional partitioning may stop once a predefined partitioning level or depth is reached from the base block. In some implementations, the aspect ratio of the partitions may be limited. For example, the aspect ratio of the partitions may not be less than 1:4 (or greater than 4:1). As such, a vertical strip partition with a vertical-to-horizontal aspect ratio of 4:1 can be further binary partitioned vertically into only upper and lower partitions, each with a vertical-to-horizontal aspect ratio of 2:1.

[0099] In some other examples, as illustrated in FIG. 13, a ternary partitioning scheme may be used to partition a base block or any intermediate block. The ternary pattern may be implemented vertically, as illustrated in 1302 of FIG. 13, or horizontally, as illustrated in 1304 of FIG. 13. The exemplary split ratio in FIG. 13 is shown as 1:2:1 vertically or horizontally, but other ratios may be predefined. In some implementations, two or more different ratios may be predefined. Since this triple-tree partitioning can capture objects located at the center of a block into a single contiguous partition, whereas quadtrees and binary trees always partition along the center of a block and thus divide objects into distinct partitions, this ternary partitioning scheme may be used to complement quadtree or binary partitioning structures. In some implementations, the width and height of the partitions of the exemplary tripletrees are always powers of two to avoid additional transformations.

[0100] The above partitioning methods can be combined in any way at different partitioning levels. As an example, the quadtree and binary partitioning methods described above can be combined to partition a base block into a quadtree-binary-tree (QTBT) structure. In this manner, the base block or intermediate block / partition can be quadtree partitioned or binary partitioned according to a set of predefined conditions, where specified. A specific example is illustrated in FIG. 14. In the example of FIG. 14, as shown in 1402, 1404, 1406, and 1408, the base block is first quadtree partitioned into four partitions. Subsequently, each of the resulting partitions is quadtree partitioned into four additional partitions (e.g., 1408), binary partitioned into two additional partitions at the next level (e.g., 1402 or 1406, horizontally or vertically, both symmetrically), or not partitioned (e.g., 1404). As illustrated by the full exemplary partition pattern in 1410 and the corresponding tree structure / representation in 1420, binary or quadtree partitioning may be recursively allowed for square-shaped partitions, where solid lines indicate quadtree partitioning and dashed lines indicate binary partitioning. Flags may be used for each binary partition node (non-leaf binary partitions) to indicate whether the binary partitioning is horizontal or vertical. For example, as illustrated in 1420, consistent with the partitioning structure of 1410, flag "0" can represent a horizontal binary partition, and flag "1" can represent a vertical binary partition. For quadtree-partition partitions, there is no need to indicate the partition type, because quadtree partitions always partition a block or partition both horizontally and vertically to create four sub-blocks / partitions of the same size.In some implementations, flag "1" may indicate horizontal binary partitioning, and flag "0" may indicate vertical binary partitioning.

[0101] In some exemplary implementations of QTBT, the quadtree and binary partitioning rule set can be represented by the following predefined parameters and their associated corresponding functions:

[0102] - CTU size: Size of the quadtree's root node (size of the base block)

[0103] - MinQTSize: Minimum allowed quadtree leaf node size

[0104] - MaxBTSize: Maximum allowed binary tree root node size

[0105] - MaxBTDepth: Maximum allowed binary tree depth

[0106] - MinBTSize: Minimum allowed binary tree leaf node size

[0107] In some exemplary implementations of the QTBT partitioning structure, the CTU size can be set to 128x128 luminance samples with two corresponding 64x64 blocks of chroma samples (when exemplary chroma subsampling is considered and used), MinQTSize can be set to 16x16, MaxBTSize can be set to 64x64, MinBTSize (for both width and height) can be set to 4x4, and MaxBTDepth can be set to 4. Quadtree partitioning can first be applied to the CTU to generate quadtree leaf nodes. The quadtree leaf nodes can have a size of their minimum allowable size of 16x16 (i.e., MinQTSize) to 128x128 (i.e., CTU size). If a node is 128x128, it will not be partitioned first by a binary tree because its size exceeds MaxBTSize (i.e., 64x64). Otherwise, nodes that do not exceed MaxBTSize can be partitioned by a binary tree. In the example of Fig. 14, the base block is 128x128. The base block can only be partitioned by a quadtree according to a predefined set of rules. The base block has a partitioning depth of 0. Each of the resulting four partitions is 64x64 and does not exceed MaxBTSize, and can be further partitioned by a quadtree or binary tree at Level 1. The process continues. When the binary tree depth reaches MaxBTDepth (i.e., 4), no further partitioning can be considered. If a binary tree node has a width equal to MinBTSize (i.e., 4), no further horizontal partitioning can be considered. Similarly, if a binary tree node has a height equal to MinBTSize, no further vertical partitioning is considered.

[0108] In some exemplary implementations, the above QTBT scheme may be configured to support flexibility, allowing the Luma and Chroma to have the same QTBT structure or distinct QTBT structures. For example, in the case of P and B slices, the Luma and Chroma CTBs within a single CTU may share the same QTBT structure. However, in the case of I slices, the Luma CTBs may be partitioned into CBs by the QTBT structure, and the Chroma CTBs may be partitioned into Chroma CBs by a different QTBT structure. This means that a CU may be used to reference different color channels in the I slice; for example, the I slice may consist of coding blocks of the Luma component or coding blocks of two Chroma components, and the CU in the P or B slice may consist of coding blocks of all three color components.

[0109] In some other implementations, the QTBT method can be supplemented with the ternary method described above. These implementations may be referred to as multi-type-tree (MTT) structures. For example, in addition to binary partitioning of nodes, one of the ternary partition patterns of Fig. 13 may be selected. In some implementations, only square nodes may undergo ternary partitioning. An additional flag may be used to indicate whether the ternary partitioning is horizontal or vertical.

[0110] The design of 2-level or multi-level trees, such as QTBT implementations and QTBT implementations complemented by ternary partitioning, can be primarily motivated by complexity reduction. Theoretically, the complexity of traversing a tree is T D And, where T represents the number of partition types and D is the depth of the tree. A trade-off can be made by using multiple types (T) while reducing the depth (D).

[0111] In some implementations, the CB may be further partitioned. For example, the CB may be further partitioned into multiple prediction blocks (PBs) for the purpose of intra- or inter-frame prediction during coding and decoding processes. In other words, the CB may be further divided into different subpartitions where individual prediction decisions / configurations can be performed. At the same time, the CB may be further partitioned into multiple transformation blocks (TBs) for the purpose of describing the levels at which the transformation or inverse transformation of the video data is performed. The partitioning method of the CB into PBs and TBs may be identical or different. For example, each partitioning method may be performed using its own procedure based, for example, on various characteristics of the video data. The PB and TB partitioning methods may be independent in some exemplary implementations. The PB and TB partitioning methods and boundaries may be correlated in some other exemplary implementations. In some implementations, for example, TBs may be partitioned after PB partitions, and in particular, after the partitioning of the coding block is determined, each PB may subsequently be further partitioned into one or more TBs. For example, in some implementations, a PB may be divided into 1, 2, 4, or other number of TBs.

[0112] In some implementations, the Luma Channel and the Chroma Channel may be treated differently in order to partition the base block into coding blocks and additionally into prediction blocks and / or transformation blocks. For example, in some implementations, partitioning the coding block into prediction blocks and / or transformation blocks may be permitted for the Luma Channel, whereas partitioning these coding blocks into prediction blocks and / or transformation blocks may not be permitted for the Chroma Channel(s). Therefore, in these implementations, transformation and / or prediction of the Luma Blocks may be performed only at the coding block level. As another example, the minimum transformation block size for the Luma Channel and the Chroma Channel(s) may differ; for example, the coding blocks for the Luma Channel may be allowed to be partitioned into smaller transformation and / or prediction blocks than those for the Chroma Channels. As another example, the maximum depth for partitioning coding blocks into transformation blocks and / or prediction blocks may differ between the lumina channel and the chroma channel; for example, coding blocks for the lumina channel may be allowed to be partitioned into transformation and / or prediction blocks deeper than the chroma channel(s). In a specific example, lumina coding blocks may be partitioned into transformation blocks of multiple sizes that can be represented by recursive partitioning down to a maximum of two levels, and transformation block shapes such as square, 2:1 / 1:2, and 4:1 / 1:4, and transformation block sizes from 4x4 to 64x64 may be allowed. However, for chroma blocks, only the largest possible transformation blocks specified for the lumina blocks may be allowed.

[0113] In some exemplary implementations for partitioning coding blocks into PBs, the depth, shape, and / or other characteristics of the PB partitioning may depend on whether the PBs are intra-coded or inter-coded.

[0114] Partitioning a coding block (or prediction block) into transformation blocks can be implemented in various exemplary ways, including but not limited to quadtree partitioning and predefined pattern partitioning, recursively or non-recursively, with additional consideration for transformation blocks at the boundaries of the coding block or prediction block. Generally, the resulting transformation blocks may be at different partition levels, may not have the same size, and may not need to be square in shape (e.g., they may be rectangular with some allowed sizes and aspect ratios). Additional examples are described in more detail below in relation to FIGS. 15, 16, and 17.

[0115] However, in some other implementations, CBs obtained through any of the above partitioning schemes can be used as basic or minimal coding blocks for prediction and / or transformation. In other words, no additional partitioning is performed for inter-prediction / intra-prediction purposes and / or transformation purposes. For example, CBs obtained from the above QTBT scheme can be used directly as units for performing predictions. Specifically, this QTBT structure eliminates the concept of multiple partition types, namely the separation of CU, PU, ​​and TU, and supports greater flexibility regarding CU / CB partition shapes as previously mentioned. In this QTBT block structure, CU / CB can have a square or rectangular shape. The leaf nodes of this QTBT are used as units for prediction and transformation processing without any additional partitioning. This means that in this exemplary QTBT coding block structure, CU, PU, ​​and TU have the same block size.

[0116] The various CB partitioning schemes above and additional partitioning of CBs into PBs and / or TBs (excluding PB / TB partitioning) may be combined in any way. The following specific implementations are provided as non-limiting examples.

[0117] Specific exemplary implementations of coding block and transform block partitioning are described below. In these exemplary implementations, the base block may be partitioned into coding blocks using recursive quadtree partitioning or the aforementioned predefined partitioning patterns (such as those in FIGS. 9 and 10). At each level, whether further quadtree partitioning of a specific partition is required may be determined by local video data characteristics. The resulting CBs may be of various quadtree partitioning levels and various sizes. The decision on whether to code a picture region using inter-picture (temporal) or intra-picture (spatial) prediction may be made at the CB level (or CU level for all 3-color channels). Each CB may be further partitioned into 1, 2, 4, or other numbers of PBs according to a predefined PB partitioning type. Within a single PB, the same prediction process may be applied, and relevant information may be transmitted to the decoder in PB units. After obtaining residual blocks by applying a prediction process based on the PB partitioning type, the CBs can be partitioned into TBs according to another quadtree structure similar to the coding tree for the CBs. In this particular implementation, the CBs or TBs may be restricted to square shapes, but are not required to be. Also, in this particular example, the PBs can be square or rectangular shapes for inter-predictions, and only square for intra-predictions. The coding blocks can be partitioned, for example, into four square-shaped TBs. Each TB can be recursively further partitioned (using quadtree partitioning) into smaller TBs referred to as Residual Quadtrees (RQTs).

[0118] Other exemplary implementations for partitioning a base block into CBs, PBs, and / or TBs are further described below. For example, rather than using multi-partition unit types such as those shown in FIG. 9 or FIG. 10, a quadtree having a nested multi-type tree using binary and ternary splits segmentation structures (e.g., a QTBT as described above or a QTBT with ternary splits) may be used. The separation of CBs, PBs, and TBs (i.e., partitioning of CBs into PBs and / or TBs, and partitioning of PBs into TBs) may be abandoned except when CBs that are too large for the maximum transformation length are required, where these CBs may require additional partitioning. This exemplary partitioning method can be designed to support greater flexibility regarding CB partition shapes so that both prediction and transformation can be performed on the CB level without additional partitioning. In this coding tree structure, the CB can have a square or rectangular shape. Specifically, the coding tree block (CTB) can first be partitioned by a quadtree structure. Then, the quadtree leaf nodes can be further partitioned by a nested multi-type tree structure. An example of a nested multi-type tree structure using binary or ternary partitioning is shown in FIG. 11. Specifically, the exemplary multi-type tree structure of FIG. 11 includes four split types referred to as vertical binary split (SPLIT_BT_VER) (1102), horizontal binary split (SPLIT_BT_HOR) (1104), vertical ternary split (SPLIT_TT_VER) (1106), and horizontal ternary split (SPLIT_TT_HOR) (1108). The CBs then correspond to the leaves of the multi-type tree.In this exemplary implementation, as long as the CB is not too large for the maximum transformation length, this segmentation is used for both prediction and transformation processing without any additional partitioning. This means that, in most cases, the CB, PB, and TB have the same block size in a quadtree having a nested multi-type tree coding block structure. An exception occurs when the maximum supported transformation length is smaller than the width or height of the color component of the CB. In some implementations, in addition to binary or ternary partitioning, the nested patterns of FIG. 11 may additionally include quadtree partitioning.

[0119] A specific example of a quadtree having a nested multi-type tree coding block structure of block partitions (including quadtree, binary, and ternary partition options) for a single base block is illustrated in FIG. 12. More specifically, FIG. 12 illustrates a base block (1200) being quadtree-partitioned into four square partitions (1202, 1204, 1206, and 1208). A decision to further use the multi-type tree structure and quadtree of FIG. 11 for additional partitioning is made for each of the quadtree-partitioned partitions. In the example of FIG. 12, partition (1204) is not further partitioned. Each of the partitions (1202 and 1208) adopts a different quadtree partitioning. In the case of partition (1202), the second-level quadtree-partition top-left, top-right, bottom-left, and bottom-right partitions each adopt a third-level partition of quadtree, horizontal binary partition (1104) of FIG. 11, non-partition, and horizontal ternary partition (1108) of FIG. 11. Partition (1208) adopts a different quadtree partition, and the second-level quadtree-partition top-left, top-right, bottom-left, and bottom-right partitions each adopt a third-level partition of vertical ternary partition (1106) of FIG. 11, non-partition, non-partition, and horizontal binary partition (1104) of FIG. 11. Two of the sub-partitions of the third-level top-left partition of 1208 are each further partitioned according to the horizontal binary partition (1104) and horizontal ternary partition (1108) of FIG. 11. Partition (1206) adopts a second level partitioning pattern according to the vertical binary partitioning (1102) of FIG. 11 into two partitions, and the two partitions are further divided into a third level according to the horizontal binary partitioning (1108) and vertical binary partitioning (1102) of FIG. 11. A fourth level partitioning is further applied to one of these according to the horizontal binary partitioning (1104) of FIG. 11.

[0120] For the specific example above, the maximum luma transform size may be 64x64 and the maximum supported chroma transform size may differ from the luma at, for example, 32x32. Even if the exemplary CBs above in FIG. 12 are not generally further subdivided into smaller PBs and / or TBs, when the width or height of a luma coding block or chroma coding block is greater than the maximum transform width or height, the luma coding block or chroma coding block may be automatically subdivided in the horizontal and / or vertical directions to meet the transform size limit in that direction.

[0121] In a specific example for partitioning the above base block into CBs, and as described above, the coding tree method can support the ability for Luma and Chroma to have separate block tree structures. For example, in the case of P and B slices, Luma and Chroma CTBs within a single CTU may share the same coding tree structure. For example, in the case of I slices, Luma and Chroma may have separate coding block tree structures. When separate block tree structures are applied, Luma CTBs may be partitioned into Luma CBs by a single coding tree structure, and Chroma CTBs are partitioned into Chroma CBs by a different coding tree structure. This means that the CU in Slice I may consist of coding blocks of the Luma component or coding blocks of two Chroma components, and that the CU in Slice P or B is always composed of coding blocks of all three color components, unless the video is monochrome.

[0122] When a coding block is further partitioned into multiple transformation blocks, the transformation blocks within it may be aligned into bitstreams according to various orders or scanning methods. Exemplary implementations for partitioning a coding block or a prediction block into transformation blocks, and the coding order of the transformation blocks, are described in more detail below. In some exemplary implementations, as described above, transformation partitioning may support transformation blocks of multiple shapes, e.g., 1:1 (square), 1:2 / 2:1, and 1:4 / 4:1, and the transformation block sizes range, e.g., from 4x4 to 64x64. In some implementations, if the coding block is less than or equal to 64x64, transformation block partitioning may be applied only to the luminance component, so that for the chroma blocks, the transformation block size is the same as the coding block size. Otherwise, if the coding block width or height is greater than 64, both the luma and chroma coding blocks can be implicitly divided into multiples of min(W, 64) x min(H, 64) and min(W, 32) x min(H, 32) transformation blocks, respectively.

[0123] In some exemplary implementations of transformation block partitioning, for both intra and intercoded blocks, the coding block may be further partitioned into multiple transformation blocks having a partitioning depth of up to a predefined number of levels (e.g., 2 levels). Transformation block partitioning depths and sizes may be related. For some exemplary implementations, an exemplary mapping from the transformation size of the current depth to the transformation size of the next depth is shown in Table 1 as follows.

[0124] Table 1: Convert Partition Size Settings

[0125]

[0126] Based on the exemplary mapping in Table 1, for a 1:1 square block, the next level transformation partition can generate four 1:1 square sub-transform blocks. The transformation partition can stop at, for example, 4x4. In this way, the transformation size for the current depth of 4x4 corresponds to the same size of 4x4 for the next depth. In the example in Table 1, for a 1:2 / 2:1 non-square block, the next level transformation partition can generate two 1:1 square sub-transform blocks, and for a 1:4 / 4:1 non-square block, the next level transformation partition can generate two 1:2 / 2:1 sub-transform blocks.

[0127] In some exemplary implementations, additional restrictions may be applied to the luma component of an intra-coded block regarding transformation block partitioning. For example, for each level of transformation partitioning, all sub-transformation blocks may be restricted to having the same size. For example, in the case of a 32x16 coding block, Level 1 transformation partitioning creates two 16x16 sub-transformation blocks, and Level 2 transformation partitioning creates eight 8x8 sub-transformation blocks. In other words, Level 2 partitioning must be applied to all Level 1 sub-blocks to maintain the transformation units of the same size. An example of transformation block partitioning for an intra-coded square block according to Table 1 is illustrated in FIG. 15 with the coding order indicated by arrows. Specifically, 1502 illustrates a square coding block. Level 1 partitioning into four transformation blocks of the same size according to Table 1 is illustrated in 1504 with the coding order indicated by arrows. The second level division of all first-level blocks of the same size into 16 conversion blocks of the same size according to Table 1 is shown in 1506 with the coding order indicated by the arrow.

[0128] In some exemplary implementations, the above restrictions on intra-coding may not apply to the luma components of the intercoded block. For example, after the first level of transformation division, any of the sub-transformation blocks may be independently divided one more level further. Thus, the resulting transformation blocks may or may not be of the same size. An exemplary division of the intercoded block into transformation blocks, along with their coding order, is illustrated in FIG. 16. In the example of FIG. 16, the intercoded block (1602) is divided into transformation blocks at two levels according to Table 1. At the first level, the intercoded block is divided into four transformation blocks of the same size. Then, only one of the four transformation blocks (not all of them) is further divided into four sub-transformation blocks, so that, as illustrated in 1604, a total of seven transformation blocks have two different sizes. An exemplary coding sequence of these seven transformation blocks is illustrated by an arrow at 1604 in Fig. 16.

[0129] In some exemplary implementations, some additional restrictions on transformation blocks may apply to the chroma component(s). For example, for the chroma component(s), the transformation block size may be as large as the coding block size, but may not be smaller than a predefined size, e.g., 8x8.

[0130] In some other exemplary implementations, for a coding block where the width (W) or height (H) is greater than 64, both the luma and chroma coding blocks may be implicitly divided into multiples of the min(W, 64) x min(H, 64) and min(W, 32) x min(H, 32) transformation units, respectively. Here, in the present disclosure, "min(a, b)" may return the smaller value between a and b.

[0131] FIG. 17 further illustrates another alternative exemplary method for partitioning a coding block or a prediction block into transformation blocks. As illustrated in FIG. 17, instead of using recursive transformation partitioning, a predefined set of partitioning types may be applied to the coding block according to the transformation type of the coding block. In the specific example illustrated in FIG. 17, one of six exemplary partitioning types may be applied to divide the coding block into a varying number of transformation blocks. This method of generating transformation block partitioning may be applied to a coding block or a prediction block.

[0132] More specifically, the partitioning method of FIG. 17 provides up to six exemplary partition types for any given transformation type (transformation type refers to a type of first-order transformation, such as ADST, etc.). In this method, a transformation partition type may be assigned to each coding block or prediction block, for example, based on a rate-distortion cost. In one example, the transformation partition type assigned to a coding block or prediction block may be determined based on the transformation type of the coding block or prediction block. A specific transformation partition type may correspond to the transformation block partition size and pattern, as shown by the six transformation partition types exemplified in FIG. 17. Correspondence relationships between various transformation types and various transformation partition types may be predefined. An example with uppercase labels indicating transformation partition types that may be assigned to a coding block or prediction block based on a rate-distortion cost is shown below:

[0133] ● PARTITION_NONE: Allocates a transformation size equal to the block size.

[0134] ● PARTITION_SPLIT: Allocates a transformation size that is half the width of the block size and half the height of the block size.

[0135] ● PARTITION_HORZ: Assigns a transformation size equal to the block size width and half the block size height.

[0136] ● PARTITION_VERT: Assigns a transformation size that is half the width of the block size and the same height as the block size.

[0137] ● PARTITION_HORZ4: Assigns a transformation size equal to the block size width and 1 / 4 of the block size height.

[0138] ● PARTITION_VERT4: Assigns a transformation size that is 1 / 4 of the block size's width and has a height equal to the block size.

[0139] In the above example, the transformation partition types as illustrated in FIG. 17 all include uniform transformation sizes for the partitioned transformation blocks. This is merely an example rather than a limitation. In some other implementations, mixed transformation block sizes may be used for partitioned transformation blocks of a specific partition type (or pattern).

[0140] PBs obtained from any of the above partitioning methods (or CBs, also called PBs when not further partitioned into prediction blocks) can then become individual blocks for coding through intra- or inter-predictions. For inter-prediction of the current PB, the residual between the current block and the prediction block is generated, coded, and can be included in the coded bitstream.

[0141] Inter-prediction can be implemented, for example, in single-reference mode or compound-reference mode. In some implementations, a skip flag may be included first in the bitstream for the current block (or at a higher level) to indicate whether the current block is inter-coded and will not be skipped. If the current block is inter-coded, another flag may be additionally included in the bitstream as a signal to indicate whether single-reference mode or compound-reference mode is used for the prediction of the current block. In the case of single-reference mode, one reference block may be used to generate the prediction block for the current block. In the case of compound-reference mode, two or more reference blocks may be used to generate the prediction block, for example, by weighted average. Compound-reference mode may be referred to as more-than-one-reference mode, two-reference mode, or multiple-reference mode. Reference blocks or reference blocks may be identified using reference frame indices or indices and additionally using corresponding motion vectors or motion vectors indicating the position, e.g., the shift(s) between the reference block(s) and the current blocks in horizontal and vertical pixels. For example, an inter-prediction block for the current block may be generated from a single-reference block identified by a single motion vector in the reference frame as a prediction block in single-reference mode, whereas for composite-reference mode, the prediction block may be generated by the weighted average of two reference blocks in two reference frames indicated by two reference frame indices and two corresponding motion vectors. Motion vector(s) may be coded in various ways and included in the bitstream.

[0142] In some implementations, the encoding or decoding system may maintain a decoded picture buffer (DPB). Some images / pictures may be maintained in the DPB awaiting display (in the decoding system), and some images / pictures within the DPB may be used as reference frames to enable inter-prediction (in the decoding or encoding system). In some implementations, reference frames within the DPB may be tagged as short-term or long-term references for the current image being encoded or decoded. For example, short-term reference frames may include frames used for inter-prediction of blocks within the current frame or within a predefined number (e.g., 2) of video frames closest to the current frame in the decoding order. Long-term reference frames may include frames within the DPB that may be used to predict image blocks within frames located further away from the current frame than a predefined number of frames in the decoding order. Information regarding these tags for short and long reference frames may be referred to as a Reference Picture Set (RPS) and may be added to the header of each frame within the encoded bitstream. Each frame within the encoded video stream may be numbered in an absolute manner according to the playback sequence, or identified by a Picture Order Counter (POC) associated with a picture group, for example, starting from the I-frame.

[0143] In some exemplary implementations, one or more reference picture lists, including the identification of short-term and long-term reference frames for inter-prediction, may be formed based on information within the RPS. For example, a single picture reference list may be formed for unidirectional inter-prediction, denoted as the L0 reference (or reference list 0), whereas two picture reference lists may be formed for bidirectional inter-prediction, denoted as L0 (or reference list 0) and L1 (or reference list 1) for each of the two prediction directions. The reference frames included in the L0 and L1 lists may be ordered in various predetermined ways. The lengths of the L0 and L1 lists may be signaled in the video bitstream. Unidirectional inter-prediction may be in a single-reference mode, or in a composite-reference mode when multiple references for generating a prediction block by weighted average in a composite prediction mode are on the same side of the block to be predicted. Bidirectional inter-prediction can only be a composite mode in that bidirectional inter-prediction involves at least two reference blocks.

[0144] Block partitioning

[0145] Figure 18 illustrates an exemplary partition tree for block partitioning. VP9 uses a 4-way partition tree starting from a 64x64 level and going down to a 4x4 level, with additional restrictions on 8x8 and lower blocks. This is exemplified in Figure 18. Partitions designated as R can be recursive in that the same partition tree is repeated at lower scales until it reaches the lowest 4x4 level. AV1 extends the partition tree to a 10-way structure as illustrated in the figure, but also increases the largest size (referred to as the superblock in VP9 / AV1) to start at 128x128. This includes 4:1 / 1:4 rectangular partitions that did not exist in VP9. None of the rectangular partitions can be further subdivided. Additionally, AV1 adds more flexibility to the use of partitions below the 8x8 level. For example, 2x2 chroma inter-prediction is now possible in certain cases.

[0146] In HEVC, Coding Tree Units (CTUs) are partitioned into Coding Units (CUs) using a quadtree structure represented by a coding tree to adapt to various local characteristics. A CU can also be considered a block containing a prediction block or a coding block. The decision on whether to code a picture region using inter-picture (temporal) or intra-picture (spatial) prediction is made at the CU level. Each CU can be further partitioned into one, two, or four prediction units (PUs) depending on the PU partitioning type. Within a single PU, the same prediction process is applied, and relevant information is transmitted to the decoder based on the PU. After acquiring residual blocks by applying the prediction process based on the PU partitioning type, the CU can be partitioned into Transform Units (TUs) according to another quadtree structure, such as the coding tree for the CU. The HEVC structure has multiple partition concepts including CUs, PUs, and TUs. In HEVC, CUs or TUs can be just square shapes, while PUs can be square or rectangular shapes for inter-predicted blocks. In HEVC, a single coding block can be further subdivided into four square sub-blocks, and a transformation is performed for each sub-block (i.e., TU). Each TU can be recursively subdivided (using quad-tree splitting) into smaller TUs called Residual Quad-Tree (RQT). At picture boundaries, HEVC uses implicit quad-tree splitting to ensure that the block maintains the quad-tree split until its size fits within the picture boundary.

[0147] A block partitioning structure utilizing a quadtree (QT) plus a binary tree (BT) may exist. In HEVC, CTUs are partitioned into CUs using a quadtree structure represented as a coding tree to adapt to various local characteristics. The decision on whether to code a picture region using inter-picture (temporal) or intra-picture (spatial) prediction is made at the CU level. In some embodiments, each CU may be further partitioned into one, two, or four PUs depending on the PU partitioning type. Within a single PU, the same prediction process may be applied, and relevant information is transmitted to the decoder based on the PU. After obtaining residual blocks by applying a prediction process based on the PU partitioning type, the CU may be partitioned into transformation units (TUs) according to another quadtree structure, such as the coding tree for the CU. The HEVC structure may have multiple partition concepts including CUs, PUs, and TUs.

[0148] FIG. 19 illustrates exemplary partitions and trees for a QTBT (quad-tree plus binary tree) structure. The QTBT structure may not have similar concepts to multiple partition types and may eliminate the separation of CU, PU, ​​and TU concepts. The QTBT structure may support increased flexibility regarding CU partition shapes. In some embodiments of the QTBT block structure, the CU may have a square or rectangular shape.

[0149] As illustrated in FIG. 19, Coding Tree Units (CTUs) are first partitioned by a quadtree structure. The quadtree leaf nodes are further partitioned by a binary tree structure. For binary tree partitioning, there may be two types of partitioning: symmetric horizontal partitioning and symmetric vertical partitioning. The binary tree leaf nodes are called Coding Units (CUs), and their segmentation can be used for prediction and transformation processing without any additional partitioning. CUs, PUs, and TUs may have the same block size in the QTBT coding block structure. In JEM, a CU may include coding blocks (CBs) of different color components (e.g., one CU includes one luminance CB and two chroma CBs for P and B slices of a 4:2:0 chroma format). In other embodiments, it may include a CB of a single component (e.g., one CU includes one luminance CB or two chroma CBs for I slices).

[0150] The following parameters can be defined for the QTBT partitioning method:

[0151] ● CTU Size: The size of the quadtree's root node; the same concept as in HEVC.

[0152] ● MinQTSize: Minimum allowed quadtree leaf node size

[0153] ● MaxBTSize: Maximum allowed binary tree root node size

[0154] ● MaxBTDepth: Maximum allowed binary tree depth

[0155] ● MinBTSize: Minimum allowed binary tree leaf node size

[0156] In one embodiment of the QTBT partitioning structure, the CTU size can be set to 128x128 luminance samples and two corresponding 64x64 blocks of chroma samples. In one embodiment, MinQTSize can be set to 16x16, MaxBTSize can be set to 64x64, MinBTSize (for both width and height) can be set to 4x4, and MaxBTDepth can be set to 4. Quadtree partitioning can first be applied to the CTU to generate quadtree leaf nodes. Quadtree leaf nodes can have a size of 16x16 (i.e., MinQTSize) to 128x128 (i.e., CTU size). If a leaf quadtree node is 128x128, it may not be further partitioned by the binary tree because its size exceeds MaxBTSize (i.e., 64x64). Otherwise, leaf quadtree nodes can be further partitioned by the binary tree. A quadtree leaf node can also be the root node of the binary tree and has a binary tree depth of 0. When the binary tree depth reaches MaxBTDepth (i.e., 4), no further partitioning can be considered. If a binary tree node has a width equal to MinBTSize (i.e., 4), no further horizontal partitioning can be considered. Similarly, if a binary tree node has a height equal to MinBTSize, no further vertical partitioning may be considered. The leaf nodes of the binary tree can be further processed by prediction and transformation operations without any additional partitioning. In JEM, the maximum CTU size can be 256x256 luminance samples.

[0157] Figure 19 illustrates an example of block partitioning using QTBT (left side of Figure 19) and the corresponding tree representation (right side of Figure 19). Solid lines indicate quadtree partitioning, and dotted lines indicate binary tree partitioning. At each partition (i.e., non-leaf) node of the binary tree, a flag is signaled to indicate which partition type (i.e., horizontal or vertical) is used, where 0 indicates horizontal partitioning and 1 indicates vertical partitioning. In the case of quadtree partitioning, there is no need to indicate the partition type, because quadtree partitioning divides the block both horizontally and vertically to produce four sub-blocks of the same size.

[0158] Additionally, the QTBT method supports flexibility for luminance and chroma to have separate QTBT structures. For example, in the case of P and B slices, luminance and chroma CTBs within a single CTU share the same QTBT structure. However, in the case of I slices, the luminance CTB is partitioned into CUs by the QTBT structure, and the chroma CTBs are partitioned into chroma CUs by a different QTBT structure. In this example, the CU in slice I contains the coding blocks of the luminance component or the coding blocks of two chroma components, and the CU in slice P or B contains the coding blocks of all three color components. In HEVC, inter-prediction for small blocks may be restricted to reduce memory access for motion compensation; thus, positive prediction is not supported for 4x8 and 8x4 blocks, and inter-prediction is not supported for 4x4 blocks. In QTBT as implemented in JEM-7.0, these restrictions can be removed.

[0159] Block partitioning can have a ternary tree (TT) structure. Fig. 20 illustrates exemplary triple-tree partitioning. In VVC, a multi-type-tree (MTT) structure can add horizontal and vertical center-side triple-trees to the top of the QTBT, as illustrated in Fig. 20 (a and b, respectively). Triple-tree partitioning can complement quad-tree and binary-tree partitioning. Triple-tree partitioning can capture objects located at the block center, while quad-trees and binary-trees are partitioned along the block center. In the case of triple-tree partitioning, the width and height of the partitions of the proposed triple trees can be powers of 2, so no additional transformations are required. The design of a 2-level tree can be motivated by complexity reduction, and the complexity of traversing the tree is T D And, where T represents the number of partition types and D is the depth of the tree.

[0160] Template Matching (TM)

[0161] FIG. 21 illustrates an example of template matching (TM) on a search area around an initial motion vector (MV). Template matching may be a decoder-side MV derivation method for refining motion information of a current coding unit (CU) or block by finding the closest match between a template within the current picture (i.e., top and / or left neighbor blocks of the current CU) and a block within a reference (i.e., of the same size as the template). The motion information may include spatial reference motion information (or spatial motion vectors) and / or temporal reference motion information (or temporal motion vectors). As exemplified in FIG. 21, a better MV is searched around the initial motion of the current CU within a [-8, +8]-pel search range.

[0162] In AMVP mode, the MVP candidate with the minimum TM error between the current block template and the referenced block template is selected on the encoder and decoder sides. TM is performed on this MVP candidate for MV refinement. In merge mode, a similar search method can be applied to merge candidates indicated by the merge index.

[0163] Decoder-side motion vector refinement (DMVR)

[0164] Figure 22 illustrates an example of decoder-side motion vector refinement. To increase the accuracy of MVs in merge mode, bilateral-matching (BM)-based decoder-side motion vector refinement can be applied in VVC. In positive prediction operation, refined MVs are searched around the initial MVs in reference picture list L0 and reference list L1. The BM method calculates the distortion between two candidate blocks in reference list L0 and list L1. As exemplified in Figure 22, the SAD(MV 1') between the two blocks is calculated based on each MV candidate around the initial MV. The MV candidate with the lowest SAD becomes the refined MV and is used to generate the positive prediction signal.

[0165] The refined MV derived by the DMVR process is used to generate inter-prediction samples for the current block and also for temporal motion vector prediction for future picture coding. Meanwhile, the original MV is used in the deblocking and spatial motion vector prediction processes for future CU coding within the current frame. In DMVR, search points surround the initial MV, and the MV offset can follow the MV difference mirroring rule. In other words, any point examined by DMVR, denoted as a candidate MV pair (MV0, MV1), can follow the following two mathematical equations:

[0166]

[0167] Here, MV_offset represents the refinement offset between the initial MV(MV0, MV1) and the refined MV(MV0',MV1') in one of the reference pictures. The refinement search range is two integer luminance samples from the initial MV. The search includes an integer sample offset search stage and a fractional sample refinement stage.

[0168] In one embodiment, a 25-point full search is applied for integer sample offset search. The SAD of the initial MV pair is calculated first. If the SAD of the initial MV pair is less than a threshold, the integer sample stage of the DMVR is terminated. Otherwise, the SAD of the remaining 24 points is calculated and examined in raster scanning order. The point with the minimum SAD is selected as the output of the integer sample offset search stage. To reduce the penalty of uncertainty in DMVR refinement, the original MV during the DMVR process may be preferred. The SAD between reference blocks referenced by the initial motion vector may be reduced by 1 / 4 of the SAD value.

[0169] Fractional sample refinement may follow integer sample search. To reduce computational complexity, fractional sample refinement is derived by using the parametric error surface equation instead of an additional search by SAD comparison. In parametric error surface-based sub-pixel offset estimation, the cost at the center position and the costs at the four integer positions adjacent to the center are used to fit a 2-D parabolic surface equation of the following form:

[0170]

[0171] Here, (x min , y min) corresponds to the coordinates of the fractional position with the minimum cost, and C corresponds to the minimum cost value. Solving the above mathematical equations with the cost values ​​of the 5 search points, (x min , y min ) is calculated as follows:

[0172]

[0173] Since all cost values ​​are positive and the smallest value is E(0,0), x min and y min The value of is clipped between -8 and 8. This corresponds to a 1 / 2 pixel offset with 1 / 16-pel MV accuracy in VVC. Calculated (x min , y min ) is added to the integer distance refinement MV to obtain the final refined delta MV with sub-pixel accuracy.

[0174] Reference frame coding in AV1

[0175] In AV1, for each coded block in an inter-frame, if the mode of the current block is not skip mode but inter-coded mode, a flag may be signaled to indicate whether single-reference mode or composite-reference mode is used for the current block. In single-reference mode, a prediction block is generated by a single motion vector, whereas in composite-reference mode, a prediction block is generated by weighted averaging two prediction blocks derived from two motion vectors. For the single-reference case, a reference frame index having a value from 1 to 7 is signaled in the bitstream to indicate which reference frame is used for the current block. Reference frame indices 1 through 4 may be assigned to reference frames preceding the current frame in the display order, while indices 5 through 7 may be assigned to reference frames following the current frame in the display order.

[0176] For complex reference cases, another flag may be signaled to indicate whether it is a unidirectional or bidirectional complex reference mode. In the case of a unidirectional complex reference mode, the two reference frames are either before or after the current frame in the display order. In the case of a bidirectional complex reference mode, one reference frame is before the current frame and the other reference frame is after the current frame in the display order. Additionally, the allowed unidirectional reference frame combinations are limited to only four possible pairs, namely (1, 2), (1, 3), (1, 4), and (5, 7), while all 12 combinations of the bidirectional case are supported. The rationale is that if the number of reference frames on both sides of the current frame in the display order is balanced, bidirectional reference predictions are more likely to provide better predictions. When most reference frames are on one side of the current frame, extrapolations involving the closest may be more relevant to the current frame.

[0177] Motion Vector Reference Methods in AV1 and CWG-B049

[0178] Bits for signaling motion vectors account for a significant portion of the total bit rate. Modern video codecs adopt predictive coding for motion vectors and can code differences using entropy coding. Prediction accuracy has a significant impact on coding efficiency. AV1 utilizes a dynamic motion vector reference scheme that acquires candidate motion vectors from spatial and temporal neighbors and ranks them for efficient entropy coding. In one embodiment, using a spatial motion vector reference, a coding block will search its spatial neighbors in units of 8x8 luminance samples to find those having the same reference frame index as the current block. In the case of complex inter-prediction modes, identical reference frame pairs may be included. FIG. 23 illustrates an exemplary spatial motion vector search pattern. As illustrated in FIG. 23, the search area includes three 8x8 block rows above the current block and three 8x8 block columns to the left of the current block, and the search order is indicated by an index. In CWG-B049, the number of rows in the upper 8x8 block can be reduced from 3 to 1, and the number of columns on the left is maintained at 3.

[0179] In another embodiment, the Temporal Motion Vector Reference may first include motion trajectories between the current frame and a previously coded frame constructed by utilizing motion vectors from previously coded frames through linear interpolation or extrapolation. Next, the motion field between the current frame and a given reference frame may be formed by extending motion trajectories from the current frame toward the reference frame.

[0180] Due to the current design of the signaling method for reference frames, the order of reference frames in reference frame list 0 and reference frame list 1 is fixed for all blocks within a single frame, regardless of the reference frames in neighboring blocks. Therefore, spatial and / or temporal correlations of reference frame selection are not utilized.

[0181] In the case of a composite reference mode, if the POCs of two reference frames for a pair of motion vectors are both greater than or both smaller than the POC of the current frame, the orientations of the two reference frames are the same. Otherwise, if the POC of one reference frame is greater than the POC of the current frame while the POC of the other reference frame is smaller than the POC of the current frame, the orientations of the two reference frames are different.

[0182] The proposed embodiments may be used individually or combined in any order. Additionally, each of the methods (or embodiments), the encoder, and the decoder may be implemented by a processing circuit (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored on a non-transient computer-readable medium. Hereinafter, the term block may be interpreted as a prediction block, a coding block, or a coding unit, i.e., a CU. The orientation of the reference frame may be determined by whether the reference frame is before the current frame in the display order or after the current frame in the display order.

[0183] FIG. 24 illustrates a flowchart of a method according to an exemplary embodiment of the present disclosure. In the case of a single or multiple reference mode, this template matching (TM) method may be used to reorder reference frames or pairs of reference frames for each block. In block 2402, the template of the current block and the template of the reference block are compared. The comparison may be for the difference between the template of the reference block referenced by the motion information of spatial reference motion information (or spatial motion vectors) and / or temporal reference motion information (or temporal motion vectors) and the template of the current block (i.e., top and / or left neighbor blocks). Based on the comparison, the difference between the templates is calculated in block 2404. The calculation may be based on SAD (sum of absolute differences), SSD (sum of squared differences), MSE (mean squared error), or SATD (sum of absolute transform differences). In one embodiment, for each spatial reference motion information and / or temporal reference motion information, the difference between the template of the current block and the template of the reference block is calculated and marked as the score value of the associated reference frames or pairs of reference frames, as in block 2406. A score for each reference frame is determined based on the calculated difference. Then, in block 2408, all available reference frames are ranked based on the score value. The ranking can be used to reorder the frames in block 2410.

[0184] Although described as reference frames, reference frame pairs may also be included. In one embodiment, if multiple motion vectors point to a single reference frame or a reference frame pair, the least different value referenced by these motion vectors may be marked as the score value of the single reference frame or a reference frame pair. In another embodiment, if there are no motion vectors pointing to a single reference frame or a reference frame pair, the score value of that reference frame or reference frame pair is marked as the maximum allowable value. In another embodiment, if the score values ​​of multiple reference frames or multiple reference frame pairs are identical, the ranking order for these reference frames or reference frame pairs is identical to the scan order of the spatial reference motion information and / or the temporal reference motion information. In another embodiment, if the score values ​​of multiple reference frames or multiple reference frame pairs are identical, the ranking order for these reference frames or reference frame pairs depends on the frequency of occurrence of these reference frames or reference frame pairs used in the spatial reference motion information and / or the temporal reference motion information.

[0185] In one embodiment, all allowed unidirectional and bidirectional composite reference frame pairs are ranked together using the TM method, and the index of the reference frame pair for the current block in this ranking order is signaled in the bitstream. In one embodiment, all allowed single reference frames are ranked together using the TM method, and the index of the reference frame for the current block in this ranking order is signaled in the bitstream.

[0186] In one embodiment, a method for reordering reference frames for each block by template matching (TM) comprises: comparing a template of the current block with a template of the reference block for motion information—the motion information includes spatial reference motion information or temporal reference motion information—; calculating the difference between the template of the current block and the template of the reference block; determining a score value for the associated reference frame based on the calculated difference; and reordering the reference frames based on the determined score. The reference frames further include reference frame pairs. The TM includes decoder-side motion vector derivation for refining the motion information of the current block. The reordering step further comprises ranking available reference frames based on the score value for each of the reference frames. When the score values ​​of a plurality of reference frames are equal, the ranking corresponds to the scan order of the spatial reference motion information or temporal reference motion information. When the score values ​​of multiple reference frames are equal, ranking is based on the frequency of occurrence of these reference frames used in spatial reference motion information or temporal reference motion information. The calculation includes at least one of SAD (sum of absolute differences), SSD (sum of squared differences), MSE (mean squared error), or SATD (sum of absolute transform differences). The template includes the top or left neighbor block. Spatial reference motion information includes one or more spatial motion vectors. Temporal reference motion information includes one or more temporal motion vectors.When multiple motion vectors point to one of the reference frames, the single motion vector with the smallest difference among the motion vectors is used to determine the score value. When no motion vector points to one of the reference frames, the score value is determined to be the maximum allowed value. Allowed unidirectional and bidirectional composite reference frames are ranked together using TM, and the index of the reference frames for the current block is signaled in the bitstream. Allowed single reference frames are ranked together using TM, and the index of the reference frame for the current block is signaled in the bitstream.

[0187] The embodiments of the present disclosure may be used individually or combined in any order. Additionally, each of the methods (or embodiments), the encoder, and the decoder may be implemented by a processing circuit (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored on a non-transient computer-readable medium. The term block may include a prediction block, a coding block, or a coding unit, i.e., a CU. The embodiments of the present disclosure may be applied to a luminance block or a chroma block.

[0188] The techniques described above may be implemented as computer software using computer-readable instructions and physically stored on one or more computer-readable media. For example, FIG. 25 illustrates a computer system (2500) suitable for implementing specific embodiments of the disclosed subject matter.

[0189] Computer software may be coded using any suitable machine code or computer language in which assembly, compilation, linking, or similar mechanisms can be performed to generate code containing instructions that can be executed directly or through interpretation, micro-code execution, etc., by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc.

[0190] The commands can be executed on various types of computers or their components, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet of Things devices, etc.

[0191] The components of the computer system (2500) illustrated in FIG. 25 are exemplary in nature and are not intended to imply any limitation on the scope of use or functionality of the computer software implementing the embodiments of the present disclosure. The configuration of the components should not be interpreted as having any dependency or requirement in relation to any one or a combination thereof of the components illustrated in the exemplary embodiments of the computer system (2500).

[0192] The computer system (2500) may include specific human interface input devices. Such human interface input devices may respond to input by one or more human users through, for example, tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, clapping), visual input (e.g., gestures), and olfactory input (not described). Human interface devices may also be used to capture specific media that are not necessarily directly related to conscious input by humans, such as audio (e.g., voice, music, ambient sound), images (e.g., scanned images, photographic images obtained from a still image camera), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0193] Input human interface devices may include one or more of: a keyboard (2501), a mouse (2502), a trackpad (2503), a touch screen (2510), a data glove (not shown), a joystick (2505), a microphone (2506), a scanner (2507), and a camera (2508) (each only one of which is depicted).

[0194] The computer system (2500) may also include specific human interface output devices. Such human interface output devices may stimulate one or more human user's senses through, for example, tactile output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., touch-screen (2510), data-glove (not shown), or haptic feedback via joystick (2505), but there may also be haptic feedback devices that do not serve as input devices), audio output devices (e.g., speakers (2509), headphones (not depicted)), visual output devices (e.g., screens (2510) including CRT screens, LCD screens, plasma screens, and OLED screens, each with or without touch-screen input capability and with or without haptic feedback capability—some of which may output two-dimensional visual output or output beyond three dimensions through means such as stereographic output—; virtual reality glasses (not depicted), holographic displays and smoke tanks (not depicted)), and printers (not depicted).

[0195] The computer system (2500) may also include human-accessible storage devices and their associated media, such as optical media including a CD / DVD ROM / RW (2520) having a media (2521) such as a CD / DVD, a thumb-drive (2522), a removable hard drive or solid-state drive (2523), legacy magnetic media such as tape and floppy disk (not described), specialized ROM / ASIC / PLD-based devices such as a security dongle (not described).

[0196] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the subject matter disclosed herein does not include transmission media, carriers, or other transient signals.

[0197] The computer system (2500) may also include an interface (2554) to one or more communication networks (2555). The networks may be, for example, wireless, wired, optical. The networks may additionally be local, wide-area, metropolitan, automotive and industrial, real-time, delay-tolerant, etc. Examples of networks include local area networks, such as Ethernet, wireless LANs, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., TV wired or wireless wide-area digital networks including cable TV, satellite TV, and terrestrial broadcast TV, automotive and industrial including CAN buses, etc. Certain networks generally require external network interface adapters attached to certain general-purpose data ports or peripheral buses (2549) (e.g., USB ports of the computer system (2500)); Others are generally integrated into the core of the computer system (2500) by attachment to a system bus (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system) as described below. Using any of these networks, the computer system (2500) can communicate with other entities. Such communication may be unidirectional receive-only (e.g., broadcast TV), unidirectional transmit-only (e.g., CANbus to specific CANbus devices), or bidirectional with other computer systems using, for example, local area or wide area digital networks. Specific protocols and protocol stacks may be used for each of the networks and network interfaces described above.

[0198] The aforementioned human interface devices, human-accessible storage devices, and network interfaces can be attached to the core (2540) of the computer system (2500).

[0199] The core (2540) may include one or more central processing units (CPUs) (2541), specialized programmable processing units in the form of graphics processing units (GPUs) (2542) and field programmable gate areas (FPGAs) (2543), hardware accelerators (2544) for specific tasks, graphics adapters (2550), etc. These devices may be connected via a system bus (2548), along with internal mass storage (2547), such as read-only memory (ROM) (2545), random access memory (2546), internal non-user accessible hard drives, SSDs, etc. In some computer systems, the system bus (2548) may be accessible in the form of one or more physical plugs to enable expansion by additional CPUs, GPUs, etc. Peripheral devices may be attached directly to the core's system bus (2548) or via a peripheral bus (2549). In one example, the screen (2510) can be connected to a graphics adapter (2550). Architectures for peripheral buses include PCI, USB, etc.

[0200] CPUs (2541), GPUs (2542), FPGAs (2543), and accelerators (2544) can be combined to execute specific instructions that can constitute the aforementioned computer code. The computer code may be stored in ROM (2545) or RAM (2546). Transient data may also be stored in RAM (2546), while persistent data may be stored, for example, in internal mass storage (2547). High-speed storage and retrieval of any of the memory devices may be made possible through the use of cache memory, which may be closely associated with one or more CPUs (2541), GPUs (2542), mass storage (2547), ROM (2545), RAM (2546), etc.

[0201] A computer-readable medium may have computer code for performing various computer-implemented operations. The medium and the computer code may be those specifically designed and configured for the purposes of this disclosure, or they may be of a kind well known and available to those skilled in the field of computer software technology.

[0202] As a non-limiting example, a computer system (2500) having an architecture, and specifically a core (2540), may provide functionality as a result of processor(s) (including CPUs, GPUs, FPGAs, accelerators, etc.) executing software implemented on one or more types of tangible computer-readable media. Such computer-readable media may be media associated with specific storage of the core (2540) that is of a non-transient nature, such as core-internal mass storage (2547) or ROM (2545), as well as user-accessible mass storage as described above. Software implementing various embodiments of the present disclosure may be stored on such devices and executed by the core (2540). The computer-readable media may include one or more memory devices or chips as needed. Software may enable the core (2540) and, specifically, the processors within it (including a CPU, GPU, FPGA, etc.) to execute specific processes or specific parts of specific processes described herein, including defining data structures stored in RAM (2546) and modifying such data structures according to processes defined by the software. Additionally, or alternatively, the computer system may provide functionality as a result of logic hardwired or otherwise implemented in a circuit (e.g., accelerator (2544)) that can operate instead of or with the software to execute specific processes or specific parts of specific processes described herein. Reference to software may include logic, where appropriate, and vice versa. Reference to a computer-readable medium may include, where appropriate, a circuit storing software for execution (e.g., an integrated circuit (IC)), or a circuit implementing logic for execution, or both. The present disclosure includes any suitable combination of hardware and software.

[0203] Although the present disclosure describes various exemplary embodiments, modifications, substitutions, and various alternative equivalents exist within the scope of the present disclosure. Accordingly, it will be acknowledged that those skilled in the art will be able to devise a number of systems and methods that implement the principles of the present disclosure and are therefore within the true meaning and scope thereof, even though they are not expressly illustrated or described in this specification.

[0204] Appendix A: Acronyms

[0205] ALF: Adaptive Loop Filter

[0206] AMVP: Advanced Motion Vector Prediction

[0207] APS: Adaptation Parameter Set

[0208] ASIC: Application-Specific Integrated Circuit

[0209] AV1: AOMedia Video 1

[0210] AV2: AOMedia Video 2

[0211] BCW: Bi-prediction with CU-level Weights

[0212] BM: Bilateral Matching

[0213] BMS: benchmark set

[0214] CANBus: Controller Area Network Bus

[0215] CC-ALF: Cross-Component Adaptive Loop Filter

[0216] CCSO: Cross-Component Sample Offset

[0217] CD: Compact Disc

[0218] CDEF: Constrained Directional Enhancement Filter

[0219] CDF: Cumulative Density Function

[0220] CfL: Chroma from Luma

[0221] CIIP: Combined intra-inter prediction

[0222] CPUs: Central Processing Units

[0223] CRT: Cathode Ray Tube

[0224] CTBs: Coding Tree Blocks

[0225] CTU: Coding Tree Unit

[0226] CTUs: Coding Tree Units

[0227] CU: Coding Unit

[0228] DMVR: Decoder-side Motion Vector Refinement

[0229] DPB: Decoded Picture Buffer

[0230] DPS: Decoding Parameter Set

[0231] DVD: Digital Video Disc

[0232] FPGA: Field Programmable Gate Areas

[0233] GBI: Generalized Bi-prediction

[0234] GOPs: Groups of Pictures

[0235] GPUs: Graphics Processing Units

[0236] GSM: Global System for Mobile communications

[0237] HDR: high dynamic range

[0238] HEVC: High Efficiency Video Coding

[0239] HRD: Hypothetical Reference Decoder

[0240] IBC (또는 IntraBC): Intra Block Copy

[0241] IC: Integrated Circuit

[0242] ISP: Intra Sub-Partitions

[0243] JEM: joint exploration model

[0244] JVET: Joint Video Exploration Team

[0245] LAN: Local Area Network

[0246] LCD: Liquid-Crystal Display

[0247] LR: Loop Restoration Filter

[0248] LSO: Local Sample Offset

[0249] LTE: Long-Term Evolution

[0250] MMVD: Merge Mode with Motion Vector Difference

[0251] MPM: most probable mode

[0252] MV: Motion Vector

[0253] MV: Motion Vector

[0254] MVD: Motion Vector difference

[0255] MVD: Motion vector difference

[0256] MVP: Motion Vector Predictor

[0257] OLED: Organic Light-Emitting Diode

[0258] PBs: Prediction Blocks

[0259] PCI: Peripheral Component Interconnect

[0260] PDPC: Position Dependent Prediction Combination

[0261] PLD: Programmable Logic Device

[0262] POC: Picture Order Count

[0263] PPS: Picture Parameter Set

[0264] PU: Prediction Unit

[0265] PUs: Prediction Units

[0266] RAM: Random Access Memory

[0267] ROM: Read-Only Memory

[0268] RPS: Reference Picture Set

[0269] SAD: Sum of Absolute Difference

[0270] SAO: Sample Adaptive Offset

[0271] SB: Super Block

[0272] SCC: Screen Content Coding

[0273] SDP: Semi Decoupled Partitioning

[0274] SDR: standard dynamic range

[0275] SDT: Semi Decoupled Tree

[0276] SEI: Supplementary Enhancement Information

[0277] SNR: Signal Noise Ratio

[0278] SPS: Sequence Parameter Setting

[0279] SSD: solid-state drive

[0280] SST: Semi Separate Tree

[0281] TM: Template Matching

[0282] TU: Transform Unit

[0283] TUs: Transform Units,

[0284] USB: Universal Serial Bus

[0285] VPS: Video Parameter Set

[0286] VUI: Video Usability Information

[0287] VVC: versatile video coding

[0288] WAIP: Wide-Angle Intra Prediction

Claims

Claim 1 A method for reordering reference frames for each block by template matching (TM), comprising: a step of comparing a first template of a current block and a second template of a reference block for motion information, wherein the motion information includes one or more of spatial reference motion information and temporal reference motion information; a step of calculating the difference between the first template of the current block and the second template of the reference block; a step of determining a value for an associated reference frame based on the calculated difference; and a step of reordering the reference frames based on the determined value, wherein when a plurality of motion vectors point to one of the reference frames, one motion vector having the smallest difference among the motion vectors is used to determine the value. Claim 2 A method according to claim 1, wherein the reference frames further include a pair of reference frames. Claim 3 A method according to claim 1, wherein the TM includes decoder-side motion vector derivation for refining motion information of the current block. Claim 4 A method according to any one of claims 1 to 3, wherein the reordering step further comprises: a step of ranking available reference frames based on a value for each of the reference frames. Claim 5 A method according to claim 4, wherein when the values ​​of a plurality of reference frames are equal, the ranking corresponds to the scan order of the spatial reference motion information or the temporal reference motion information. Claim 6 A method according to claim 4, wherein when the values ​​of a plurality of reference frames are equal, the ranking is based on the frequency of occurrence of these reference frames used in the spatial reference motion information or temporal reference motion information. Claim 7 A method according to any one of claims 1 to 3, wherein calculating the difference comprises at least one of SAD (sum of absolute differences), SSD (sum of squared differences), MSE (mean squared error), or SATD (sum of absolute transform differences). Claim 8 A method according to any one of paragraphs 1 to 3, wherein each of the templates includes an upper or left neighbor block. Claim 9 A method according to any one of claims 1 to 3, wherein the spatial reference motion information comprises one or more spatial motion vectors. Claim 10 A method according to any one of claims 1 to 3, wherein the temporal reference motion information comprises one or more temporal motion vectors. Claim 11 delete Claim 12 A method in which, in any one of claims 1 to 3, when there is no motion vector pointing to one of the reference frames, the value is determined to be the maximum allowed value. Claim 13 A method according to any one of claims 1 to 3, wherein allowed unidirectional and bidirectional composite reference frames are ranked together using the TM, and the index of the reference frames for the current block is signaled in the bitstream. Claim 14 A method according to any one of claims 1 to 3, wherein allowed single reference frames are ranked together using the TM, and the index of the reference frame for the current block is signaled in the bitstream. Claim 15 A device for decoding a video bitstream, comprising: a memory storing instructions; and a processor communicating with said memory, wherein when said processor executes said instructions, said processor is configured to cause said device to perform the method of any one of said claims 1 to 3. Claim 16 A non-transient computer-readable storage medium storing instructions, wherein when the instructions are executed by a processor, the instructions are configured to cause the processor to perform the method of any one of claims 1 to 3. Claim 17 delete Claim 18 delete Claim 19 delete Claim 20 delete

Images