Joint motion vector differential coding

Abstract

To provide a method for encoding and decoding a motion vector difference for inter-predicting a video block.SOLUTION: A method includes the steps of receiving a coded video bitstream, extracting an inter prediction mode and a joint delta motion vector (MV) of a current block in a current frame, extracting a flag indicating whether a first delta MV for a first reference frame and a second delta MV for a second reference frame are signaled together, deriving a first delta MV and a second delta MV on the basis of the joint delta MV in response to the flag indicating that the first delta MV and the second delta MV are signaled together, and decoding the current block in the current frame on the basis of the first delta MV and the second delta MV.SELECTED DRAWING: Figure 18

Description

Technical Field

[0001] Cross - Reference to Related Applications This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 245,655, filed on September 17, 2021, which is hereby incorporated by reference in its entirety. This application also claims the benefit of priority to U.S. Non - Provisional Application No. 17 / 700,745, filed on March 22, 2022, which is hereby incorporated by reference in its entirety.

[0002] The present disclosure relates to video coding and / or decoding techniques, and more particularly, to improved design and signaling of joint motion vector differences for coding and / or decoding.

Background Art

[0003] The description of the background art provided herein is for the purpose of generally presenting the context of the present disclosure. The inventors' research is not admitted as prior art to the present disclosure, either expressly or implicitly, to the extent that the research is described in this background art section and to the extent that aspects of the description that may not be recognized as prior art at the time of filing of the present application are concerned.

[0004] Video coding and decoding can be performed using inter-picture prediction with motion compensation. Uncompressed digital video can include a series of pictures, each picture having spatial dimensions of, for example, 1920×1080 luminance samples and associated full-sampled or subsampled chrominance samples. The series of pictures can have a fixed or variable picture rate (or frame rate, also called), for example, 60 pictures per second or 60 frames per second. Uncompressed video has specific bitrate requirements for streaming or data processing. For example, video having a pixel resolution of 1920×1080, a frame rate of 60 frames / second, and 4:2:0 chroma subsampling with 8 bits per pixel per color channel requires a bandwidth close to 1.5 Gbit / s. One hour of such video requires storage space exceeding 600 GByte.

[0005] One purpose of video coding and decoding can be the reduction of redundancy in the uncompressed input video signal by compression. Compression can help reduce the aforementioned bandwidth and / or storage space requirements, in some cases by more than two orders of magnitude. Both reversible compression and irreversible compression, and combinations thereof, can be used. Reversible compression refers to a technique in which an exact copy of the original signal can be reconstructed from the compressed original signal by the decoding process. Irreversible compression refers to a coding / decoding process in which the original video information is not fully retained during coding and cannot be fully restored during decoding. When using irreversible compression, the reconstructed signal may not be identical to the original signal, but the distortion between the original signal and the reconstructed signal is small enough to make the reconstructed signal useful for its intended purpose, even with some information loss. In the case of video, irreversible compression is widely adopted in many applications. The amount of tolerable distortion depends on the application. For example, a user of certain consumer video streaming applications may tolerate higher distortion than a user of movie or television broadcast applications. The compression ratio achievable by a particular coding algorithm can be selected or adjusted to reflect various distortion tolerances. That is, generally, the higher the distortion tolerance, the more possible it is to have a coding algorithm that results in high loss and a high compression ratio.

[0006] Video encoders and decoders can utilize techniques from several broad categories and steps, including, for example, motion compensation, Fourier transform, quantization, and entropy coding.

[0007] Video coding technology may include techniques known as intra coding. In intra coding, sample values are represented without reference to samples or other data from previously reconstructed reference pictures. In some video coders, a picture is spatially subdivided into blocks of samples. If all blocks of samples are coded in an intra mode, that picture may be called an intra picture. Intra pictures and their derivatives such as independent decoder refresh pictures can be used to reset the decoder state and thus can be used as the first picture in a coded video bitstream and video session or as a still image. The samples of the blocks after intra prediction can then undergo a transform to the frequency domain, and the transform coefficients so generated can be quantized prior to entropy coding. Intra prediction represents techniques for minimizing sample values in the pre-transform domain. In some cases, the smaller the DC value after transformation and the smaller the AC coefficients, the fewer bits are required with a given quantization step size to represent the block after entropy coding.

[0008] For example, conventional intra coding, such as known from MPEG-2 generation coding technology, does not use intra prediction. However, some newer video compression technologies include techniques that attempt to code / decode blocks based on surrounding sample data and / or metadata that precede in decoding order blocks of intra-coded or intra-decoded data obtained, for example, during spatial adjacent encoding and / or decoding. Such techniques are hereinafter referred to as "intra prediction" techniques. Note that in at least some cases, intra prediction uses reference data only from the current picture being reconstructed and does not use reference data from other reference pictures.

[0009] Intra prediction can have many different forms. If two or more of such techniques are available in a given video coding technique, the technique used can be called an intra prediction mode. One or more intra prediction modes can be provided in a particular codec. In certain cases, a mode can have sub-modes and / or can be associated with various parameters. The mode / sub-mode information and the intra coding parameters of a video block can be coded individually or can be included together in the codeword of the mode. Which codeword to use for a given combination of mode, sub-mode, and / or parameters can affect the coding efficiency improvement via intra prediction, and thus can also affect the entropy coding technique used to convert the codeword into the bitstream.

[0010] A certain mode of intra prediction was introduced in H.264, improved in H.265, and further improved in more recent coding techniques such as the Joint Exploration Model (JEM), Versatile Video Coding (VVC), and Benchmark Set (BMS). Generally, in intra prediction, a predictor block can be formed using the available adjacent sample values. For example, the available values of a particular set of adjacent samples along a certain direction and / or line can be copied into the predictor block. The reference to the direction used can be coded within the bitstream or can itself be predicted.

[0011] Referring to FIG. 1A, shown at the lower right is a subset of nine predictor directions specified in 33 possible intra predictor directions of H.265 (corresponding to 33 of the 35 intra modes specified in H.265). The point (101) where the arrows converge represents the sample being predicted. The arrows represent the direction in which adjacent samples are used to predict sample 101 therefrom. For example, arrow (102) indicates that sample (101) is predicted at an angle of 45 degrees from the horizontal, upward and to the right, from one or more adjacent samples. Similarly, arrow (103) indicates that sample (101) is predicted at an angle of 22.5 degrees from the horizontal, downward and to the left of sample (101), from one or more adjacent samples.

[0012] Referring further to FIG. 1A, depicted at the upper left is a square block (104) of 4×4 samples (indicated by the thick dashed line). Square block (104) contains 16 samples, each labeled with an "S", its position in the Y dimension (e.g., row index), and its position in the X dimension (e.g., column index). For example, sample S21 is the second sample from the top in the Y dimension and the first sample from the left in the X dimension. Similarly, sample S44 is the fourth sample in both the Y and X dimensions within block (104). Since the size of the block is 4×4 samples, S44 is at the lower right. Further examples of reference samples following a similar numbering scheme are shown. The reference samples are labeled with an "R", its Y position (e.g., row index) and X position (column index) relative to block (104). In both H.264 and H.265, predicted samples adjacent to the block being reconstructed are used.

[0013] The intra-picture prediction of block 104 may start by copying the reference sample value from adjacent samples according to the signaled prediction direction. For example, a coded video bitstream may include signaling indicating the prediction direction of arrow (102) for this block 104, i.e., it is assumed that the samples are predicted from one or more prediction samples at an angle of 45 degrees from the horizontal direction towards the upper right. In such a case, samples S41, S32, S23, S14 are predicted from the same reference sample R05. Then, sample S44 is predicted from reference sample R08.

[0014] In certain cases, in order to calculate the reference samples, especially when the direction is not evenly divisible by 45 degrees, the values of multiple reference samples may be combined, for example, by interpolation.

[0015] The number of possible directions has been increasing as video coding technology continues to evolve. In H.264 (2003), for example, nine different directions are available for intra prediction. This has increased to 33 in H.265 (2013), and JEM / VVC / BMS can support up to 65 directions at the time of this disclosure. Experimental studies have been conducted to help identify the most appropriate intra prediction direction, and certain techniques of entropy coding can be used to accept a certain bit penalty for the directions and encode those most appropriate directions with a small number of bits. Further, the direction itself can be predicted from adjacent directions used in the intra prediction of decoded adjacent blocks.

[0016] FIG. 1B shows a schematic diagram (180) showing 65 intra prediction directions by JEM to illustrate the increasing number of prediction directions in various coding technologies that have evolved over time.

[0017] Methods for mapping bits representing intra prediction directions in a coded video bitstream to prediction directions can vary depending on video coding technology and can range from simple direct mappings of prediction directions to intra prediction modes to complex adaptive schemes including codewords, most probable modes, and similar techniques. However, in all cases, there may be certain directions of intra prediction that are statistically less likely to occur in video content than other specific directions. Since the purpose of video compression is redundancy reduction, in well-designed video coding technologies, those less likely directions may be represented with more bits than the more likely directions.

[0018] Interpicture prediction, or inter prediction, may be based on motion compensation. In motion compensation, sample data from a previously reconstructed picture or a part thereof (reference picture) is spatially shifted in the direction indicated by a motion vector (hereinafter MV) and then used for prediction of a newly reconstructed picture or picture part (e.g., block). In some cases, the reference picture may be the same as the picture currently being reconstructed. The MV may have two dimensions X and Y, or three dimensions, with the third dimension being an indication of the reference picture used (similar to the temporal dimension).

[0019] In some video compression techniques, the current MV applicable to a particular area of sample data can be predicted from other MVs, for example, from other MVs related to other areas of sample data that are spatially adjacent to the area being reconstructed and precede the current MV in decoding order. By doing so, the overall amount of data required to code the MVs can be significantly reduced by relying on the removal of redundancy of correlated MVs, thereby increasing the compression efficiency. MV prediction can function effectively, for example, when coding an input video signal derived from a camera (known as natural video), because areas larger than the area to which a single MV is applicable have a statistical likelihood of moving in the same direction in the video sequence and can thus, in some cases, be predicted using similar motion vectors derived from the MVs of adjacent areas. As a result, the actual MV of a given area becomes similar or identical to the predicted MV from surrounding MVs. Such MVs can further be represented with fewer bits than the number of bits that would be used if the MVs were coded directly rather than predicted from one or more adjacent MVs after entropy coding. In some cases, MV prediction can be considered an example of lossless compression of the signal (i.e., the MV) derived from the original signal (i.e., the sample stream). In other cases, for example, due to rounding errors when calculating predictors from several surrounding MVs, MV prediction itself can be lossy.

[0020] Various MV prediction mechanisms are described in H.265 / HEVC (ITU-T Rec. H.265, "High Efficiency Video Coding", December 2016). Among the many MV prediction mechanisms specified by H.265, the one described below is a technique hereinafter referred to as "spatial merge".

[0021] Specifically, referring to FIG. 2, the current block (201) contains samples that have been detected by the encoder during the motion search process to be predictable from the previous block of the same size that has been spatially shifted. Instead of directly coding the MV, the MV can be derived from metadata associated with one or more reference pictures, for example, from the last reference picture (in decoding order), using an MV associated with any one of five surrounding samples represented by A0, A1, and B0, B1, B2 (202 to 206 respectively). In H.265, the MV prediction can use predictors from the same reference picture that the adjacent blocks are using. SUMMARY OF THE INVENTION MEANS FOR SOLVING THE PROBLEM

[0022] The present disclosure describes various embodiments of methods, apparatuses, and computer-readable storage media for video encoding and / or decoding.

[0023] According to one aspect, an embodiment of the present disclosure provides a method for decoding an inter-predicted video block. The method includes receiving, by a device, a coded video bitstream. The device includes a memory storing instructions and a processor communicating with the memory. The method also includes extracting, by the device, an inter-prediction mode and a joint delta motion vector (MV) of a current block in a current frame from the coded video bitstream, extracting, by the device, a flag indicating whether a first delta MV for a first reference frame and a second delta MV for a second reference frame are signaled together from the coded video bitstream, deriving, by the device, the first delta MV and the second delta MV based on the joint delta MV in response to the flag indicating that the first delta MV and the second delta MV are signaled together, and decoding, by the device, the current block in the current frame based on the first delta MV and the second delta MV.

[0024] According to other aspects, an embodiment of the present disclosure provides an apparatus for video encoding and / or decoding. The apparatus includes a memory storing instructions and a processor communicating with the memory. When the processor executes the instructions, the processor is configured to cause the apparatus to perform the above method for video decoding and / or encoding.

[0025] Aspects of the present disclosure also provide a video encoding or decoding device or apparatus including a circuit configured to perform any of the above method embodiments.

[0026] In another aspect, one embodiment of the present disclosure provides a non-transitory computer-readable medium storing instructions that, when executed by a computer for video decoding and / or encoding, cause the computer to execute the above method for video decoding and / or encoding.

[0027] The above and other aspects and their implementations are described in more detail in the drawings, the specification, and the claims.

[0028] Further features, properties, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings.

Brief Description of the Drawings

[0029]

Figure 1A

Figure 1B

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

DETAILED DESCRIPTION OF THE INVENTION

[0030] Next, with reference to the accompanying drawings that form part of the present invention and illustrate specific examples of embodiments, the present invention will be described in detail below. However, it should be noted that the present invention may be embodied in various different forms, and thus it is intended that the subject matter included or claimed is not limited to any of the embodiments described below. It should also be noted that the present invention may be embodied as a method, device, component, or system. Accordingly, embodiments of the present invention may take the form of, for example, hardware, software, firmware, or any combination thereof.

[0031] Throughout this specification and the claims, terms may have subtle meanings suggested or implied in the context beyond the explicitly described meaning. The phrases "in one embodiment" or "in some embodiments" as used herein do not necessarily refer to the same embodiment, and the phrases "in other embodiments" or "in other some embodiments" as used herein do not necessarily refer to different embodiments. Similarly, the phrases "in one implementation" or "in some implementations" as used herein do not necessarily refer to the same implementation, and the phrases "in other implementations" or "in other some implementations" as used herein do not necessarily refer to different implementations. For example, the claimed subject matter is intended to include combinations of all or part of the exemplary embodiments / implementations.

[0032] Generally, terms can be understood at least in part from their usage in context. For example, terms such as "and," "or," or "and / or" as used herein can include a variety of meanings that can depend at least in part on the context in which such terms are used. Typically, "or" as used to associate a list such as A, B, or C is intended here to mean A, B, and C in an inclusive sense, as well as A, B, or C in an exclusive sense here. Further, the terms "one or more" or "at least one" as used herein can, at least in part, depend on context, be used to describe any feature, structure, or characteristic in a singular sense, or be used to describe a combination of features, structures, or characteristics in a plural sense. Similarly, terms such as "a," "an," or "the" can also, at least in part, depend on context, be understood to convey a singular usage or a plural usage. Further, the terms "based on" or "determined by" are understood not to necessarily convey a set of exclusive factors, and instead, can, at least in part, also depend on context and allow for the presence of additional factors that are not necessarily explicitly described.

[0033] Figure 3 shows a simplified block diagram of a communication system (300) according to an embodiment of the present disclosure. The communication system (300) includes, for example, a plurality of terminal devices that can communicate with each other via a network (350). For example, the communication system (300) includes a first pair of terminal devices (310) and (320) interconnected via the network (350). In the example of Figure 3, the first pair of terminal devices (310) and (320) can perform unidirectional transmission of data. For example, the terminal device (310) can code video data (e.g., a stream of video pictures captured by the terminal device (310)) for transmission to the other terminal device (320) via the network (350). The encoded video data can be transmitted in the form of one or more encoded video bitstreams. The terminal device (320) can receive the encoded video data from the network (350), decode the encoded video data to restore the video pictures, and display the video pictures according to the restored video data on an accessible display device. Unidirectional data transmission can be implemented in media serving applications and the like.

[0034] In other examples, the communication system (300) includes a second pair of terminal devices (330) and (340) that perform bidirectional transmission of encoded video data, which can be implemented, for example, during video conferencing applications. For bidirectional data transmission, in one example, each of the terminal devices (330) and (340) can code video data (e.g., a stream of video pictures captured by that terminal device) for transmission to the other of the terminal devices (330) and (340) via the network (350). Each of the terminal devices (330) and (340) can also receive the encoded video data transmitted by the other of the terminal devices (330) and (340), decode the encoded video data to restore the video pictures, and display the video pictures on an accessible display device according to the restored video data.

[0035] In the example of FIG. 3, the terminal devices (310), (320), (330), and (340) can be implemented as servers, personal computers, and smartphones, but the applicability of the underlying principles of the present disclosure is not so limited. Embodiments of the present disclosure can be implemented in desktop computers, laptop computers, tablet computers, media players, wearable computers, dedicated video conferencing devices, and the like. The network (350) represents any number and type of network that transmits coded video data between the terminal devices (310), (320), (330), and (340), including, for example, a wired (wired) and / or wireless communication network. The communication network (350)9 may exchange data over a circuit-switched channel, a packet-switched channel, and / or other types of channels. Representative networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For the purposes of this discussion, the architecture and topology of the network (350) may not be important for the operation of the present disclosure, unless explicitly described herein.

[0036] FIG. 4 shows the arrangement of a video encoder and a video decoder in a video streaming environment as an example of the use of the subject matter of the present disclosure. The subject matter of the present disclosure can be equally applied to other video-related applications, including, for example, video conferencing, digital television broadcasting, gaming, virtual reality, storage of compressed video on digital media including CDs, DVDs, memory sticks, and the like.

[0037] A video streaming system may include a video source (401), such as a digital camera, for creating a stream (402) of uncompressed video pictures or images, which may include a video capture subsystem (413). In one example, the stream (402) of video pictures includes samples recorded by the digital camera of the video source 401. The stream (402) of video pictures is shown in bold to emphasize the high data volume when compared to the encoded video data (404) (or encoded video bitstream), and may be processed by an electronic device (420) including a video encoder (403) coupled to the video source (401). The video encoder (403) can include hardware, software, or a combination thereof to enable or implement aspects of the subject matter of this disclosure, as described in more detail below. The encoded video data (404) (or encoded video bitstream (404)) is shown in thin lines to emphasize the low data volume when compared to the stream (402) of uncompressed video pictures, and may be stored directly in a streaming server (405) or in a downstream video device (not shown) for future use. One or more streaming client subsystems, such as client subsystems (406) and (408) of FIG. 4, can access the streaming server (405) to obtain copies (407) and (409) of the encoded video data (404). The client subsystem (406) can include, for example, a video decoder (410) within an electronic device (430). The video decoder (410) decodes an input copy (407) of the encoded video data and creates an output stream (411) of video pictures that can be rendered uncompressed on a display (412) (e.g., a display screen) or other rendering device (not shown). The video decoder 410 can be configured to perform some or all of the various functions described in this disclosure.In some streaming systems, the encoded video data (404), (407), and (409) (e.g., video bitstreams) can be encoded according to specific video coding / compression standards. Examples of such standards include ITU-T Recommendation H.265. In one example, a video coding standard under development is informally known as Versatile Video Coding (VVC). The subject matter of the present disclosure can be used in the context of VVC and other video coding standards.

[0038] Note that the electronic devices (420) and (430) can include other components (not shown). For example, the electronic device (420) can include a video decoder (not shown), and the electronic device (430) can also include a video encoder (not shown).

[0039] FIG. 5 shows a block diagram of a video decoder (510) according to any embodiment of the present disclosure below. The video decoder (510) can be included in an electronic device (530). The electronic device (530) can include a receiver (531) (e.g., receiving circuitry). The video decoder (510) can be used in place of the video decoder (410) in the example of FIG. 4.

[0040] Receiver (531) may receive one or more coded video sequences to be decoded by video decoder (510). In the same or other embodiments, one coded video sequence may be decoded at a time, and the decoding of each coded video sequence is independent of other coded video sequences. Each video sequence may be associated with a plurality of video frames or images. The coded video sequence can be received from channel (501), and channel (501) can be a hardware / software link to a storage device storing the encoded video data, or a streaming source transmitting the encoded video data. Receiver (531) may receive the encoded video data together with other data such as a coded audio data and / or an auxiliary data stream that can be transferred to respective processing circuits (not shown). Receiver (531) may separate the coded video sequence from other data. To counter network jitter, buffer memory (515) may be disposed between receiver (531) and entropy decoder / parser (520) (hereinafter referred to as "parser (520)"). For certain applications, buffer memory (515) may be implemented as part of video decoder (510). For other applications, buffer memory (515) may be external and separated from video decoder (510) (not shown). For still other applications, for example, there may be buffer memory (not shown) external to video decoder (510) to counter network jitter, and for example, there may be other additional buffer memory (515) inside video decoder (510) to handle playback timing. When receiver (531) is receiving data from a storage / transfer device with sufficient bandwidth and controllability, or from an isochronous network, buffer memory (515) may be unnecessary or can be made small.For use in a best effort packet network such as the Internet, a buffer memory (515) of sufficient size may be required, and its size can be relatively large. Such buffer memory may be implemented in an adaptive size or may be implemented at least partially in an operating system external to the video decoder (510) or a similar element (not shown).

[0041] The video decoder (510) may include a parser (520) to recover symbols (521) from the coded video sequence. The categories of those symbols include information used to manage the operation of the video decoder (510) and, potentially, information for controlling a rendering device such as a display (512) (e.g., a display screen) that may or may not be an integral part of the electronic device (530) as shown in FIG. 5 but can be coupled to the electronic device (530). The control information for the (one or more) rendering devices can be in the form of supplementary enhancement information (SEI messages) or video usability information (VUI) parameter set fragments (not shown). The parser (520) can parse / entropy-decode the coded video sequence received by the parser (520). The entropy coding of the coded video sequence can be in accordance with a video coding technology or standard and can be in accordance with various principles including variable length coding, Huffman coding, arithmetic coding with or without context dependence, etc. The parser (520) can extract a set of subgroup parameters of at least one subgroup of pixels in the video decoder based on at least one parameter corresponding to a subgroup from the coded video sequence. Subgroups can include Group of Pictures (GOP), picture, tile, slice, macroblock, coding unit (CU), block, transform unit (TU), prediction unit (PU), etc. The parser (520) can also extract information such as transform coefficients (e.g., Fourier transform coefficients), quantization parameter values, motion vectors, etc. from the coded video sequence.

[0042] The parser (520) can perform an entropy decoding / parsing operation on the video sequence received from the buffer memory (515) to create symbols (521).

[0043] The reconstruction of symbol (521) can include multiple different processing units or functional units depending on the type of the coded video picture or a portion thereof (such as inter-picture and intra-picture, inter-block and intra-block, etc.), as well as other factors. The units included and how the units are included can be controlled by subgroup control information parsed from the coded video sequence by parser (520). Such a flow of subgroup control information between parser (520) and the following multiple processing units or functional units is not illustrated for simplicity.

[0044] In addition to the functional blocks already described, video decoder (510) can be conceptually subdivided into several functional units as described below. In an actual implementation operating under commercial constraints, many of these functional units interact closely with each other and can be at least partially integrated with each other. However, for the purpose of clearly explaining the various functions of the subject matter of the present disclosure, a conceptual subdivision into functional units is adopted in the following disclosure.

[0045] The first unit can include a scaler / inverse transform unit (551). The scaler / inverse transform unit (551) can receive, as (one or more) symbols (521) from parser (520), quantization transform coefficients, as well as control information including information indicating which type of inverse transform to use, block size, quantization coefficient / parameter, quantization scaling matrix, etc. The scaler / inverse transform unit (551) can output a block including sample values that can be input to aggregator (555).

[0046] In some cases, the output samples of the scaler / inverse transform (551) may relate to blocks that are intra-coded, i.e., blocks that do not use prediction information from a previously reconstructed picture but can use prediction information from a previously reconstructed portion of the current picture. Such prediction information may be provided by the intra-picture prediction unit (552). In some cases, the intra-picture prediction unit (552) may use the information of surrounding blocks that have already been reconstructed and stored in the current picture buffer (558) to generate blocks of the same size and shape as the block being reconstructed. The current picture buffer (558) buffers, for example, a partially reconstructed current picture and / or a fully reconstructed current picture. The aggregator (555) may, in some implementations, add, for each sample, the prediction information generated by the intra prediction unit (552) to the output sample information provided by the scaler / inverse transform unit (551).

[0047] In other cases, the output samples of the scaler / inverse transform unit (551) may be related to inter-coded and potentially motion-compensated blocks. In such cases, the motion compensation prediction unit (553) can access the reference picture memory (557) to fetch the samples used for inter-picture prediction. After motion-compensating the samples fetched according to the symbols (521) related to the block, these samples can be added by the aggregator (555) to the output of the scaler / inverse transform unit (551) to generate output sample information (the output of unit 551 can be called the residual sample or residual signal). The address in the reference picture memory (557) from which the motion compensation prediction unit (553) fetches the prediction samples can be controlled by a motion vector, in the form of a symbol (521) that can have, for example, an X component, a Y component (shift), and a reference picture component (time), which is available to the motion compensation prediction unit (553). Motion compensation may also include interpolation of the sample values fetched from the reference picture memory (557) when an exact motion vector of sub-samples is used, and may be associated with a motion vector prediction mechanism or the like.

[0048] The output samples of the aggregator (555) can undergo various loop filtering techniques in the loop filter unit (556). The video compression technology is controlled by the parameters included in the coded video sequence (also called the coded video bitstream), and can include in-loop filter techniques made available to the loop filter unit (556) as symbols (521) from the parser (520), but can also react not only to the meta-information obtained during the decoding of the previous part (in decoding order) of the coded picture or coded video sequence, but also to the previously reconstructed and loop-filtered sample values. As will be explained in more detail below, several types of loop filters can be included as part of the loop filter unit 556 in various orders.

[0049] The output of the loop filter unit (556) can be output to the rendering device (512) and can be a sample stream that can be stored in the reference picture memory (557) for use in future inter-picture prediction.

[0050] When a particular coded picture is completely reconstructed, it can be used as a reference picture for future inter-picture prediction. For example, when the coded picture corresponding to the current picture is completely restored and the coded picture is identified as a reference picture (e.g., by the parser (520)), the current picture buffer (558) can become part of the reference picture memory (557), and the unused current picture buffer can be reallocated before starting the restoration of the next coded picture.

[0051] The video decoder (510) may perform a decoding operation in accordance with a predetermined video compression technique adopted in a standard such as ITU-T Rec.H.265. In the sense that the coded video sequence conforms to both the syntax of the video compression technique or standard and the profile documented in the video compression technique or standard, the coded video sequence may conform to the syntax specified by the video compression technique or standard being used. Specifically, the profile can select specific tools from all the tools available in the video compression technique or standard as tools that can only be used under that profile. To comply with the standard, the complexity of the coded video sequence may be within the range defined by the level of the video compression technique or standard. In some cases, the level limits the maximum picture size, the maximum frame rate, the maximum reconstruction sample rate (e.g., measured in megasamples per second), the maximum reference picture size, etc. The limits set by the level may, in some cases, be further restricted by the virtual reference decoder (HRD) specification and the metadata for HRD buffer management signaled in the coded video sequence.

[0052] In some exemplary embodiments, the receiver (531) may receive additional (redundant) data along with the encoded video. The additional data may be included as part of the (one or more) coded video sequences. The additional data may be used by the video decoder (510) to properly decode the data and / or to more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0053] FIG. 6 shows a block diagram of a video encoder (603) according to an exemplary embodiment of the present disclosure. The video encoder (603) may be included in an electronic device (620). The electronic device (620) may further include a transmitter (640) (e.g., a transmission circuit). The video encoder (603) may be used in place of the video encoder (403) of the example of FIG. 4.

[0054] The video encoder (603) may receive video samples from a video source (601) (not part of the electronic device (620) in the example of FIG. 6) that may capture the (one or more) video images to be coded by the video encoder (603). In another example, the video source (601) may be implemented as a part of the electronic device (620).

[0055] The video source (601) may provide the source video sequence to be coded by the video encoder (603) in the form of a digital video sample stream that can be of any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit,...), any color space (e.g., BT.601 YCrCb, RGB, XYZ,...), and any suitable sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media serving system, the video source (601) may be a storage device that can store previously prepared videos. In a video conferencing system, the video source (601) may be a camera that captures local image information as a video sequence. The video data may be provided as a plurality of individual pictures or images that give motion when viewed in sequence. The picture itself may be organized as a spatial array of pixels, and each pixel may include one or more samples depending on the sampling structure, color space, etc. being used. One skilled in the art can easily understand the relationship between pixels and samples. The following description focuses on samples.

[0056] According to some exemplary embodiments, the video encoder (603) may code and compress pictures of a source video sequence into a coded video sequence (643) in real time or under any other arbitrary time constraints required by the application. Enforcing an appropriate coding speed constitutes one function of the controller (650). In some embodiments, the controller (650) may be functionally coupled to and control other functional units, as described below. For simplicity, the couplings are not shown. Parameters set by the controller (650) may include rate control related parameters (such as picture skip, quantizer, lambda value of rate distortion optimization method, etc.), picture size, Group of Pictures (GOP) layout, maximum motion vector search range, etc. The controller (650) may be configured to have other appropriate functions related to the video encoder (603) optimized for a certain system design.

[0057] In some exemplary embodiments, the video encoder (603) may be configured to operate in a coding loop. As an overly simplified explanation, in one example, the coding loop may include a source coder (630) (e.g., which is responsible for creating symbols such as a symbol stream based on an input picture to be coded and one or more reference pictures) and a (local) decoder (633) incorporated in the video encoder (603). The decoder (633) can reconstruct the symbols and create sample data in a manner similar to that which would be created by a (remote) decoder, even if the incorporated decoder 633 processes a video stream coded by the source coder 630 without entropy coding (since in the video compression techniques contemplated by the subject matter of the present disclosure, any compression between the symbols and the coded video bitstream can be reversible). The reconstructed sample stream (sample data) is input into the reference picture memory (634). Since the decoding of the symbol stream leads to bit - accurate results regardless of the location of the decoder (local or remote), the content in the reference picture memory (634) is also bit - accurate between the local encoder and the remote encoder. In other words, the prediction part of the encoder "sees" the same sample values as reference picture samples as the decoder would "see" when using the prediction during decoding. This basic principle of reference picture synchronization (and the resulting drift if synchronization cannot be maintained, for example, due to channel errors) is used to improve coding quality.

[0058] The operation of the "local" decoder (633) can be the same as that of a "remote" decoder such as the video decoder (510), which has already been described in detail above with reference to FIG. 5. Referring briefly to FIG. 5, however, since symbols are available and the encoding / decoding of symbols to the coded video sequence by the entropy encoder (645) and the parser (520) can be reversible, the entropy decoding part of the video decoder (510) including the buffer memory (515) and the parser (520) may not be fully implemented in the local decoder (633) within the encoder.

[0059] What can be said at this point is that any decoder technology other than parsing / entropy decoding that can only exist within the decoder may also necessarily need to exist in substantially the same functional form in the corresponding encoder. For this reason, the subject matter of the present disclosure may focus on decoder operations, which are similar to the decoding part of the encoder. Therefore, the description of encoder technology can be omitted since it is the reverse of the decoder technology that is comprehensively described. A more detailed description of the encoder is provided below only in certain areas or aspects.

[0060] During operation, in some exemplary implementations, the source coder (630) may perform motion compensation prediction coding that predictively codes an input picture by referring to one or more previously coded pictures from the video sequence designated as "reference pictures". In this way, the coding engine (632) codes the difference (or residue) in color channels between a pixel block of the input picture and a pixel block of the (one or more) reference pictures that can be selected as the (one or more) prediction references to the input picture. The terms "residue" and its adjective form "residual" can be used interchangeably.

[0061] The local video decoder (633) can decode the coded video data of a picture that can be specified as a reference picture based on the symbols created by the source coder (630). The operation of the coding engine (632) may advantageously be an irreversible process. When the coded video data can be decoded by a video decoder (not shown in FIG. 6), the reconstructed video sequence can usually be a replica of the source video sequence with some errors. The local video decoder (633) can replicate the decoding process that can be executed by the video decoder for the reference picture and store the reconstructed reference picture in the reference picture cache (634). In this way, the video encoder (603) can locally store a copy of the reconstructed reference picture having the same content as the reconstructed reference picture obtained by the remote video decoder (without transmission errors).

[0062] The predictor (635) can perform a prediction search for the coding engine (632). That is, for a new picture to be coded, the predictor (635) can search the reference picture memory (634) for specific metadata such as sample data (as candidate reference pixel blocks) or reference picture motion vectors, block shapes, etc. that can serve as an appropriate prediction reference for the new picture. The predictor (635) can operate on the sample blocks for each pixel block to find an appropriate prediction reference. In some cases, the input picture can have a prediction reference drawn from a plurality of reference pictures stored in the reference picture memory (634) as determined by the search result obtained by the predictor (635).

[0063] The controller (650) can manage the coding operation of the source coder (630), including, for example, setting the parameters and subgroup parameters used for encoding the video data.

[0064] The outputs of all the aforementioned functional units can undergo entropy coding within the entropy coder (645). The entropy coder (645) converts the symbols generated by various functional units into a coded video sequence by reversibly compressing the symbols according to techniques such as Huffman coding, variable-length coding, arithmetic coding, etc.

[0065] The transmitter (640) can buffer the coded video sequence created by the entropy coder (645) to prepare for transmission via the communication channel (660), and the communication channel (660) may be a hardware / software link to a storage device that stores the encoded video data. The transmitter (640) can merge the coded video data from the video coder (603) with other data to be transmitted, such as coded audio data and / or an auxiliary data stream (the source is not shown).

[0066] The controller (650) can manage the operation of the video encoder (603). During coding, the controller (650) can assign a specific coded picture type to each coded picture, which may affect the coding techniques applicable to each picture. For example, a picture can often be assigned as one of the following picture types.

[0067] An intra picture (I picture) can be coded and decoded without using other pictures in the sequence as a prediction source. Some video codecs allow for different types of intra pictures, for example, independent decoder refresh (「IDR」) pictures. Those skilled in the art are aware of those variations of I pictures as well as their respective uses and characteristics.

[0068] A predicted picture (P picture) can be a picture that can be coded and decoded using intra prediction or inter prediction that uses at most one motion vector and a reference index to predict the sample values of each block.

[0069] A bi-directionally predicted picture (B picture) can be a picture that can be coded and decoded using intra prediction or inter prediction that uses at most two motion vectors and reference indices to predict the sample values of each block. Similarly, multiple predicted pictures can use three or more reference pictures and associated metadata for the reconstruction of a single block.

[0070] A source picture is generally spatially subdivided into a plurality of sample coding blocks (e.g., blocks of 4×4, 8×8, 4×8, or 16×16 samples each) and can be coded block by block. The blocks can be coded predictively by referring to other (already coded) blocks as determined by the coding assignment applied to each block of the picture. For example, blocks of an I picture can be coded non-predictively or predictively (spatial prediction or intra prediction) by referring to already coded blocks of the same picture. Pixel blocks of a P picture can be coded predictively via spatial prediction or via temporal prediction by referring to one previously coded reference picture. Blocks of a B picture can be coded predictively by spatial prediction or by temporal prediction by referring to one or two previously coded reference pictures. The source picture or an intermediate processed picture may be subdivided into other types of blocks for other purposes. The division of coding blocks and other types of blocks may or may not follow the same method as will be explained in more detail below.

[0071] The video encoder (603) can perform a coding operation according to a predetermined video coding technology or standard such as ITU-T Rec.H.265. In that operation, the video encoder (603) can perform various compression operations including a predictive coding operation that utilizes the temporal and spatial redundancies in the input video sequence. Thus, the coded video data can conform to the syntax specified by the video coding technology or standard being used.

[0072] In some exemplary embodiments, the transmitter (640) can transmit additional data along with the encoded video. The source coder (630) can include such data as part of the encoded video sequence. The additional data can include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures or slices, SEI messages, VUI parameter set fragments, and the like.

[0073] Video can be captured in time series as a plurality of source pictures (video pictures). Intra-picture prediction (often abbreviated as intra prediction) utilizes the spatial correlation within a given picture, and inter-picture prediction utilizes the temporal or other correlations between pictures. For example, a particular picture being encoded / decoded, called the current picture, can be divided into blocks. A block within the current picture can be coded by a vector called a motion vector if it is similar to a reference block within a reference picture that has been previously coded and buffered within the video. The motion vector points to the reference block within the reference picture and can have a third dimension that identifies the reference picture if multiple reference pictures are being used.

[0074] In some exemplary embodiments, dual prediction techniques may be used for inter-picture prediction. According to such dual prediction techniques, two reference pictures, such as a first reference picture and a second reference picture, are used, both of which advance the current picture in the video in decoding order (however, in display order, they can be in the past or future respectively). A block in the current picture can be coded by a first motion vector indicating a first reference block in the first reference picture and a second motion vector indicating a second reference block in the second reference picture. The block can be predicted cooperatively by a combination of the first reference block and the second reference block.

[0075] Furthermore, merge mode techniques may be used to improve coding efficiency in inter-picture prediction.

[0076] According to some exemplary embodiments of the present disclosure, predictions such as inter-picture prediction and intra-picture prediction are performed in block units. For example, pictures in a sequence of video pictures are divided into coding tree units (CTUs) for compression, and the CTUs in a picture may have the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. Generally, a CTU may include three parallel coding tree blocks (CTBs), namely, one luma CTB and two chroma CTBs. Each CTU may be recursively quadtree-divided into one or more coding units (CUs). For example, a 64×64 pixel CTU may be divided into one 64×64 pixel CU, or four 32×32 pixel CUs. Each of one or more of the 32×32 blocks may be further divided into four 16×16 pixel CUs. In some exemplary embodiments, each CU may be analyzed during encoding to determine the prediction type of that CU from various prediction types such as inter-prediction type or intra-prediction type. A CU may be divided into one or more prediction units (PUs) according to its temporal and / or spatial predictability. Generally, each PU includes one luma prediction block (PB) and two chroma PBs. In one embodiment, the prediction operation in coding (encoding / decoding) is performed in prediction block units. The division of a CU into PUs (or PBs of different color channels) may be performed in various spatial patterns. A luma PB or chroma PB may include a matrix of sample values (e.g., luma values), such as 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, etc.

[0077] FIG. 7 shows a diagram of a video encoder (703) according to another exemplary embodiment of the present disclosure. The video encoder (703) receives a processing block (e.g., a prediction block) of sample values within a current video picture in a sequence of video pictures, and is configured to encode the processing block into a coded picture that is part of a coded video sequence. An exemplary video encoder (703) may be used in place of the video encoder (403) of the example of FIG. 4.

[0078] For example, the video encoder (703) receives a matrix of sample values of a processing block such as an 8×8 sample prediction block. The video encoder (703) then determines, using for example rate distortion optimization (RDO), whether the processing block is best coded using an intra mode, an inter mode, or a bi-prediction mode. If it is determined that the processing block is to be coded in the intra mode, the video encoder (703) encodes the processing block into the coded picture using intra prediction techniques, and if it is determined that the processing block is to be coded in the inter mode or the bi-prediction mode, the video encoder (703) may encode the processing block into the coded picture using inter prediction techniques or bi-prediction techniques, respectively. In some exemplary embodiments, as a sub-mode of inter-picture prediction, a merge mode may be used in which one or more motion vector predictors derive motion vectors without the benefit of coded motion vector components outside the predictor. In some other exemplary embodiments, there may be motion vector components applicable to the target block. Thus, the video encoder (703) may include components not explicitly shown in FIG. 7, such as a mode decision module, to determine the prediction mode of the processing block.

[0079] In the example of FIG. 7, video encoder (703) includes an inter-encoder (730), an intra-encoder (722), a residual calculator (723), a switch (726), a residual encoder (724), a general-purpose controller (721), and an entropy encoder (725) coupled to each other as shown in the exemplary configuration of FIG. 7.

[0080] The inter-encoder (730) receives samples of the current block (e.g., processing block), compares the block with one or more reference blocks (e.g., blocks in the previous picture and the subsequent picture in display order) in the reference picture, generates inter-prediction information (e.g., description of redundant information, motion vectors, merge mode information by inter-encoding techniques), and is configured to calculate an inter-prediction result (e.g., predicted block) based on the inter-prediction information using any suitable technique. In some examples, the reference picture is a decoded reference picture decoded based on the encoded video information using a decoding unit 633 incorporated in the exemplary encoder 620 of FIG. 6 (shown as the residual decoder 728 of FIG. 7 and described in more detail below).

[0081] The intra-encoder (722) receives samples of the current block (e.g., processing block), compares the block with already-coded blocks in the same picture, generates quantized coefficients after transformation, and optionally also generates intra-prediction information (e.g., intra-prediction direction information by one or more intra-encoding techniques). The intra-encoder (722) may calculate an intra-prediction result (e.g., predicted block) based on the intra-prediction information and reference blocks in the same picture.

[0082] The general-purpose controller (721) may be configured to determine general-purpose control data and control other components of the video encoder (703) based on the general-purpose control data. In one example, the general-purpose controller (721) determines the prediction mode of a block and provides a control signal to the switch (726) based on the prediction mode. For example, when the prediction mode is the intra mode, the general-purpose controller (721) controls the switch (726) to select the intra mode result for use by the residual calculator (723), controls the entropy encoder (725) to select the intra prediction information and include the intra prediction information in the bitstream. When the prediction mode of the block is the inter mode, the general-purpose controller (721) controls the switch (726) to select the inter prediction result for use by the residual calculator (723), controls the entropy encoder (725) to select the inter prediction information and include the inter prediction information in the bitstream.

[0083] The residual calculator (723) may be configured to calculate the difference (residual data) between the received block and the prediction result for the block selected from the intra encoder (722) or the inter encoder (730). The residual encoder (724) may be configured to encode the residual data to generate a conversion coefficient. For example, the residual encoder (724) may be configured to convert the residual data from the spatial domain to the frequency domain to generate a conversion coefficient. The conversion coefficient then undergoes quantization processing to obtain a quantized conversion coefficient. In various exemplary embodiments, the video encoder (703) also includes a residual decoder (728). The residual decoder (728) is configured to perform inverse conversion to generate decoded residual data. The decoded residual data may be appropriately used by the intra encoder (722) and the inter encoder (730). For example, the inter encoder (730) can generate a decoded block based on the decoded residual data and the inter prediction information, and the intra encoder (722) can generate a decoded block based on the decoded residual data and the intra prediction information. The decoded block is appropriately processed to generate a decoded picture, and the decoded picture can be buffered in a memory circuit (not shown) and used as a reference picture.

[0084] The entropy encoder (725) may be configured to format the bitstream to include the encoded block and perform entropy coding. The entropy encoder (725) is configured to include various information in the bitstream. For example, the entropy encoder (725) may be configured to include general control data, selected prediction information (e.g., intra prediction information or inter prediction information), residual information, and other appropriate information in the bitstream. When coding a block in either the inter mode or the merge sub-mode of the bi-prediction mode, there may be no residual information.

[0085] FIG. 8 shows a diagram of an exemplary video decoder (810) according to another embodiment of the present disclosure. The video decoder (810) is configured to receive a coded picture that is part of a coded video sequence and decode the coded picture to generate a reconstructed picture. In one example, the video decoder (810) can be used in place of the video decoder (410) of the example of FIG. 4.

[0086] In the example of FIG. 8, the video decoder (810) includes an entropy decoder (871), an inter decoder (880), a residual decoder (873), a reconstruction module (874), and an intra decoder (872) coupled to each other as shown in the exemplary configuration of FIG. 8.

[0087] The entropy decoder (871) can be configured to reconstruct from the coded picture specific symbols that represent the syntax elements that the coded picture is composed of. Such symbols can include, for example, the mode in which a block is coded (e.g., intra mode, inter mode, bi-prediction mode, merge sub-mode or another sub-mode), prediction information (e.g., intra prediction information or inter prediction information) that can identify specific samples or metadata used for prediction by the intra decoder (872) or the inter decoder (880), and residual information such as in the form of quantized transform coefficients. In one example, when the prediction mode is inter mode or bi-prediction mode, the inter prediction information is provided to the inter decoder (880), and when the prediction type is intra prediction type, the intra prediction information is provided to the intra decoder (872). The residual information can undergo inverse quantization and is provided to the residual decoder (873).

[0088] The inter decoder (880) can be configured to receive inter prediction information and generate an inter prediction result based on the inter prediction information.

[0089] The intra decoder (872) may be configured to receive intra prediction information and generate a prediction result based on the intra prediction information.

[0090] The residual decoder (873) may be configured to perform inverse quantization to extract inverse quantization transform coefficients, and process the inverse quantization transform coefficients to convert the residual from the frequency domain to the spatial domain. The residual decoder (873) may also use certain control information (to include quantization parameter (QP)), and that information may be provided by the entropy decoder (871) (since this may be only a small amount of control information, the data path is not shown).

[0091] The reconstruction module (874) may be configured to combine, in the spatial domain, the residual as the output by the residual decoder (873) and the prediction result (optionally, as the output by the inter prediction module or the intra prediction module) to form a reconstructed block that forms a part of the reconstructed picture as a part of the reconstructed video. Note that other appropriate operations such as deblocking operations may be performed to improve visual quality.

[0092] Note that the video encoders (403), (603), and (703), and the video decoders (410), (510), and (810) may be implemented using any suitable technique. In some exemplary embodiments, the video encoders (403), (603), and (703), and the video decoders (410), (510), and (810) may be implemented using one or more integrated circuits. In other embodiments, the video encoders (403), (603), and (603), and the video decoders (410), (510), and (810) may be implemented using one or more processors that execute software instructions.

[0093] Turning to block partitioning for coding and decoding, a general partitioning may start from a base block or may follow a given set of rules, a specific pattern, a partitioning tree, or any partitioning structure or scheme. The partitioning may be hierarchical and recursive. After partitioning or segmenting the base block according to any of the above partitioning procedures or other procedures, a final set of partitions or coding blocks can still be obtained. Each of these partitions may be at one of various partitioning levels within the partition hierarchy and may have various shapes. Each partition may be called a coding block (CB). In each of the various exemplary partitioning implementations above, each resulting CB can be of any allowable size and partitioning level. Such a partition can have some basic coding / decoding decisions made for it, and the coding / decoding parameters can form a unit that can be optimized, determined, and signaled in an encoded video bitstream, so it is called a coding block. The highest or deepest level in the final partition represents the depth of the coding block partitioning of the tree. The coding block can be a luma coding block or a chroma coding block. The CB tree structure for each color may be called a coding block tree (CBT).

[0094] The coding blocks for all color channels may collectively be called a coding unit (CU). The hierarchical structure for all color channels may be collectively called a coding tree unit (CTU). The partitioning patterns or structures for the various color channels within a CTU may or may not be the same.

[0095] The coding split tree method used for the luma channel and the chroma channel may not be the same. In other words, the luma channel and the chroma channel may have separate coding tree structures. Further, whether the luma channel and the chroma channel use the same coding split tree structure or a different coding split tree structure, and the actual coding split tree structure to be used, may depend on whether the slice being coded is a P slice, a B slice, or an I slice. For example, in the case of an I slice, the chroma channel and the luma channel may have separate coding split tree structures or coding split tree structure modes, but in the case of a P slice or a B slice, the luma channel and the chroma channel may share the same coding split tree method. When separate coding split tree structures or modes are applied, the luma channel can be split into CBs by one coding split tree structure, and the chroma channel can be split into chroma CBs by another coding split tree structure.

[0096] In some exemplary implementations, a predetermined partitioning pattern can be applied to a base block. As shown in FIG. 9, an exemplary four-way partitioning tree may start from a first predetermined level (e.g., as a base block size, 64×64 block level or other size), and the base block may be hierarchically divided down to a predetermined lowest level (e.g., 4×4 level). For example, the base block can follow four predetermined partitioning options or patterns shown as 902, 904, 906, and 908, and the partitions represented by R can be recursively partitioned in that the same partitioning options shown in FIG. 9 can be repeated at a lower scale down to the lowest level (e.g., 4×4 level). In some implementations, additional restrictions may be applied to the partitioning method of FIG. 9. In the implementation of FIG. 9, rectangular partitions (e.g., 1:2 / 2:1 rectangular partitions) are possible but not recursive, while square partitions can be recursive. Optionally, the subsequent partitioning following FIG. 9 by recursion generates a final set of coding blocks. A coding tree depth can be further defined to indicate the depth of partitioning from the root node or root block. For example, the coding tree depth of a root node or root block of a 64×64 block may be set to 0, and after the root block is further partitioned once following FIG. 9, the coding tree depth increases by 1. The maximum level or deepest level from a 64×64 base block to a 4×4 minimum partition is 4 in the above method (starting from level 0). Such a partitioning method can be applied to one or more of the color channels. Each color channel can be independently partitioned according to the method of FIG. 9 (e.g., the partitioning pattern or option among the predetermined patterns may be determined independently for each of the color channels at each hierarchical level). Alternatively, two or more color channels may share the same hierarchical pattern tree of FIG. 9 (e.g., the same partitioning pattern or option among the predetermined patterns may be selected for two or more color channels at each hierarchical level).

[0097] FIG. 10 shows another exemplary predetermined partitioning pattern that enables the formation of a partitioning tree by recursive partitioning. As shown in FIG. 10, an exemplary 10-way partitioning structure or pattern may be predefined. The root block may start from a predetermined level (e.g., from the base block at the 128×128 level, or the 64×64 level). The exemplary partitioning structure of FIG. 10 includes various 2:1 / 1:2 and 4:1 / 1:4 rectangular partitions. The partitioning type having three sub-partitions shown at 1002, 1004, 1006, and 1008 in the second column of FIG. 10 may be referred to as a "T-shaped" partition. The "T-shaped" partitions 1002, 1004, 1006, and 1008 may be referred to as left T-shaped, upper T-shaped, right T-shaped, and lower T-shaped, respectively. In some exemplary implementations, none of the rectangular partitions of FIG. 10 can be further subdivided. A coding tree depth may be further defined to indicate the depth of partitioning from the root node or root block. For example, the coding tree depth of the root node or root block of a 128×128 block may be set to 0, and after the root block is further partitioned one more time following FIG. 10, the coding tree depth increases by 1. In some implementations, only all square partitions of 1010 may enable recursive partitioning to the next level of the partitioning tree following the pattern of FIG. 10. In other words, recursive partitioning is not possible for the square partitions of the T-shaped patterns 1002, 1004, 1006, and 1008. Optionally, the subsequent partitioning procedure following FIG. 10 by recursion generates a final set of coding blocks. Such a scheme may be applied to one or more of the color channels. In some implementations, more flexibility may be added by using partitions less than the 8×8 level. For example, in some cases, 2×2 chroma inter prediction may be used.

[0098] In some other exemplary implementations for coding block splitting, a quadtree structure may be used to divide a base block or an intermediate block into a quadtree partition. Such quadtree splitting can be applied hierarchically and recursively to any square partition. Whether a base block or an intermediate block or partition is further quadtree split can be adapted to various local characteristics of the base block or intermediate block / partition. Quadtree splitting at the picture boundary can be further adapted. An implicit quadtree splitting can be performed at the picture boundary such that the block continues to be quadtree split until its size fits within the picture boundary.

[0099] In some other exemplary implementations, a hierarchical binary splitting from a base block may be used. In such a case, a base block or an intermediate level block can be divided into two partitions. The binary splitting can be either horizontal or vertical. For example, a horizontal binary splitting can divide a base block or an intermediate block into equal left and right partitions. Similarly, a vertical binary splitting can divide a base block or an intermediate block into equal upper and lower partitions. Such binary splitting can be hierarchical and recursive. Whether the binary splitting scheme should continue, and if the scheme continues further, which of the horizontal or vertical binary splitting should be used can be determined for each of the base blocks or intermediate blocks. In some implementations, further splitting can stop at a predetermined minimum partition size (in one or both dimensions). Alternatively, further splitting can be stopped when a predetermined splitting level or depth from the base block is reached. In some implementations, the aspect ratio of the partition may be limited. For example, the aspect ratio of the partition may not be less than 1:4 (or greater than 4:1). Thus, a vertical strip partition having a vertical-to-horizontal aspect ratio of 4:1 can only be further vertically binary split into an upper partition and a lower partition each having a vertical-to-horizontal aspect ratio of 2:1.

[0100] In some further examples, as shown in FIG. 13, a ternary splitting scheme can be used to split a base block or any intermediate block. The ternary pattern may be implemented vertically as shown at 1302 in FIG. 13 or horizontally as shown at 1304 in FIG. 13. The exemplary split ratios in FIG. 13 show either vertical or horizontal as 1:2:1, although other ratios may be predefined. In some implementations, two or more different ratios may be predefined. Such a ternary splitting scheme can be used to complement a quadtree or binary tree splitting structure, and such a ternary tree split can capture an object located at the center of a block within one continuous partition, whereas a quadtree and a binary tree always split along the block center and thus split the object into separate partitions. Further, since the proposed width and height of the ternary tree split are always a power of two, no additional conversion is necessary.

[0101] The above splitting methods can be combined in any way at different splitting levels. As an example, the above-mentioned quadtree and binary tree splitting methods may be combined to split a base block into a quadtree-binary tree (QTBT) structure. In such a method, a base block or an intermediate block / partition may be either quadtree split or binary tree split, subject to a set of specified conditions if specified. A specific example is shown in FIG. 14. In the example of FIG. 14, the base block is first quadtree split into four partitions as shown by 1402, 1404, 1406, and 1408. Each of the resulting partitions is then either quadtree split into four further partitions (such as 1408) or binary split into two further partitions (e.g., either horizontally or vertically, e.g., 1402 or 1406, both of which are symmetric) at the next level or not split (such as 1404). The binary or quadtree splitting can be made recursively for square partitions as shown by the overall exemplary partition pattern of 1410 and the corresponding tree structure / representation of 1420, where solid lines represent quadtree splits and dashed lines represent binary tree splits. A flag can be used for each binary split node (non-leaf binary partition) to indicate whether the binary split is horizontal or vertical. For example, as shown in 1420 and consistent with the split structure of 1410, the flag "0" can represent a horizontal binary split and the flag "1" can represent a vertical binary split. In the case of a quadtree split partition, since the quadtree split always splits the block in both the horizontal and vertical directions to generate four sub-blocks of the same size, there is no need to indicate the split type. In some implementations, the flag "1" can represent a horizontal binary split and the flag "0" can represent a vertical binary split.

[0102] In some exemplary implementations of QTBT, the quadtree and binary split rule sets may be represented by the following predetermined parameters and their corresponding associated functions. -CTU size: The size of the root node of the quadtree (the size of the base block) -MinQTSize: The minimum allowable quadtree leaf node size -MaxBTSize: The maximum allowable binary tree root node size -MaxBTDepth: The maximum allowable binary tree depth -MinBTSize: The minimum allowable binary tree leaf node size In some exemplary embodiments of the QTBT partitioning structure, the CTU size may be set as 128×128 luma samples having two corresponding 64×64 blocks of chroma samples (when exemplary chroma subsampling is considered and used), the MinQTSize may be set as 16×16, the MaxBTSize may be set as 64×64, the MinBTSize (for both width and height) may be set as 4×4, and the MaxBTDepth may be set as 4. Quadtree partitioning can be first applied to the CTU to generate quadtree leaf nodes. The quadtree leaf nodes can have sizes ranging from its minimum allowable size of 16×16 (i.e., MinQTSize) to 128×128 (i.e., CTU size). If the node is 128×128, since its size exceeds MaxBTSize (i.e., 64×64), it will not be first divided by the binary tree. Otherwise, nodes not exceeding MaxBTSize can be divided by the binary tree. In the example of FIG. 14, the base block is 128×128. The base block can only be quad-tree partitioned according to a given set of rules. The base block has a partition depth of 0. Each of the resulting four partitions is 64×64 and does not exceed MaxBTSize, and can be further quad-tree or binary-tree partitioned at level 1. The process continues. When the binary tree depth reaches MaxBTDepth (i.e., 4), no further partitioning needs to be considered. If the width of the binary tree node is equal to MinBTSize (i.e., 4), no further horizontal partitioning needs to be considered. Similarly, if the height of the binary tree node is equal to MinBTSize, no further vertical partitioning is considered.

[0103] In some exemplary implementations, the above QTBT scheme may be configured to support the flexibility for luma and chroma to have the same QTBT structure or separate QTBT structures. For example, in the case of P slices and B slices, the luma CTB and chroma CTB within one CTU may share the same QTBT structure. However, in the case of I slices, the luma CTB can be divided into CUs by the QTBT structure, and the chroma CTB can be divided into chroma CUs by another QTBT structure. This means that CUs can be used to reference different color channels within an I slice. For example, an I slice can consist of a coding block of the luma component or coding blocks of two chroma components, and a CU within a P or B slice can consist of coding blocks of all three color components.

[0104] In some other implementations, the QTBT scheme may be complemented by the ternary scheme described above. Such an implementation may be referred to as a multi-type tree (MTT) structure. For example, in addition to the binary splitting of nodes, one of the ternary partitioning patterns in FIG. 13 may be selected. In some implementations, only square nodes may be subject to ternary splitting. An additional flag may be used to indicate whether the ternary splitting is horizontal or vertical.

[0105] The design of two-level or multi-level trees, such as QTBT implementations and QTBT implementations complemented by ternary splitting, can be mainly motivated by complexity reduction. Theoretically, the complexity of traversing the tree is T D where T represents the number of splitting types and D represents the depth of the tree. A trade-off can be made by using multiple types (T) while reducing the depth (D).

[0106] In some implementations, the CB can be further divided. For example, the CB can be further divided into a plurality of prediction blocks (PBs) for the purpose of intra-frame prediction or inter-frame prediction during the coding and decoding processes. In other words, the CB may be further divided into different sub-partitions where individual prediction decisions / configurations can be made. In parallel, the CB can be further divided into a plurality of transform blocks (TBs) for the purpose of describing the level at which the video data is transformed or inverse-transformed. The way of dividing the CB into PBs and TBs may or may not be the same. For example, each division method can be performed using its own procedure based on various characteristics of the video data, for example. The division methods of PBs and TBs may be independent in some exemplary implementations. The division methods and boundaries of PBs and TBs may be correlated in some other exemplary implementations. In some implementations, for example, the TB may be divided after the PB division, particularly, each PB may be further divided into one or more TBs after being determined following the division of the coding block. For example, in some implementations, the PB can be divided into one, two, four, or other numbers of TBs.

[0107] In some implementations, the luma and chroma channels may be processed differently in order to split a base block into coding blocks, and further into prediction blocks and / or transform blocks. For example, in some implementations, splitting of a coding block into prediction blocks and / or transform blocks may be allowed for the luma channel, but such splitting of the coding block into prediction blocks and / or transform blocks may not be allowed for the (one or more) chroma channels. In such implementations, thus, the transformation and / or prediction of luma blocks may be performed only at the coding block level. In another example, the minimum transform block sizes of the luma channel and the (one or more) chroma channels may be different. For example, a coding block of the luma channel may be allowed to be split into smaller transform blocks and / or prediction blocks than those of the chroma channel. In yet another example, the maximum depth of splitting of a coding block into transform blocks and / or prediction blocks may be different between the luma channel and the chroma channel. For example, a coding block of the luma channel may be allowed to be split into deeper transform blocks and / or prediction blocks than the (one or more) chroma channels. As a specific example, a luma coding block may be split into transform blocks of multiple sizes that can be represented by recursive splitting that goes down a maximum of two levels, and transform block shapes such as square, 2:1 / 1:2, 4:1 / 1:4, etc., and transform block sizes from 4×4 to 64×64 may be allowed. However, for chroma blocks, only the maximum possible transform blocks specified for luma blocks may be allowed.

[0108] In some exemplary implementations for splitting a coding block into PBs, the depth, shape, and / or other characteristics of the PB split may depend on whether the PB is intra-coded or inter-coded.

[0109] The splitting of a coding block (or prediction block) into transform blocks can be performed recursively or non-recursively in various exemplary ways including, but not limited to, quadtree splitting and predetermined pattern splitting, further considering the transform blocks at the boundaries of the coding block or prediction block. Generally, the resulting transform blocks may be at different splitting levels, may not be of the same size, and may not be square in shape (e.g., those blocks can be rectangles having several allowed sizes and aspect ratios). Further examples are described in more detail below in connection with FIGS. 15, 16, and 17.

[0110] However, in some other implementations, the CBs obtained via any of the above splitting methods can be used as basic or minimum coding blocks for prediction and / or transformation. In other words, no further splitting is performed for inter prediction / intra prediction and / or for transformation. For example, the CBs obtained from the above QTBT method may be used as is as the units for prediction. Specifically, such a QTBT structure eliminates the concept of multiple partition types, i.e., eliminates the separation of CU, PU, and TU, and supports higher flexibility in the CU / CB partition shape as described above. In such a QTBT block structure, the CU / CB can be either square or rectangular in shape. Such leaf nodes of QTBT are used as units for prediction and transformation processing without further splitting. This means that the CU, PU, and TU have the same block size in such a QTBT coding block structure.

[0111] The various CB splitting methods described above, as well as further splitting of the CBs into PBs and / or TBs (including no PB / TB splitting), can be combined in any way. The following specific implementations are provided as non-limiting examples.

[0112] Specific exemplary implementations of the division of coding blocks and transform blocks are described below. In such exemplary embodiments, a base block can be divided into coding blocks using a recursive quadtree division, or the above-described predetermined division patterns (such as those in FIGS. 9 and 10). At each level, whether to continue the further quadtree division of a particular partition can be determined by local video data characteristics. The resulting CBs can be at various quadtree division levels of various sizes. The decision of whether to code a picture area using inter-picture (temporal) prediction or intra-picture (spatial) prediction can be made at the CB level (or, in the case of a three-color channel, at the CU level). Each CB can be further divided into one, two, four, or other numbers of PBs according to a predetermined PB division type. Within one PB, the same prediction process may be applied, and the relevant information can be sent to the decoder on a PB basis. After obtaining the residual block by applying the prediction process based on the PB division type, the CB can be divided into TBs according to another quadtree structure similar to the coding tree of the CB. In this specific implementation, the CB or TB does not have to be limited to a square shape. Further, in this specific example, the PB may be square or rectangular in inter prediction and square only in intra prediction. The coding block can be divided, for example, into four square-shaped TBs. Each TB can be recursively divided (using quadtree division) into smaller TBs called residual quadtree (RQT).

[0113] Other exemplary implementations for splitting a base block into CB, PB, and / or TB are further described below. For example, instead of using multiple partition unit types as shown in FIGS. 9 or 10, a quadtree having a nested multi-type tree using a binary and ternary segmentation structure (e.g., QTBT with ternary segmentation as described above) may be used. The separation of CB, PB, and TB (i.e., the splitting of CB into PB and / or TB, and the splitting of PB into TB) may be waived, except when necessary for a CB having a size that is too large for the maximum transform length where the CB may require further splitting. This exemplary splitting method may be designed to support a higher flexibility of the CB splitting shape so that both prediction and transformation can be performed at the CB level without further splitting. In such a coding tree structure, the CB may have either a square or rectangular shape. Specifically, a coding tree block (CTB) may first be split by a quadtree structure. Then, the leaf nodes of the quadtree may be further split by a nested multi-type tree structure. An example of a nested multi-type tree structure using binary or ternary splitting is shown in FIG. 11. Specifically, the exemplary multi-type tree structure of FIG. 11 includes four splitting types: vertical binary split (SPLIT_BT_VER) (1102), horizontal binary split (SPLIT_BT_HOR) (1104), vertical ternary split (SPLIT_TT_VER) (1106), and horizontal ternary split (SPLIT_TT_HOR) (1108). The CB then corresponds to the leaf of the multi-type tree. In this exemplary implementation, this segmentation is used for both prediction and transformation processing without further splitting, as long as the CB is not too large for the maximum transform length. This means that in most cases, CB, PB, and TB have the same block size in a quadtree having a nested multi-type tree coding block structure. An exception occurs when the maximum supported transform length is smaller than the width or height of the color components of the CB. In some implementations, in addition to binary or ternary splitting, the nested pattern of FIG. 11 may further include quadtree splitting.

[0114] A specific example of a quadtree having a nested multi-type tree coding block structure of block partitions (including quadtree, binary, and ternary split options) for one base block is shown in FIG. 12. More specifically, FIG. 12 shows that the base block 1200 is quadtree split into four square partitions 1202, 1204, 1206, and 1208. The decision to further use the multi-type tree structure and quadtree of FIG. 11 for the split is made for each of the quadtree split partitions. In the example of FIG. 12, partition 1204 is not split further. Partition 1202 and partition 1208 each adopt another quadtree split. In the case of partition 1202, the second-level quadtree split upper left, upper right, lower left, and lower right partitions adopt the third-level splits of quadtree, the horizontal binary split 1104 of FIG. 11, non-split, and the horizontal ternary split 1108 of FIG. 11, respectively. Partition 1208 adopts another quadtree split, and the second-level quadtree split upper left, upper right, lower left, and lower right partitions adopt the third-level splits of the vertical ternary split 1106, non-split, non-split, and the horizontal binary split 1104 of FIG. 11, respectively. Two of the sub-partitions of the third-level upper left partition of 1208 are further split according to the horizontal binary split 1104 and the horizontal ternary split 1108 of FIG. 11, respectively. Partition 1206 adopts a second-level split pattern following the vertical binary split 1102 of FIG. 11 to become two partitions that are further split at the third level according to the horizontal ternary split 1108 and the vertical binary split 1102 of FIG. 11. A fourth-level split is further applied to one of them according to the horizontal binary split 1104 of FIG. 11.

[0115] In the above specific example, the maximum luma transform size may be 64×64, and the maximum chroma transform size supported may be different from that of luma, for example, 32×32. The above exemplary CB in FIG. 12 is generally not further divided into smaller PBs and / or TBs, but when the width or height of a luma coding block or a chroma coding block is larger than the maximum transform width or height, the luma coding block or the chroma coding block can be automatically divided horizontally and / or vertically in that direction to satisfy the transform size limit in that direction.

[0116] In a specific example for dividing the above base block into CBs, as described above, the coding tree method can support the ability that luma and chroma have separate block tree structures. For example, in the case of P slices and B slices, the luma CTB and the chroma CTB within one CTU can share the same coding tree structure. In the case of I slices, for example, luma and chroma can have separate coding block tree structures. When separate block tree structures are applied, the luma CTB may be divided into luma CBs by one coding tree structure, and the chroma CTB is divided into chroma CBs by another coding tree structure. This means that a CU within an I slice can consist of a coding block of the luma component or coding blocks of two chroma components, and a CU within a P slice or a B slice always consists of coding blocks of all three color components unless the video is monochrome.

[0117] If a coding block is further divided into a plurality of transform blocks, the transform blocks therein can be ordered in the bitstream according to various orders or scanning methods. Exemplary implementations for dividing a coding block or a prediction block into transform blocks, and the coding order of the transform blocks, are described in more detail below. In some exemplary implementations, as described above, the transform division may support a plurality of shapes of transform blocks having a transform block size in the range of, for example, from 4×4 to 64×64, such as 1:1 (square), 1:2 / 2:1, and 1:4 / 4:1. In some implementations, when the coding block is 64×64 or less, the transform block division may be applied only to the luma component such that the transform block size is the same as the coding block size for the chroma block. Otherwise, when the width or height of the coding block is greater than 64, both the luma coding block and the chroma coding block may be implicitly divided into multiples of transform blocks of min(W,64)×min(H,64) and min(W,32)×min(H,32), respectively.

[0118] In some exemplary implementations of transform block division, for both intra-coded blocks and inter-coded blocks, a coding block can be further divided into a plurality of transform blocks having a division depth up to a predetermined number of levels (e.g., 2 levels). The division depth and size of the transform blocks can be related. For some exemplary implementations, the mapping from the transform size of the current depth to the transform size of the next depth is shown in Table 1 below.

[0119] [Table 1]

[0120] Based on the exemplary mapping in Table 1, for a 1:1 square block, the next level of transform splitting can create four 1:1 square sub-transform blocks. The transform splitting can stop, for example, at 4×4. Thus, the transform size of the current depth of 4×4 corresponds to the same size of 4×4 at the next depth. In the example of Table 1, for a 1:2 / 2:1 non-square block, the next level of transform splitting can create two 1:1 square sub-transform blocks, and for a 1:4 / 4:1 non-square block, the next level of transform splitting can create two 1:2 / 2:1 sub-transform blocks.

[0121] In some exemplary implementations, further restrictions may apply to the transform block splitting for the luma component of an intra-coded block. For example, for each level of transform splitting, all sub-transform blocks may be restricted to have equal sizes. For example, for a coding block of 32×16, the level 1 transform splitting creates two 16×16 sub-transform blocks, and the level 2 transform splitting creates eight 8×8 sub-transform blocks. In other words, to keep the transform units of equal size, the level 2 splitting must be applied to all level 1 sub-blocks. An example of transform block splitting for an intra-coded square block according to Table 1 is shown in FIG. 15 along with the coding order indicated by the arrows. Specifically, 1502 shows a square coding block. The level 1 splitting into four equal-sized transform blocks according to Table 1 is shown at 1504 along with the coding order indicated by the arrows. The level 2 splitting of all level 1 equal-sized blocks into 16 equal-sized transform blocks according to Table 1 is shown at 1506 along with the coding order indicated by the arrows.

[0122] In some exemplary implementations, the above restrictions on intra - coding may not apply to the luma components of the inter - coded blocks. For example, after the first - level transform split, any one of the sub - transform blocks can be further independently split at another level. Thus, the resulting transform blocks may or may not be of the same size. An exemplary split into transform blocks having the coding order of the inter - coded block is shown in FIG. 16. In the example of FIG. 16, the inter - coded block 1602 is split into transform blocks at two levels according to Table 1. At the first level, the inter - coded block is split into four transform blocks of equal size. Then, only one (not all) of the four transform blocks is further split into four sub - transform blocks, resulting in a total of seven transform blocks having two different sizes, as shown at 1604. An exemplary coding order of these seven transform blocks is indicated by arrows at 1604 in FIG. 16.

[0123] In some exemplary implementations, some additional restrictions on the transform blocks may apply to the (one or more) chroma components. For example, for the (one or more) chroma components, the transform block size can be the same size as the coding block size, but cannot be made smaller than a predetermined size, such as 8×8.

[0124] In some other exemplary implementations, for coding blocks with a width (W) or height (H) greater than 64, both the luma coding block and the chroma coding block can be implicitly split into multiples of transform units of min(W,64)×min(H,64) and min(W,32)×min(H,32), respectively. Here, in the present disclosure, "min(a,b)" can return the smaller value between a and b.

[0125] FIG. 17 further shows another alternative exemplary manner for dividing a coding block or a prediction block into transform blocks. As shown in FIG. 17, instead of using recursive transform splitting, a set of predetermined splitting types may be applied to the coding block according to the transform type of the coding block. In the specific example shown in FIG. 17, one of six exemplary splitting types may be applied to divide the coding block into various numbers of transform blocks. A manner of generating such transform block splitting may be applied to either the coding block or the prediction block.

[0126] More specifically, the splitting manner of FIG. 17 provides up to six exemplary partition types for any given transform type (the transform type refers to a type of primary transform such as, for example, ADST). In this manner, a transform type may be assigned to all coding blocks or prediction blocks, for example, based on a rate-distortion cost. In one example, the transform partition type assigned to a coding block or a prediction block may be determined based on the transform type of the coding block or the prediction block. As shown by the six transform partition types illustrated in FIG. 17, a particular transform partition type may correspond to the split size and pattern of the transform blocks. A correspondence between various transform types and various transform partition types may be predefined. An example is shown below with capital letter labels indicating the transform partition types that may be assigned to a coding block or a prediction block based on a rate-distortion cost.

[0127] ·PARTITION_NONE: Assign a transform size equal to the block size.

[0128] ·PARTITION_SPLIT: Assign a transform size that is half the width of the block size and half the height of the block size.

[0129] ·PARTITION_HORZ: Allocate a conversion size with the same width as the block size and a height that is half of the block size.

[0130] ·PARTITION_VERT: Allocate a conversion size with a width that is half of the block size and the same height as the block size.

[0131] ·PARTITION_HORZ4: Allocate a conversion size with the same width as the block size and a height that is one-fourth of the block size.

[0132] ·PARTITION_VERT4: Allocate a conversion size with a width that is one-fourth of the block size and the same height as the block size.

[0133] In the above examples, all the conversion partition types shown in FIG. 17 include a uniform conversion size for the divided conversion blocks. This is merely an example and not a limitation. In some other implementations, mixed conversion block sizes may be used for the divided conversion blocks in a particular division type (or pattern).

[0134] A PB (or CB which is also called PB if it is not further divided into prediction blocks) obtained from any of the above division methods can become individual blocks for coding via either intra prediction or inter prediction. For the inter prediction of the current PB, a residual between the current block and the prediction block is generated, coded, and can be included in the coded bitstream.

[0135] Inter prediction can be performed, for example, in a single-reference mode or a multiple-reference mode. In some implementations, a skip flag may first be included in the bitstream of the current block (or at a higher level) to indicate whether the current block is inter-coded and not skipped. If the current block is inter-coded, another flag may be further included in the bitstream as a signal indicating whether a single-reference mode or a multiple-reference mode is used for the current block. In the single-reference mode, one reference block may be used to generate the predicted block of the current block. In the multiple-reference mode, two or more reference blocks may be used to generate the predicted block, for example, by weighted averaging. The multiple-reference mode may be referred to as a multi-reference mode, a two-reference mode, or a multiple-reference mode. One or more reference blocks may be identified using one or more reference frame indices and further using one or more corresponding motion vectors indicating a shift between one or more reference blocks and the current block at positions within, for example, horizontal and vertical pixels. For example, the inter-predicted block of the current block may be generated from a single reference block identified by one motion vector in a reference frame as a predicted block in the single-reference mode, whereas in the multiple-reference mode, the predicted block may be generated by weighted averaging of two reference blocks in two reference frames indicated by two motion vectors. Motion vectors may be coded in various ways and may be included in the bitstream.

[0136] In some implementations, an encoding or decoding system can maintain a decoded picture buffer (DPB). Some images / pictures may be maintained in the DPB waiting to be displayed (in the decoding system), and some images / pictures in the DPB may be used as reference frames to enable inter prediction. In some implementations, the reference frames in the DPB can be tagged as either short-term or long-term references to the current image being encoded or decoded. For example, the short-term reference frames can include, in decoding order, the frames used for inter prediction of blocks within the current frame or a predetermined number (e.g., two) of subsequent video frames closest to the current frame. The long-term reference frames can include the frames within the DPB that can be used to predict image blocks within frames that are a predetermined number of frames beyond the current frame in decoding order. Information regarding such tags for short-term and long-term reference frames is referred to as a reference picture set (RPS) and can be added to the header of each frame in the encoded bitstream. Each frame in the encoded video stream can be identified by a picture order counter (POC), which is numbered according to the playback sequence, either in an absolute manner or in relation to a picture group starting from an I-frame, for example.

[0137] In some exemplary implementations, one or more reference picture lists including identification of short-term and long-term reference frames for inter prediction may be formed based on information within the RPS. For example, a single picture reference list may be formed for uni-directional inter prediction shown as L0 reference (or reference list 0), and two picture reference lists may be formed for bi-directional inter prediction shown as L0 (or reference list 0) and L1 (or reference list 1) for each of the two prediction directions. The reference frames included in the L0 list and the L1 list may be ordered in various predetermined ways. The lengths of the L0 list and the L1 list may be signaled in the video bitstream. Uni-directional inter prediction may be either in a single-reference mode or a multiple-reference mode if the multiple references for generation of a predicted block by weighted averaging in a combined prediction mode are on the same side of the block being predicted. Bi-directional inter prediction may be only in a combined mode in that the bi-directional inter prediction includes at least two reference blocks.

[0138] In some implementations, a merge mode (MM) for inter prediction may be implemented. Generally, in the case of the merge mode, one or more of the motion vectors in the single-reference prediction of the current PB or the motion vectors in the multiple-reference prediction are not calculated and signaled independently, but may be derived from other motion vectors. For example, in an encoding system, the current motion vector of the current PB can be reduced to the difference between the current motion vector and one or more other already-encoded motion vectors (referred to as reference motion vectors). Such a difference of the motion vector rather than the entire current motion vector may be encoded and included in the bitstream and may be linked to the reference motion vector. Correspondingly, in a decoding system, the motion vector corresponding to the current PB can be derived based on the decoded difference of the motion vectors and the decoded reference motion vector linked thereto. As a specific form of the general merge mode (MM) inter prediction, such inter prediction based on the difference of the motion vectors is sometimes referred to as merge mode by motion vector difference (MMVD). Therefore, the general MM or particularly MMVD can be implemented to improve the coding efficiency by leveraging the correlation between the motion vectors associated with different PBs. For example, adjacent PBs may have similar motion vectors. In other examples, the motion vectors can be temporally (between frames) correlated for blocks that are similarly positioned / arranged in space.

[0139] In some exemplary implementations, an MM flag can be included in the bitstream during the encoding process to indicate whether the current PB is in merge mode. Additionally or alternatively, an MMVD flag can be included and signaled in the bitstream during the encoding process to indicate whether the current PB is in MMVD mode. The MM and / or MMVD flag or indicator can be provided at the PB level, CB level, CU level, CTB level, CTU level, slice level, picture level, etc. In a particular example, both an MM flag and an MMVD flag can be included for the current CU, and the MMVD flag can be signaled immediately after the skip flag and the MM flag to specify whether the MMVD mode is used for the current CU.

[0140] In some exemplary implementations of MMVD, a list of merge candidates for motion vector prediction can be formed for the predicted block. The list of merge candidates can include a predetermined number (e.g., two) of MV predictor candidate blocks whose motion vectors can be used to predict the current motion vector. The MVD candidate blocks can include blocks selected from adjacent blocks and / or temporal blocks (e.g., blocks located identically in the current frame in progress or subsequent frames) within the same frame. These options represent blocks at spatial or temporal positions relative to the current block that are likely to have a motion vector similar or identical to the current block. The size of the list of MV predictor candidates may be predetermined. For example, the list can include two candidates. In order to be on the list of merge candidates, the candidate blocks need to have, for example, the same reference frame (or multiple frames) as the current block, must exist (e.g., if the current block is near the edge of the frame, boundary checks need to be performed), must already have been encoded during the encoding process, and / or must already have been decoded during the decoding process. In some implementations, the list of merge candidates is available and, if the above conditions are met, is first filled with spatially adjacent blocks (scanned in a particular predetermined order) and then, if there is still room in the list, with temporal blocks. The adjacent candidate blocks can be selected, for example, from the blocks to the left and above the current block. The list of merge MV predictor candidates can be signaled within the bitstream.

[0141] In some implementations, the actual merge candidates that are used as reference motion vectors for predicting the motion vector of the current block may be signaled. If two candidates are included in the merge candidate list, a 1-bit flag called a merge candidate flag may be used to indicate the selection of the reference merge candidate. For a current block that is predicted in a composite mode, each of the plurality of motion vectors predicted using an MV predictor may be associated with a reference motion vector from the merge candidate list.

[0142] In some exemplary implementations of MMVD, after a merge candidate is selected and used as a base motion vector predictor for the predicted motion vector, a motion vector difference (MVD or delta MV, which represents the difference between the motion vector to be predicted and the reference candidate motion vector) may be calculated in the encoding system. Such an MVD can include information representing the magnitude of the MV difference and the direction of the MV difference, which can be signaled within the bitstream. The magnitude of the motion difference and the direction of the motion difference can be signaled in various ways.

[0143] In some exemplary implementations of MMVD, a distance index can be used to specify the magnitude information of the motion vector difference and indicate one of a set of predetermined offsets that represent a predetermined motion vector difference from the starting point (reference motion vector). Then, an MV offset corresponding to the signaled index can be added to either the horizontal or vertical component of the starting (reference) motion vector. Which of the horizontal or vertical components of the reference motion vector should be offset is determined by exemplary direction information of the MVD. An exemplary predetermined relationship between the distance index and the predetermined offset is specified in Table 2.

[0144] [Table 2]

[0145] In some exemplary implementations of MMVD, a direction index may be further signaled and used to represent the direction of the MVD relative to the reference motion vector. In some implementations, the direction may be limited to either the horizontal or vertical direction. An exemplary 2-bit direction index is shown in Table 3. In the example of Table 3, the interpretation of the MVD may vary according to the start / reference MV information. For example, if the start / reference MV corresponds to a single prediction block, or both reference frame lists correspond to a bi-prediction block that points to the same side of the current picture (i.e., the POCs of both reference pictures are greater than the POC of the current picture or both are less than the POC of the current picture), the signs in Table 3 can specify the sign (direction) of the MV offset added to the start / reference MV. If the start / reference MV corresponds to a bi-prediction block having two reference pictures on different sides of the current picture (i.e., the POC of one reference picture is greater than the POC of the current picture and the POC of the other reference picture is less than the POC of the current picture), and the difference between the reference POC in picture reference list 0 and the current frame is greater than the difference between the reference POC in picture reference list 1 and the current frame, the signs in Table 3 can specify the sign of the MV offset added to the reference MV corresponding to the reference picture in picture reference list 0, and the sign of the offset of the MV corresponding to the reference picture in picture reference list 1 can have the opposite value (opposite sign of the offset). Otherwise, if the difference between the reference POC in picture reference list 1 and the current frame is greater than the difference between the reference POC in picture reference list 0 and the current frame, the signs in Table 3 can specify that the sign of the MV offset added to the reference MV associated with picture reference list 1 and the sign of the offset to the reference MV associated with picture reference list 0 have opposite values.

[0146]

Table 3

[0147] In some exemplary implementations, the MVD may be scaled according to the difference in POC in each direction. If the difference in POC in both lists is the same, no scaling is required. Otherwise, if the difference in POC in reference list 0 is greater than that in reference list 1, the MVD of reference list 1 is scaled. If the POC difference in reference list 1 is greater than that in list 0, the MVD of list 0 may be scaled as well. When a single prediction is made for the starting MV, the MVD is added to the available or reference MV.

[0148] In some exemplary implementations of MVD coding and signaling for bidirectional composite prediction, in addition to or instead of coding and signaling two MVDs separately, symmetric MVD coding may be implemented such that only one MVD requires signaling and the other MVD can be derived from the signaled MVD. In such implementations, motion information including reference picture indices for both list 0 and list 1 is signaled. However, for example, only the MVD associated with reference list 0 is signaled and the MVD associated with reference list 1 is not signaled but derived. Specifically, at the slice level, a flag called "mvd_l1_0_flag" may be included in the bitstream to indicate whether reference list 1 is signaled within the bitstream. If this flag is 1, indicating that reference list 1 is equal to 0 (and thus not signaled), the bidirectional prediction flag called "BiDirPredFlag" may be set to 0, which means there is no bidirectional prediction. Otherwise, if mvd_l1_0_flag is 0 and the closest reference picture in list 0 and the closest reference picture in list 1 form a pair of reference pictures in the forward and reverse directions or a pair of reference pictures in the reverse and forward directions, BiDirPredFlag may be set to 1 and the reference pictures for both list 0 and list 1 are both short-term reference pictures. Otherwise, BiDirPredFlag is set to 0. A BiDirPredFlag of 1 indicates that a symmetric mode flag is additionally signaled within the bitstream. The decoder can extract the symmetric mode flag from the bitstream if BiDirPredFlag is 1. The symmetric mode flag is signaled, for example, at the CU level (if necessary), to indicate whether the symmetric MVD coding mode is being used for the corresponding CU.When the symmetric mode flag is 1, it indicates the use of the symmetric MVD coding mode, meaning that only the reference picture indexes for both list 0 and list 1 (referred to as "mvp_l0_flag" and "mvp_l1_flag") are signaled by the MVD associated with list 0 (referred to as "MVD0"), and the other motion vector difference "MVD1" should be derived rather than signaled. For example, MVD1 can be derived as -MVD0. Thus, in an exemplary symmetric MVD mode, only one MVD is signaled. In some other exemplary implementations for MV prediction, a harmonic method may be used for both single-reference mode MV prediction and composite-reference mode MV prediction, to implement the general merge mode MMVD and several other types of MV prediction. Various syntax elements can be used to signal how the MV of the current block is predicted.

[0149] For example, in the case of the single-reference mode, the following MV prediction modes can be signaled.

[0150] Use one of the motion vector predictors (MVPs) within the list indicated by the direct DRL (dynamic reference list) index without NEARMV-MVD.

[0151] Use one of the motion vector predictors (MVPs) within the list notified by the NEWMV-DRL index as a reference and apply a delta to the MVP (e.g., use MVD).

[0152] GLOBALMV - Use the motion vector based on the global motion parameters at the frame level.

[0153] Similarly, in the case of the composite-reference inter-prediction mode that uses two reference frames corresponding to the two MVs to be predicted, the following MV prediction modes can be signaled.

[0154] For each of the two MVs to be predicted, use one of the motion vector predictors (MVPs) in the list signaled by the DRL index without MVD.

[0155] NEAR_NEWMV - To predict the first of the two motion vectors, use one of the motion vector predictors (MVPs) in the list signaled by the DRL index as the reference MV with the MVD, and to predict the second of the two motion vectors, use one of the motion vector predictors (MVPs) in the list signaled by the DRL index as the reference MV together with the additionally signaled delta MV (MVD).

[0156] NEW_NEARMV - To predict the second of the two motion vectors, use one of the motion vector predictors (MVPs) in the list signaled by the DRL index as the reference MV with the MVD, and to predict the first of the two motion vectors, use one of the motion vector predictors (MVPs) in the list signaled by the DRL index as the reference MV together with the additionally signaled delta MV (MVD).

[0157] NEW_NEWMV - Use one of the motion vector predictors (MVPs) in the list signaled by the DRL index as the reference MV, and use it together with the additionally signaled delta MV to predict each of the two MVs.

[0158] GLOBAL_GLOBALMV - Use the MV from each reference based on the global motion parameters at the frame level.

[0159] Thus, the term "NEAR" refers to MV prediction that uses a reference MV without MVD as a general merge mode, while the term "NEW" refers to MV prediction that uses a reference MV and offsets it with an MVD signaled like in the MMVD mode. In the case of composite inter prediction, both the reference-based motion vector and the above-mentioned motion vector delta may be used even if they are correlated and such correlation can be utilized to reduce the amount of information required to signal the two motion vector deltas, and they may generally be different or independent between the two references. In such a situation, joint signaling of the two MVDs can be implemented and shown in the bitstream.

[0160] The above-mentioned dynamic reference list (DRL) can be used to dynamically maintain and hold a set of indexed motion vectors that are regarded as candidate motion vector predictors.

[0161] In some exemplary implementations, an optical flow-based approach can be used to improve the motion vector (MV) for each sub-block for composite prediction. In particular, the optical flow equation may be applied to formulate a least-squares problem, from which fine motions may be derived from the gradients of the composite inter prediction samples. With these fine motions, the MV for each sub-block can be improved within the predicted block, which can improve the inter prediction quality. Some coding features may be an extension of the concept of bidirectional optical flow (BDOF) since they support MV improvement when the two reference blocks have an arbitrary temporal distance to the current block.

[0162] In some implementations, four additional inter-composite modes listed below may be added: NEAR_NEARMV_OPTFLOW, NEAR_NEWMV_OPTFLOW, NEW_NEARMV_OPTFLOW, and / or NEW_NEWMV_OPTFLOW.

[0163] These modes can be referred to as optical flow modes, and the reference MV types can be defined similarly to the conventional composite modes (e.g., NEAR_NEWMV_OPTFLOW having the same reference MV type as NEAR_NEWMV). The composite prediction may be performed based on the improved MV for each sub-block instead of the original MV.

[0164] The various embodiments and / or implementations described in this disclosure may be used separately or combined in any order. Further, some, all, or any partial or whole combination of these embodiments and / or implementations may be embodied as part of an encoder and / or decoder and may be implemented in hardware and / or software. For example, they may be hard-coded in a dedicated processing circuit (e.g., one or more integrated circuits). In another example, they may be implemented by one or more processors executing a program stored on a non-transitory computer-readable medium.

[0165] For example, there may be some problems / issues related to some implementations of the signaling method for the motion vector difference, such as how the delta MV is signaled in the NEW_NEARMV mode, NEAR_NEWMV mode, or NEW_NEWMV mode. One of the problems / issues may be that the correlation of the difference between the motion vectors in the two reference lists is not utilized, thus reducing the coding / decoding efficiency and performance.

[0166] This disclosure describes various embodiments for signaling the motion vector difference (MVD or delta MV) for inter-prediction mode coding and / or decoding, addressing at least one of the above problems / issues, and achieving an efficient software / hardware implementation for improved inter-prediction mode coding / decoding.

[0167] In various embodiments, referring to FIG. 18, method 1800 decodes an inter-predicted video block. Method 1800 can include some or all of the following steps: Step 1810, i.e., receiving a coded video bitstream by a device comprising a memory storing instructions and a processor communicating with the memory; Step 1820, i.e., extracting, by the device, an inter-prediction mode and a joint delta motion vector (MV) of a current block within a current frame from the coded video bitstream; Step 1830, i.e., extracting, by the device, a flag indicating whether a first delta MV for a first reference frame and a second delta MV for a second reference frame are signaled together from the coded video bitstream; Step 1840, i.e., deriving, by the device, the first delta MV and the second delta MV based on the joint delta MV in response to the flag indicating that the first delta MV and the second delta MV are signaled together; and / or Step 1850, i.e., decoding, by the device, a current block within the current frame based on the first delta MV and the second delta MV.

[0168] In some implementations, the joint delta MV can be represented as joint_delta_mv, which can be an element indicating the joint delta MV. In some implementations, the flag indicating whether the first delta MV for the first reference frame and the second delta MV for the second reference frame are signaled together can be represented as joint_mvd_flag. In some implementations, the first reference frame may be a frame within a reference list (reference list 0), and / or the second reference frame may be a frame within another reference list (reference list 1).

[0169] In some implementations, step 1830 can include the device extracting a flag (joint_mvd_flag) indicating whether the first delta MV for the first reference frame in reference list 0 and the second delta MV for the second reference frame in reference list 1 are signaled together from the coded video bitstream.

[0170] In various embodiments of the present disclosure, the size of a block (e.g., but not limited to, a coding block, a prediction block, or a transform block) can refer to the width or height of the block. The width or height of the block can be an integer in pixel units. In various embodiments of the present disclosure, the size of the block can refer to the area size of the block. The area size of the block can be an integer calculated by multiplying the width of the block by the height of the block in pixel units. In some various embodiments of the present disclosure, the size of the block can refer to the maximum value of the width or height of the block, the minimum value of the width or height of the block, or the aspect ratio of the block. The aspect ratio of the block can be calculated as the width of the block divided by the height, or as the height of the block divided by the width.

[0171] Here, in some embodiments of the present disclosure, the "first" reference frame can refer not only to "one" reference frame, but also to the "first" reference frame among a plurality of reference frames (e.g., having the smallest index or appearing earliest in the sequence), and the "second" reference frame can refer not only to "another" reference frame, but also to the "second" reference frame among a plurality of reference frames (e.g., having the second smallest index or appearing second earliest in the sequence).

[0172] Here, in various embodiments of the present disclosure, "XYZ is signaled" can refer to XYZ being encoded into a bitstream encoded during an encoding process, and / or after the encoded bitstream is transmitted from one device to another device, "XYZ is signaled" can refer to XYZ being decoded / extracted from the encoded bitstream during a decoding process.

[0173] Here, in various embodiments of the present disclosure, the direction of a reference frame can be determined by whether the reference frame is before or after the current frame in the display order. In some implementations of the composite reference mode, when the picture order count (POC) of both reference frames for one pair of motion vectors is greater than or less than the POC of the current frame, the directions of the two reference frames are the same. Otherwise, when the POC of one reference frame is greater than the POC of the current frame and the POC of the other reference frame is less than the POC of the current frame, the directions of the two reference frames are different.

[0174] Here, in various embodiments of the present disclosure, "block" can refer to a prediction block, a coding block, a transform block, or a coding unit (CU).

[0175] Referring to step 1810, the device can be the electronic device (530) of FIG. 5 or the video decoder (810) of FIG. 8. In some implementations, the device can be the decoder (633) within the encoder (620) of FIG. 6. In some implementations, the device can be a part of the electronic device (530) of FIG. 5, a part of the video decoder (810) of FIG. 8, or a part of the decoder (633) within the encoder (620) of FIG. 6. The encoded video bitstream can be the encoded video sequence of FIG. 8, or the intermediate encoded data of FIGS. 6 or 7.

[0176] Referring to step 1820, the device may extract the inter prediction mode and the joint delta motion vector (MV) of the current block in the current frame from the coded video bitstream. The current block may be in a composite reference mode. The inter prediction mode may include one of the NEAR_NEAR mode, the NEW_NEARMV mode, the NEAR_NEWMV mode, or the NEW_NEWMV mode. The joint delta MV may sometimes be referred to as the MV difference (MVD).

[0177] Referring to step 1830, the device may extract a flag from the coded video bitstream indicating whether the first delta MV for the first reference frame and the second delta MV for the second reference frame are signaled together.

[0178] In some implementations of some inter prediction modes, the flag is encoded within the coded video bitstream, and the device may extract the flag from the coded video bitstream.

[0179] In some implementations of certain inter prediction modes, the flag is not encoded within the coded video bitstream, and the device may derive the flag according to a specific inter prediction model based on a default value. For example, the inter prediction mode of the current block is NEAR_NEARMV, and step 1830 may include determining the flag as a default value. In some implementations, the default value is 0, indicating that the first delta MV for the first reference frame and the second delta MV for the second reference frame are not signaled together.

[0180] In some implementations of the composite reference mode, a flag, which may be named joint_mvd_flag, may be sent to the device to indicate whether the deltas MV of the first reference list (reference list 0) and the second reference list (reference list 1) are signaled together.

[0181] In some other implementations, depending on the value of a flag (joint_mvd_flag) indicating that the deltas MV of reference list 0 and reference list 1 are signaled together, only one joint delta MV, which may be named joint_delta_mv, is signaled and sent to the decoder, and the deltas MV of reference list 0 and reference list 1 may be derived from the joint delta MV (joint_delta_mV). Depending on the value of a flag (joint_mvd_flag) indicating that the deltas MV of reference list 0 and reference list 1 are not signaled together, zero or one or two delta MVs may be signaled separately for reference list 0 and / or reference list 1 based on the inter prediction mode.

[0182] In some implementations, a flag value of 0 can indicate that the deltas MV of reference list 0 and reference list 1 are signaled together and only one joint delta MV is signaled and sent, and a flag value of 1 can indicate that the deltas MV of reference list 0 and reference list 1 are not signaled together and zero (or one or two) joint delta MVs can be signaled and sent. Conversely, in some other implementations, a flag value of 1 can indicate that the deltas MV of reference list 0 and reference list 1 are signaled together and only one joint delta MV is signaled and sent, and a flag value of 0 can indicate that the deltas MV of reference list 0 and reference list 1 are not signaled together and zero (or one or two) joint delta MVs can be signaled and sent.

[0183] Referring to step 1840, in response to the flag indicating that the first delta MV and the second delta MV are signaled together, the device can derive the first delta MV and the second delta MV based on the joint delta MV.

[0184] In some implementations, the inter prediction mode of the current block is NEW_NEWMV. When a flag (e.g., joint_mvd_flag) indicates that the delta MVs of reference list 0 and reference list 1 are signaled together, the delta MVs of reference list 0 and / or reference list 1 can be derived from joint_delta_mv based on the POC distances of the first and second reference frames with respect to the current frame and the directions of the two reference frames.

[0185] In some other embodiments, the inter prediction mode of the current block is NEW_NEWMV. Step 1840 may include a step of determining the first delta MV as the joint delta MV and a step of determining the second delta MV by scaling the joint delta MV according to at least one of the following: the first picture order count (POC) distance between the first reference frame and the current frame, the second POC distance between the second reference frame and the current frame, or the directional relationship between the first reference frame and the second reference frame with respect to the current frame.

[0186] In some other embodiments, the inter prediction mode of the current block is NEW_NEWMV. Step 1840 may include a step of determining the second delta MV as the joint delta MV and a step of determining the first delta MV by scaling the joint delta MV according to at least one of the following: the first picture order count (POC) distance between the first reference frame and the current frame, the second POC distance between the second reference frame and the current frame, or the directional relationship between the first reference frame and the second reference frame with respect to the current frame.

[0187] In some embodiments, the delta MV in reference list 0 (or list 1) may always be set equal to joint_delta_mv, and the delta MV in reference list 1 (or list 0) may be scaled from joint_delta_mv according to the POC distance to the current frame of the reference frame and / or the direction of the two reference frames.

[0188] In some other embodiments, the inter prediction mode of the current block is NEW_NEWMV, and step 1840 includes determining a first delta MV as the joint delta MV in response to the first absolute POC distance between the first reference frame and the current frame being greater than the second absolute POC distance between the second reference frame and the current frame, and determining a second delta MV by scaling the joint delta MV according to at least one of the following: the first POC distance between the first reference frame and the current frame, the second POC distance between the second reference frame and the current frame, or the directional relationship between the first reference frame and the second reference frame with respect to the current frame.

[0189] In some other embodiments, the inter prediction mode of the current block is NEW_NEWMV, and step 1840 includes determining a second delta MV as the joint delta MV in response to the first absolute POC distance between the first reference frame and the current frame being less than the second absolute POC distance between the second reference frame and the current frame, and determining a first delta MV by scaling the joint delta MV according to at least one of the following: the first POC distance between the first reference frame and the current frame, the second POC distance between the second reference frame and the current frame, or the directional relationship between the first reference frame and the second reference frame with respect to the current frame.

[0190] In some embodiments, if the absolute POC distance between reference list 0 (or list 1) and the current frame is greater than the absolute POC distance between reference list 1 (or list 0) and the current frame, the delta MV in reference list 0 (or list 1) having the greater absolute POC distance can be set equal to joint_delta_mv. The delta MV in reference list 1 (or list 0) which is the other one having a smaller absolute POC distance can be scaled from joint_delta_mv according to the POC distance from the current frame of the reference frame and / or the directions of the two reference frames.

[0191] In some other embodiments, the inter prediction mode of the current block is NEW_NEWMV, and step 1840 includes determining the first delta MV as the joint delta MV in response to the first absolute POC distance between the first reference frame and the current frame being equal to the second absolute POC distance between the second reference frame and the current frame, determining the second delta MV as the joint delta MV in response to the direction relationships of the first reference frame and the second reference frame with respect to the current frame being the same, and determining the second delta MV as the joint delta MV multiplied by -1 in response to the direction relationships of the first reference frame and the second reference frame with respect to the current frame being opposite.

[0192] In some embodiments, if the absolute POC distance between reference list 1 and the current frame is the same as the absolute POC distance between reference list 0 and the current frame, the delta MV in reference list 0 can be set equal to joint_delta_mv. If the directions of the two reference frames are the same, the delta MV in reference list 1 may be set equal to joint_delta_mv. On the other hand, if the directions of the two reference frames are different, the delta MV of reference list 1 is set to joint_delta_mv multiplied by -1.

[0193] In various embodiments / implementations of the present disclosure, scaling the joint delta MV to obtain a second delta MV according to at least one of a first POC distance between a first reference frame and a current frame, a second POC distance between a second reference frame and the current frame, or a directional relationship between the first reference frame and the second reference frame with respect to the current frame can include a linear scaling method, i.e., the absolute value of the scaled delta MV can be proportional to the ratio of the second POC distance divided by the first POC distance, and the sign of the scaled delta MV can be determined according to the directional relationship. For example, if the first POC distance is 4, the second POC distance is 8, and the directional relationship is the same direction for the first reference frame and the second reference frame with respect to the current frame, the second delta MV is obtained by scaling / multiplying the joint delta MV by 2 (=8 / 4), and the second delta MV has the same sign as the joint delta MV because the directional relationship is the same direction. In another example, if the first POC distance is 3, the second POC distance is -9, and the directional relationship is the opposite direction for the first and second reference frames with respect to the current frame, the second delta MV is obtained by scaling / multiplying the joint delta MV by -3 (=-9 / 3), and the second delta MV has the opposite sign to the joint delta MV because the directional relationship is the opposite direction.

[0194] In some other implementations, the inter prediction mode of the current block is NEW_NEARMV, and the second delta MV may be slightly adjusted by a predetermined weight. Step 1840 can include determining the first delta MV as the joint delta MV and determining the second delta MV by scaling the joint delta MV according to at least one of the following: a first POC distance between the first reference frame and the current frame, a second POC distance between the second reference frame and the current frame, a directional relationship between the first reference frame and the second reference frame with respect to the current frame, or a predetermined weighting factor.

[0195] In some other implementations, the inter-prediction mode of the current block is NEAR_NEWMV, and the first delta MV may be slightly adjusted by a predetermined weight. Step 1840 may include determining the second delta MV as the joint delta MV and determining the first delta MV by scaling the joint delta MV according to at least one of the following: the first POC distance between the first reference frame and the current frame, the second POC distance between the second reference frame and the current frame, the directional relationship between the first reference frame and the second reference frame with respect to the current frame, or a predetermined weighting coefficient.

[0196] In some other embodiments, the predetermined weighting coefficient is a fraction between -1 and 1.

[0197] In some other implementations, the predetermined weighting coefficient is signaled in a high-level syntax including at least one of a sequence parameter set (SPS), a video parameter set (VPS), a picture parameter set (PPS), a picture header, a tile header, a slice header, a frame header, a coding tree unit (CTU) header, or a superblock header.

[0198] In some embodiments, when the current block is in the NEW_NEAR mode (or NEAR_NEW mode) and the joint_mvd_flag indicates that the delta MVs of reference list 0 and reference list 1 are signaled together, the delta MV of reference list 0 (or list 1) may be set equal to joint_delta_mv, and the delta MV of list 1 (or list 0) is scaled from joint_delta_mv based on one or more coded information including, but not limited to, the POC distance from the current frame of the reference frame, the direction of the two reference frames, the difference between the MV predictors of the two MVs, and / or a predetermined weighting factor w. In some implementations, this predetermined weighting factor may be a number between -1 and 1, for example 1 / 2. In some other implementations, this predetermined weighting factor may be signaled in a high-level syntax including, but not limited to, SPS, VPS, PPS, picture header, tile header, slice header, frame header, CTU (or superblock) header.

[0199] In various embodiments / aspects of the present disclosure, the step of scaling the joint delta MV to obtain a delta MV weighted according to at least one of the following, namely, the first POC distance between the first reference frame and the current frame, the second POC distance between the second reference frame and the current frame, the directional relationship of the first reference frame and the second reference frame with respect to the current frame, or a predetermined weighting coefficient, can include a linear scaling method, that is, the absolute value of the scaled delta MV is proportional to the ratio of the second POC distance divided by the first POC distance and can then be multiplied by a predetermined weighting coefficient. The sign of the scaled delta MV can be determined according to the directional relationship. For example, when the first POC distance is 4, the second POC distance is 8, the directional relationship is the same direction for the first and second reference frames with respect to the current frame, and the predetermined weighting coefficient is 1 / 2, the weighted delta MV is obtained by scaling / multiplying the joint delta MV by a total coefficient of 1, which is calculated by multiplying the coefficient 2 (=8 / 4) by the weighting coefficient 1 / 2. Since the directional relationship is the same direction, the weighted delta MV has the same sign as the joint delta MV. In another example, when the first POC distance is 3, the second POC distance is -9, the directional relationship is the opposite direction for the first and second reference frames with respect to the current frame, and the predetermined weighting coefficient is 1 / 2, the weighted delta MV is obtained by scaling / multiplying the joint delta MV by a total coefficient of -3 / 2, and the total coefficient is calculated by multiplying the coefficient -3 (=-9 / 3) by the weighting coefficient 1 / 2. The weighted delta MV has the opposite sign to the joint delta MV because the directional relationship is the reverse direction.

[0200] Embodiments of the present disclosure may be used separately or may be combined in any order. Any steps and / or operations in any embodiment of the present disclosure may be combined or arranged in any quantity or order as needed. Two or more of the steps and / or operations in any embodiment of the present disclosure may be performed in parallel. Further, each of the methods (or embodiments), encoders, and decoders may be implemented by a processing circuit (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored on a non-transitory computer-readable medium. Embodiments of the present disclosure may be applied to luma blocks or chroma blocks.

[0201] The techniques described above may be implemented as computer software using computer-readable instructions and may be physically stored on one or more computer-readable media. For example, FIG. 19 shows a computer system (2000) suitable for implementing certain embodiments of the disclosed subject matter.

[0202] The computer software can be coded using any suitable machine code or computer language that can undergo mechanisms such as assembly, compilation, linking, etc., and can create code containing instructions that can be executed directly, or via interpretation, microcode execution, etc., by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc.

[0203] The instructions may be executed on various types of computers or their components, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet-of-things devices, etc.

[0204] The components shown in FIG. 19 with respect to the computer system (2000) are illustrative in nature and are not intended to suggest any limitation as to the use or functionality of the computer software implementing the embodiments of the present disclosure. The configuration of the components should not be construed as having any dependency or requirement with respect to any one or combination of the components shown in the exemplary embodiments of the computer system (2000).

[0205] The computer system (2000) may include a specific human interface input device. Such a human interface input device may respond to input by one or more human users via, for example, tactile input (e.g., keystrokes, swipes, movement of a data glove), audio input (e.g., voice, clapping), visual input (e.g., gesture), olfactory input (not shown). The human interface device may also be used to capture a specific medium that is not necessarily directly involved in conscious input by a person, such as audio (utterance, music, ambient sound, etc.), images (scanned images, photographic images obtained by a still image camera, etc.), video (2D video, 3D video including stereoscopic video, etc.).

[0206] The input human interface device may include one or more (only one of each shown) of a keyboard (2001), a mouse (2002), a track pad (2003), a touch screen (2010), a data glove (not shown), a joystick (2005), a microphone (2006), a scanner (2007), a camera (2008).

[0207] The computer system (2000) may also include certain human interface output devices. Such human interface output devices can stimulate the senses of one or more human users, for example, by tactile output, sound, light, and smell / taste. Such human interface output devices include tactile output devices (e.g., tactile feedback by a touch screen (2010), a data glove (not shown), or a joystick (2005), although there may also be tactile feedback devices that do not function as input devices), audio output devices (e.g., speakers (2009), headphones (not shown)), visual output devices (e.g., screens (2010) including CRT screens, LCD screens, plasma screens, OLED screens, each with or without a touch screen input function and each with or without a tactile feedback function, some of which can provide 2D visual output, or stereoscopic image output, virtual reality glasses (not shown), holographic displays, and means such as a smoke tank (not shown) that can provide output in four or more dimensions), and may also include a printer (not shown).

[0208] The computer system (2000) may also include human-accessible storage devices and their associated media, such as optical media including CD / DVD ROM / RW (2020) with media (2021) such as CD / DVD, thumb drives (2022), removable hard drives or solid state drives (2023), legacy magnetic media such as tapes and floppy disks (not shown), and dedicated ROM / ASIC / PLD-based devices such as security dongles (not shown), etc.

[0209] One of ordinary skill in the art should also understand that the term "computer-readable medium" as used in connection with the present disclosure does not include transmission media, carrier waves, or other transient signals.

[0210] The computer system (2000) can also include an interface (2054) to one or more communication networks (2055). The network can be, for example, wireless, wired, optical. The network can further be local, wide area, metropolitan, vehicle and industrial, real-time, delay tolerant, etc. Examples of networks include local area networks such as Ethernet, wireless LANs, cellular networks including GSM, 3G, 4G, 5G, LTE, etc., wired or wireless wide area digital networks for television including cable television, satellite television, and terrestrial television, vehicle and industrial including CAN bus, etc. A particular network typically requires an external network interface adapter connected to a particular general-purpose data port or peripheral bus (2049) (e.g., the USB port of the computer system (2000)), and others are generally integrated into the core of the computer system (2000) by connecting to the system bus as described below (e.g., an Ethernet interface for a PC computer system, or a cellular network interface for a smartphone computer system). Using any of these networks, the computer system (2000) can communicate with other entities. Such communication can be, for example, one-way reception only (e.g., television broadcast), one-way transmission only (e.g., CAN bus to a particular CANbus device), or two-way with respect to other computer systems using local or wide area digital networks. Particular protocols and protocol stacks can be used for each of those networks and network interfaces as described above.

[0211] The aforementioned human interface device, human-accessible storage device, and network interface can be attached to the core (2040) of the computer system (2000).

[0212] The core (2040) can include one or more central processing units (CPUs) (2041), a graphics processing unit (GPU) (2042), a dedicated programmable processing device in the form of a field programmable gate array (FPGA) (2043), a hardware accelerator for specific tasks (2044), a graphics adapter (2050), and the like. These devices can be connected via a system bus (2048) together with a read-only memory (ROM) (2045), a random access memory (2046), an internal large-capacity storage such as an internal non-user-accessible hard drive, SSD (2047). In some computer systems, the system bus (2048) can be made accessible in the form of one or more physical plugs to allow for expansion by additional CPUs, GPUs, etc. Peripheral devices can be attached directly to the core's system bus (2048) or via a peripheral bus (2049). In one example, a screen (2010) can be connected to a graphics adapter (2050). The architecture of the peripheral bus includes PCI, USB, and the like.

[0213] The CPU (2041), GPU (2042), FPGA (2043), and accelerator (2044) can execute specific instructions that can be combined to form the aforementioned computer code. That computer code can be stored in the ROM (2045) or RAM (2046). Also, temporary data can be stored in the RAM (2046), and persistent data can be stored, for example, in the internal large-capacity storage (2047). Fast storage and retrieval to any of the memory devices can be enabled by the use of cache memory that can be closely associated with one or more CPUs (2041), GPUs (2042), large-capacity storage (2047), ROM (2045), RAM (2046), etc.

[0214] A computer-readable medium can have computer code for performing various computer-implemented operations. The medium and the computer code may be specially designed and configured for the purposes of the present disclosure, or may be of the kinds available to and well known to those of ordinary skill in the computer software arts.

[0215] As a non-limiting example, a computer system (2000) having an architecture, and in particular a core (2040), can provide functionality as a result of one or more processors (including, e.g., a CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media can be associated with user-accessible mass storage as described above, as well as media associated with specific storage of the core (2040) that is non-transitory in nature, such as core internal mass storage (2047) and ROM (2045). The software implementing various embodiments of the present disclosure can be stored on such devices and executed by the core (2040). The computer-readable media can include one or more memory devices or chips, as needed. The software can cause the core (2040), specifically the processor(s) therein (including, e.g., a CPU, GPU, FPGA, etc.), to define data structures stored in RAM (2046) and modify such data structures according to processes defined by the software, thereby executing specific processes or specific portions of specific processes described herein. Additionally or alternatively, the computer system can provide functionality as a result of logic hardwired or otherwise embodied in a circuit (e.g., an accelerator (2044)), which can operate instead of or in conjunction with software to execute specific processes or specific portions of specific processes described herein. Optionally, references to software can include logic, and vice versa. Optionally, references to computer-readable media can include circuits (such as integrated circuits (ICs)) that store software for execution, circuits that embody logic for execution, or both. The present disclosure encompasses any suitable combination of hardware and software.

[0216] Although a particular invention has been described with reference to exemplary embodiments, this description is not intended to be limiting. Reading this description will make various modifications of the exemplary embodiments of the present invention and additional embodiments apparent to those skilled in the art. It will be readily understood by those skilled in the art that various other modifications can be made to the exemplary embodiments illustrated and described herein without departing from the spirit and scope of the present invention. Accordingly, the appended claims are intended to cover any such modifications and alternative embodiments. Certain portions within the figures may be exaggerated and other portions may be minimized. Therefore, the present disclosure and the drawings should be regarded as illustrative rather than limiting.

[0217] The following is a list of acronyms, some of which may appear in the present disclosure. JEM: Joint Exploration Model VVC: Versatile Video Coding BMS: Benchmark Set MV: Motion Vector HEVC: High Efficiency Video Coding SEI: Supplementary Enhancement Information VUI: Video User Information GOP: Group of Pictures TU: Transform Unit PU: Prediction Unit CTU: Coding Tree Unit CTB: Coding Tree Block PB: Prediction Block HRD: Hypothetical Reference Decoder SNR: Signal-to-Noise Ratio CPU: Central Processing Unit GPU: Graphics Processing Unit CRT: Cathode Ray Tube LCD: Liquid Crystal Display OLED: Organic Light-Emitting Diode CD: Compact Disc DVD: Digital Video Disc ROM: Read-Only Memory RAM: Random Access Memory ASIC: Application Specific Integrated Circuit PLD: Programmable Logic Device LAN: Local Area Network GSM: Global System for Mobile Communications LTE: Long Term Evolution CANBus: Controller Area Network Bus USB: Universal Serial Bus PCI: Peripheral Component Interconnect FPGA: Field Programmable Gate Array SSD: Solid State Drive IC: Integrated Circuit HDR: High Dynamic Range SDR: Standard Dynamic Range JVET: Joint Video Exploration Team MPM: Most Probable Mode WAIP: Wide Angle Intra Prediction CU: Coding Unit PU: Prediction Unit TU: Transform Unit CTU: Coding Tree Unit PDPC: Position Dependent Prediction Combination ISP: Intra Sub - partition SPS: Sequence Parameter Set PPS: Picture Parameter Set APS: Adaptive Parameter Set VPS: Video Parameter Set DPS: Decoding Parameter Set ALF: Adaptive Loop Filter SAO: Sample Adaptive Offset CC - ALF: Cross Component Adaptive Loop Filter CDEF: Constraint Directed Enhancement Filter CCSO: Cross Component Sample Offset LSO: Local Sample Offset LR: Loop Restoration Filter AV1: AOMedia Video 1 AV2: AOMedia Video 2 MVD: Motion Vector Difference CfL: Chroma from Luma SDT: Semi-Detached Tree SDP: Semi-Detached Partition SST: Semi-Detached Tree SB: Super Block IBC (or IntraBC): Intra Block Copy CDF: Cumulative Distribution Function SCC: Screen Content Coding GBI: Generalized Bi-Prediction BCW: Bi-Prediction with CU-Level Weight CIIP: Combined Intra-Inter Prediction POC: Picture Order Count RPS: Reference Picture Set DPB: Decoded Picture Buffer MMVD: Merge Mode with Motion Vector Difference

Explanation of Signs

[0218] 101 Samples 102 Arrow 103 Arrow 104 Block 201 Current Block 300 Communication System 310 Terminal Device 320 Terminal Device 330 Terminal Device 340 Terminal Device 350 Network 400 Communication System 401 Video Source 402 Video Picture or Image Stream 403 Video Encoder 404 Encoded Video Data 405 Streaming Server 406 Client Subsystem Copy of Encoded Video Data 408 Client Subsystem Copy of Encoded Video Data 410 Video Decoder 411 Output Stream of Video Picture 412 Display 413 Video Capture Subsystem 420 Electronic Device 430 Electronic Device 501 Channel 510 Video Decoder 512 Display 515 Buffer Memory 520 Entropy Decoder / Parser 521 Symbol 530 Electronic Device 531 Receiver 551 Scaler / Inverse Transformation Unit 552 Intra-Picture Prediction Unit 553 Motion Compensation Prediction Unit 555 Aggregator 556 Loop Filter Unit 557 Reference Picture Memory 558 Current Picture Buffer 601 Video Source 603 Video Encoder 620 Electronic Device 630 Source Coder 632 Coding Engine 633 Decoder 634 Reference Picture Memory 635 Predictor 640 Transmitter 643 Encoded Video Sequence 645 Entropy Encoder 650 Controller 660 Communication Channel 703 Video Encoder 721 General-Purpose Controller 722 Intra Encoder 723 Residual Calculator 724 Residual Encoder 725 Entropy Encoder 726 Switch 728 Residual Decoder 730 Inter Encoder 810 Video Decoder 871 Entropy Decoder 872 Intra Decoder 873 Residual Decoder 874 Reconstruction Module 880 Inter Decoder 902 Partition Option or Pattern 904 Partition Option or Pattern 906 Partition Option or Pattern 908 Partition Option or Pattern 1002 Sub - Partition 1004 Sub - Partition 1006 Sub - Partition 1008 Sub - Partition 1200 Base Block 1202 Square Partition 1204 Square Partition 1206 Square Partition 1208 Square Partition 1402 Partition 1404 Partition 1406 Partition 1408 Partition 1502 Square Coding Block 1504 Transformation Block 1506 Transformation Block 1602 Inter - Coded Block 1604 Inter - Coded Block 2000 Computer System 2001 Keyboard 2002 Mouse 2003 Trackpad 2005 Joystick 2006 Microphone 2007 Scanner 2008 Camera 2009 Speaker 2010 Touch Screen 2020 CD / DVD ROM / RW 2021 Media 2022 Thumb Drive 2023 Removable Hard Drive or Solid State Drive 2040 Core 2041 Central Processing Unit (CPU) 2042 Graphics Processing Unit (GPU) 2043 Field Programmable Gate Array (FPGA) 2044 Hardware Accelerator 2045 Read Only Memory (ROM) 2046 Random Access Memory 2047 Internal Mass Storage 2048 System Bus 2049 General Purpose Data Port or Peripheral Bus 2050 Graphics Adapter 2054 Interface 2055 Communication Network

Claims

1. A method for encoding video data, performed by a device having a processor, comprising: determining whether an inter prediction mode and a delta motion vector (MV) should be applied to a current block within a current frame of the video data; generating a flag (joint_mvd_flag) indicating whether a first delta MV for a first reference frame in Reference List 0 and a second delta MV for a second reference frame in Reference List 1 are jointly signaled for the current block; encoding the flag in a bitstream for the video data; A method comprising:

2. When it is determined that the inter prediction mode of the current block is NEW_NEWMV and the first delta MV and the second delta MV are signaled together, determining the first delta MV as a joint delta MV; the first delta MV; and a first Picture Order Count (POC) distance between the first reference frame and the current frame; a second POC distance between the second reference frame and the current frame; A directional relationship between the first reference frame and the second reference frame with respect to the current frame. determining the second delta MV based on at least one of The method of claim 1 further comprising:

3. The method of claim 2, further comprising a step of signaling the joint delta MV within the bitstream for the video data.

4. When it is determined that the inter prediction mode of the current block is NEW_NEWMV, and the first delta MV and the second delta MV are signaled together, and a first absolute POC distance between the first reference frame and the current frame is greater than a second absolute POC distance between the second reference frame and the current frame, determining the first delta MV as a joint delta MV; The joint delta MV, a first POC distance between the first reference frame and the current frame; a second POC distance between the second reference frame and the current frame; A directional relationship between the first reference frame and the second reference frame with respect to the current frame. determining the second delta MV by scaling it according to at least one of: The method of claim 1 further comprising:

5. When it is determined that the inter prediction mode of the current block is NEW_NEWMV, and the first delta MV and the second delta MV are signaled together, and a first absolute POC distance between the first reference frame and the current frame is smaller than a second absolute POC distance between the second reference frame and the current frame, determining the second delta MV as a joint delta MV; The joint delta MV, a first POC distance between the first reference frame and the current frame; a second POC distance between the second reference frame and the current frame; A directional relationship between the first reference frame and the second reference frame with respect to the current frame. determining the first delta MV by scaling it according to at least one of: The method of claim 1 further comprising:

6. When it is determined that the inter prediction mode of the current block is NEW_NEWMV and the first delta MV and the second delta MV are signaled together, determining the first delta MV as the joint delta MV if a first absolute POC distance between the first reference frame and the current frame is equal to a second absolute POC distance between the second reference frame and the current frame; determining the second delta MV as the joint delta MV in response to the first and second reference frames having the same orientation relative to the current frame; determining the second delta MV as the joint delta MV multiplied by −1 in response to the directional relationships of the first and second reference frames relative to the current frame being opposite; The method of claim 2, comprising:

7. When it is determined that the inter prediction mode of the current block is NEAR_NEWMV and the first delta MV and the second delta MV are signaled together, determining the first delta MV as the joint delta MV; The joint delta MV, a first POC distance between the first reference frame and the current frame; a second POC distance between the second reference frame and the current frame; a directional relationship between the first reference frame and the second reference frame with respect to the current frame; Predetermined weighting factors determining the second delta MV by scaling it according to at least one of:

3. The method of claim 2, comprising:

8. The method of claim 7, wherein the inter prediction mode of the current block is NEAR_NEWMV and the first delta MV and the second delta MV are signaled together. determining the second delta MV as the joint delta MV; The joint delta MV, a first POC distance between the first reference frame and the current frame; a second POC distance between the second reference frame and the current frame; a directional relationship between the first reference frame and the second reference frame with respect to the current frame; Predetermined weighting factors determining the first delta MV by scaling it according to at least one of:

3. The method of claim 2, comprising:

9. The method of claim 8, wherein the predetermined weighting coefficient is a fraction between -1 and 1.

10. The method described in claim 8, wherein the predetermined weighting coefficients are signaled in high-level syntax including at least one of a sequence parameter set (SPS), a video parameter set (VPS), a picture parameter set (PPS), a picture header, a tile header, a slice header, a frame header, a coding tree unit (CTU) header, or a superblock header.

11. The method of claim 1, comprising a step of determining the flag as a default value when it is determined that the inter prediction mode of the current block is NEAR_NEARMV and the first delta MV and the second delta MV are signaled together.

12. The method described in claim 11, wherein the default value is 0 indicating that the first delta MV for the first reference frame and the second delta MV for the second reference frame are not signaled together.

13. An apparatus for encoding an inter-predicted video block, comprising: a memory for storing instructions; a processor in communication with the memory, the processor configured, when executing the instructions, to cause the apparatus to perform the method of any one of claims 1 to 12; An apparatus comprising:

14. A computer program for causing a computer to carry out the method according to any one of claims 1 to 12.

Images