Context design for encoding and decoding syntax elements in inter prediction mode - Patent Application 20070122997

Abstract

[0001] The present disclosure relates generally to video coding, and more particularly to methods and systems for determining contexts for encoding and decoding various syntax elements of a video stream in inter-prediction mode. Various example implementations are disclosed for limiting the number of possible coding contexts for syntax elements related to inter-prediction mode and for selecting an coding context for a current block based on coding information and temporal motion prediction information of spatially neighboring blocks.

Description

[Technical Field]

[0001] Incorporation by Reference

[0001] This application is based on and claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 420,071, filed October 27, 2022, and U.S. Provisional Patent Application No. 63 / 419,689, filed October 26, 2022, both entitled "Inter Mode Context Simplification," and is based on and claims the benefit of priority to U.S. Non-Provisional Patent Application No. 18 / 188,041, filed March 22, 2023, both of which are incorporated by reference in their entireties.

[0002] FIELD

[0002] The present disclosure relates generally to video coding, and more particularly to methods and systems for determining contexts for encoding and decoding various syntax elements of a video stream in inter prediction mode. [Background technology]

[0003]

[0003] This background discussion provided herein is intended to generally present the context for this disclosure. Work of the inventors currently identified to the extent that their work is described in this background section, and aspects of the discussion that may not otherwise be considered prior art at the time of filing this application, are not expressly or implicitly admitted to be prior art to the present disclosure.

[0004]

[0004] By way of introduction, one goal of video encoding and decoding can be the reduction of redundancy in an uncompressed input video signal through compression. Compression can, in some cases, help reduce the aforementioned bandwidth and / or storage space requirements by two or more orders of magnitude. Both lossless and lossy compression, as well as combinations thereof, can be employed. Lossless compression refers to a technique in which an exact copy of the original signal can be reconstructed from the compressed original signal through a decoding process. Lossy compression refers to an encoding / decoding process in which the original video information is not fully preserved during encoding and therefore cannot be fully restored during decoding. When using lossy compression, the reconstructed signal may not be identical to the original signal, but even with some information loss, the distortion between the original and reconstructed signal is small enough to make the reconstructed signal useful for the intended application. In the case of video, lossy compression is widely adopted in many applications. The amount of tolerable distortion depends on the application. For example, users of certain consumer video streaming applications can tolerate greater distortion than users of movie or television broadcasting applications. The compression ratio achievable by a particular encoding algorithm can be selected or adjusted to reflect different distortion tolerances; generally, the more distortion that can be tolerated, the more lossy and the greater the compression ratio that is possible with an encoding algorithm.

[0005] Video encoders and decoders can utilize techniques from several broad categories and processes, including, for example, motion compensation, Fourier transform, quantization, and entropy coding.

[0006]

[0006] Video codec technology can include a technique known as intra-coding. In intra-coding, sample values ​​are represented without reference to samples or other data from previously reconstructed reference pictures. In some video codecs, a picture is spatially subdivided into blocks of samples. When all blocks of samples are coded in intra mode, the picture can be called an intra-picture. Intra-pictures and derivatives of intra-pictures, such as independent decoder refresh pictures, can be used to reset the decoder state and can therefore be used as the first picture in a coded video bitstream and video session, or as a still image. The samples of the block after intra-prediction can then be transformed into the frequency domain, and the transform coefficients thus generated can be quantized before entropy coding. Intra-prediction refers to a technique that minimizes sample values ​​in the pre-transform domain. In some cases, the smaller the DC value and the smaller the AC coefficients after the transformation, the fewer bits are required to represent the block after entropy coding at a given quantization step size.

[0007]

[0007] There can be many different forms of intra prediction. When two or more such techniques are available within a given video coding technique, the technique in use can be referred to as an intra prediction mode. In a particular codec, one or more intra prediction modes may be provided. In certain cases, a mode can have sub-modes and / or can be associated with various parameters, and the mode / sub-mode information and the intra coding parameters of a video block can be coded separately or collectively included in a mode codeword. Which codeword to use for a given mode, sub-mode, and / or parameter combination can also affect coding efficiency gains through intra prediction, and therefore, entropy coding techniques can be used to turn the codeword into a bitstream.

[0008]

[0008] In general, for intra-prediction, a predictor block can be formed using available neighboring sample values. For example, available values ​​of a specific set of neighboring samples along a specific direction and / or line can be copied to the predictor block. A reference to the direction in use can be coded in the bitstream or can be predicted itself.

[0009] 1A, a subset of nine predictor directions specified by the 33 possible intra predictor directions of H.265 (corresponding to the 33 angle modes of the 35 intra modes specified in H.265) is depicted in the lower right. The point (101) where the arrows converge represents the sample to be predicted. The arrows represent the direction in which neighboring samples are used to predict the sample at 101. For example, arrow (102) indicates that sample (101) is predicted from one or more neighboring samples located to the upper right and at an angle of 45 degrees from horizontal. Similarly, arrow (103) indicates that sample (101) is predicted from one or more neighboring samples located to the lower left of sample (101) and at an angle of 22.5 degrees from horizontal.

[0010]

[0010] With further reference to FIG. 1A, a square block (104) of 4x4 samples (indicated by a thick dotted line) is depicted at the upper left. The square block (104) contains 16 samples, each labeled with an "S," the block's position in the Y dimension (e.g., row index), and the block's position in the X dimension (e.g., column index). For example, sample S21 is the second sample (from the top) in the Y dimension and the first sample (from the left) in the X dimension. Similarly, sample S44 is the fourth sample in block (104) in both the Y and X dimensions. Because the block is 4x4 samples in size, S44 is located at the bottom right. An example reference sample is also shown following a similar numbering scheme. The reference sample is labeled R, the sample's Y position (e.g., row index) and X position (column index) relative to block (104). In both H.264 and H.265, prediction samples that are adjacent to the block being reconstructed are used.

[0011]

[0011] Intra-picture prediction of block 104 may begin by copying reference sample values ​​from adjacent samples according to the signaled prediction direction. For example, assume that the coded video bitstream includes signaling indicating the prediction direction of the arrow (102) for this block 104—that is, the sample is predicted from one or more prediction samples located to the upper right and at a 45-degree angle from the horizontal. In such a case, samples S41, S32, S23, and S14 are predicted from the same reference sample R05. Thus, sample S44 is predicted from reference sample R08.

[0012] In certain cases, especially when the direction is not evenly divisible by 45 degrees, the values ​​of multiple reference samples may be combined, for example through interpolation, to calculate the reference sample.

[0013]

[0013] Figure 1B shows a schematic diagram (180) depicting 65 intra prediction directions according to JEM to illustrate the increasing number of prediction directions in various encoding techniques developed over time.

[0014]

[0014] Inter-picture prediction or inter-prediction may be based on motion compensation. In motion compensation, sample data (reference picture) from a previously reconstructed picture or part thereof may be used for predicting a newly reconstructed picture or part of a picture (e.g., a block) after being spatially shifted in a direction indicated by a motion vector (hereinafter, MV). In some cases, the reference picture may be the same as the picture currently being reconstructed. The MV may have two dimensions, X and Y, or three dimensions, with the third dimension indicating the reference picture in use (similar to a temporal dimension).

[0015]

[0015] In some video compression techniques, the current MV applicable to a particular area of ​​sample data can be predicted from other MVs, such as other MVs related to other areas of sample data that are spatially adjacent to the area being reconstructed and precede the current MV in decoding order. Doing so can substantially reduce the total amount of data required to encode the MV by relying on the elimination of redundancy in correlated MVs, thereby improving compression efficiency. For example, when encoding an input video signal derived from a camera (known as raw video), there is a statistical possibility that an area larger than the area to which a single MV is applicable moves in a similar direction in the video sequence, and therefore, in some cases, MV prediction can work effectively because it can be predicted using similar motion vectors derived from MVs of neighboring areas. This makes the actual MV of a given area similar or identical to the MV predicted from surrounding MVs. Then, after entropy encoding, such an MV may be represented with fewer bits than would be used if the MV were encoded directly rather than predicted from neighboring MVs. In some cases, the MV prediction may be an example of lossless compression of a signal (i.e., the MV) derived from the original signal (i.e., the sample stream). In other cases, the MV prediction itself may be lossy, for example, due to rounding errors when computing the predictor from several surrounding MVs.

[0016] 2, a current block (201) contains samples that the encoder finds during the motion search process to be predictable from a spatially shifted previous block of the same size. Instead of directly encoding this MV, the MV can be derived from metadata associated with one or more reference pictures, such as the most recent reference picture (in decoding order), using the MV associated with any one of five surrounding samples denoted A0, A1, and B0, B1, B2 (202 to 206, respectively). In H.265, MV prediction can use predictors from the same reference picture used by neighboring blocks. Summary of the Invention [Problem to be solved by the invention]

[0017]

[0017] This disclosure relates generally to video coding, and more particularly to methods and systems for determining contexts for encoding and decoding various syntax elements of a video stream in inter-prediction mode. [Means for solving the problem]

[0018] In one example implementation, a method for decoding a current predictive block within a current coding block of a video stream may include determining whether the current predictive block is associated with an inter-prediction mode, deriving a first parameter in response to the current predictive block being associated with the inter-prediction mode, the first parameter representing whether the current predictive block shares an inter-prediction reference frame with a spatially neighboring predictive block of the current coding block, deriving a second parameter associated with utilization of motion vector prediction in the spatially neighboring predictive block of the current coding block, and selecting a coding context for at least one syntax associated with the inter-prediction mode of the current predictive block in response to the first parameter and the second parameter.

[0019]

[0019] In another example implementation, a method for decoding a current predictive block within a current coding block of a video stream is disclosed. The method may include determining whether the current predictive block is associated with an inter-prediction mode, deriving a context parameter based on coding mode information associated with spatially neighboring predictive blocks of the current coding block in response to the current predictive block being associated with the inter-prediction mode, and selecting a coding context for syntax associated with inter-prediction of the current predictive block in response to the context parameter being independent of coding mode information of any of the temporal predictive blocks associated with the current coding block.

[0020]

[0020] In another example implementation, a method for decoding a current predictive block in a current coding block of a video stream is disclosed. The method may include: determining whether the current predictive block is associated with an inter-prediction mode and whether a motion vector of the current predictive block is predicted by a reference motion vector; in response to the current predictive block being associated with the inter-prediction mode and the motion vector of the current predictive block being predicted by the reference motion vector, deriving one or more coding contexts for a dynamic motion vector reference list (DRL) index that identifies the reference motion vector; applying the coding contexts to process first N bits of the DRL index, where N is a predefined or configured positive integer; and applying an equal probability distribution as a context for processing remaining bits of the DRL index.

[0021] Aspects of the present disclosure also provide an electronic device or apparatus including circuitry or a processor configured to perform any of the implementations of the above methods.

[0022]

[0022] An aspect of the present disclosure also provides a non-transitory computer-readable medium storing instructions that, when executed by an electronic device, cause the electronic device to perform any one of the implementation forms of the above methods.

[0023]

[0023] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings. [Brief explanation of the drawings]

[0024] [Figure 1A] FIG. 1 is a schematic diagram of an exemplary subset of intra-prediction direction modes. [Figure 1B]

[0025] FIG. 1 is a diagram of exemplary intra-prediction directions. [Figure 2]

[0026] FIG. 1 is a schematic diagram of a current block and its surrounding spatial merge candidates for motion vector prediction in one example. [Figure 3]

[0027] FIG. 3 is a simplified block diagram schematic of a communication system (300) according to an example embodiment. [Figure 4]

[0028] FIG. 4 is a simplified block diagram schematic of a communication system (400) according to an example embodiment. [Figure 5]

[0029] FIG. 2 is a schematic diagram of a simplified block diagram of a video decoder in accordance with an example embodiment; [Figure 6]

[0030] FIG. 2 is a schematic diagram of a simplified block diagram of a video encoder in accordance with an example embodiment; [Figure 7]

[0031] FIG. 10 is a block diagram of a video encoder according to another example embodiment. [Figure 8]

[0032] FIG. 10 is a block diagram of a video decoder according to another example embodiment. [Figure 9]

[0033] FIG. 10 is a diagram of spatial neighboring blocks of a current coding block for inter-prediction mode coding context derivation. [Figure 10]

[0034] 1 is a flowchart of a method according to an example embodiment of the present disclosure. [Figure 11]

[0035] 10 is a flowchart of a method according to another example embodiment of the present disclosure. [Figure 12]

[0036] 10 is a flowchart of a method according to yet another example embodiment of the present disclosure. [Figure 13]

[0037] FIG. 1 is a schematic diagram of a computer system according to an example embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0025]

[0038] FIG. 3 illustrates a simplified block diagram of a communication system (300) according to an embodiment of the present disclosure. The communication system (300) includes multiple terminal devices capable of communicating with each other, e.g., via a network (350). For example, the communication system (300) includes a first pair of terminal devices (310) and (320) interconnected via the network (350). In the example of FIG. 3, the first pair of terminal devices (310) and (320) may perform unidirectional transmission of data. For example, the terminal device (310) may encode video data (e.g., a stream of video pictures captured by the terminal device (310)) for transmission to another terminal device (320) via the network (350). The encoded video data may be transmitted in the form of one or more coded video bitstreams. The terminal device (320) may receive the coded video data from the network (350), decode the coded video data to reconstruct the video pictures, and display the video pictures according to the reconstructed video data. One-way data transmission may be implemented in media serving applications, etc.

[0026]

[0039] In another example, the communication system (300) includes a second pair of terminal devices (330) and (340) that implement bidirectional transmission of encoded video data, such as may be implemented during a video conferencing application. For bidirectional transmission of data, in one example, each of the terminal devices (330) and (340) may encode video data (e.g., of a stream of video pictures captured by the terminal device) for transmission to the other of the terminal devices (330) and (340) over the network (350). Each of the terminal devices (330) and (340) may also receive encoded video data transmitted by the other of the terminal devices (330) and (340), decode the encoded video data to reconstruct the video pictures, and display the video pictures on an accessible display device in response to the reconstructed video data.

[0027]

[0040] In the example of FIG. 3 , the terminal devices (310), (320), (330), and (340) may be implemented as a server, a personal computer, and a smartphone, although the applicability of the underlying principles of the present disclosure may not be so limited. Embodiments of the present disclosure may also be implemented in desktop computers, laptop computers, tablet computers, media players, wearable computers, dedicated videoconferencing equipment, and the like. The network (350) represents any number or type of network that conveys coded video data between the terminal devices (310), (320), (330), and (340), including, for example, wireline and / or wireless communication networks. The communication network (350) 9 may exchange data over circuit-switched, packet-switched, and / or other types of channels. Exemplary networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For the purposes of this discussion, the architecture and topology of the network (350) may not be important to the operation of the present disclosure unless explicitly described herein.

[0028]

[0041] 4 illustrates the arrangement of a video encoder and a video decoder in a video streaming environment as an example of application of the disclosed subject matter. The disclosed subject matter may be equally applicable to other video applications including, for example, video conferencing, digital TV broadcasting, gaming, virtual reality, storage of compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0029]

[0042] The video streaming system may include a video capture subsystem (413), which may include a video source (401), such as a digital camera, for producing a stream of uncompressed video pictures or images (402). In one example, the stream of video pictures (402) includes samples recorded by the digital camera of the video source 401. The stream of video pictures (402), depicted as a bold line to emphasize its larger data volume when compared to the encoded video data (404) (or coded video bitstream), can be processed by an electronic device (420) that includes a video encoder (403) coupled to the video source (401). The video encoder (403) may include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter, as described in more detail below. The encoded video data (404) (or encoded video bitstream (404)), depicted as a thin line to emphasize its smaller data volume when compared to the stream of uncompressed video pictures (402), can be stored on the streaming server (405) for future use or directly on a downstream video device (not shown). One or more streaming client subsystems, such as the client subsystems (406) and (408) of FIG. 4, can access the streaming server (405) to retrieve copies (407) and (409) of the encoded video data (404). The client subsystem (406) can include a video decoder (410), for example, within the electronic device (430). The video decoder (410) decodes the incoming copy of the encoded video data (407) to produce an outgoing stream of video pictures (411) that is uncompressed and ready to be rendered on a display (412) (e.g., a display screen) or other rendering device (not shown). Video decoder 410 may be configured to perform some or all of the various functions described in this disclosure.In some streaming systems, the encoded video data (404), (407), and (409) (e.g., video bitstreams) can be encoded according to a particular video encoding / compression standard. Examples of these standards include ITU-T Recommendation H.265. In one example, a developing video encoding standard is informally known as Versatile Video Coding (VVC). The disclosed subject matter may also be used in the context of VVC, as well as other video encoding standards.

[0030]

[0043] It should be noted that the electronic devices (420) and (430) may include other components (not shown). For example, the electronic device (420) may include a video decoder (not shown), and the electronic device (430) may also include a video encoder (not shown).

[0031]

[0044] 5 shows a block diagram of a video decoder (510) according to any of the embodiments of the present disclosure described below. The video decoder (510) can be included in an electronic device (530). The electronic device (530) can include a receiver (531) (e.g., receiving circuitry). The video decoder (510) can be used in place of the video decoder (410) in the example of FIG. 4.

[0032]

[0045] The receiver (531) may receive one or more coded video sequences to be decoded by the video decoder (510). In the same or another embodiment, one coded video sequence may be decoded at a time, with the decoding of each coded video sequence being independent of other coded video sequences. Each video sequence may be associated with multiple video frames or images. The coded video sequences may be received from a channel (501), which may be a hardware or software link to a storage device that stored the coded video data or a streaming source that transmitted the coded video data. The receiver (531) may receive the coded video data with other data, such as coded audio data and / or auxiliary data streams, and the coded video data may be forwarded to its respective processing circuitry (not shown). The receiver (531) may separate the coded video sequences from other data. A buffer memory (515) may be located between the receiver (531) and the entropy decoder / parser (520) (hereinafter "parser (520)") to prevent network jitter. In certain applications, the buffer memory (515) may be implemented as part of the video decoder (510). In other applications, the buffer memory (515) may be external to and separate from the video decoder (510) (not shown). In still other applications, there may be a buffer memory (not shown) external to the video decoder (510), for example, to prevent network jitter, or another additional buffer memory (515) internal to the video decoder (510), for example, to handle playback timing. When the receiver (531) is receiving data from a store / forward device with sufficient bandwidth and controllability or from an isosynchronous network, the buffer memory (515) may not be needed or may be small.For use with best-effort packet networks such as the Internet, a sufficiently sized buffer memory (515) may be required, and may be significantly larger. Such a buffer memory may be provided in an adaptable size and may be implemented at least in part in an operating system or similar element (not shown) external to the video decoder (510).

[0033]

[0046] The video decoder (510) may include a parser (520) for reconstructing symbols (521) from the encoded video sequence. These symbol categories include information used to manage the operation of the video decoder (510) and possibly information for controlling a rendering device, such as a display (512) (e.g., a display screen), which may or may not be an integral part of the electronic device (530), but may be coupled to the electronic device (530) as shown in FIG. 5. The control information for the rendering device may be in the form of a Supplemental Enhancement Information (SEI) message or a Video Usability Information (VUI) parameter set fragment (not shown). The parser (520) may parse / entropy decode the encoded video sequence received by the parser (520). The entropy coding of the coded video sequence may be in accordance with a video coding technique or standard and may follow various principles, including variable length coding, Huffman coding, context-independent or non-context-dependent arithmetic coding, etc. The parser (520) may extract a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder from the coded video sequence based on at least one parameter corresponding to the subgroup. The subgroup may include a group of pictures (GOP), a picture, a tile, a slice, a macroblock, a coding unit (CU), a block, a transform unit (TU), a prediction unit (PU), etc. Additionally, the parser (520) may extract coded video sequence information, such as transform coefficients (e.g., Fourier transform coefficients), quantization parameter values, motion vectors, etc.

[0034]

[0047] The parser (520) may perform entropy decoding / parsing operations on the video sequence received from the buffer memory (515) to produce symbols (521).

[0035]

[0048] The reconstruction of the symbols (521) may require several different processing or functional units, depending on the type of coded video picture or portion thereof (inter and intra pictures, inter and intra blocks, etc.), as well as other factors. The units required, and how they are required, may be controlled by subgroup control information parsed from the coded video sequence by the parser (520). The flow of such subgroup control information between the parser (520) and the following processing or functional units is not depicted for the sake of simplicity.

[0036]

[0049] In addition to the functional blocks already mentioned, the video decoder (510) can be conceptually subdivided into several functional units, as described below. In a practical implementation operating under commercial constraints, many of these functional units will interact closely with each other and may be at least partially integrated with each other. However, to clearly explain the various functions of the disclosed subject matter, a conceptual subdivision into functional units will be adopted in the following disclosure.

[0037]

[0050] The first unit may include a scalar / inverse transform unit (551). The scalar / inverse transform unit (551) may receive quantized transform coefficients and control information from the parser (520), including information indicating which type of inverse transform to use, block size, quantization factors / parameters, quantization scaling matrices, and lies as symbols (521). The scalar / inverse transform unit (551) may output blocks having sample values ​​that can be input to an aggregator (555).

[0038]

[0051] In some cases, the output samples of the scaler / inverse transform (551) may be for intra-coded blocks, i.e., blocks that can use prediction information from a previously reconstructed portion of the current picture rather than using prediction information from a previously reconstructed picture. Such prediction information can be provided by an intra-picture prediction unit (552). In some cases, the intra-picture prediction unit (552) may generate blocks of the same size and shape as the block being reconstructed using surrounding block information already reconstructed and stored in the current picture buffer (558). The current picture buffer (558), for example, buffers a partially reconstructed and / or fully reconstructed current picture. In some implementations, the aggregator (555) may add, on a sample-by-sample basis, the prediction information generated by the intra-prediction unit (552) to the output sample information as provided by the scaler / inverse transform unit (551).

[0039]

[0052] In other cases, the output samples of the scalar / inverse transform unit (551) may be for an inter-coded and potentially motion-compensated block. In such cases, the motion-compensated prediction unit (553) may access the reference picture memory (557) to fetch samples used for inter-picture prediction. After motion-compensating the fetched samples according to the symbols (521) for the block, these samples may be added to the output of the scalar / inverse transform unit (551) by the aggregator (555) to generate the output sample information (the output of unit 551 may be referred to as residual samples or residual signals). The addresses in the reference picture memory (557) from which the motion-compensated prediction unit (553) fetches the prediction samples may be controlled by a motion vector, available to the motion-compensated prediction unit (553) in the form of a symbol (521), which may have, for example, X and Y components (shift) and a reference picture component (time). Motion compensation may further include interpolation of sample values ​​as fetched from the reference picture memory (557) when sub-sample accurate motion vectors are in use, may be further associated with a motion vector prediction mechanism, etc.

[0040]

[0053] The output samples of the aggregator (555) may be subjected to various loop filtering techniques in the loop filter unit (556). Video compression techniques can include in-loop filter techniques controlled by parameters contained in the coded video sequence (also called the coded video bitstream) and made available to the loop filter unit (556) as symbols (521) from the parser (520), but can also include those that respond to meta-information obtained during decoding of previous portions (in decoding order) of the coded picture or coded video sequence, and those that respond to previously reconstructed, loop-filtered sample values. As described in more detail below, several types of loop filters may be included as part of the loop filter unit 556, in various orders.

[0041]

[0054] The output of the loop filter unit (556) may be a sample stream that can be output to a rendering device (512) or stored in a reference picture memory (557) for use during future inter-picture prediction.

[0042]

[0055] A particular coded picture can be used as a reference picture for future inter-picture prediction as soon as it is fully reconstructed. For example, once the coded picture corresponding to the current picture is fully reconstructed and the coded picture is identified as a reference picture (e.g., by the parser (520)), the current picture buffer (558) can become part of the reference picture memory (557), and an unused current picture buffer can be reallocated before beginning reconstruction of the next coded picture.

[0043]

[0056] The video decoder (510) may perform decoding operations according to a predetermined video compression technique adopted in a standard such as ITU-T Recommendation H.265. An encoded video sequence may conform to the syntax specified by the video compression technique or standard used, in the sense that the encoded video sequence conforms to both the syntax of the video compression technique or standard and the profile as documented in the video compression technique or standard. Specifically, a profile may select a specific tool from all tools available in the video compression technique or standard as the only tool available for use under this profile. To be standard-compliant, the complexity of the encoded video sequence may be within a range as defined by the level of the video compression technique or standard. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstructed sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. In some cases, the limits set by the levels can be further restricted through the Hypothetical Reference Decoder (HRD) specification and metadata for HRD buffer management signaled in the coded video sequence.

[0044]

[0057] In some example embodiments, the receiver (531) may receive additional (redundant) data along with the encoded video. The additional data may be included as part of the encoded video sequence. The additional data may be used by the video decoder (510) to properly decode the data and / or to more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0045]

[0058] 6 shows a block diagram of a video encoder (603) according to an example embodiment of the present disclosure. The video encoder (603) may be included in an electronic device (620). The electronic device (620) may further include a transmitter (640) (e.g., transmission circuitry). The video encoder (603) can be used in place of the video encoder (403) in the example of FIG. 4.

[0046]

[0059] The video encoder (603) may receive video samples from a video source (601) (not part of the electronic device (620) in the example of FIG. 6) that may capture video images to be encoded by the video encoder (603). In another example, the video source (601) may be provided as part of the electronic device (620).

[0047]

[0060] The video source (601) may provide a source video sequence to be encoded by the video encoder (603) in the form of a digital video sample stream, which can be of any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, etc.), any color space (e.g., BT.601 YCrCb, RGB, XYZ, etc.), and any suitable sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4).

[0048]

[0061] According to some example embodiments, the video encoder (603) may encode and compress pictures of a source video sequence into an encoded video sequence (643) in real time or under any other time constraints, as needed by the application. Enforcing an appropriate encoding speed constitutes one function of the controller (650). In some embodiments, the controller (650) may be functionally coupled to and control other functional units, such as those described below. The coupling is not depicted for simplicity. Parameters set by the controller (650) may include rate control-related parameters (picture skip, quantizer, lambda value for rate-distortion optimization techniques, etc.), picture size, group-of-picture (GOP) layout, maximum motion vector search range, etc. The controller (650) can be configured to have other appropriate functions for the video encoder (603) optimized for a particular system design.

[0049]

[0062] In some example embodiments, the video encoder (603) may be configured to operate within an encoding loop. As an overly simplified explanation, in one example, the encoding loop may include a source coder (630) (e.g., responsible for producing symbols, such as a symbol stream, based on an input picture to be encoded and a reference picture) and a (local) decoder (633) embedded in the video encoder (603). The decoder (633) reconstructs the symbols to produce sample data in a manner similar to that which a (remote) decoder would produce even if the embedded decoder 633 processed the video stream encoded by the source coder 630 without entropy encoding (since any compression between the symbols and the encoded video bitstream in entropy encoding may be lossless in the video compression techniques considered in the disclosed subject matter). The reconstructed sample stream (sample data) is input to a reference picture memory (634). Because decoding the symbol stream results in bit-exact results independent of the decoder location (local or remote), the contents of the reference picture memory (634) are also bit-exact between the local and remote encoders. In other words, the predictive portion of the encoder "sees" the reference picture sampling exactly the same sample values ​​that the decoder "sees" when using prediction during decoding. This basic principle of reference picture synchrony (and the resulting drift when synchrony cannot be maintained, e.g., due to channel errors) is used to improve coding quality.

[0050]

[0063] The operation of the "local" decoder (633) can be the same as a "remote" decoder, such as the video decoder (510) already described in detail above in conjunction with Figure 5. Furthermore, and briefly referring to Figure 5, however, the entropy decoding portion of the video decoder (510), including the buffer memory (515) and parser (520), need not be entirely implemented in the local decoder (633) within the encoder, because symbols are available and the encoding / decoding of the symbols into an encoded video sequence by the entropy coder (645) and parser (520) can be lossless.

[0051]

[0064] An observation that can be made at this point is that any decoder technology other than parsing / entropy decoding, which may only be present in the decoder, may not necessarily be present in substantially identical functional form in the corresponding encoder. For this reason, the disclosed subject matter may sometimes focus on decoder operations in cooperation with the decoding portion of the encoder. Therefore, a description of the encoder technology may be omitted, as it is the reverse of the decoder technology, which is described generically. Only in certain areas or aspects is a more detailed description of the encoder provided below.

[0052]

[0065] During operation in some example implementations, the source coder (630) may perform motion-compensated predictive coding, which predictively codes an input picture with reference to one or more previously coded pictures from a video sequence designated as "reference pictures." In this manner, the coding engine (632) codes color channel differences (or residuals) between pixel blocks of the input picture and pixel blocks of reference pictures that may be selected as prediction references for the input picture. The terms "residual" and its adjective form "residual" are sometimes used interchangeably.

[0053]

[0066] The local video decoder (633) may decode the coded video data of pictures that may be designated as reference pictures based on symbols produced by the source coder (630). The operation of the coding engine (632) may advantageously be a lossy process. When the coded video data may be decoded in a video decoder (not shown in FIG. 6), the reconstructed video sequence may be a copy of the source video sequence, typically containing some errors. The local video decoder (633) may replicate the decoding process that may be performed on reference pictures by the video decoder and store the reconstructed reference pictures in a reference picture cache (634). In this way, the video encoder (603) may locally store copies of reconstructed reference pictures that have content in common with reconstructed reference pictures that will be retrieved by a far-end (remote) video decoder (without transmission errors).

[0054]

[0067] The predictor (635) may perform prediction searches for the coding engine (632). That is, to encode a new picture, the predictor (635) may search the reference picture memory (634) for sample data (as candidate reference pixel blocks) or for specific metadata, such as reference picture motion vectors, block shapes, etc., that may serve as appropriate prediction references for the new picture. The predictor (635) may operate on a sample block-by-pixel block basis to find appropriate prediction references. In some cases, as determined by search results obtained by the predictor (635), the input picture may have prediction references created from multiple reference pictures stored in the reference picture memory (634).

[0055]

[0068] The controller (650) may manage the coding operations of the source coder (630), including, for example, setting the parameters and subgroup parameters used to encode the video data.

[0056]

[0069] The output of all the aforementioned functional units may undergo entropy coding in an entropy coder (645), which transforms the symbols as produced by the various functional units into an encoded video sequence by lossless compression of the symbols through techniques such as Huffman coding, variable length coding, or arithmetic coding.

[0057]

[0070] The transmitter (640) may buffer the encoded video sequence as produced by the entropy coder (645) in preparation for transmission over a communication channel (660), which may be a hardware / software link to a storage device that will store the encoded video data. The transmitter (640) may merge the encoded video data from the video coder (603) with other data to be transmitted, such as encoded audio data and / or auxiliary data streams (sources not shown).

[0058]

[0071] The controller (650) may manage the operation of the video encoder (603). During encoding, the controller (650) may assign a particular coded picture type to each coded picture, which may affect the coding technique that may be applied to the respective picture. For example, pictures may often be assigned as one of the following picture types:

[0059]

[0072] An intra-picture (I-picture) may be coded and decoded without using any other pictures in a sequence as a source of prediction. Some video codecs allow different types of intra-pictures, including, for example, Independent Decoder Refresh ("IDR") pictures. Those skilled in the art are aware of these variations of I-pictures and their respective uses and characteristics.

[0060]

[0073] A predicted picture (P picture) may be coded and decoded using intra- or inter-prediction, using at most one motion vector and reference index to predict the sample values ​​of each block.

[0061]

[0074] Bidirectionally predicted pictures (B-pictures) may be coded and decoded using intra- or inter-prediction, using at most two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, multiple predicted pictures can use more than two reference pictures and associated metadata for the reconstruction of a single block.

[0062]

[0075] A source picture may generally be spatially subdivided into multiple sample coding blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples each) and coded block by block. Blocks may be predictively coded with reference to other (already coded) blocks as determined by the coding assignment applied to the block's respective picture. For example, blocks of an I-picture may be non-predictively coded or predictively coded with reference to previously coded blocks of the same picture (spatial prediction or intra-prediction). Pixel blocks of a P-picture may be predictively coded via spatial prediction or temporal prediction with reference to one previously coded reference picture. Blocks of a B-picture may be predictively coded via spatial prediction or temporal prediction with reference to one or two previously coded reference pictures. Source pictures or intermediate processed pictures may also be subdivided into other types of blocks for other purposes. The division of coding blocks and other types of blocks may or may not follow the same manner as described in more detail below.

[0063]

[0076] In some example embodiments, the transmitter (640) may transmit additional data along with the encoded video. The source coder (630) may include data such as portions of the encoded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

[0064]

[0077] Video may be captured as a time sequence of multiple source pictures (video pictures). Intra-picture prediction (often abbreviated to intra-prediction) exploits spatial correlations within a given picture, while inter-picture prediction exploits temporal or other correlations between pictures. For example, a particular picture being encoded / decoded, called the current picture, may be partitioned into blocks. When a block in the current picture resembles a reference block in a previously coded and still buffered reference picture in the video, it may be coded by a vector, called a motion vector. A motion vector points to a reference block within a reference picture and may have a third dimension that identifies the reference picture in cases where multiple reference pictures are in use.

[0065]

[0078] In some example embodiments, bi-prediction techniques can be used for inter-picture prediction. Such bi-prediction techniques use two reference pictures, such as a first reference picture and a second reference picture, that both advance the current picture in the video in decoding order (but may be in the past or future, respectively, in display order). A block in the current picture can be coded by a first motion vector that points to a first reference block in the first reference picture and a second motion vector that points to a second reference block in the second reference picture. A block can be jointly predicted by combining the first reference block and the second reference block.

[0066]

[0079] Furthermore, to improve coding efficiency, merge mode techniques may be used during inter-picture prediction.

[0067]

[0080] According to some example embodiments of the present disclosure, prediction, such as inter-picture prediction and intra-picture prediction, is performed in units of blocks. For example, pictures in a sequence of video pictures are partitioned into coding tree units (CTUs) for compression, and the CTUs within a picture may have the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. In general, a CTU may include three parallel coding tree blocks (CTBs): one luma CTB and two chroma CTBs. Each CTU may be recursively quadtree split into one or more coding units (CUs). For example, a 64×64 pixel CTU may be split into one CU of 64×64 pixels or four CUs of 32×32 pixels. One or more of the 32×32 blocks may each be further split into four CUs of 16×16 pixels. In some example embodiments, each CU may be analyzed during encoding to determine a prediction type for the CU from various prediction types, such as an inter-prediction type or an intra-prediction type. The CU may be divided into one or more prediction units (PUs) according to temporal and / or spatial predictability. Generally, each PU includes one luma prediction block (PB) and two chroma PBs. In one embodiment, prediction operations during coding (encoding / decoding) are performed in units of prediction blocks. The division of a CU into PUs (or PBs of different color channels) may be performed in various spatial patterns. For example, a luma or chroma PB may include a matrix of sample values ​​(e.g., luma values), such as 8x8 pixels, 16x16 pixels, 8x16 pixels, 16x8 samples, etc.

[0068]

[0081] 7 shows a diagram of a video encoder (703) according to another example embodiment of this disclosure. The video encoder (703) is configured to receive a processed block (e.g., a predictive block) of sample values ​​in a current video picture in a sequence of video pictures and encode the processed block into an encoded picture that is part of an encoded video sequence. The example video encoder (703) may be used in place of the example video encoder (403) of FIG. 4.

[0069]

[0082] For example, the video encoder (703) receives a matrix of sample values ​​for a processing block, such as a predicted block of 8x8 samples. The video encoder (703) then determines, for example, using rate-distortion optimization (RDO), whether the processing block is best coded using intra mode, inter mode, or bi-predictive mode. When it is determined that the processing block is coded in intra mode, the video encoder (703) may encode the processing block into a coded picture using intra prediction techniques, and when it is determined that the processing block is coded in inter mode or bi-predictive mode, the video encoder (703) may encode the processing block into a coded picture using inter prediction or bi-predictive techniques, respectively. In some example embodiments, merge mode may be used as a sub-mode of inter-picture prediction, in which a motion vector is derived from one or more motion vector predictors without the benefit of coded motion vector components outside the predictors. In some other example embodiments, there may be motion vector components applicable to the subject block. Thus, the video encoder (703) may include components not explicitly shown in FIG. 7, such as a mode decision module, to determine the prediction mode of a processing block.

[0070]

[0083] In the example of Figure 7, the video encoder (703) includes an inter-encoder (730), an intra-encoder (722), a residual calculator (723), a switch (726), a residual encoder (724), a general controller (721), and an entropy encoder (725), linked together as shown in the example arrangement of Figure 7.

[0071]

[0084] The inter-encoder (730) is configured to receive a sample of a current block (e.g., a processing block), compare the block with one or more reference blocks in a reference picture (e.g., blocks in previous and subsequent pictures in display order), generate inter-prediction information (e.g., a description of redundant information due to inter-coding techniques, motion vectors, merge mode information), and calculate an inter-prediction result (e.g., a predicted block) based on the inter-prediction information using any suitable technique. In some examples, the reference picture is a decoded reference picture decoded based on the encoded video information using a decoding unit 633 incorporated in the example encoder 620 of FIG. 6 (such as shown as residual decoder 728 of FIG. 7, described in further detail below).

[0072]

[0085] The intra encoder (722) is configured to receive samples of a current block (e.g., a processing block), compare the block with previously encoded blocks in the same picture, generate transformed quantized coefficients, and in some cases, further generate intra prediction information (e.g., intra prediction direction information using one or more intra encoding techniques). The intra encoder (722) may calculate intra prediction results (e.g., predicted blocks) based on the intra prediction information and reference blocks in the same picture.

[0073]

[0086] The general controller (721) may be configured to determine general control data and control other components of the video encoder (703) based on the general control data. In one example, the general controller (721) determines a prediction mode for a block and sends a control signal to the switch (726) based on the prediction mode. For example, when the prediction mode is intra-mode, the general controller (721) controls the switch (726) to select an intra-mode result for use by the residual calculator (723) and controls the entropy encoder (725) to select intra-prediction information and include the intra-prediction information in the bitstream. When the block's predicate mode is inter-mode, the general controller (721) controls the switch (726) to select an inter-prediction result for use by the residual calculator (723) and controls the entropy encoder (725) to select inter-prediction information and include the inter-prediction information in the bitstream.

[0074]

[0087] The residual calculator (723) may be configured to calculate the difference (residual data) between the received block and a prediction result of the block selected from the intra-encoder (722) or the inter-encoder (730). The residual encoder (724) may be configured to encode the residual data to generate transform coefficients. For example, the residual encoder (724) may be configured to convert the residual data from the spatial domain to the frequency domain to generate the transform coefficients. The transform coefficients then undergo a quantization process to obtain quantized transform coefficients. In various illustrative embodiments, the video encoder (703) further includes a residual decoder (728). The residual decoder (728) is configured to perform an inverse transform and generate decoded residual data. The decoded residual data can be used by the intra-encoder (722) and the inter-encoder (730) as appropriate. For example, the inter-encoder (730) can generate decoded blocks based on the decoded residual data and inter-prediction information, and the intra-encoder (722) can generate decoded blocks based on the decoded residual data and intra-prediction information. The decoded blocks are appropriately processed to generate decoded pictures, which can be buffered in a memory circuit (not shown) and used as reference pictures.

[0075]

[0088] The entropy encoder (725) may be configured to format a bitstream to include the encoded block and perform entropy encoding. The entropy encoder (725) may be configured to include various information in the bitstream. For example, the entropy encoder (725) may be configured to include general control data, selected prediction information (e.g., intra-prediction information or inter-prediction information), residual information, and other appropriate information in the bitstream. When encoding a block in an inter mode or a merged sub-mode of a bi-prediction mode, the residual information may be absent.

[0076]

[0089] 8 shows a diagram of an example video decoder (810) according to another embodiment of the present disclosure. The video decoder (810) is configured to receive coded pictures that are part of a coded video sequence and decode the coded pictures to generate reconstructed pictures. In one example, the video decoder (810) may be used in place of the video decoder (410) in the example of FIG. 4.

[0077]

[0090] In the example of Figure 8, the video decoder (810) includes an entropy decoder (871), an inter decoder (880), a residual decoder (873), a reconstruction module (874), and an intra decoder (872) linked together as shown in the example arrangement of Figure 8.

[0078]

[0091] The entropy decoder (871) can be configured to reconstruct, from the coded picture, certain symbols that represent syntax elements that make up the coded picture. Such symbols can include, for example, prediction information (e.g., intra- or inter-prediction information) that can identify the mode in which the block is coded (e.g., intra-mode, inter-mode, bi-prediction mode, merged sub-mode, or another sub-mode), certain samples or metadata used for prediction by the intra decoder (872) or inter decoder (880), residual information in the form of, for example, quantized transform coefficients, etc. In one example, when the prediction mode is an inter- or bi-prediction mode, the inter-prediction information is provided to the inter decoder (880), and when the prediction type is an intra-prediction type, the intra-prediction information is provided to the intra decoder (872). The residual information can undergo inverse quantization and is provided to the residual decoder (873).

[0079]

[0092] The inter decoder (880) may be configured to receive inter prediction information and generate inter prediction results based on the inter prediction information.

[0080]

[0093] The intra decoder (872) may be configured to receive intra prediction information and generate prediction results based on the intra prediction information.

[0081]

[0094] The residual decoder (873) may be configured to perform inverse quantization to extract dequantized transform coefficients, process the dequantized transform coefficients, and convert the residual from the frequency domain to the spatial domain. Additionally, the residual decoder (873) may utilize certain control information (to include quantization parameters (QPs)) that may be provided by the entropy decoder (871) (this may be only small data volume control information, so the data path is not depicted).

[0082]

[0095] The reconstruction module (874) may be configured to combine, in the spatial domain, the residual, as output by the residual decoder (873), and the prediction result (as output by the inter- or intra-prediction module, as the case may be), to form a reconstructed block that forms part of the reconstructed picture as part of the reconstructed video. Note that other appropriate operations, such as deblocking operations, may also be performed to improve visual quality.

[0083]

[0096] It should be noted that the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) may be implemented using any suitable technique. In some example embodiments, the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) may be implemented using one or more integrated circuits. In another embodiment, the video encoders (403), (603), and (603) and the video decoders (410), (510), and (810) may be implemented using one or more processors executing software instructions.

[0084]

[0097] Returning to inter prediction, for example, in single reference mode or combined reference mode. In some implementations, a skip flag may be initially included in the bitstream for the current block (or at a higher level) to indicate whether the current block is inter-coded and should not be skipped. If the current block is inter-coded, another flag may be further included in the bitstream to signal whether single reference mode or combined reference mode is used for predicting the current block. In single reference mode, one reference block may be used to generate a prediction block for the current block. In combined reference mode, two or more reference blocks may be used to generate a prediction block, for example, by weighted averaging. Combined reference mode is sometimes also referred to as two or more reference mode, two reference mode, or multiple reference mode. One or more reference blocks may be identified using one or more reference frame indices and, additionally, using one or more corresponding motion vectors, which indicate a shift between the reference block and the current block in location relative to the frame, e.g., horizontal and vertical pixels. For example, an inter-predicted block for a current block may be generated from a single reference block identified by one motion vector in a reference frame as the predictive block in single reference mode, while in mixed reference mode, the predictive block may be generated by a weighted average of two reference blocks in two reference frames indicated by two reference frame indices and two corresponding motion vectors. Motion vectors may be coded and included in the bitstream in various manners.

[0085]

[0098] In some implementations, an encoding or decoding system may maintain a decoded picture buffer (DPB). Some images / pictures may be kept in the DPB waiting to be displayed (in the decoding system), and some images / pictures in the DPB may be used as reference frames to enable inter-prediction (in the decoding or encoding system). In some implementations, reference frames in the DPB may be tagged as short-term or long-term references for the current image being encoded or decoded. For example, short-term reference frames may include frames used for inter-prediction of blocks in the current frame or in a predefined number (e.g., two) of subsequent video frames closest to the current frame in decoding order. Long-term reference frames may include frames in the DPB that can be used to predict image blocks in frames that are more than a predefined number of frames away from the current frame in decoding order. Information about such tags for short-term and long-term reference frames may be called a Reference Picture Set (RPS) and may be added to the header of each frame in the encoded bitstream. Each frame in the encoded video stream may be identified by a Picture Order Counter (POC), which may be absolutely numbered according to the playback sequence or may relate to a group of pictures starting from, for example, an I-frame.

[0086]

[0099] In some example implementations, one or more reference picture lists including identification of short-term and long-term reference frames for inter prediction may be formed based on information in the RPS. For example, a single picture reference list may be formed for unidirectional inter prediction and denoted as L0 reference (or reference list 0), while two picture reference lists may be formed for bidirectional inter prediction and denoted as L0 (or reference list 0) and L1 (or reference list 1), for each of the two prediction directions. The reference frames included in the L0 and L1 lists may be ordered in various predetermined manners. The lengths of the L0 and L1 lists may be signaled in the video bitstream. Unidirectional inter prediction may be a single reference mode, or may be a combined reference mode when multiple references for generating a prediction block by weighted averaging in combined prediction mode are on the same side of a frame. Bidirectional inter prediction may be a combined mode only, since bidirectional inter prediction involves at least two reference blocks.

[0087]

[0100] In some implementations, a merge mode (MM) for inter prediction may be implemented. In general, in the merge mode, one or more of the motion vectors in single-reference prediction or the motion vectors in mixed-reference prediction for the current PB may be derived from other motion vectors rather than being calculated and signaled independently. For example, in an encoding system, the current motion vector of the current PB may be represented by the difference between the current motion vector and one or more other already-encoded motion vectors (referred to as reference motion vectors). Such a motion vector difference, rather than the entire current motion vector, may be encoded and included in the bitstream and linked to the reference motion vector. Correspondingly, in a decoding system, the motion vector corresponding to the current PB may be derived based on the decoded motion vector difference and the decoded reference motion vector linked thereto. As a specific form of merge mode (MM) inter prediction in general, such inter prediction based on the motion vector difference may be referred to as merge mode with motion vector difference (MMVD). Therefore, MM in general, or MMVD in particular, may be implemented to exploit the correlation between motion vectors associated with different PBs to improve coding efficiency. For example, neighboring PBs may have similar motion vectors, and therefore the MVD may be small and efficiently codeable. As another example, motion vectors may be temporally correlated (between frames) for blocks that are similarly located / placed in space.

[0088]

[0101] In some example implementations, an MM flag may be included in the bitstream during the encoding process to indicate whether the current PB is in merge mode. Additionally or alternatively, an MMVD flag may be included during the encoding process and signaled in the bitstream to indicate whether the current PB is in MMVD mode. MM and / or MMVD flags or indicators may be provided at the PB level, CB level, CU level, CTB level, CTU level, slice level, frame level, picture level, sequence level, etc. As a specific example, both an MM flag and an MMVD flag may be included for the current CU, and the MMVD flag may be signaled immediately after the skip flag and MM flag to specify whether MMVD mode is used for the current CU.

[0089]

[0102] In some example implementations of MMVD, a list of reference motion vectors (RMVs), or MV predictor candidates for motion vector prediction, may be formed for the block being predicted. The list of RMV candidates may include a predetermined number (e.g., two) of MV predictor candidate blocks, and the motion vectors of the MV predictor candidate blocks may be used to predict the current motion vector. The RMV candidate blocks may include blocks selected from neighboring blocks in the same frame (called spatial motion vector predictors, or SMVPs) and / or temporal blocks (e.g., co-located blocks in a following or subsequent frame of the current frame, called temporal motion vector predictors, or TMVPs). These options represent blocks in spatial or temporal locations relative to the current block that are likely to have a motion vector similar or identical to the current block. The size of the list of MV predictor candidates may be predetermined. For example, the list may include two or more candidates. To be on the list of RMV candidates, a candidate block may be required, for example, to have the same reference frame (or multiple reference frames) as the current block, must exist (e.g., a boundary check needs to be performed when the current block is near the edge of a frame), must have already been encoded during the encoding process, and / or must have already been decoded during the decoding process. In some implementations, the list of merge candidates may be populated with spatially neighboring blocks (scanned in a specific predefined order) first, if available and if they satisfy the above conditions, and then with temporal blocks if space is still available in the list. Neighboring RMV candidate blocks may be selected, for example, from the blocks to the left and top of the current block. The list of RMV predictor candidates may be dynamically formed at various levels (sequence, picture, frame, slice, superblock, etc.) as a Dynamic Reference List (DRL). The DRL may be signaled in the bitstream.

[0090]

[0103] In some implementations, the actual MV predictor candidate used as the reference motion vector for predicting the motion vector of the current block may be signaled. In cases where the RMV candidate list contains two candidates, a one-bit flag called a merge candidate flag may be used to indicate the selection of the reference merge candidate. If the current block is predicted in mixed mode, each of the multiple motion vectors predicted using the MV predictor may be associated with a reference motion vector from the merge candidate list. The encoder may determine which of the RMV candidates more closely predicts the MV of the current coding block and signal the selection as an index to the DRL.

[0091]

[0104] In some example implementations of MMVD, after an RMV candidate is selected and used as a base motion vector predictor for predicting a motion vector, a motion vector difference (MVD or delta MV representing the difference between the motion vector to be predicted and a reference candidate motion vector) may be calculated in the encoding system. Such MVD may include information representing the magnitude and direction of the MV difference, and both the magnitude and direction of the MV difference may be signaled in the bitstream. The magnitude and direction of the motion difference may be signaled in various ways.

[0092]

[0105] In some example implementations of MMVD, a distance index may be used to specify magnitude information of the motion vector difference and to indicate one of a set of predefined offsets that represent predefined motion vector differences from a starting point (reference motion vector). The MV offset according to the signaled index may then be added to the horizontal or vertical component of the start (reference) motion vector. Whether the horizontal or vertical component of the reference motion vector should be offset may be determined by the direction information of the MVD. Example predefined relationships between the distance index and the predefined offsets are specified in Table 2.

[0093] [Table 1]

[0094]

[0106] In some example implementations of MMVD, a direction index may be further signaled and used to indicate the direction of the MVD relative to the reference motion vector. In some implementations, the direction may be limited to one of horizontal and vertical. An example 2-bit direction index is shown in Table 3. In the example of Table 3, the interpretation of the MVD may differ depending on the information of the start / reference MV. For example, when the start / reference MV corresponds to a uni-prediction block or a bi-prediction block where both reference frame lists point to the same side of the current picture (i.e., when the POCs of the two reference pictures are both greater than or both less than the POC of the current picture), the code in Table 3 may specify the sign (direction) of the MV offset added to the start / reference MV. When the start / reference MV corresponds to a bi-predictive block with two reference pictures on different sides of the current picture (i.e., the POC of one reference picture is greater than the POC of the current picture and the POC of the other reference picture is less than the POC of the current picture), and when the difference between the reference POC in picture reference list 0 and the current frame is greater than the difference between the reference POC in picture reference list 1 and the current frame, the code in Table 3 may specify the sign of the MV offset added to the reference MV corresponding to the reference picture in picture reference list 0, and the sign of the offset of the MV corresponding to the reference picture in picture reference list 1 may have the opposite value (the opposite sign of the offset). Otherwise, if the difference between the reference POC in picture reference list 1 and the current frame is greater than the difference between the reference POC in picture reference list 0 and the current frame, the code in Table 3 may specify the sign of the MV offset added to the reference MV associated with picture reference list 1, and the sign of the offset to the reference MV associated with picture reference list 0 has the opposite value.

[0095] [Table 2]

[0096]

[0107] In some example implementations, the MVD may be scaled according to the POC difference per direction. If the POC difference in both lists is the same, no scaling is necessary. Otherwise, if the POC difference in reference list 0 is greater than the one in reference list 1, the MVD for reference list 1 is scaled. If the POC difference in reference list 1 is greater than list 0, the MVD for list 0 may be scaled in the same manner. If the start MV is uni-predicted, the MVD is added to the available MV or reference MV.

[0097]

[0108] In some other example implementations of MV prediction, a harmonized scheme may be used to implement general merge-mode MMVD, as well as some other types of MV prediction, for both single-reference mode and combined-reference mode MV prediction. Various syntax elements may be used to signal the manner in which the MV of the current block is predicted.

[0098]

[0109] For example, for single criterion mode, the following MV prediction modes may be signaled:

[0099]

[0110] NEARMV - Use one of the motion vector predictors (MVP) in the list directly pointed to by the DRL (Dynamic Reference List) index with no MVD.

[0100]

[0111] NEWMV - Use one of the motion vector predictors (MVP) in the list signaled by the DRL index as a reference and apply a delta to the MVP (e.g., using MVD).

[0101]

[0112] GLOBALMV - Use motion vectors based on frame-level global motion parameters.

[0102]

[0113] Similarly, if a combined reference inter prediction mode uses two reference frames corresponding to two MVs to be predicted, the following MV prediction modes may be signaled:

[0103]

[0114] NEAR_NEARMV - Use one of the motion vector predictors (MVPs) in the list signaled by the DRL index without MVD for each of two of the MVs to be predicted.

[0104]

[0115] NEAR_NEWMV - Use one of the motion vector predictors (MVP) in the list signaled by the DRL index as a reference MV without MVD to predict the first of two motion vectors, and use one of the motion vector predictors (MVP) in the list signaled by the DRL index as a reference MV with an additionally signaled delta MV (MVD) to predict the second of two motion vectors.

[0105]

[0116] NEW_NEARMV - Use one of the motion vector predictors (MVP) in the list signaled by the DRL index as a reference MV without MVD to predict the second of two motion vectors, and use one of the motion vector predictors (MVP) in the list signaled by the DRL index as a reference MV with the additionally signaled delta MV (MVD) to predict the first of two motion vectors.

[0106]

[0117] NEW_NEWMV - Use one of the motion vector predictors (MVPs) in the list signaled by the DRL index as the reference MV and use this together with the additionally signaled delta MV to predict for each of the two MVs.

[0107]

[0118] GLOBAL_GLOBALMV - Use the MV from each reference based on its frame-level global motion parameters.

[0108]

[0119] Thus, the term "NEAR" above refers to MV prediction using a reference MV without any MVD as a general merge mode, while the term "NEW" refers to MV prediction using a reference MV and offsetting the reference MV with a signaled or derived MVD, as in the MMVD mode. In the case of hybrid inter prediction, for example, the two MVDs may be correlated, and both the reference base motion vector and the motion vector delta may generally be different or unrelated between the two references or two MVDs, even though such correlation may be exploited to reduce the amount of information required to signal the two motion vector deltas. To exploit such correlation, joint signaling of the two MVDs may be implemented and indicated in the bitstream, as described in more detail below.

[0109]

[0120] Moving on to entropy coding, various syntax elements of a video stream may be coded based on context-adaptive binary arithmetic coding (CABAC). In some other implementations, an M-ary arithmetic coding engine may be implemented to encode a sequence of multi-level symbols, rather than binary symbols. For example, each syntax element may be associated with an alphabet of M elements, where M can be any integer value, e.g., from 2 to 16. The input to the encoding is a sequence of M-ary symbols. The coding context may correspondingly consist of a set of M probabilities represented by a cumulative distribution function (CDF). The probabilities in the coding context may be updated after encoding / parsing each syntax element. The cumulative distribution function may be represented by an array of M 15-bit integers (in fixed-point precision) as follows: C=[c0,c1,...,c (M-2) ,2 15 ] (1)

[0110]

[0121] where c n / 32768 represents the probability of a symbol being less than or equal to n. In some example implementations, the probability update may be performed using the following equation:

[0111]

number

[0112]

[0122] where α is the probability update rate that is adapted based on the number of times a symbol is decoded (up to a maximum, e.g., 32), and m is the index of the element in the CDF. This adaptation of α allows for faster probability updates at the beginning of encoding / parsing a syntax element. The M-ary arithmetic coding process follows conventional arithmetic coding engine design. However, in some example implementations, only the most significant bits, e.g., 9 bits, of a 15-bit probability value are input to the arithmetic encoder / decoder.

[0113]

[0123] In some example implementations, for each transform block, entropy coding of transform coefficients may start by coding an all_zero flag, which indicates whether all quantized transform coefficients in the transform block are zero. If the all_zero flag is signaled as 0, this may be followed by signaling a primary transform kernel type, an end-of-block (EOB) position, and a secondary transform kernel index. The transform coefficient values ​​may then be coded in multiple level maps along with their sign values. For example, the level map may be coded as three level planes, e.g., a low-level plane, a mid-level plane, and a high-level plane, and the signs of the transform coefficients may be coded as separate planes. The low-level plane, the mid-level plane, and the high-level plane may correspond to different ranges of coefficient magnitudes (e.g., 0 to 2, 3 to 14, and 15 or greater, respectively). In some example implementations, the three level planes may be coded as follows: After the EOB value is coded, the low-level plane and the mid-level plane may be coded together in a reverse scan order (the scan order refers to the scan applied to the transform coefficients of the current transform block). The scan order may be, but is not limited to, one of a zigzag scan, a diagonal scan, a horizontal scan, and a vertical scan, depending on the transform block size and the applied transform type. Then, the sign plane and the high-level plane may be coded together in a forward scan order. After that, the remainder (coefficient magnitude minus 14) may be entropy coded, for example, using an Exp-Golomb code.

[0114]

[0124] In some example implementations, the context model applied to low-level planes may depend on 1) the primary transform direction, i.e., one of bidirectional, horizontal, and vertical transform directions; 2) the transform block size; and 3) up to five neighboring transform domain coefficients. Mid-level planes may use similar context models, but the number of neighboring coefficients that determine the context may be reduced, for example, from five to two. High-level planes may be coded using Exp-Golomb codes without using context modeling. For code planes, with the exception of the DC code, which may be coded using the DC code from a neighboring transform unit, the code values ​​of all other transform coefficients may be coded directly without using context modeling.

[0115]

[0125] Turning specifically to entropy coding of syntax elements related to inter-prediction modes, such syntax elements may be coded based on context-adaptive arithmetic coding. Such syntax elements may include, but are not limited to, an indicator for single-mode inter prediction (represented by "inter_single_mode"), an indicator for mixed-mode inter prediction (represented by "inter_comp_mode"), an indicator for optical flow mode (represented by "use_optical_flow"), and a dynamic motion vector reference list (DRL) index (represented by "drl_index") for specifying a motion vector predictor in the DRL with or without motion vector difference (MVD). Some of these syntax elements are binary. Others may be non-binary. For example, a DRL index for a DRL list longer than two may not be binary. Non-binary syntax elements may be binarized and coded based on two-symbol context-adaptive binary arithmetic coding or multi-symbol context-adaptive arithmetic coding.

[0116]

[0126] Each of these syntax elements for inter prediction may be coded using an appropriately selected context for optimal coding efficiency. In some implementations, a set of contexts may be provided for selection during the coding / decoding process of a particular syntax element. In some implementations, the number of possible contexts and their selection during the coding / decoding process of a predictive block may depend on the coding mode and coding information of spatially neighboring blocks of the current coding block within the same frame, or the temporal motion information of the current coding block, reflecting a statistical correlation between the coding mode and motion vectors of the spatially neighboring blocks of the current coding block and the coding mode and motion vectors of the current coding block.

[0117]

[0127] In some example implementations, the context derivation (including both the number of possible contexts and the selection of a particular context depending on the dynamic coding information) may be designed to be based on or derived from one or more context parameters, which may further depend on coding information related to spatially neighboring blocks and / or a temporal motion vector predictor of the current coding block.

[0118]

[0128] In some example implementations, a first context parameter, denoted "newmv_ctx," may be derived to represent an estimated likelihood that the current prediction (or transform) block uses motion vector prediction with both a motion vector predictor and a motion vector differential. A second context parameter, denoted "refmv_ctx," may be derived to represent an estimated likelihood that the current prediction (or transform) block directly uses a motion vector predictor without a motion vector differential. A third context parameter, denoted "zeromv_tx," may be derived to represent an estimated likelihood that the current prediction (or transform) block uses a global motion vector predictor.

[0119]

[0129] In some example implementations, these context parameters may be derived from the coding properties and information of spatially neighboring blocks and / or temporal motion vector predictor information of the current coding block. An example is shown in FIG. 9. FIG. 9 illustrates a coding block (e.g., a superblock) 902, which may include multiple prediction or transform blocks distributed row-wise and column-wise. In the example of FIG. 9, the coding block 902 includes six rows and six columns of prediction blocks (prediction blocks are assumed to be the same as transform blocks and are referred to as "blocks"). Each block may be associated with a block position represented by a pair of column and row coordinates (e.g., relative to the frame or other reference objects within the frame). The top-leftmost block 906 may be associated with block coordinates (L,T). The top-rightmost block 908 may be associated with block coordinates (R,T). The bottom-leftmost block 910 may be associated with block coordinates (L,B). The bottom-rightmost block 912 may be associated with block coordinates (R,B). The current block is the block coordinate (C col ,C row ) and designated as 904.

[0120]

[0130] In some implementations, a predefined or signaled set of spatial neighboring blocks of the current coding block 902 may be scanned and examined to determine one or more of the above context parameters. The predefined or signaled locations of the set of neighboring blocks to be scanned and the order in which these blocks are scanned may be based on general statistics or real-time statistics of the correlation between the coding characteristics of the neighboring blocks and the coding characteristics of the current block 904. In the example implementation illustrated in FIG. 9, the sets of neighboring blocks immediately to the left and immediately above the current coding block 902 may be scanned in the order specified in FIG. 9. These are the nearest neighboring blocks that are coded / decoded before the current coding block 902. The scanning of these nearest spatial neighboring blocks may follow the same manner as the scanning of an MVP list (DRL list structure), for example. For example, the scanning may follow the following: 1. The next block 920 at the bottom and immediately to the left of the coding block 902, with block coordinates (L-1, B). 2. The neighboring block 922 immediately to the right of and above the coding block 902, with block coordinates (R, T-1). 3. The neighboring block 924 at the top and immediately to the left of the coding block 902, with block coordinates (L-1,T). 4. The neighboring block 926 located immediately above and to the left of the coding block 902 and having block coordinates (L, T-1). 5. The neighboring block 928 immediately below and to the left of the lower left corner of the coding block 902, with block coordinates (L-1, B-1). 6. A neighboring block 930 immediately above and to the right of the upper right corner of the coding block 902, with block coordinates (R-1, T-1). 7. The block in the same row as the current block 904 and immediately to the left of the coding block 902, with block coordinates (L-1, C row ) adjacent block 932. 8. The block in the same column as the current block 904 and immediately above the coding block 902, with block coordinates (Ccol , T-1).

[0121]

[0131] In some example implementations, a scan through these spatially neighboring blocks may be performed to check whether these blocks share at least one reference frame for inter-prediction with the current block 904. The number of blocks among these neighboring blocks that share a reference frame with the current block 904 may be tracked. The number of reference frame matching blocks in the left neighbor column (among blocks 920, 924, 928, and 932 of FIG. 9) and in the above neighbor row (among blocks 922, 926, 930, and 934) may be tracked separately and are represented by col_match_count and row_match_count, respectively.

[0122]

[0132] In some example implementations, a scan through these spatially neighboring blocks immediately to the left and above may also be performed to check the number of these blocks that are inter-predicted based on MVD (e.g., blocks in which the new MV mode involves a motion vector predictor and MVD, as described above), the number or count being represented as newmv_count.

[0123]

[0133] In some example implementations, the reference frame matching parameters may be derived based on the row and column counts of the nearest neighboring block having a reference frame match with the current coding block 904. For example, such reference frame matching parameters may be derived as follows: nearest_match=(row_match_count>0)+(col_match_count>0) (3)

[0124]

[0134] Here, the () operator returns "1" or "0" depending on whether the logical expression enclosed in the () operator is true or false, respectively. Therefore, nearest_match has three possible values ​​(0, 1, and 2) depending on the derivation above.

[0125]

[0135] In some example implementations, two of the above three context parameters, newmv_ctx and refmv_ctx, may be derived from nearest_match and newmv_count. For example, as follows: newmv_ctx=2*nearest_match+(newmv_count>0) (4) refmv_ctx=2*nearest_match+(newmv_count<3) (5)

[0126]

[0136] In the above derivation, the "2*" operation is effectively a left bit shift. Therefore, newmv_ctx and refmv_ctx each have six possible values ​​(0-5).

[0127]

[0137] In some example implementations, to derive the third context parameter, a temporal motion vector prediction (TMVP) candidate (representing a global MV context) zeromv_ctx may be checked at the top-left position inside the spatially co-located coding block of the current coding block 902 in a different time frame. If this TMVP candidate is not available or if one component (horizontal or vertical) of the motion vector difference between this TMVP candidate and the global MV is greater than a threshold (e.g., 2pel), the zeromv_ctx context parameter may be set as 1; otherwise, it may be set as 0.

[0128]

[0138] In some example implementations, the context for various inter-prediction-related syntax elements may be derived based on one or more of the above context parameters. In particular, the coding context for syntax elements including, but not limited to, drl_index, inter_single_mode, inter_comp_mode, and use_optical_flow may be derived from one or more of these context parameters. For example, the coding context for these syntax elements may be derived according to newmv_ctx, refmv_ctx, and zeromv_ctx as follows: drl_index_ctx0=2*newmv_ctx+zeromv_ctx(12 possible contexts) (6) drl_index_ctx1=2*newmv_ctx+zeromv_ctx(12 possible contexts) (7) drl_index_ctx2=2*newmv_ctx+zeromv_ctx(12 possible contexts) (8) inter_single_mode_ctx=2*2*newmv_ctx+2*zeromv_ctx+isrefmv_ctx[refmv_ctx] (24 possible contexts) (9) where isrefmv[6]={0,0,1,0,1,0} inter_comp_mode_ctx=comp_mode_ctx_map[refmv_ctx>>1][MIN(newmv_ctx,4)] (8 possible contexts) (10) where comp_mode_ctx_map[3][5]={ {0,1,1,1,1}, {1,2,3,4,4}, {4,4,5,6,7} } use_optical_flow_ctx=comp_mode_ctx_map[refmv_ctx>>1][MIN(newmv_ctx,4)](8 possible contexts) (11) where comp_mode_ctx_map[3][5]={ {0,1,1,1,1}, {1,2,3,4,4}, {4,4,5,6,7} }

[0129]

[0139] In the above, it is assumed that drl_index includes three bits, and each of the three bits may be coded with a different context, as specified by drl_index_ctx0, drl_index_ctx1, and drl_index_ctx2. The values ​​derived above may be used to specify the context to be used for the corresponding syntax element or bit of the syntax element (e.g., as an index into a group of contexts). The number of possible context groups for each of the syntax elements is also indicated in the above equation.

[0130]

[0140] In the various example implementations described above, there are a relatively large number of possible contexts for each of the various inter-prediction related syntax elements for the encoder / decoder to maintain. The complexity can negatively affect the efficiency of the parsing process. Furthermore, the scanning process in avoiding the various counts described above is also of a certain complexity. To provide simplified example implementations that reduce the number of contexts or simplify the scanning process, the following additional disclosure is included.

[0131]

[0141] In some implementations, the third context parameter zeromv_ctx may not need to be derived, and correspondingly, the inter mode context derivations above may be modified to remove the dependency of these derivations on zeromv_ctx, insofar as these derivations depend on zeromv_ctx. For example, the coding context derivations for drl_index and inter_single_mode among various syntax elements may be modified to the following: drl_index_ctx0=newmv_ctx(6 possible contexts) (12) drl_index_ctx1=newmv_ctx(6 possible contexts) (13) drl_index_ctx2=newmv_ctx(6 possible contexts) (14) inter_single_mode_ctx=2*newmv_ctx+isrefmv_ctx[refmv_ctx] (12 possible contexts) (15) where isrefmv[6]={0,0,1,0,1,0}.

[0132]

[0142] The number of possible coding contexts for each of these syntax elements is therefore reduced two-fold: the "2*" operator for drl_index in equations (7) and (8), as well as one of the two "2*" operators in equation (9), are not needed and therefore, furthermore, are removed in the above implementation in equations (12)-(15) because they do not affect the number of possible contexts once the dependency on zeromv_ctx is removed.

[0133]

[0143] Furthermore, in the above example implementation, the inter single-mode context may be designed to be derived only according to newmv_ctx among the above context parameters. inter_single_mode_ctx=newmv_ctx(6 possible contexts) (16)

[0134]

[0144] This reduces the possible contexts for inter_single_mode from 12 or 24 to 6. Again, the double "2*" operator in equation (9) is unnecessary since it does not affect the number of possible contexts once the dependency on zeromv_ctx is removed.

[0135]

[0145] In some example implementations, the coding context for inter_comp_mode and / or use_optical_flow may be designed to be derived solely dependent on newmv_ctx. inter_comp_mode_ctx=newmv_ctx(6 possible contexts) (17) use_optical_flow_ctx=newmv_ctx(possible context) (18)

[0136]

[0146] This eliminates the mapping design in equations (10) and (11) and reduces the number of possible contexts for inter_comp_mode and use_optical_flow, respectively, from 8 to 6.

[0137]

[0147] In some example implementations, the drl_index context may be designed to be derived only depending on newmv_ctx. drl_index_ctx0=newmv_ctx(6 possible contexts) (19) drl_index_ctx1=newmv_ctx(6 possible contexts) (20) drl_index_ctx2=newmv_ctx(6 possible contexts) (21)

[0138]

[0148] This achieves the same conclusion as for the design of equations (12-14) above: the drl_index context is designed to remove the dependency on zeromv_ctx, and the number of possible contexts, drl_index contexts, is reduced to 6 each. Again, the "2*" operator in drl_index in equations (7) and (8) is unnecessary because it does not affect the number of contexts in equations (20) and (21).

[0139]

[0149] In some example implementations, the coding context for various syntax elements related to inter-prediction modes may be designed to depend only on the mode information of spatial neighboring blocks. For example, among the context parameters of the above three examples, newmv_ctx and refmv_ctx depend on spatial neighboring blocks, while zeromv_ctx depends on TMVP information. Therefore, according to these implementations of the context design for various syntax elements related to inter-prediction modes, the dependency on newmv_ctx and refmv_ctx may be retained, while the dependency on zeromv_ctx may be removed. Unnecessary "2*" operators may also be removed as appropriate.

[0140]

[0150] In some example embodiments, the context design and selection for drl_index may be reduced by context encoding only the first N bits that need to be signaled or have unequal probability distributions, and encoding the remaining bits using equal probability, where N is a positive integer. For example, if N is equal to 2, only drl_index_ctx0 and drl_index_ctx1 are used, drl_index_ctx2 is discarded, and equal probability is used to encode drl_index[2]. The value of N can be predefined at both the encoder and decoder sides, or signaled in the bitstream in a high-level syntax.

[0141]

[0151] In some example implementations, a reduction in the number of possible contexts for various inter-prediction mode syntaxes may be achieved by a modified design for the derivation of the various parameters mentioned above, including but not limited to nearest_match. For example, the number of possible values ​​of nearest_match may be reduced. The number of possible values ​​of context parameters that depend on parameters such as newmv_ctx and refmv_ctx should be correspondingly reduced. For example, in some implementations, instead of calculating the sum of two Boolean values ​​regarding whether the top row and immediately left column of a neighboring block have at least one matched reference frame with the current block (i.e., equation (3), nearest_match=row_match_count>0)+(col_match_count>0)), an "OR" operation may be performed between these Boolean values, such as nearest_match=row_match_count>0) / / (col_match_count>0). In other words, in such example implementations, nearest_match becomes binary, with only two possible values: "0" or "1."

[0142]

[0152] Specifically, in some example implementations, the context parameters newmv_ctx and refmv_ctx may be derived as follows: nearest_match=(row_match_count>0)||(col_match_count>0) (22) newmv_ctx=2*nearest_match+(newmv_count>0) (23) refmv_ctx=2*nearest_match+(newmv_count<3) (24)

[0143]

[0153] This reduces the possible values ​​of newmv_ctx and refmv_ctx to 0-3 (4 possible values).

[0144]

[0154] In some example implementations, the possible values ​​of the newmv_ctx and refmv_ctx parameters may be further reduced by removing the "2*" operator above by using the example derivation below. nearest_match=(row_match_count>0)||(col_match_count>0) (twenty five) newmv_ctx=nearest_match+(newmv_count>0) (26) refmv_ctx=nearest_match+(newmv_count<3) (27)

[0145]

[0155] This reduces the possible values ​​of newmv_ctx and refmv ctx to 0-2 (3 possible values).

[0146]

[0156] In some further example implementations, the refmv_ctx parameter may be further simplified to remove its dependency on newmv_count. nearest_match=(row_match_count>0)||(col_match_count>0) (28) newmv_ctx=2*nearest_match+(newmv_count>0) (29-1) refmv_ctx=nearest_match (30-1)

[0147]

[0157] This further reduces the value of refmv_ctx to 0 to 1 (a binary with two possible values). The "2*" operator may or may not be included.

[0148]

[0158] Similarly, in some alternative example implementations, the newmv_ctx parameter may instead be further simplified to remove its dependency on newmv_count. nearest_match=(row_match_count>0)||(col_match_count>0) newmv_ctx=nearest_match (29-2) refmv_ctx=2*nearest_match+(newmv_count>0) (30-2)

[0149]

[0159] This further reduces the value of newmv_ctx to 0 to 1 (a binary with two possible values). The "2*" operator may or may not be included.

[0150]

[0160] In some alternative example implementations, during the derivation of newmv_ctx and refmv_ctx, instead of calculating a weighted sum of nearest_match and a Boolean value for whether newmv_count is greater than 0 or less than 3, an alternative derivation formula may be used to reduce the number of possible values ​​for the newmv_ctx and refmv_ctx context parameters.

[0151]

[0161] For example, newmv_ctx and refmv_ctx may be derived by removing the "2*" operator from equations (4) and (5) as follows: newmv_ctx=nearest_match+(newmv_count>0) (31) refmv_ctx=nearest_match+(newmv_count<3) (32)

[0152]

[0162] This reduces the possible values ​​to 0 to 3 (4 possible values) when nearest_match takes the above sum form for (row_match_count>0) and (col_match_count>0), and to 0 to 2 (3 possible values) when nearest_match takes the above OR form for (row_match_count>0) and (col_match_count>0).

[0153]

[0163] In some example implementations, the newmv_ctx and refmv_ctx context parameters may be derived by ORing rather than summing their derived components as follows: newmv_ctx=nearest_match||(newmv_count>0) (33) refmv_ctx=nearest_match||(newmv_count<3) (34)

[0154]

[0164] This makes them binary and reduces their possible values ​​to 0 to 1 (two possible values) when nearest_match takes the above OR form for (row_match_count>0) and (col_match_count>0). The above sum form of nearest_match for (row_match_count>0) and (col_match_count>0) may not be applicable due to the OR operation in equations (33) and (34).

[0155]

[0165] In some alternative example implementations, the newmv_ctx and refmv_ctx context parameters may be derived by removing their dependency on newmv_count. newmv_ctx=nearest_match (35) refmv_ctx=nearest_match (36)

[0156]

[0166] This reduces the possible values ​​to 0 to 2 (3 possible values) when nearest_match takes the sum form above for (row_match_count>0) and (col_match_count>0), and to 0 to 1 (2 possible values) when nearest_match is binary and takes the OR form above for (row_match_count>0) and (col_match_count>0).

[0157]

[0167] The above example implementations provide ways to reduce the number of possible values ​​of various parameters used to derive context parameters. In some other example implementations, alternatively or additionally, the number of spatially neighboring blocks to be scanned may be reduced, and the order in which these blocks are scanned may be designed to reduce the scanning complexity and / or the number of possible values ​​of the various parameters for deriving coding context for syntax related to inter-prediction modes.

[0158]

[0168] In some example implementations, during the determination of col_match_count, row_match_count, and newmv_count above, only the first M adjacent spatial candidate blocks may be scanned, and the order in which the spatial candidate blocks are scanned may be predefined or signaled. M is a positive integer. The value of M may be predefined at both the encoder and decoder sides or signaled in the bitstream in a high-level syntax.

[0159]

[0169] For example, the scanning order of the M spatial candidates may be the same as the scanning order of the MVP list structure, as shown in FIG.

[0160]

[0170] For example, only the first M=2 candidates are scanned to derive newmv_ctx and refmv_ctx. As another example, only the first 4 candidates are scanned to derive newmv_ctx and refmv_ctx.

[0161]

[0171] In some example implementations, a scan order other than that shown in Figure 9 may be followed. For example, the nearest neighbors in Figure 9 may be scanned in the order 2, 1, 4, 3 for M=4.

[0162]

[0172] In some example implementations, a value M may be applied to each scan of col_match_count, row_match_count, and newmv_count. For example, up to M nearest top-row neighbor blocks may be scanned to determine row_match_count. Similarly, up to M nearest left-row neighbor blocks may be scanned to determine col_match_count. Similarly, up to M nearest top-row and left columns of a neighbor block may be scanned to determine newmv_count in a predefined or signaled scan order. In some alternative implementations, M may be applied in aggregate for determining col_match_count and row_match_count. That is, the total number of top-rows and left columns of a neighbor block scanned to determine these counts may not exceed M. For example, if M=4, up to two top-rows of a neighbor block may be scanned to determine row_match_count, and up to two left columns of a neighbor block may be scanned to determine col_match_count.

[0163]

[0173] In some example implementations, the first M adjacent spatial candidate blocks may be scanned in a predefined or signaled scanning order to determine the parameters necessary to derive newmv_ctx and refmv_ctx. From the M scanned candidate blocks, K of them may be used to determine the parameters for deriving newmv_ctx, while L of them may be used to determine the parameters for deriving refmv_ctx. L and K may be positive integers not greater than M.

[0164]

[0174] Moving on to determining the third context parameter zeromv_ctx related to TMVP, instead of setting the value of zeromv_ctx to "0" or "1," a TMVP at another position may be used by checking the TMVP candidate at the top-left position inside a spatially co-located coding block of the current coding block 902 in Figure 9 in a different time frame to see if it is available, and comparing one component (horizontal or vertical) of the motion vector difference between this TMVP candidate and the global MV to see if it is greater than a threshold value (e.g., 2pels). In some implementations, the TMVP corresponding to the top-left position of the current coding block and one or more TMVP candidates at the following positions may be checked for availability and compared to the global MV. 1. TMVP candidate in the central position 2. The TMVP candidate at the bottom right

[0165]

[0175] In some implementations, the TMVPs at one or more of the above-listed locations may be checked for availability and compared to the global MV to determine the zeromv_ctx parameter. In other words, any combination of the TMVs at the top-left, center, and bottom-right locations may be checked for availability and compared to the global MV to determine the zeromv_ctx parameter.

[0166]

[0176] 10 shows a flowchart 1000 of an example method according to the principles underlying the above implementation. The flow of the example method starts at S1001. At S1010, it is determined whether a current prediction block is associated with an inter-prediction mode. At S1020, in response to the current prediction block being associated with the inter-prediction mode, a first parameter is derived representing whether the current prediction block shares an inter-prediction reference frame with a spatially neighboring prediction block of the current coding block. At S1030, a second parameter associated with the utilization of motion vector prediction in the spatially neighboring prediction block of the current coding block is further derived. At S1040, a coding context for at least one syntax associated with the inter-prediction mode of the current prediction block is selected according to the first parameter and the second parameter. The example method stops at S1099.

[0167]

[0177] 11 shows a flowchart 1100 of an example method according to the principles underlying the above implementation. The flow of the example method starts at S1101. At S1110, it is determined whether the current prediction block is associated with an inter prediction mode. At S1120, in response to the current prediction block being associated with the inter prediction mode, a context parameter is derived based on coding mode information associated with spatially neighboring prediction blocks of the current coding block. At S1130, in response to the context parameter being independent of the coding mode information of any of the temporal prediction blocks associated with the current coding block, a coding context for syntax associated with inter prediction of the current prediction block is selected. The example method stops at S1199.

[0168]

[0178] FIG. 12 shows a flowchart 1200 of an example method according to the principles underlying the above implementation. The flow of the example method starts at S1201. At S1210, it is determined whether a current prediction block is associated with an inter prediction mode and whether a motion vector of the current prediction block is predicted by a reference motion vector. At S1220, in response to the current prediction block being associated with an inter prediction mode and the motion vector of the current prediction block being predicted by the reference motion vector, one or more coding contexts for a dynamic motion vector reference list (DRL) index that identifies the reference motion vector are derived. At S1230, the coding contexts are applied to decode the first N bits of the DRL index, where N is a predefined or configured positive integer. At S1240, an equal probability distribution is applied as a context for processing the remaining bits of the DRL index. The example method stops at S1299.

[0169]

[0179] In the embodiments and implementations of the present disclosure, any steps and / or operations may be combined or arranged in any quantity or order as desired. Two or more of the steps and / or operations may be performed in parallel. The embodiments and implementations of the present disclosure may be used separately or in combination in any order. Furthermore, each of the method (or embodiment), encoder, and decoder may be implemented by processing circuitry (e.g., one or more processors or one or more integrated circuits). In one example, the one or more processors execute a program stored on a non-transitory computer-readable medium. The embodiments of the present disclosure may apply to luma blocks or chroma blocks. The term block may be interpreted as a prediction block, a coding block, or a coding unit (CU). The term block here may also be used to refer to a transform block. In the following sections, when referring to a block size, the block size may refer to the block width or height, the maximum value of the width and height, the minimum value of the width and height, the area size (width * height), or the aspect ratio of the block (width:height or height:width).

[0170]

[0180] The techniques described above can be implemented as computer software using computer-readable instructions and physically stored on one or more computer-readable media. For example, Figure 13 illustrates a computer system (1300) suitable for implementing certain embodiments of the disclosed subject matter.

[0171]

[0181] Computer software can be coded using any suitable machine or computer language, which may be subjected to mechanisms such as assembling, compiling, linking, etc. to produce code comprising instructions that can be executed by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., either directly or through translation, microcode execution, etc.

[0172]

[0182] The instructions may be executable by various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet of Things devices, and the like.

[0173]

[0183] 13 for computer system (1300) are exemplary in nature and are not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing embodiments of the present disclosure. The arrangement of components should not be interpreted as having any dependency or requirement regarding any one or combination of components illustrated in the exemplary embodiment of computer system (1300).

[0174]

[0184] The computer system (1300) may include certain human interface input devices that can respond to input by one or more human users through, for example, tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, clapping), visual input (e.g., gestures), or olfactory input (not shown). The human interface devices can also be used to capture certain media that are not necessarily directly related to conscious human input, such as audio (e.g., speech, music, ambient sounds), images (e.g., scanned images, photographic images obtained from a still-image camera), and video (e.g., two-dimensional video, three-dimensional video, including stereoscopic video).

[0175]

[0185] The input human interface devices may include one or more (only one of each depicted) of a keyboard (1301), a mouse (1302), a trackpad (1303), a touchscreen (1310), a data glove (not shown), a joystick (1305), a microphone (1306), a scanner (1307), and a camera (1308).

[0176]

[0186] The computer system (1300) may further include certain human interface output devices. Such human interface output devices may stimulate one or more of the human user's senses, for example, through tactile output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touchscreen (1310), data gloves (not shown), or joystick (1305), although haptic feedback devices that do not function as input devices are also possible), audio output devices (such as speakers (1309), headphones (not shown)), visual output devices (such as screens (1310), to include CRT screens, LCD screens, plasma screens, and OLED screens, each with or without touchscreen input capability and each with or without haptic feedback capability—some of which may be capable of outputting two-dimensional visual output or three-dimensional or higher-dimensional output through means such as stereoscopic output, virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).

[0177]

[0187] The computer system (1300) may also include human-accessible storage devices and their associated media, such as optical media including CD / DVD ROM / RW (1320), including media such as CDs / DVDs (1321), thumb drives (1322), removable hard drives or solid-state drives (1323), older magnetic media such as tape and floppy disks (not shown), and specialized ROM / ASIC / PLD-based devices (not shown) such as security dongles.

[0178]

[0188] Furthermore, those skilled in the art will understand that the term "computer-readable medium" as used in connection with the subject matter of this disclosure does not encompass transmission media, carrier waves, or other transient signals.

[0179]

[0189] The computer system 1300 may also include an interface 1354 to one or more communications networks 1355. For example, the networks may be wireless, wireline, or optical. Furthermore, the networks may be local, wide-area, metropolitan, vehicular, and industrial, real-time, delay-tolerant, and the like. Examples of networks include local area networks such as Ethernet; wireless LANs; cellular networks, such as to include GSM, 3G, 4G, 5G, LTE, and the like; TV wireline or wireless wide-area digital networks, such as to include cable TV, satellite TV, and terrestrial television broadcast; and vehicular and industrial networks, such as to include CAN buses. Particular networks generally require external network interface adapters attached to particular general-purpose data ports or peripheral buses 1349 (e.g., USB ports on the computer system 1300), while others are generally integrated into the core of the computer system 1300 by attachment to a system bus, as described below (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system (1300) can communicate with other entities. Such communication can be one-way receive-only (e.g., television broadcast), one-way transmit-only (e.g., CANbus to a specific CANbus device), or bidirectional, e.g., to other computer systems using local or wide-area digital networks. Specific protocols and protocol stacks are available for use with each of these networks and network interfaces, as described above.

[0180]

[0190] The aforementioned human interface devices, human-accessible storage devices, and network interfaces may be attached to the core (1340) of the computer system (1300).

[0181]

[0191] The core (1340) may include one or more central processing units (CPUs) (1341), graphics processing units (GPUs) (1342), specialized programmable processing units (1343) in the form of field programmable gate areas (FPGAs), hardware accelerators for specific tasks (1344), graphics adapters (1350), etc. These devices may be connected through a system bus (1348), along with read-only memory (ROM) (1345), random access memory (1346), internal mass storage (1347) such as an internal non-user-accessible hard drive, SSD, etc. In some computer systems, the system bus (1348) may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may be attached directly to the core's system bus (1348) or through a peripheral bus (1349). In one example, the screen (1310) can be connected to a graphics adapter (1350). Peripheral bus architectures include PCI, USB, etc.

[0182]

[0192] The CPU (1341), GPU (1342), FPGA (1343), and accelerator (1344) can execute specific instructions that, in combination, can make up the aforementioned computer code. This computer code can be stored in ROM (1345) or RAM (1346). Temporary data can also be stored in RAM (1346), while permanent data can be stored, for example, in internal mass storage (1347). Rapid storage and retrieval to and from any of the memory devices can be enabled through the use of cache memory, which can be closely associated with one or more of the CPU (1341), GPU (1342), mass storage (1347), ROM (1345), RAM (1346), etc.

[0183]

[0193] The computer-readable medium can bear computer code for performing various computer-executed operations. The medium and computer code can be those specially designed and constructed for the purposes of the present disclosure, or they can be of the kind well known and available to those skilled in the computer software arts.

[0184]

[0194] As a non-limiting example, the computer system (1300) having the architecture, and specifically the core (1340), can provide functionality as a result of a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media can be user-accessible mass storage as introduced above, as well as media associated with specific storage of the core (1340) of a non-transitory nature for storing software for implementing various embodiments of the present disclosure. The computer-readable media can include one or more memory devices or chips, depending on specific needs. The software can cause the core (1340) and specifically the core's processor (including a CPU, GPU, FPGA, etc.) to perform specific processes, or specific portions of specific processes, described herein, including defining data structures stored in RAM (1346) and modifying such data structures in accordance with the software-defined processes. Additionally or alternatively, a computer system may provide functionality as a result of logic hardwired or otherwise embodied in circuitry (e.g., accelerator (1344)) that can operate in place of or together with software to perform particular processes, or particular portions of particular processes, described herein. References to software may encompass logic, and vice versa, where appropriate. References to computer-readable media may encompass, where appropriate, circuitry (such as an integrated circuit (IC)) that stores software for execution, circuitry that embodies logic for execution, or both. The present disclosure encompasses any appropriate combination of hardware and software.

[0185]

[0195] While this disclosure has described several exemplary embodiments, there are alterations, permutations, and various substitute equivalents that are within the scope of this disclosure. It will thus be recognized that those skilled in the art will be capable of devising numerous systems and methods that, while not explicitly shown or described herein, embody the principles of the present disclosure and are therefore within the spirit and scope of the present disclosure.

Claims

1. 1. A method for decoding a current prediction block within a current coding block of a video stream, comprising: determining whether the current prediction block is associated with an inter prediction mode; deriving a first parameter in response to the current prediction block being associated with the inter prediction mode, the first parameter representing whether the current prediction block shares an inter prediction reference frame with a spatially neighboring prediction block of the current coding block; deriving a second parameter associated with the use of motion vector prediction in the spatially neighboring predicted block of the current coding block; selecting a coding context for at least one syntax associated with the inter prediction mode of the current predictive block according to the first parameter and the second parameter; A method comprising:

2. The method of claim 1 , wherein the first parameter comprises a binary parameter.

3. The first parameter, represented by "nearest_match", a binary "1" when any one of a set of spatially neighboring prediction blocks within a predetermined set of neighboring positions relative to the current coding block is associated with a reference frame that matches that of the current prediction block; and otherwise, binary "0" is derived as the second parameter, denoted "newmv_count", indicates the number of prediction blocks in the set of spatially neighboring prediction blocks having at least one motion vector predicted by the reference motion vector; The method of claim 2.

4. 4. The method of claim 3, wherein the set of spatially neighboring predictive blocks includes a first subset of spatially neighboring predictive blocks located immediately to the left of the current coding block and a second subset of spatially neighboring predictive blocks located immediately above the current coding block.

5. the first parameter is derived as nearest_match=(row_match_count>0) / / (column_match_count>0), where " / / " represents a Boolean OR operator, and the () operator is configured to generate 1 and 0 when the logical expression enclosed by the () operator is true and false, respectively; column_match_count represents the number of prediction blocks in the first subset of spatially neighboring prediction blocks having reference frames that match the current prediction block; row_match_count represents the number of prediction blocks in the second subset of spatially neighboring prediction blocks having reference frames that match the current prediction block; The method of claim 4.

6. deriving a first context index, denoted "newmv_ctx", from the first parameter and the second parameter according to a first relationship; deriving a second context index, denoted "refmv_ctx", from the first parameter and the second parameter according to a second relationship; selecting the coding context in response to the first context index and the second context index; The method of claim 5 further comprising:

7. the first relationship comprises newmv_ctx=2*nearest_match+(newmv_count>0); the second relationship comprises refmv_ctx=2*nearest_match+(newmv_count<A) or refmv_ctx=nearest_match+(newmv_count<A), where A is a predefined or signaled positive integer; The method of claim 6.

8. the first relationship comprises newmv_ctx=nearest_match+(newmv_count>0); the second relationship comprises refmv_ctx=nearest_match+(newmv_count<A) or refmv_ctx=2*nearest_match+(newmv_count<A), where A is a predefined or signaled positive integer; The method of claim 6.

9. the first relationship comprises newmv_ctx=2*nearest_match+(newmv_count>0); the second relationship includes refmv_ctx=nearest_match; The method of claim 6.

10. the first relationship comprises newmv_ctx=nearest_match+(newmv_count>0); the second relationship includes refmv_ctx=nearest_match; The method of claim 6.

11. the first parameter, denoted "nearest_match", indicates the extent to which the spatially neighboring prediction block immediately to the left or immediately above the current coding block matches the current prediction block relative to a reference frame; the second parameter, denoted "newmv_count", indicates the number of prediction blocks among the spatially neighboring prediction blocks immediately to the left or immediately above the current coding block that have at least one motion vector predicted by at least one reference motion vector; The method according to any one of claims 1 to 10.

12. deriving a first context index, denoted "newmv_ctx", from the first parameter and the second parameter according to a first relationship; deriving a second context index, denoted "refmv_ctx", from the first parameter and the second parameter according to a second relationship; selecting the coding context in response to the first context index and the second context index; The method of claim 11 further comprising:

13. the first relation comprises newmv_ctx=nearest_match+(newmv_count>0), wherein the () operator is configured to generate 1 and 0 when a logical expression enclosed by the () operator is true and false, respectively; the second relationship comprises refmv_ctx=nearest_match+(newmv_count<A), where A is a predefined or signaled positive integer; The method of claim 12.

14. the first relation comprises newmv_ctx=nearest_match / / (newmv_count>0), where " / / " represents a Boolean OR operator, and the () operator is configured to generate 1 and 0 when the logical expression enclosed by the () operator is true and false, respectively; the second relationship includes refmv_ctx=nearest_match / / (newmv_count<A), where A is a predefined or signaled positive integer; The method of claim 12.

15. the first relationship includes newmv_ctx=nearest_match; the second relationship includes refmv_ctx=nearest_match; The method of claim 12.

16. 1. A method for decoding a current prediction block within a current coding block of a video stream, comprising: determining whether the current prediction block is associated with an inter prediction mode; deriving a context parameter based on coding mode information associated with a spatially neighboring predictive block of the current coding block in response to the current predictive block being associated with the inter prediction mode; selecting a coding context for syntax associated with the inter prediction of the current predictive block in response to the context parameter being independent of coding mode information of any temporal predictive block associated with the current coding block; A method comprising:

17. 1. An electronic device for decoding a current prediction block within a current coding block of a video stream, the electronic device comprising: determining whether the current prediction block is associated with an inter prediction mode; deriving a first parameter in response to the current prediction block being associated with the inter prediction mode, the first parameter representing whether the current prediction block shares an inter prediction reference frame with a spatially neighboring prediction block of the current coding block; deriving a second parameter associated with the use of motion vector prediction in the spatially neighboring predicted block of the current coding block; selecting a coding context for at least one syntax associated with the inter prediction mode of the current predictive block according to the first parameter and the second parameter; and a processor for executing said instructions.

18. the first parameter comprises a binary parameter; The first parameter, represented by "nearest_match", a binary "1" when any one of a set of spatially neighboring prediction blocks within a predetermined set of neighboring positions relative to the current coding block is associated with a reference frame that matches that of the current prediction block; and otherwise, binary "0" is derived as the second parameter, denoted "newmv_count", indicates the number of prediction blocks in the set of spatially neighboring prediction blocks having at least one motion vector predicted by a reference motion vector; the set of spatially neighboring prediction blocks includes a first subset of spatially neighboring prediction blocks located immediately to the left of the current coding block and a second subset of spatially neighboring prediction blocks located immediately above the current coding block.

18. The electronic device of claim 17.

19. the first parameter is derived as nearest_match=(row_match_count>0) / / (column_match_count>0), where " / / " represents a Boolean OR operator, and the () operator is configured to generate 1 and 0 when the logical expression enclosed by the () operator is true and false, respectively; column_match_count represents the number of prediction blocks in the first subset of spatially neighboring prediction blocks having reference frames that match the current prediction block; row_match_count represents the number of prediction blocks in the second subset of spatially neighboring prediction blocks having reference frames that match the current prediction block; the processor: deriving a first context index, denoted "newmv_ctx", from the first parameter and the second parameter according to a first relationship; deriving a second context index, denoted "refmv_ctx," from the first parameter and the second parameter according to a second relationship; determining the coding context in response to the first context index and the second context index; and further configured to execute the instructions to cause the electronic device to:

20. The electronic device of claim 18.

20. 1. A method for decoding a current prediction block within a current coding block of a video stream, comprising: determining whether the current prediction block is associated with an inter prediction mode and whether a motion vector of the current prediction block is predicted by a reference motion vector; deriving one or more coding contexts for a dynamic motion vector reference list (DRL) index that identifies the reference motion vector in response to the current prediction block being associated with an inter prediction mode and the motion vector of the current prediction block being predicted by the reference motion vector; applying the coding context to decode the first N bits of the DRL index, where N is a predefined or configured positive integer; applying an equal probability distribution as a context for processing the remaining bits of the DRL index; A method comprising:

21. 21. A computer program causing a computer to execute a method for decoding a current prediction block within a current coding block of a video stream, the method being the method of any one of claims 1 to 10, 16 and 20.

Images