Method, apparatus, and program for AMVP merge mode with SBTMVP

Abstract

The processing circuit receives an encoded video bitstream including coded information for a current block in a current picture. The coded information indicates an AMVP merge mode, which uses a bidirectional predictor including an AMVP predictor in a first direction and a merge predictor in a second direction. The processing circuit constructs an AMVP candidate list including multiple AMVP candidates for the AMVP predictor and a merge candidate list including multiple merge candidates for the merge predictor. The merge candidate list includes SbTMVP candidates. The processing circuit determines an AMVP predictor from the AMVP candidate list, determines a merge predictor from the merge candidate list, and reconstructs the current block according to a combination of a first prediction according to the AMVP predictor in the first direction and a second prediction according to the merge predictor in the second direction.

Description

[Technical Field]

[0001] Incorporation by Reference This application claims the benefit of priority to U.S. Patent Application No. 18 / 406,411, entitled "AMVP Merge Mode with SBTMVP," filed January 8, 2024. This application claims the benefit of priority to U.S. Provisional Application No. 63 / 437,986, entitled "AMVP Merge Mode with SbTMVP," filed January 9, 2023. The disclosures of the prior applications are incorporated herein by reference in their entireties.

[0002] Technical Field This disclosure describes embodiments generally related to video coding. [Background technology]

[0003] The background discussion provided herein is intended to generally set forth the context of the present disclosure. The work of the inventors identified in this application, to the extent that their work is described in this background section, as well as aspects of this specification that may not qualify as prior art as of the filing date, are not admitted expressly or impliedly as prior art to the present disclosure.

[0004] Image / video compression can help transmit image / video data across different devices, storage, and networks with minimal quality degradation. In some examples, video codec technology can compress video based on spatial and temporal redundancy. In one example, a video codec can use a technique called intra-prediction, which can compress images based on spatial redundancy. For example, intra-prediction can use reference data from the current picture being reconstructed for sample prediction. In another example, a video codec can use a technique called inter-prediction, which can compress images based on temporal redundancy. For example, inter-prediction can predict samples in a current picture from a previously reconstructed picture using motion compensation. Motion compensation can be indicated by a motion vector (MV). Summary of the Invention [Means for solving the problem]

[0005] Aspects of the present disclosure include methods and apparatus for video encoding / decoding. In some examples, an apparatus for video decoding includes a processing circuit. The processing circuit receives an encoded video bitstream including encoded information for a current block in a current picture. The encoded information indicates an advanced motion vector prediction-merge (AMVP merge) mode, which uses a bidirectional predictor including an AMVP predictor in a first direction and a merge predictor in a second direction different from the first direction. The processing circuit constructs an AMVP candidate list including multiple AMVP candidates for the AMVP predictor and a merge candidate list including multiple merge candidates for the merge predictor. The merge candidate list includes at least subblock-based temporal motion vector prediction (SbTMVP) candidates. The processing circuit determines an AMVP predictor from the AMVP candidate list, determines a merge predictor from the merge candidate list, and reconstructs the current block according to a combination of a first prediction according to the AMVP predictor in a first direction and a second prediction according to the merge predictor in a second direction.

[0006] In some examples, the merge predictor is determined to be an SbTMVP candidate. The processing circuit determines a co-located block for the current block according to the displacement vector of the SbTMVP candidate in a collocated picture for the current picture, where the current block includes a plurality of sub-blocks, and the co-located block includes a plurality of co-located sub-blocks corresponding to the plurality of sub-blocks. The processing circuit determines a plurality of motion vectors corresponding to the plurality of sub-blocks in the current block according to motion information of each of the co-located sub-blocks in the co-located block, and generates a plurality of reconstructed sub-blocks corresponding to the plurality of sub-blocks in the current block according to the plurality of motion vectors corresponding to the plurality of sub-blocks in the current block. The reconstructed sub-blocks form a second prediction.

[0007] In some examples, to construct the merge candidate list, the processing circuitry derives SbTMVP candidates according to motion vectors of spatially neighboring blocks of the current block. In one example, the processing circuitry determines a co-located block in a co-located picture for the current picture according to the motion vectors of the spatially neighboring blocks of the current block, and uses the motion vector of the spatially neighboring block as a displacement vector (DV) for the SbTMVP candidate in response to the availability of sub-block-based motion information in the co-located block.

[0008] In one example, the availability of sub-block-based motion information for a co-located block is determined based on the availability of a motion vector in a central sub-block of the co-located block.

[0009] In another example, the availability of sub-block-based motion information for the co-located block is determined based on the availability of a motion vector associated with a second direction in a central sub-block of the co-located block.

[0010] In another example, the availability of sub-block-based motion information for the co-located block is determined based on the availability of a motion vector associated with a second direction for at least one sub-block in the co-located block.

[0011] In some examples, to derive the SbTMVP candidate, the processing circuit checks a plurality of spatially neighboring blocks in a predetermined order and identifies a first spatially neighboring block among the plurality of spatially neighboring blocks having a first motion vector. A co-located block according to the first motion vector has available subblock-based motion information. The processing circuit uses the first motion vector as a displacement vector (DV) for the SbTMVP candidate.

[0012] In some examples, to construct the merge candidate list, a zero displacement vector (DV) is used as the SbTMVP candidate in the merge candidate list. In one example, the processing circuitry checks multiple spatial neighboring blocks in a predetermined order, determines that none of the multiple spatial neighboring blocks has sub-block-based motion information available for the corresponding co-located block, and uses the zero displacement vector (DV) for the SbTMVP candidate.

[0013] In some examples, to determine a plurality of motion vectors for each of a plurality of sub-blocks in a current block, for a first sub-block in the plurality of sub-blocks, in response to a lack of motion information in a second direction in a co-located sub-block corresponding to the first sub-block, the processing circuit determines a first motion vector for the first sub-block according to at least one of a second motion vector of a neighboring sub-block of the first sub-block in the second direction, a third motion vector of a central sub-block of the co-located block in the second direction, and a default motion vector (e.g., zero DV).

[0014] In some examples, to determine a plurality of motion vectors for each of a plurality of sub-blocks in the current block, the processing circuit generates a binary mask for each of the plurality of sub-blocks in the current block, wherein binary bins in the binary mask for each of the sub-blocks indicate availability of a motion vector in a second direction.

[0015] In one example, in response to the binary bin for the subblock indicating that a motion vector in a second direction is available, a first reconstructed subblock is generated according to an AMVP predictor in the first direction, and a second reconstructed subblock is reconstructed according to a motion vector in the second direction, and the subblock is reconstructed based on a combination of the first reconstructed subblock and the second reconstructed subblock.

[0016] In another example, in response to the binary bin for the subblock indicating that a motion vector in a second direction is unavailable, uni-prediction is used for the subblock. For example, the processing circuit generates a first reconstructed subblock in a first direction according to an AMVP predictor and reconstructs the subblock based on the first reconstructed subblock.

[0017] In some examples, the merge candidate list includes an SbTMVP candidate, one or more spatially neighboring merge candidates, and one or more temporally neighboring merge candidates in a predefined order. In one example, the merge candidate list includes the SbTMVP candidate in the first position in the merge candidate list.

[0018] Aspects of the present disclosure also provide a non-transitory computer-readable medium storing instructions that, when executed by a computer, cause the computer to perform a method for video decoding / encoding. [Brief explanation of the drawings]

[0019] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings.

[0020] [Figure 1] FIG. 1 is a schematic diagram of an exemplary block diagram of a communication facility (100).

[0021] [Figure 2] FIG. 2 is a schematic diagram of an exemplary block diagram of a decoder.

[0022] [Figure 3] FIG. 2 is a schematic diagram of an exemplary block diagram of an encoder.

[0023] [Figure 4] Several examples show the subblock-based temporal motion vector prediction (SbTMVP) process used in SbTMVP mode.

[0024] [Figure 5] 1 shows a flowchart outlining another process according to some embodiments of the present disclosure.

[0025] [Figure 6] 1 shows a flowchart outlining a process according to some embodiments of the present disclosure.

[0026] [Figure 7] 1 is a schematic diagram of a computer system according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0027] 1 illustrates a block diagram of a video processing system (100) in some examples. The video processing system (100) is an example of an application of the disclosed subject matter, which is a video encoder and video decoder in a streaming environment. The disclosed subject matter is equally applicable to other video-enabled applications, including, for example, video conferencing, digital TV, streaming services, and storing compressed video on digital media, including CDs, DVDs, memory sticks, etc.

[0028] The video processing system (100) includes a capture subsystem (113), which may include a video source (101), such as a digital camera, generating a stream of uncompressed video pictures (102). In one example, the stream of video pictures (102) includes samples captured by the digital camera. The stream of video pictures (102), shown with a thick line to emphasize its large amount of data compared to the encoded video data (104) (or encoded video bitstream), may be processed by an electronic device (120) including a video encoder (103) coupled to the video source (101). The video encoder (103) may include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter, as described in more detail below. The encoded video data (104) (or encoded video bitstream), shown with a thin line to emphasize its small amount of data compared to the stream of video pictures (102), may be stored on a streaming server (105) for future use. One or more streaming client subsystems, such as the client subsystems (106) and (108) of FIG. 1, can access the streaming server (105) to retrieve copies (107) and (109) of the encoded video data (104). The client subsystem (106) can include a video decoder (110), for example, within an electronic device (130). The video decoder (110) decodes the incoming copy (107) of the encoded video data and generates an outgoing stream of video pictures (111) that can be rendered on a display (112) (e.g., a display screen) or other rendering device (not shown). In some streaming systems, the encoded video data (104), (107), and (109) (e.g., a video bitstream) can be encoded according to some video encoding / compression standard.Examples of these standards include ITU-T Recommendation H.265. In one example, a video coding standard under development is informally known as Versatile Video Coding (VVC). The disclosed subject matter may be used in the context of VVC.

[0029] It should be noted that the electronic devices (120) and (130) may include other components (not shown). For example, the electronic device (120) may include a video decoder (not shown), and the electronic device (130) may include a video encoder (not shown).

[0030] 2 shows an exemplary block diagram of a video decoder (210). The video decoder (210) can be included in an electronic device (230). The electronic device (230) can include a receiver (231) (e.g., a receiving circuit). The video decoder (210) can be used in place of the video decoder (110) in the example of FIG. 1.

[0031] The receiver (231) can receive one or more coded video sequences, e.g., included in a bitstream, to be decoded by the video decoder (210). In some embodiments, one coded video sequence is received at a time, and the decoding of each coded video sequence is independent of the decoding of the other coded video sequences. The coded video sequences can be received from a channel (201), which can be a hardware / software link to a storage device that stores the encoded video data. The receiver (231) can receive the encoded video data along with other data, e.g., coded audio data and / or auxiliary data streams, which can be forwarded to respective usage entities (not shown). The receiver (231) can separate the coded video sequences from other data. To address network jitter, a buffer memory (215) can be coupled between the receiver (231) and the entropy decoder / parser (220) (hereinafter, "parser (220)"). In some applications, the buffer memory (215) is part of the video decoder (210). In other applications, the buffer memory may be external to the video decoder (210) (not shown). In still other applications, there may be a buffer memory (not shown) external to the video decoder (210), for example, to deal with network jitter, and another buffer memory (215) internal to the video decoder (210), for example, to handle playback timing. If the receiver (231) receives data from a storage / forwarding device with sufficient bandwidth and controllability or from an isosynchronous network, the buffer memory (215) may be unnecessary or may be small. For use with best-effort packet networks such as the Internet, the buffer memory (215) may be necessary, may be relatively large, may be advantageously adaptively sized, and may be implemented, at least in part, in an operating system or similar element (not shown) external to the video decoder (210).

[0032] The video decoder (210) may include a parser (220) that reconstructs symbols (221) from the coded video sequence. These symbol categories, as shown in FIG. 2, include information used to manage the operation of the video decoder (210) and, potentially, information for controlling a rendering device, such as a rendering device (212) (e.g., a display screen) that is not an integral part of the electronic device (230) but may be coupled to the electronic device (230). The control information for the rendering device may be in the form of a Supplemental Enhancement Information (SEI) message or a Video Usability Information (VUI) parameter set fragment (not shown). The parser (220) can parse and entropy decode the received coded video sequence. The coding of the coded video sequence may follow a variety of video coding techniques or standards, including variable-length coding, Huffman coding, arithmetic coding with or without context sensitivity, and the like. The parser (220) can extract from the coded video sequence a set of subgroup parameters for at least one subgroup of pixels in the video decoder based on at least one parameter corresponding to the group. The subgroup can include a group of pictures (GOP), a picture, a tile, a slice, a macroblock, a coding unit (CU), a block, a transform unit (TU), a prediction unit (PU), etc. The parser (220) can also extract information from the coded video sequence, such as transform coefficients, quantizer parameter values, and motion vectors.

[0033] The parser (220) can perform entropy decoding / parsing operations on the video sequence received from the buffer memory (215) to generate symbols (221).

[0034] The reconstruction of the symbols (221) can involve several different units, depending on the type of coded video picture or part thereof (e.g., inter / intra picture, inter / intra block, etc.) and other factors. Which units are involved and how can be controlled by subgroup control information parsed from the coded video sequence by the parser (220). The flow of such subgroup control information between the parser (220) and the following units is not shown for clarity.

[0035] In addition to the functional blocks already mentioned, the video decoder (210) can be conceptually subdivided into several functional units, as described below. In a practical implementation operating under commercial constraints, many of these units will interact closely with each other and may be, at least partially, integrated with each other. However, for purposes of describing the disclosed subject matter, the following conceptual subdivision into functional units is appropriate.

[0036] The first unit is a scaler / inverse transform unit (251), which receives quantized transform coefficients and control information from the parser (220) as symbols (221), including which transform to use, block size, quantization factor, quantization scaling matrix, etc. The scaler / inverse transform unit (251) can output blocks containing sample values ​​that can be input to an aggregator (255).

[0037] In some cases, the output samples of the scaler / inverse transform unit (251) may belong to intra-coded blocks. Intra-coded blocks are blocks that do not use prediction information from a previously reconstructed picture, but can use prediction information from a previously reconstructed portion of the current picture. Such prediction information may be provided by an intra-picture prediction unit (252). In some cases, the intra-picture prediction unit (252) generates a block of the same size and shape as the block being reconstructed using surrounding, already reconstructed information fetched from the current picture buffer (258). The current picture buffer (258), for example, buffers a partially reconstructed current picture and / or a fully reconstructed current picture. The aggregator (255) may add, on a sample-by-sample basis, the prediction information generated by the intra-prediction unit (252) to the output sample information provided by the scaler / inverse transform unit (251).

[0038] In other cases, the output samples of the scaler / inverse transform unit (251) may relate to an inter-coded, potentially motion-compensated, block. In such cases, the motion-compensated prediction unit (253) may access a reference picture memory (257) to fetch samples used for prediction. After motion-compensating the fetched samples according to the symbols (221) associated with the block, these samples may be added by an aggregator (255) to the output of the scaler / inverse transform unit (251) (in this case, referred to as residual samples or residual signals) to generate output sample information. The addresses in the reference picture memory (257) from which the motion-compensated prediction unit (253) fetches the prediction samples may be controlled by motion vectors, which are available to the motion-compensated prediction unit (253) in the form of symbols (221), which may have, for example, X, Y, and reference picture components. Motion compensation may also include interpolation of sample values ​​fetched from the reference picture memory (257) when sub-sample accurate motion vectors are used, motion vector prediction mechanisms, and the like.

[0039] The output samples of the aggregator (255) can be subjected to various loop filtering techniques in a loop filtering unit (256). Video compression techniques can include in-loop filtering techniques controlled by parameters contained in the coded video sequence (also called the coded video bitstream) and made available to the loop filter unit (256) as symbols (221) from the parser (220). Video compression can also respond to meta-information obtained during decoding of previous portions (in decoding order) of the coded picture or coded video sequence, and can also respond to previously reconstructed, loop-filtered sample values.

[0040] The output of the loop filter unit (256) can be a sample stream that can be output to a rendering device (212) and can also be stored in a reference picture memory (257) for use in future inter-picture prediction.

[0041] Certain coded pictures, once fully reconstructed, can be used as reference pictures for future prediction. For example, once the coded picture corresponding to the current picture is fully reconstructed and the coded picture is identified as a reference picture (e.g., by the parser (220)), the current picture buffer (258) can become part of the reference picture memory (257), and a new current picture buffer can be reallocated before starting reconstruction of the next coded picture.

[0042] The video decoder (210) can perform decoding operations according to a given video compression technology or standard, such as ITU-T Recommendation H.265. The coded video sequence can conform to the syntax specified by the video compression technology or standard being used. This means that the coded video sequence conforms to both the syntax of the video compression technology or standard and the profile described in the video compression technology or standard. Specifically, the profile can select certain tools from all tools available in the video compression technology or standard as the only tools available under that profile. Also required for compliance is that the complexity of the coded video sequence must be within a range defined by the level of the video compression technology or standard. In some cases, the level constrains the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. The limits set by the level can, in some cases, be further constrained through a hypothetical reference decoder (HRD) specification and metadata for HRD buffer management signaled in the coded video sequence.

[0043] In some embodiments, the receiver (231) can receive additional (redundant) data along with the encoded video. The additional data may be included as part of the coded video sequence(s). The additional data may be used by the video decoder (210) to properly decode the data and / or more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) improvement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0044] 3 shows an example block diagram of a video encoder (303). The video encoder (303) is included in an electronic device (320). The electronic device (320) includes a transmitter (340) (e.g., a transmission circuit). The video encoder (303) can be used in place of the video encoder (103) in the example of FIG. 1.

[0045] The video encoder (303) can receive video samples from a video source (301) (which is not part of the electronic device (320) in the example of Figure 3) that can capture video images to be encoded by the video encoder (303). In another example, the video source (301) is part of the electronic device (320).

[0046] The video source (301) can provide a source video sequence to be encoded by the video encoder (303) in the form of a digital video sample stream, which can be of any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, etc.), any color space (e.g., BT.601 YCrCB, RGB, etc.), and any suitable sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media serving system, the video source (301) can be a storage device that stores prepared video. In a video conferencing system, the video source (301) can be a camera that captures local image information as a video sequence. The video data can be provided as multiple individual pictures that, when viewed in sequence, give the impression of motion. The pictures themselves can be organized as a spatial array of pixels, each of which can contain one or more samples, depending on the sampling structure, color space, etc., in use. The following discussion focuses on samples.

[0047] According to some embodiments, the video encoder (303) may encode and compress pictures of a source video sequence into a coded video sequence (343) in real time, or under any other time constraints as needed. Enforcing an appropriate coding rate is one function of the controller (350). In some embodiments, the controller (350) controls and is operatively coupled to other functional units, as described below. Coupling is not shown for clarity. Parameters set by the controller (350) may include rate control-related parameters (picture skip, quantizer, lambda value for rate-distortion optimization techniques, etc.), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (350) may be configured with other appropriate functions for optimizing the video encoder (303) for a particular system design.

[0048] In some embodiments, the video encoder (303) is configured to operate in an encoding loop. As a very simplified explanation, in one example, the encoding loop can include a source coder (330) (e.g., responsible for generating symbols, such as a symbol stream, based on an input picture to be encoded and reference picture(s)) and a (local) decoder (333) embedded in the video encoder (303). The decoder (333) reconstructs the symbols to generate sample data in a manner similar to that generated by a (remote) decoder. The reconstructed sample stream (sample data) is input to a reference picture memory (334). Because decoding of the symbol stream produces bit-accurate results independent of the decoder location (local or remote), the contents of the reference picture memory (334) are also bit-accurate between the local and remote encoders. In other words, the predictive portion of the encoder "sees" the exact same sample values ​​as the decoder "sees" when using prediction during decoding. This basic principle of reference picture synchronism (and the resulting drift if synchronism cannot be maintained, eg, due to channel errors) is also used in several related techniques.

[0049] The operation of the "local" decoder (333) may be the same as a "remote" decoder, such as the video decoder (210) already described in detail in connection with Figure 2. However, briefly referring also to Figure 2, because symbols are available and the encoding / decoding of symbols into an encoded video sequence by the entropy coder (345) and parser (220) may be lossless, the entropy decoding portion of the video decoder (210), including the buffer memory (215) and parser (220), may not be fully implemented in the local decoder (333).

[0050] In some embodiments, decoder technology, with the exception of parsing / entropy decoding, present in a decoder is present in the same or substantially identical functional form in the corresponding encoder. Thus, the disclosed subject matter focuses on the operation of the decoder. A description of the encoder technology can be omitted, as it is the inverse of the decoder technology, which is comprehensively described. In certain areas, more detailed descriptions are provided below.

[0051] In operation, in some examples, the source coder (330) may perform motion-compensated predictive coding, which predictively codes an input picture with reference to one or more previously coded pictures from a video sequence designated as “reference pictures.” In this manner, the coding engine (332) codes differences between pixel blocks of the input picture and pixel blocks of reference pictures that may be selected as predictive references for the input picture.

[0052] The local video decoder (333) can decode the coded video data of a picture that may be designated as a reference picture based on the symbols generated by the source coder (330). The operation of the coding engine (332) can advantageously be a lossy process. When the coded video data is decoded by a video decoder (not shown in FIG. 3), the reconstructed video sequence may be a replica of the source video sequence, typically with some errors. The local video decoder (333) can replicate the decoding process that may be performed on the reference picture by the video decoder and store the reconstructed reference picture in the reference picture memory (334). In this way, the video encoder (303) can locally store a copy of the reconstructed reference picture that has common content with the reconstructed reference picture that would be obtained by the far-end video decoder (in the absence of transmission errors).

[0053] The predictor (335) can perform the prediction search for the coding engine (332). That is, for a new picture to be coded, the predictor (335) can search the reference picture memory (334) for sample data (as candidate reference pixel blocks) or types of metadata, such as reference picture motion vectors, block shapes, etc., that may serve as suitable prediction references for the new picture. The predictor (335) can operate on a sample block or pixel block basis to find suitable prediction references. In some cases, as determined by the search results obtained by the predictor (335), the input picture can have prediction references drawn from multiple reference pictures stored in the reference picture memory (334).

[0054] The controller (350) can manage the coding operations of the source coder (330), including, for example, setting the parameters and subgroup parameters used to encode the video data.

[0055] The output of all the functional units described above may undergo entropy coding in an entropy coder (345), which converts the symbols produced by the various functional units into a coded video sequence by applying lossless compression to the symbols according to techniques such as Huffman coding, variable length coding, or arithmetic coding.

[0056] The transmitter (340) can buffer the coded video sequence produced by the entropy coder (345) and prepare it for transmission over a communication channel (360), which can be a hardware or software link to a storage device that stores the encoded video data. The transmitter (340) can merge the coded video data from the video encoder (303) with other data to be transmitted, such as coded audio data and / or auxiliary data streams (sources not shown).

[0057] The controller (350) can manage the operation of the video encoder (303). During encoding, the controller (350) can assign a certain coding picture type to each coded picture, which can affect the coding technique that can be applied to each picture. For example, pictures may often be assigned as one of the following picture types:

[0058] Intra pictures (I pictures) can be coded and decoded without using other pictures in the sequence as a source of prediction. Some video coders allow various types of intra pictures, including, for example, Independent Decoder Refresh (IDR) pictures.

[0059] Predictive pictures (P pictures) may be encoded and decoded using intra prediction or inter prediction, using motion vectors and reference indices to predict the sample values ​​of each block.

[0060] Bidirectionally predicted pictures (B pictures) may be coded and decoded using intra- or inter-prediction, using two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, multi-predictive pictures may use three or more reference pictures and associated metadata for the reconstruction of a single block.

[0061] A source picture is typically spatially subdivided into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples each) and may be coded block by block. Blocks may be predictively coded with reference to other (already coded) blocks, as determined by the coding assignment applied to the block's respective picture. For example, blocks of an I picture may be non-predictively coded or predictively coded with reference to previously coded blocks of the same picture (spatial prediction or intra-prediction). Pixel blocks of a P picture may be predictively coded via spatial prediction or via temporal prediction with reference to one previously coded reference picture. Blocks of a B picture may be predictively coded via spatial prediction or via temporal prediction with reference to one or two previously coded reference pictures.

[0062] The video encoder (303) may perform encoding operations in accordance with a predetermined video encoding technique or standard, such as ITU-T Recommendation H.265. In its operations, the video encoder (303) may perform various compression operations, including predictive encoding operations that exploit temporal and spatial redundancies in the input video sequence. Thus, the encoded video data may conform to the syntax specified by the video encoding technique or standard being used.

[0063] In some embodiments, the transmitter (340) can transmit additional data along with the encoded video. The source coder (330) can include such data as part of the coded video sequence. The additional data may include other forms of redundant data, such as temporal / spatial / SNR enhancement layers, redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

[0064] Video may be captured as multiple source pictures (video pictures) in a time sequence. Intra-picture prediction (often abbreviated as intra-prediction) exploits spatial correlation within a given picture, while inter-picture prediction exploits correlation (temporal or otherwise) between pictures. In one example, a particular picture being encoded / decoded, called the current picture, is divided into blocks. If a block in the current picture is similar to a reference block in a previously coded and still buffered reference picture in the video, the block in the current picture can be coded by a vector called a motion vector. The motion vector points to a reference block in the reference picture and may have a third dimension that identifies the reference picture if multiple reference pictures are used.

[0065] In some embodiments, inter-picture prediction may use a bi-prediction technique. According to the bi-prediction technique, two reference pictures, such as a first reference picture and a second reference picture, are used, both of which precede a current picture in a video in decoding order (but may be past and future in display order, respectively). A block in the current picture may be coded by a first motion vector pointing to a first reference block in the first reference picture and a second motion vector pointing to a second reference block in the second reference picture. A block may be predicted by a combination of the first and second reference blocks.

[0066] Furthermore, merge mode techniques can be used in inter-picture prediction to improve coding efficiency.

[0067] According to some embodiments of the present disclosure, prediction, such as inter-picture prediction and intra-picture prediction, is performed block-by-block. For example, according to the HEVC standard, pictures in a sequence of video pictures are divided into coding tree units (CTUs) for compression, and the CTUs within a picture have the same size, such as 64x64 pixels, 32x32 pixels, or 16x16 pixels. Generally, a CTU includes three coding tree blocks (CTBs), one luma CTB and two chroma CTBs. Each CTU can be recursively quadtree-decomposed into one or more coding units (CUs). For example, a 64x64 pixel CTU can be divided into one CU of 64x64 pixels, four CUs of 32x32 pixels, or 16 CUs of 16x16 pixels. In one example, each CU is analyzed to determine a prediction type for the CU (e.g., inter prediction type or intra prediction type). The CU is divided into one or more prediction units (PUs) depending on temporal and / or spatial predictability. Generally, each PU includes a luma prediction block (PB) and two chroma PBs. In an embodiment, prediction operations in coding (encoding / decoding) are performed in units of prediction blocks. Using a luma prediction block as an example of a prediction block, the prediction block includes a matrix of values ​​(e.g., luma values) for pixels of 8x8 pixels, 16x16 pixels, 8x16 pixels, 16x8 pixels, etc.

[0068] It should be noted that the video encoders (103) and (303) and the video decoders (110) and (210) may be implemented using any suitable technology. In one embodiment, the video encoders (103) and (303) and the video decoders (110) and (210) may be implemented using one or more integrated circuits. In another embodiment, the video encoders (103) and (303) and the video decoders (110) and (210) may be implemented using one or more processors executing software instructions.

[0069] Aspects of this disclosure provide techniques for a particular inter-prediction mode called advanced motion vector prediction (AMVP) merge mode that uses subblock-based temporal motion vector prediction (SbTMVP).

[0070] For example, various inter-prediction modes can be used in VVC. For an inter-predicted CU, motion parameters can include MV(s), one or more reference picture indices, a reference picture list usage index, and additional information about certain coding features to be used for inter-predicted sample generation. Motion parameters can be signaled explicitly or implicitly. When a CU is coded in skip mode, the CU can be associated with a PU, have no significant residual coefficients, and may not have coded motion vector deltas or MV differences (e.g., MVDs), or reference picture indices. A merge mode can be specified, in which motion parameters for the current CU are obtained from neighboring CU(s), including spatial and / or temporal candidates, and optionally additional information as introduced in VVC. The merge mode can be applied not only to skip mode but also to inter-predicted CUs. In one example, an alternative to merge mode is explicit transmission of motion parameters, where the MV(s), corresponding reference picture index and reference picture list usage flag for each reference picture list, and other information are explicitly signaled per CU.

[0071] In embodiments such as VVC, the VVC Test Model (VTM) reference software supports enhanced merge prediction, merge motion vector difference (MMVD) mode, adaptive motion vector prediction (AMVP) mode with symmetric MVD signaling, affine motion compensation prediction, subblock-based temporal motion vector prediction (SbTMVP), adaptive motion vector resolution (AMVR), motion field storage (1 / 16 luma sample MV storage and 8x8 motion field compression), bi-prediction with CU-level weights (BCW), bi-directional optical flow (BDOF), prediction refinement using optical flow (PROF), decoder side motion vector refinement (DMVR), combined inter and intra prediction (CIIP), geometric partitioning mode, and more. The inter-prediction coding method includes one or more inter-prediction coding tools, including a generalized partitioning mode (GPM), etc. Some inter-prediction and related methods are described in more detail below.

[0072] Specifically, in some examples, in merge mode, the video coder constructs a candidate list of motion parameters (e.g., reference pictures and motion vectors) using candidates from neighboring blocks, including spatial and temporal neighboring blocks. In some examples, a maximum number of candidates is defined, such as 5 or 7. In one example, the maximum number of candidates is 5, with four of the candidates being from spatial neighboring blocks and one of the candidates being from temporal neighboring blocks. The selected motion parameters are signaled from the video encoder to the video decoder by transmitting an index of the selected candidate from the candidate list. In the video decoder, once the index is decoded, the motion parameters of the selected candidate's corresponding neighboring block are inherited by the current block. The video encoder and video decoder are configured to construct the same list based on already-coded neighboring blocks. Thus, based on the index, the video decoder can identify the motion parameters of the candidate selected by the video encoder.

[0073] Skip mode generally operates in the same manner as described above for merge mode, except that in skip mode, residual data is not added to the predicted block, whereas in merge mode, residual data is added to the predicted block. The list construction and the transmission of indices to identify candidates in the list, described above for merge mode, are generally also performed in skip mode. Some video codecs may treat skip mode as a separate mode from merge mode, while other video coding standards may treat skip mode as a special case of merge mode.

[0074] Another mode for predicting motion parameters is called advanced motion vector prediction (AMVP) mode. In AMVP mode, a candidate list of motion vector predictors for each motion hypothesis is derived based on an encoded reference index. The candidate list includes spatial motion vector predictors of neighboring blocks associated with the same reference index as the current block and a temporal motion vector predictor derived based on motion parameters of neighboring blocks of a co-located block in a temporal reference picture. In one example, the candidate list includes two motion vector predictors from spatial neighboring blocks in the current picture as the current block and one motion vector predictor from a temporal neighboring block in a temporal reference picture for the current picture. In one example, the selected motion vector predictor is signaled by transmitting an index into the candidate list, along with a reference index value and a motion vector differential. In another example, a template matching technique is used to select the motion vector predictor with the smallest template matching error, and decoder-side motion vector refinement (DMVR) can be applied to the selected motion vector predictor. In some cases, the motion vector predictor and the motion vector used for inter prediction may be the same, while in other cases the motion vector predictor and the motion vector used for inter prediction may be different. In AMVP mode, for example, the motion vector used for inter prediction corresponds to the combination of the motion vector predictor and the motion vector differential value.

[0075] In some examples (e.g., HEVC), AMVP may be used to predict the motion vector of a current block by exploiting the spatial and temporal correlation of neighboring partitions. At the encoder side, a rate distortion optimization (RDO) process is used to select the best motion vector predictor from a candidate list of candidates. An index for the selected candidate is then encoded and transmitted to the decoder. At the decoder side, the same candidate list as the encoder side may be constructed, for example, in a defined manner. In some examples, the candidate list may be constructed in three steps. In the first step, the decoder retrieves spatial and temporal motion vectors from a memory buffer to form the candidate list. In the second step, a redundancy check process is used to remove duplicate motion vectors from the candidate list. In the third step, a zero motion check process is optionally used to check for the presence of zero motion in the candidate list. Note that candidate list construction removes duplicate motion vectors, shortening the candidate list, and fewer bits may be used to signal the index of the selected candidate in the candidate list for AMVP.

[0076] Note that for AMVP, in some examples, the encoder also signals a reference picture index to specify the reference picture to which the motion vector predictor pointed by the index of the selected candidate in the candidate list points. Additionally, for AMVP, the encoder can determine a motion vector difference (MVD) for the current block, where the MVD is the difference between the motion vector predictor and the true motion vector, or disparity motion vector, used to perform inter prediction for the current block. For AMVP, in addition to the reference picture index and the index of the selected candidate in the candidate list, the encoder also signals the MVD for the current block in the bitstream in some examples.

[0077] In some examples, the number of spatial neighbor candidates in an AMVP candidate list is limited to no more than a threshold number, and the number of temporal candidates is limited to no more than a threshold number. For example, there are at most two spatial neighbor candidates and one co-located temporal candidate in an AMVP candidate list. Potential AMVP spatial neighbor candidates are located at the bottom-left, left, top-right, top, and top-left positions of the current block. In one example, the potential AMVP spatial neighbor candidates are classified into two classes. The left and bottom-left candidates are classified into a first class, and the top-right, top, and top-left candidates are classified into a second class. In one example, a scanning order is used to place the potential AMVP spatial neighbor candidates in the candidate list. For example, the scanning order is bottom-to-top for the first class and right-to-left for the second class, respectively.

[0078] To improve coding efficiency and reduce MV transmission overhead, subblock-level MV refinement can be applied to extend CU-level temporal motion vector prediction (TMVP). In one example, subblock-based TMVP (SbTMVP) mode allows subblock-level motion information inheritance from a co-located reference picture. The co-located reference picture can be indicated by a reference index in syntax such as a high-level syntax (e.g., a picture header or a slice header). Each sub-block of multiple sub-blocks in a current CU (e.g., a current CU with a large size) in a current picture can have its own motion information without explicitly transmitting a block partition structure or its own motion information. In SbTMVP mode, the motion information of each sub-block can be obtained, for example, in three steps as follows: In the first step, a displacement vector (DV) of the current CU can be derived. The DV can indicate a block in the co-located reference picture, for example, the DV points from the current block in the current picture to the block in the co-located reference picture. Thus, the block indicated by DV is considered to be collocated with the current block and is called the co-located block of the current block. In the second step, the availability of SbTMVP candidates can be checked, and then the central motion (e.g., the central motion of the current CU) can be derived. In the third step, sub-block motion information can be derived from corresponding sub-blocks in the co-located block using DV. The three steps can be combined into one or two steps, and / or the order of the three steps may be adjusted.

[0079] Unlike TMVP candidate derivation, which derives temporal MVs from co-located blocks in a reference frame or reference picture, in SbTMVP mode, for a sub-block in a current CU in a current picture, a DV (e.g., a DV derived from the MV of the current CU's left neighboring CU) can be applied to locate the corresponding sub-block in the co-located reference picture. In some examples, when the corresponding sub-block is not inter-coded, the motion information of the current sub-block can be set to be the central motion of the co-located block.

[0080] The SbTMVP mode may be supported by various video coding standards, including, for example, VVC. Similar to the TMVP mode, for example, in HEVC, the SbTMVP mode may use motion fields (also called motion information fields or MV fields) in a co-located reference picture to improve MV prediction and merge mode for the CU in the current picture. In one example, the same co-located reference picture used by the TMVP mode is used in the SbTMVP mode. In one example, the SbTMVP mode differs from the TMVP mode in the following ways: (i) the TMVP mode predicts motion information at the CU level, while the SbTMVP mode predicts motion information at the sub-CU level; and (ii) the TMVP mode fetches temporal MVs from a co-located block in the co-located reference picture (e.g., the co-located block is the bottom-right or center block relative to the current CU), while the SbTMVP mode may apply a motion shift before fetching temporal motion information from the co-located reference picture. In one example, the motion shift used in the SbTMVP mode is obtained from the MV of one of the spatial neighboring blocks of the current CU.

[0081] Figure 4 shows the SbTMVP process used in the SbTMVP mode. The SbTMVP process can predict the motion vector (MV) of a sub-CU (e.g., a sub-block) in a current CU (e.g., a current block) (401) in a current picture (411), for example, in two steps. In the first step, a spatial neighboring block (e.g., A1) of the current block (401) in Figure 4 is examined. If the spatial neighboring block (e.g., A1) has a motion vector (MV) (421) that uses a co-located reference picture (412) as the reference picture for the spatial neighboring block (e.g., A1), the motion vector (MV) (421) may be selected to be the motion shift (or DV) to be applied to the current block (401). If no such motion vector (e.g., a MV that uses the co-located reference picture (412) as the reference picture) is identified, the motion shift or DV may be set to a zero motion vector (e.g., (0,0)). In some examples, if no such MV is identified for spatial neighboring block A1, the MV(s) of additional spatial neighboring blocks, such as A0, B0, B1, etc., are checked.

[0082] In a second step, the motion shift or DV (421) identified in the first step can be applied to the current block (401) (e.g., adding the DV (421) to the coordinates of the current block) to obtain sub-CU level motion information (e.g., including MV and reference index) from the co-located reference picture (412). In the example shown in FIG. 4, the motion shift or DV (421) is set to be the MV of a spatially neighboring block A1 (e.g., block A1) of the current block (401). Motion information of a sub-block in the corresponding co-located block (402) in the co-located reference picture (412) can be used to derive motion information for the sub-CU or sub-block (431) in the current picture (411). For example, after the motion information of the co-located sub-CU (432) in the co-located block (402) is identified, the motion information of the co-located sub-CU (432) may be converted into motion information (e.g., MV(s) and one or more reference indices) of the current sub-CU (431), for example, using a scaling method, in a manner similar to the TMVP process used in HEVC, where temporal motion scaling is applied to align the reference picture of the temporal MV with the reference picture of the current CU.

[0083] The motion field of the current block (401) derived based on the DV (421) may include motion information for each sub-block (431) within the current block (401), such as MV(s) and one or more associated reference indices. The motion field of the current block (401), also referred to as an SbTMVP candidate, corresponds to the DV (421).

[0084] For example, the motion information of the bi-predicted sub-block (431(1)) includes a first motion vector (MV) and a first index indicating a first reference picture in reference picture list 0 (L0), a second motion vector (MV) and a second index indicating a second reference picture in reference picture list 1 (L1). In one example, the motion information of the uni-predicted sub-block (431(2)) includes a motion vector (MV) and an index indicating a reference picture in L0 or L1.

[0085] In one example, the DV (421) is applied to the center position of the current block (401) to locate the displaced center position in the co-located reference picture (412). If the block containing the displaced center position is not inter-coded, the SbTMVP candidate is considered unavailable. Alternatively, if the block containing the displaced center position (e.g., the co-located block (402)) is inter-coded, motion information for the center position of the current block (401), referred to as the central motion of the current block (401), may be derived from motion information for the co-located block (402) containing the displaced center position in the co-located reference picture (412). In one example, a scaling process may be used to derive the central motion of the current block (401) from motion information for the co-located block (402) containing the displaced center position in the co-located reference picture (412). When SbTMVP candidates are available, the DV (421) may be applied to find the corresponding sub-block (431) in the co-located reference picture (412) for each sub-block (432) of the current block (401). The motion information of the corresponding sub-block (432) may be used to derive motion information for the sub-block (431) within the current block (401), such as in the same manner as it is used to derive the central motion of the current block (401). In one example, if the corresponding sub-block (432) is not inter-coded, the motion information of the current sub-block (431) is set to be the central motion of the current block (401).

[0086] In some examples, such as VVC, a combined subblock-based merge list containing an SbTMVP candidate and one or more affine merge candidates is used to signal the subblock-based merge mode. SbTMVP mode can be enabled or disabled by a sequence parameter set (SPS) flag. When SbTMVP mode is enabled, an SbTMVP candidate (or SbTMVP predictor) is added as the first entry in a subblock-based merge list containing subblock-based merge candidates, which can be followed by one or more affine merge candidates. The size of the subblock-based merge list can be signaled in the SPS. In one example, the maximum allowed size of a subblock-based merge list is 5 in VVC. In one example, multiple SbTMVP candidates are included in a subblock-based merge list.

[0087] In some examples, such as VVC, the sub-CU size used in SbTMVP mode is fixed at 8x8, as used for affine merge mode. In one example, SbTMVP mode is only applicable to CUs whose width and height are both 8 or greater. The sub-block size (e.g., 8x8) may be configurable to other sizes, such as 4x4 in the ECM software model used for explorations beyond VVC. In one example, multiple co-located reference pictures, such as two co-located frames, are utilized to provide temporal motion information for SbTMVP and / or TMVP.

[0088] In some examples, a specific prediction mode called AMVP merge mode is used for bi-prediction. The bidirectional predictor in AMVP merge mode includes an AMVP predictor in one direction and a merge predictor in the other direction. In some examples, AMVP merge mode can be enabled for a coding block when the selected merge predictor and AMVP predictor satisfy the DMVR condition, where there is at least one reference picture from the past and at least one reference picture from the future for the current picture, the distances from the two reference pictures to the current picture are the same, and bilateral matching MV refinement is applied to the merge MV candidate (e.g., merge predictor) and AMVP MVP (e.g., AMVP predictor) used as the starting point. Otherwise, if the template matching function is enabled, template matching MV refinement is applied to the merge predictor or AMVP predictor with a higher template matching cost.

[0089] In some examples, the AMVP portion of the mode is signaled as normal unidirectional AMVP, e.g., a reference index and MVD are signaled. In some examples, the AMVP portion can have a derived MVP index (e.g., an index to indicate a selected AMVP predictor) when template matching is used, or the MVP index is signaled when template matching is disabled.

[0090] In some examples, the AMVP direction is represented by LX, where X can be 0 or 1, and the merge portion in the other direction (1-LX) is implicitly derived by minimizing the bilateral matching cost between the AMVP predictor and merge predictor, i.e., for the pair of AMVP motion vector and merge motion vector (e.g., the bilateral matching cost between the AMVP predictor and each merge candidate is calculated). For every merge candidate in the merge candidate list with a motion vector in the other direction (1-LX), the bilateral matching cost is calculated using the merge candidate MV and the AMVP MV. The merge candidate with the minimum bilateral matching cost is selected. Bilateral matching refinement can also be applied to the coding block using the selected merge candidate MV and AMVP MV as a starting point.

[0091] In some examples, a third pass of multi-pass DMVR, which is an 8x8 sub-PU BDOF refinement of multi-pass DMVR, is enabled for blocks coded in AMVP merge mode. In one example, the multi-pass DMVR includes a first pass that performs block-based bilateral matching MV refinement. Additionally, the multi-pass DMVR includes a second pass that performs sub-block-based bilateral matching MV refinement. The multi-pass DMVR also includes a third pass that performs sub-block-based bilateral optical flow MV refinement. In the third pass, refined MVs are derived by applying BDOF to the 8x8 grid of sub-blocks.

[0092] In some examples, the AMVP merge mode may be indicated by a flag. In one example, when the AMVP merge mode is enabled, the AMVP direction LX is further indicated by a flag.

[0093] In some examples, when bilateral matching (BM) AMVP merge mode is used for the current block and template matching is enabled, MVD is not signaled. In one example, an additional pair of AMVP merge MVPs is introduced. The merge candidate list is sorted in ascending order based on BM cost. An index (0 or 1) is signaled to indicate which merge candidate in the sorted merge candidate list should be used. When there is only one candidate in the merge candidate list, the pair of AMVP MVP without bilateral matching MV refinement and merge MVP is padded.

[0094] In some examples, for bidirectional prediction, a first prediction of a current block is based on motion information in a first direction, and a second prediction of the current block is based on motion information in a second direction. Then, the bidirectional prediction of the current block is a combination of the first prediction and the second prediction, such as an average of the first prediction and the second prediction, or a weighted sum of the first prediction and the second prediction. In some examples, for a current block in AMVP merge mode, a bidirectional predictor of AMVP merge mode is used to code the current block. The bidirectional predictor includes an AMVP predictor in one direction (e.g., the first direction) and a merge predictor in the other direction (e.g., the second direction). Then, the first prediction of the current block is based on the AMVP predictor in the first direction, and the second prediction of the current block is based on the merge predictor in the second direction. In AMVP merge mode, the bidirectional prediction for the current block is a combination of the first and second predictions, such as the average of the first and second predictions, or the weighted sum of the first and second predictions.

[0095] According to certain aspects of the present disclosure, in some examples, such as ECM, AMVP merge mode is used to combine one uni-predictor from an AMVP mode in one reference list with another uni-predictor from a merge candidate in another reference list to form a final bi-predictor. However, subblock-level temporal motion vectors are not used as merge candidates in AMVP merge mode.

[0096] Some aspects of the present disclosure provide techniques for allowing subblock-level temporal motion vectors, such as SbTMVP, to be used as merge candidates in AMVP merge mode, which may result in more accurate motion field information. Therefore, by considering SbTMVP in AMVP merge mode, the coding gain of AMVP merge mode may be improved. For example, an encoder / decoder may construct an AMVP candidate list including multiple AMVP candidates for an AMVP predictor and a merge candidate list including multiple merge candidates for a merge predictor, where the merge candidate list includes SbTMVP candidates. The encoder / decoder may then select an AMVP predictor from the AMVP candidate list, select a merge predictor from the merge candidate list, and reconstruct the current block according to a combination of a first prediction by the AMVP predictor in a first direction and a second prediction by the merge predictor in a second direction.

[0097] According to some aspects of the present disclosure, SbTMVP candidates can be added to a merge candidate list during merge list construction in AMVP merge mode.

[0098] In some embodiments, the derivation of the SbTMVP candidate to be added is the same as the SbTMVP derivation in the subblock merge list. For example, the SbTMVP process described with reference to FIG. 4 is used to derive SbTMVP candidates according to spatial neighboring blocks, and the derived SbTMVP candidates are added to the merge candidate list. In one example, as shown in FIG. 4, the left neighboring block A1 is inter-coded and has a motion vector (MV) (421). The motion vector (MV) (421) is used as the differential vector (DV) (421). In one example, the differential vector (DV) (421) is applied to the center position of the current block (401) to locate the displaced center position in the co-located reference picture (412). When a block including a displaced center position (e.g., a co-located block (402), a sub-block at the center of the co-located block (402)) is inter-coded, motion information of the center position of the current block (401), referred to as the central motion of the current block (401), may be derived, for example, from motion information of the displaced center position in the co-located block (402) (e.g., a central sub-block of the co-located block (402)). The motion field of the current block (401) derived based on the DV (421) may include motion information of each sub-block (431) in the current block (401), such as MV(s) and one or more associated reference indexes. The motion field of the current block (401), also referred to as an SbTMVP candidate, corresponds to the DV (421). However, in one example, when motion information at the displaced center position in the co-located block (402) is unavailable, the SbTMVP candidate corresponding to the DV (421) is considered unavailable.

[0099] In some embodiments, unlike VVC, the left neighboring coded block is used for SbTMVP derivation, and a zero displacement vector (DV) is used to derive the co-located SbTMVP in the co-located reference picture. If the co-located central subblock motion vector in the co-located picture is not available, the SbTMVP candidate is also not available during merge list construction. In one example, an SbTMVP with zero DV is added to the merge candidate list. In another example, an SbTMVP with zero DV is added to the merge candidate list under certain conditions. For example, when an SbTMVP candidate according to the motion information of the spatial neighboring block is deemed unavailable, a zero DV, such as (0,0), is used to derive the co-located SbTMVP candidate in the co-located reference picture. When an SbTMVP candidate with zero DV is available, the SbTMVP candidate with zero DV may be added to the merge candidate list.

[0100] In some examples, multiple SbTMVP candidates may be added to the merge candidate list, for example, the SbTMVP candidate list includes a first SbTMVP candidate with a DV derived from motion information of spatially neighboring blocks and a second SbTMVP candidate with a zero DV.

[0101] In some embodiments, not only is the left neighboring coded block used for SbTMVP derivation in VVC, but multiple spatial neighboring coded blocks in a predefined order are used one by one to check the availability of SbTMVP. If the central subblock motion vectors of all spatial neighboring coded blocks are unavailable, a zero DV is used to derive SbTMVP. In some examples, spatial neighboring blocks such as A1, A0, B1, and B0 in FIG. 4 are checked in a predefined order to determine the availability of an SbTMVP candidate with a DV derived from the motion information of one of the spatial neighboring blocks. When an SbTMVP candidate is unavailable from any of the spatial neighboring blocks, an SbTMVP candidate with a zero DV is added to the merge candidate list.

[0102] In some embodiments, for the AMVP direction LX and the merge direction (1-LX), only the central subblock motion vector with a motion vector in that direction (1-LX) may be marked as "available" during SbTMVP derivation. In some other examples, each subblock is examined to determine whether it has an SbTMVP candidate with an MV associated with the direction (1-LX). If none of the subblocks has an SbTMVP candidate with an MV associated with the direction (1-LX), SbTMVP is not allowed to be applied in AMVP merge mode, and the related signaling is not applied. For example, when the motion vector of the central subblock of a co-located block has direction (1-LX) (e.g., the reference picture for the motion vector is in direction (1-LX)), the motion information of the central subblock of the co-located block is considered available for SbTMVP derivation. In some examples, each subblock in the co-located block is examined to determine whether one of the subblocks in the co-located block has a motion vector in direction (1-LX). When at least one of the sub-blocks has a motion vector in the direction (1-LX), the co-located block is used to derive an SbTMVP candidate with motion information in the direction (1-LX). When none of the sub-blocks in the co-located block has a motion vector in the direction (1-LX), in one example, no SbTMVP candidate is used in AMVP merge mode.

[0103] In some embodiments, for each subblock, when performing motion compensation, if the associated subblock motion vector does not have an MV for the direction (1-LX), an MV from a neighboring subblock with an associated reference (1-LX) is used for this subblock, or a central subblock with an associated reference (1-LX) is used for this subblock, or a default MV (0,0) is used for this subblock. In some examples of motion compensation using SbTMVP candidates, for a subblock in the current block, when a co-located subblock according to an SbTMVP candidate does not have a motion vector for the (1-LX) direction (also referred to as having reference (1-LX)), an alternative motion vector for the (1-LX) direction is preferably determined for that subblock. In one example, neighboring subblocks of the co-located block can be examined to find a motion vector for the direction (1-LX) as an alternative motion vector. In another example, the motion vector of the central subblock for the direction (1-LX) is used as an alternative motion vector. In another example, a default motion vector, such as (0,0), is used as an alternative motion vector.

[0104] In some embodiments, a binary mask is derived based on the availability of MVs for each subblock in the reference direction (1-LX). For each subblock, the mask at the associated position is marked as true if the subblock MVs in the reference direction 1-LX are available; otherwise, the mask at the associated position is marked as false. During motion compensation in AMVP merge mode by using SbTMVP, this mask is used to determine whether uni-prediction in 1-LX is available for each subblock. When the associated value in the mask for a subblock is true, bi-prediction is applied, with uni-prediction in LX from AMVP and uni-prediction in 1-LX from SbTMVP; otherwise, only uni-prediction in LX in AMVP is used.

[0105] For example, for a current block containing four sub-blocks (e.g., arranged in a 2x2 array), a 2x2 binary mask is derived, where each binary bin corresponds to a sub-block and indicates the availability of MV in the reference direction (1-LX). During motion compensation in AMVP merge mode using SbTMVP candidates, the 2x2 binary mask is used to determine whether uni-prediction in the reference direction (1-LX) is available for each sub-block. When a binary bin in the binary mask corresponding to a sub-block is true, bi-prediction using a first uni-prediction in the reference direction (LX) from the AMVP predictor and a second uni-prediction in the reference direction (1-LX) from the SbTMVP candidate can be applied to reconstruct the sub-block. When a binary bin in the binary mask corresponding to a sub-block is false, the sub-block is reconstructed based on uni-prediction in the reference direction LX from the AMVP predictor.

[0106] In some embodiments, SbTMVP candidates are derived in a predefined order during merge list construction in AMVP merge mode. For example, during merge candidate list construction, SbTMVP candidates, regular spatial merge candidates, and temporal merge candidates are derived and placed in the merge candidate list in a predefined order. In one example, the predefined order is SbTMVP candidates, regular spatial merge candidates, and temporal merge candidates. In another example, the predefined order is regular spatial merge candidates, SbTMVP candidates, and temporal merge candidates.

[0107] In some embodiments, SbTMVP candidates may be derived first when there is at least one SbTMVP mode used in spatially neighboring coded blocks during merge list construction in AMVP merge mode. For example, when one of the spatial neighboring blocks is coded in SbTMVP mode, the SbTMVP candidate is derived first and placed in the first position in the merge candidate list.

[0108] 5 shows a flowchart outlining a process (500) according to one embodiment of the present disclosure. The process (500) may be used in a video decoder. In various embodiments, the process (500) is performed by a processing circuit, such as a processing circuit that performs the functions of the video decoder (110), a processing circuit that performs the functions of the video decoder (210), or the like. In some embodiments, the process (500) is implemented with software instructions, such that the processing circuit performs the process (500) when it executes the software instructions. The process begins at (S501) and proceeds to (S510).

[0109] At (S510), coded information for a current block in a current picture is received from a coded video bitstream, the coded information indicating an AMVP merge mode, the AMVP merge mode using a bidirectional predictor including an AMVP predictor in a first direction and a merge predictor in a second direction different from the first direction.

[0110] At (S520), an AMVP candidate list is constructed that includes a plurality of AMVP candidates for the AMVP predictor.

[0111] At (S530), a merge candidate list is constructed that includes multiple merge candidates for the merge predictor. The merge candidate list includes at least the SbTMVP candidate.

[0112] At (S540), an AMVP predictor is selected from the AMVP candidate list.

[0113] At (S550), a merge predictor is selected from the merge candidate list.

[0114] At (S560), the current block is reconstructed according to a combination of a first prediction by an AMVP predictor in a first direction and a second prediction by a merge predictor in a second direction.

[0115] In some examples, the merge predictor is determined to be an SbTMVP candidate. To reconstruct the current block, a co-located block for the current block is determined according to the displacement vector of the SbTMVP candidate in a collocated picture for the current picture. The current block includes a plurality of sub-blocks, and the co-located block includes a plurality of co-located sub-blocks for each of the sub-blocks. A plurality of motion vectors for each of the sub-blocks in the current block are determined according to the motion information of each of the co-located sub-blocks in the co-located block. A plurality of reconstructed sub-blocks for each of the sub-blocks in the current block are reconstructed according to the motion vectors for each of the sub-blocks in the current block. The reconstructed sub-blocks form a second prediction.

[0116] In some examples, to construct the merge candidate list, SbTMVP candidates are derived according to the motion vectors of spatially neighboring blocks of the current block. In one example, co-located blocks in a co-located picture for the current picture are determined according to the motion vectors of spatially neighboring blocks of the current block. In response to the availability of sub-block-based motion information in the co-located blocks, the motion vectors of the spatially neighboring blocks are used as displacement vectors (DVs) for the SbTMVP candidates. In one example, the availability of sub-block-based motion information for the co-located blocks is determined based on the availability of motion vectors in a central sub-block of the co-located blocks. In another example, the availability of sub-block-based motion information for the co-located blocks is determined based on the availability of motion vectors associated with a second direction in a central sub-block of the co-located blocks. In another example, the availability of sub-block-based motion information for the co-located blocks is determined based on the availability of motion vectors associated with a second direction for at least one sub-block in the co-located blocks.

[0117] In some examples, to derive an SbTMVP candidate, multiple spatial neighboring blocks are examined in a predefined order. A first spatial neighboring block having a first motion vector is identified among the multiple spatial neighboring blocks, and a co-located block according to the first motion vector has subblock-based motion information available. The first motion vector is used as a displacement vector (DV) for the SbTMVP candidate.

[0118] In some examples, a zero displacement vector (DV) (e.g., (0,0)) is used as the SbTMVP candidate to construct the merge candidate list. In one example, multiple spatial neighboring blocks are in a predefined order. When none of the multiple spatial neighboring blocks has subblock-based motion information available for the corresponding co-located block, a zero displacement vector (DV) is used for the SbTMVP candidate.

[0119] In some examples, to determine a plurality of respective motion vectors for a plurality of sub-blocks in the current block, for a first sub-block in the plurality of sub-blocks, in response to a lack of motion information in the second direction in a co-located sub-block corresponding to the first sub-block, the first motion vector for the first sub-block is determined according to a second motion vector of a neighboring sub-block of the first sub-block in the second direction. In another example, the first motion vector for the first sub-block is determined according to a third motion vector of a central sub-block of the co-located block in the second direction. In another example, the first motion vector for the first sub-block is set according to a default motion vector.

[0120] In some examples, to determine a plurality of motion vectors for each of the plurality of sub-blocks in the current block, a binary mask is generated for the plurality of sub-blocks in the current block, the binary mask including binary bins associated with the plurality of sub-blocks in the current block, and the binary bins in the binary mask for the sub-blocks indicate availability of a motion vector in a second direction.

[0121] In one example, in response to the binary bin for the subblock indicating that a motion vector in a second direction is available, a first reconstructed subblock is reconstructed according to an AMVP predictor in the first direction and a second reconstructed subblock is reconstructed according to a motion vector in the second direction, and the subblock is reconstructed based on a combination of the first reconstructed subblock and the second reconstructed subblock.

[0122] In another example, in response to the binary bin for a subblock indicating that a motion vector in a second direction is unavailable, uni-prediction is used to reconstruct the subblock, e.g., a first reconstructed subblock is generated according to an AMVP predictor in a first direction, and the subblock is reconstructed based on the first reconstructed subblock.

[0123] In some examples, the merge candidate list includes an SbTMVP candidate, one or more spatially neighboring merge candidates, and one or more temporally neighboring merge candidates in a predefined order. In one example, the merge candidate list includes the SbTMVP candidate in the first position in the merge candidate list.

[0124] Then, the process proceeds to (S599) and ends.

[0125] Process 500 may be adapted as appropriate. Steps of process 500 may be modified and / or omitted. Additional steps may be added. Any suitable order of performance may be used.

[0126] 6 shows a flowchart outlining a process (600) according to one embodiment of the present disclosure. The process (600) may be used in a video encoder. In various embodiments, the process (600) is performed by a processing circuit, such as a processing circuit that performs the functions of the video encoder (103), a processing circuit that performs the functions of the video encoder (303), or the like. In some embodiments, the process (600) is implemented with software instructions, such that the processing circuit performs the process (600) when it executes the software instructions. The process begins at (S601) and proceeds to (S610).

[0127] At (S610), it is determined to use an AMVP merge mode for coding a current block in a current picture, the AMVP merge mode using a bidirectional predictor including an AMVP predictor in a first direction and a merge predictor in a second direction different from the first direction.

[0128] At (S620), an AMVP candidate list is constructed that includes a plurality of AMVP candidates for the AMVP predictor.

[0129] At (S630), a merge candidate list including a plurality of merge candidates for the merge predictor is constructed, where the merge candidate list includes at least sub-block-based temporal motion vector prediction (SbTMVP) candidates for the merge predictor.

[0130] At (S640), an AMVP predictor is selected from the AMVP candidate list.

[0131] At (S650), a merge predictor is selected from the merge candidate list.

[0132] At (S660), the current block is reconstructed by combining a first prediction by an AMVP predictor in a first direction and a second prediction by a merge predictor in a second direction.

[0133] In some examples, the merge predictor is determined to be an SbTMVP candidate. To reconstruct the current block, a co-located block for the current block is determined according to the displacement vector of the SbTMVP candidate in a collocated picture for the current picture, where the current block includes a plurality of sub-blocks, and the co-located block includes a plurality of co-located sub-blocks for each of the sub-blocks. A plurality of motion vectors for each of the sub-blocks in the current block are determined according to the motion information of each of the co-located sub-blocks in the co-located block. A plurality of reconstructed sub-blocks for each of the sub-blocks in the current block are reconstructed according to the motion vectors for each of the sub-blocks in the current block. The reconstructed sub-blocks form a second prediction.

[0134] In some examples, to construct the merge candidate list, SbTMVP candidates are derived according to the motion vectors of spatially neighboring blocks of the current block. In one example, co-located blocks in a co-located picture for the current picture are determined according to the motion vectors of the spatially neighboring blocks of the current block. In response to the availability of sub-block-based motion information in the co-located blocks, the motion vectors of the spatially neighboring blocks are used as displacement vectors (DVs) for the SbTMVP candidates. In one example, the availability of sub-block-based motion information for the co-located blocks is determined based on the availability of motion vectors in a central sub-block of the co-located blocks. In another example, the availability of sub-block-based motion information for the co-located blocks is determined based on the availability of motion vectors associated with a second direction in a central sub-block of the co-located blocks. In another example, the availability of sub-block-based motion information for the co-located blocks is determined based on the availability of motion vectors associated with a second direction for at least one sub-block in the co-located blocks.

[0135] In some examples, to derive an SbTMVP candidate, multiple spatial neighboring blocks are examined in a predefined order. A first spatial neighboring block having a first motion vector is identified among the multiple spatial neighboring blocks, and a co-located block according to the first motion vector has subblock-based motion information available. The first motion vector is used as a displacement vector (DV) for the SbTMVP candidate.

[0136] In some examples, a zero displacement vector (DV) (e.g., (0,0)) is used as the SbTMVP candidate to construct the merge candidate list. In one example, multiple spatial neighboring blocks are in a predefined order. When none of the multiple spatial neighboring blocks has subblock-based motion information available for the corresponding co-located block, a zero displacement vector (DV) is used for the SbTMVP candidate.

[0137] In some examples, to determine a plurality of respective motion vectors for a plurality of sub-blocks in the current block, for a first sub-block in the plurality of sub-blocks, in response to a lack of motion information in the second direction in a co-located sub-block corresponding to the first sub-block, the first motion vector for the first sub-block is determined according to a second motion vector of a neighboring sub-block of the first sub-block in the second direction. In another example, the first motion vector for the first sub-block is determined according to a third motion vector of a central sub-block of the co-located block in the second direction. In another example, the first motion vector for the first sub-block is set according to a default motion vector.

[0138] In some examples, to determine a plurality of motion vectors for each of the plurality of sub-blocks in the current block, a binary mask is generated for the plurality of sub-blocks in the current block, the binary mask including binary bins associated with the plurality of sub-blocks in the current block, and the binary bins in the binary mask for the sub-block indicate availability of a motion vector in a second direction.

[0139] In one example, in response to the binary bin for the subblock indicating that a motion vector in a second direction is available, a first reconstructed subblock is reconstructed according to an AMVP predictor in the first direction and a second reconstructed subblock is reconstructed according to a motion vector in the second direction, and the subblock is reconstructed based on a combination of the first reconstructed subblock and the second reconstructed subblock.

[0140] In another example, uni-prediction is used to reconstruct a subblock in response to the binary bin for the subblock indicating that a motion vector in a second direction is unavailable, e.g., a first reconstructed subblock is generated according to an AMVP predictor in a first direction, and the subblock is reconstructed based on the first reconstructed subblock.

[0141] In some examples, the merge candidate list includes an SbTMVP candidate, one or more spatially neighboring merge candidates, and one or more temporally neighboring merge candidates in a predefined order. In one example, the merge candidate list includes the SbTMVP candidate in the first position in the merge candidate list.

[0142] Then, the process proceeds to (S699) and ends.

[0143] Process 600 may be adapted as appropriate. Steps of process 600 may be modified and / or omitted. Additional steps may be added. Any suitable order of performance may be used.

[0144] The techniques described above can be implemented as computer software using computer-readable instructions and physically stored on one or more computer-readable media. For example, Figure 7 illustrates a computer system (700) suitable for implementing certain embodiments of the disclosed subject matter.

[0145] Computer software may be coded using any suitable machine code or computer language and may apply assembly, compilation, linking, or similar mechanisms to create code containing instructions that can be executed by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc. directly, or through interpretation, microcode execution, etc.

[0146] The instructions may be executed on various types of computers or components thereof including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet of Things devices, and the like.

[0147] 7 for computer system (700) are exemplary in nature and are not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing embodiments of the present disclosure. Neither the arrangement of components should be interpreted as having any dependency or requirement regarding any one or combination of components shown in the exemplary embodiment of computer system (700).

[0148] The computer system (700) may include certain human interface input devices that can respond to input by one or more human users through, for example, tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, claps), visual input (e.g., gestures), or olfactory input (not shown). The human interface devices may also be used to capture certain media that do not necessarily involve direct conscious human input, such as audio (e.g., speech, music, ambient sounds), images (e.g., scanned images, photographic images obtained from still cameras), and video (e.g., two-dimensional video, three-dimensional video, including stereoscopic video).

[0149] The input human interface devices may include one or more (only one of each is shown) of a keyboard (701), a mouse (702), a trackpad (703), a touchscreen (710), a data glove (not shown), a joystick (705), a microphone (706), a scanner (707), and a camera (708).

[0150] The computer system (700) may also include some type of human interface output device. Such human interface output devices may stimulate one or more of the human user's senses, for example, through tactile output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touchscreen (710), data gloves (not shown), or joystick (705) (although haptic feedback devices may also function as input devices), audio output devices (e.g., speakers (709), headphones (not shown)), visual output devices (e.g., screens (710), including CRT screens, LCD screens, plasma screens, and OLED screens; each may or may not have touchscreen input capabilities, each may or may not have haptic feedback capabilities, some of which may output two-dimensional visual output or output in greater than three dimensions through means such as stereoscopic output; virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).

[0151] The computer system (700) may also include human-accessible storage and associated media, such as optical media including CD / DVD ROM / RW (720) along with CD / DVD or similar media (721), thumb drives (722), removable hard drives or solid state drives (723), legacy magnetic media such as tape and floppy disks (not shown), specialized ROM / ASIC / PLD-based devices (not shown) such as security dongles, etc.

[0152] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the presently disclosed subject matter does not encompass transmission media, carrier waves, or other transitory signals.

[0153] The computer system (700) may also include an interface (754) to one or more communication networks (755). Networks may be, for example, wireless, wired, or optical. Networks may further be local, wide-area, metropolitan, in-vehicle, and industrial, real-time, delay-tolerant, and the like. Examples of networks include Ethernet, WLAN, cellular networks including GSM, 3G, 4G, 5G, LTE, and the like; TV wired or wireless wide-area digital networks including cable, satellite, and terrestrial broadcast television; and in-vehicle and industrial networks including CANbus. Some networks typically require an external network interface adapter attached to some kind of general-purpose data port or peripheral bus (749) (e.g., a USB port on the computer system (700)). Others are typically integrated into the core of the computer system (700) by attachment to a system bus, as described below (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system (700) can communicate with other entities. Such communication may be unidirectional, receive-only (e.g., broadcast television), unidirectional transmit-only (e.g., CANbus to certain CANbus devices), or bidirectional, for example, to other computer systems using local or wide-area digital networks. Each of these networks and network interfaces, as described above, may use certain protocols and protocol stacks.

[0154] The aforementioned human interface devices, human-accessible storage devices, and network interfaces may be attached to the core (740) of the computer system (700).

[0155] The core (740) may include one or more central processing units (CPUs) (741), graphics processing units (GPUs) (742), specialized programmable processing units in the form of field programmable gate arrays (FPGAs) (743), hardware accelerators for certain tasks (744), graphics adapters (750), etc. These devices may be connected through a system bus (748), along with read-only memory (ROM) (745), random access memory (746), and internal mass storage devices (747), such as internal non-user-accessible hard drives or solid-state drives (SSDs). In some computer systems, the system bus (748) may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may be attached directly to the core's system bus (748) or through a peripheral bus (749). In one example, a screen (710) may be connected to the graphics adapter (750). Architectures for peripheral buses include PCI, USB, etc.

[0156] The CPU (741), GPU (742), FPGA (743), and accelerator (744) may execute certain instructions that, in combination, may constitute the above-mentioned computer code. The computer code may be stored in ROM (745) or RAM (746). Temporary data may also be stored in RAM (746), while persistent data may be stored, for example, in internal mass storage device (747). Rapid storage and retrieval to any of the memory devices may be enabled through the use of cache memory, which may be closely associated with one or more of the CPU (741), GPU (742), mass storage device (747), ROM (745), RAM (746), etc.

[0157] The computer-readable medium can have computer code thereon for performing various computer-implemented operations. The medium and computer code may be those specially designed and constructed for the purposes of the present disclosure, or they may be of the kind well known and available to those having skill in the computer software arts.

[0158] By way of example and not limitation, the architecture (700), and in particular a computer system having a core (740), can provide functionality as a result of a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media can be user-accessible mass storage, as discussed above, as well as media associated with some type of storage of the core (740) that is non-transitory, such as the core's internal mass storage (747) or ROM (745). Software implementing various embodiments of the present disclosure can be stored on such devices and executed by the core (740). The computer-readable media can include one or more memory devices or chips, depending on particular needs. The software can cause the core (740), and in particular the processor (including a CPU, GPU, FPGA, etc.) therein, to perform certain processes or certain portions thereof described herein, including defining data structures stored in RAM (746) and modifying such data structures according to software-defined processes. Additionally or alternatively, a computer system may provide functionality as a result of logic hardwired or otherwise embodied in circuitry (e.g., accelerator (744)), which may operate in place of or in conjunction with software to perform particular processes or portions of particular processes described herein. Reference to software includes logic, and vice versa, as appropriate. Reference to a computer-readable medium may encompass circuitry (e.g., an integrated circuit (IC)) that stores software for execution, circuitry that embodies logic for execution, or both, as appropriate. The present disclosure encompasses any suitable combination of hardware and software.

[0159] The use of "at least one of" or "one of" in this disclosure is intended to include any one or combination of the listed elements. For example, reference to at least one of A, B, or C; at least one of A, B, and C; at least one of A, B, and / or C; and at least one of A through C is intended to include A only, B only, C only, or any combination thereof. Reference to one of A or B, and one of A and B is intended to include A or B or (A and B). The use of "one of" does not exclude any combination of the listed elements, where applicable, such as when the elements are not mutually exclusive.

[0160] While this disclosure has described several exemplary embodiments, there are alterations, permutations, and various substitute equivalents that fall within the scope of this disclosure. Thus, those skilled in the art will be able to devise numerous systems and methods that, although not explicitly shown or described herein, embody the principles of the present disclosure and are thus within its spirit and scope.

Claims

1. 1. A method of video decoding comprising: receiving an encoded video bitstream including encoded information of a current block in a current picture, the encoded information indicating an advanced motion vector prediction merge (AMVP merge) mode, the AMVP merge mode using a bidirectional predictor including an AMVP predictor in a first direction and a merge predictor in a second direction different from the first direction; constructing an AMVP candidate list including a plurality of AMVP candidates for the AMVP predictor; constructing a merge candidate list including a plurality of merge candidates for the merge predictor, the merge candidate list including at least sub-block-based temporal motion vector prediction (SbTMVP) candidates; selecting the AMVP predictor from the AMVP candidate list; selecting the merge predictor from the merge candidate list with the SbTMVP candidate; reconstructing the current block in the AMVP merge mode according to a combination of a first prediction according to the AMVP predictor in the first direction and a second prediction according to the merge predictor selected from the merge candidate list with the SbTMVP candidate in the second direction. method.

2. The merge predictor is determined to be the SbTMVP candidate, and reconstructing the current block is: determining a co-located block for a current block according to the displacement vector of the SbTMVP candidate in a collocated picture for the current picture, wherein the current block includes a plurality of sub-blocks, and the co-located block includes a plurality of co-located sub-blocks corresponding to the plurality of sub-blocks; determining a plurality of motion vectors respectively corresponding to the plurality of sub-blocks in the current block according to motion information of each of the plurality of co-located sub-blocks in the co-located block; generating a plurality of reconstructed sub-blocks corresponding to the plurality of sub-blocks in the current block according to the plurality of motion vectors corresponding to the plurality of sub-blocks in the current block, the plurality of reconstructed sub-blocks forming the second prediction; The method of claim 1 , comprising:

3. Constructing the merge candidate list: deriving the SbTMVP candidates according to motion vectors of spatially neighboring blocks of the current block; The method of claim 1.

4. Deriving the SbTMVP candidate: determining a co-located block in a co-located picture for the current picture according to the motion vectors of the spatial neighboring blocks of the current block; responsive to availability of sub-block-based motion information for the co-located block, using motion vectors of the spatially neighboring blocks as displacement vectors (DVs) for the SbTMVP candidate; The method of claim 3 further comprising:

5. The method of claim 4 , wherein the availability of the sub-block-based motion information for the co-located block is determined based on the availability of a motion vector in a central sub-block of the co-located block.

6. The method of claim 5 , wherein the availability of the sub-block-based motion information for the co-located block is determined based on the availability of a motion vector associated with the second direction in a central sub-block of the co-located block.

7. The method of claim 4 , wherein the availability of the sub-block-based motion information for the co-located block is determined based on the availability of a motion vector associated with the second direction for at least one sub-block in the co-located block.

8. Deriving the SbTMVP candidate includes: checking a plurality of spatially neighboring blocks in a predefined order; identifying a first spatial neighboring block having a first motion vector among the plurality of spatial neighboring blocks, wherein a co-located block according to the first motion vector has sub-block-based motion information available; using the first motion vector as a displacement vector (DV) for the SbTMVP candidate. The method of claim 3 further comprising:

9. Constructing the merge candidate list comprises: using a zero displacement vector (DV) for the SbTMVP candidate; The method of claim 1.

10. Checking a plurality of spatially neighboring blocks in a predefined order; determining that none of the plurality of spatially neighboring blocks has sub-block-based motion information available for a corresponding co-located block; Using a zero displacement vector (DV) for the SbTMVP candidate 10. The method of claim 9, further comprising:

11. Determining a plurality of motion vectors corresponding to the plurality of sub-blocks in the current block includes: For a first sub-block in the plurality of sub-blocks, in response to a lack of motion information in the second direction in a co-located sub-block corresponding to the first sub-block: a second motion vector of a neighboring sub-block of the first sub-block in the second direction; a third motion vector of a central sub-block of the co-located block in the second direction; and Default Motion Vectors determining a first motion vector for the first sub-block according to at least one of The method of claim 2.

12. Determining a plurality of motion vectors corresponding to the plurality of sub-blocks in the current block includes: generating a binary mask for the plurality of sub-blocks in the current block, wherein a binary bin in the binary mask for a sub-block indicates availability of a motion vector in the second direction. The method of claim 2.

13. Currently, blocks can be reconstructed as follows: in response to the binary bin for the sub-block indicating that the motion vector in the second direction is available, generating a first reconstructed sub-block according to the AMVP predictor in the first direction; generating a second reconstructed sub-block according to the motion vector in the second direction; reconstructing the sub-block based on a combination of the first reconstructed sub-block and the second reconstructed sub-block. The method of claim 12.

14. Currently, blocks can be reconstructed as follows: in response to the binary bin for the sub-block indicating that a motion vector in the second direction is unavailable, generating a first reconstructed sub-block according to the AMVP predictor in the first direction; reconstructing the sub-block based on the first reconstructed sub-block. The method of claim 12.

15. The method of claim 1 , wherein the merge candidate list includes the SbTMVP candidate, one or more spatial neighbor merge candidates, and one or more temporal neighbor merge candidates in a predefined order.

16. The method of claim 1 , wherein the merge candidate list includes the SbTMVP candidate in a first position in the merge candidate list.

17. Apparatus for video decoding, comprising processing circuitry configured to perform a method according to any one of claims 1 to 16.

18. A non-transitory computer readable medium storing instructions for causing a computer to perform the method of any one of claims 1 to 16.

19. 1. A method of video encoding comprising: determining to use an advanced motion vector prediction merge (AMVP merge) mode for encoding a current block in a current picture, the AMVP merge mode using a bidirectional predictor including an AMVP predictor in a first direction and a merge predictor in a second direction different from the first direction; constructing an AMVP candidate list including a plurality of AMVP candidates for the AMVP predictor; constructing a merge candidate list including a plurality of merge candidates for the merge predictor, the merge candidate list including at least sub-block-based temporal motion vector prediction (SbTMVP) candidates; selecting the AMVP predictor from the AMVP candidate list; selecting the merge predictor from the merge candidate list with the SbTMVP candidate; reconstructing the current block according to a combination of a first prediction according to the AMVP predictor in the first direction and a second prediction according to the merge predictor selected from the merge candidate list with the SbTMVP candidate in the second direction; method.

20. 20. An apparatus for video encoding, comprising processing circuitry configured to perform the method of claim 19.

21. 20. A non-transitory computer readable medium storing instructions for causing a computer to perform the method of claim 19.

Images