METHOD, APPARATUS AND COMPUTER PROGRAM FOR VIDEO CODING - Patent application

Abstract

Aspects of the present disclosure provide a method and an apparatus including a processing circuit, and determine, based on a syntax element in an encoded video bitstream, that a current block including a plurality of sub-blocks is coded in a sub-block-based temporal motion vector prediction (SbTMVP) mode. Motion vector offset (MVO) information indicating a MVO is received. The MVO indicates a motion offset of a displacement vector (DV) used to adjust a position of the co-located block in a co-located reference picture. An updated DV of the current block is determined based on the DV and the MVO. SbTMVP information of each sub-block of the plurality of sub-blocks is derived based on motion information of a corresponding sub-block in the co-located block indicated by the updated DV. The plurality of sub-blocks in the SbTMVP mode are reconstructed based on the SbTMVP information of the plurality of sub-blocks.

Description

[Technical field]

[0001] This application claims the benefit of priority to U.S. Provisional Application No. 63 / 332,131 entitled "SUBBLOCK BASED MOTION VECTOR PREDICTOR WITH MOTION VECTOR OFFSET," filed April 18, 2022, which claims the benefit of priority to U.S. Provisional Application No. 17 / 984,123 entitled "A SUB-BLOCK BASED TEMPORAL MOTION VECTOR PREDICTOR WITH AN MOTION VECTOR OFFSET," filed November 9, 2022. The disclosures of these prior applications are incorporated herein by reference in their entireties.

[0002] This disclosure describes embodiments generally relating to video coding. [Background technology]

[0003] The background presented herein is intended to provide a general overview of the context of the disclosure. To the extent described in this Background section, the work of the named inventors, as well as aspects of the disclosure that may not otherwise qualify as prior art at the time of filing, are not admitted, explicitly or implicitly, as prior art to the present disclosure.

[0004] An uncompressed digital image and / or video includes a sequence of pictures, each having spatial dimensions of, for example, 1920x1080 luminance samples and associated chrominance samples. The sequence of pictures may have a fixed or variable picture rate (also informally known as frame rate), for example, 60 pictures per second, i.e., a picture rate of 60 Hz. An uncompressed image and / or video has specific bitrate requirements. For example, 1080p60 4:2:0 video (1920x1080 luminance sample resolution at a frame rate of 60 Hz) with 8 bits per sample requires a bandwidth approaching 1.5 Gbit / s. One hour of such video requires more than 600 GByte of storage space.

[0005] One objective of image and / or video encoding and decoding may be the reduction of redundancy in the input image and / or video signal through compression. Compression can help reduce the aforementioned bandwidth and / or storage space requirements, possibly by more than one order of magnitude. The description herein uses video encoding / decoding as an illustrative example, but the same techniques can be applied to image encoding / decoding in a similar manner without departing from the spirit of the present disclosure. Both lossless and lossy compression, as well as combinations thereof, can be used. Lossless compression refers to techniques where an exact replica of the original signal can be reconstructed from a compressed original signal. When lossy compression is used, the reconstructed signal may not be the same as the original signal, but the distortion between the original signal and the reconstructed signal will be small enough to make the reconstructed signal useful for the intended application. For video, lossy compression is widely used. The amount of acceptable distortion depends on the application, for example, a user of a particular consumer streaming application may tolerate higher distortion than a user of a television distribution application. The achievable compression ratio reflects this, and higher acceptable / tolerable distortion can result in a higher compression ratio.

[0006] Video encoders and decoders can utilize techniques from several broad categories, including, for example, motion compensation, transform processing, quantization, and entropy coding.

[0007] Video codec techniques may include a technique known as intra-coding. In intra-coding, sample values ​​are represented without reference to samples or other data from previously reconstructed reference pictures. In some video codecs, a picture is spatially subdivided into blocks of samples. If all the samples of a block are coded in intra mode, the picture may be an intra picture. Intra pictures and their derivatives, such as independent decoder refresh pictures, may be used to reset the decoder state and therefore may be used as the first picture in a coded video bitstream and video session or as a still image. Samples of an intra block may be subjected to a transform, and the transform coefficients may be quantized before entropy coding. Intra prediction may be a technique that minimizes sample values ​​in the pre-transform domain. In some cases, the smaller the DC value after the transform and the smaller the AC coefficients, the fewer bits are needed for a given quantization step size to represent the block after entropy coding.

[0008] Traditional intra-coding, for example as used in MPEG-2 generation coding techniques, does not use intra-prediction. However, some newer video compression techniques include techniques that attempt to make predictions based on surrounding sample data and / or metadata obtained during the encoding and / or decoding of a block of data. Such techniques are hereinafter referred to as "intra-prediction" techniques. Note that, at least in some cases, intra-prediction uses only reference data from the current picture being reconstructed, and not from reference pictures.

[0009] There may be many different forms of intra prediction. If more than one such technique may be used in a given video coding technique, the particular technique used may be coded as a particular intra prediction mode that uses that particular technique. In certain cases, an intra prediction mode may have sub-modes and / or parameters that may be coded separately or may be included in a mode codeword that defines the prediction mode used. Which codeword is used for a given mode, sub-mode, and / or parameter combination may affect the coding efficiency gain through intra prediction, as may the entropy coding technique used to convert the codeword into a bitstream.

[0010] A specific mode of intra prediction was introduced in H.264, improved in H.265, and further improved in newer coding techniques, such as the Joint Exploration Model (JEM), Versatile Video Coding (VVC), and Benchmark Set (BMS). A predictor block may be formed using neighboring sample values ​​of already available samples. The sample values ​​of the neighboring samples are copied to the predictor block according to a direction. The reference of the direction to use may be coded into the bitstream or predicted itself.

[0011] Referring to FIG. 1A, in the lower right corner, a subset of 9 predictor directions known from the 33 possible predictor directions (corresponding to the 33 angle modes of the 35 intra modes) defined in H.265 is depicted. The confluence of the arrows (101) represents the sample being predicted. The arrow indicates the direction from which the sample is predicted. For example, the arrow (102) indicates that the sample (101) is predicted from one or more samples on the upper right at an angle of 45 degrees from the horizontal. Similarly, the arrow (103) indicates that the sample (101) is predicted from one or more samples on the lower left at an angle of 22.5 degrees from the horizontal.

[0012] Still referring to FIG. 1A, a square block (104) of 4×4 samples (indicated by a dashed bold line) is depicted at the top left. The square block (104) contains 16 samples, with each sample labeled with an “S” and its position in the Y dimension (e.g., row index) and its position in the X dimension (e.g., column index). For example, sample S21 is the second sample (from the top) in the Y dimension and the first sample (from the left) in the X dimension. Similarly, sample S44 is the fourth sample in the block (104) in both the Y and X dimensions. Since the block is 4×4 samples in size, S44 is at the bottom right. Additionally, reference samples are shown following a similar numbering scheme. The reference samples are labeled with R and their Y position (e.g., row index) and X position (column index) relative to the block (104). In both H.264 and H.265, the prediction samples are adjacent to the block being reconstructed, and therefore there is no need to use negative values.

[0013] Intra-picture prediction works by copying reference sample values ​​from adjacent samples indicated by the signaled prediction direction. For example, assume that for this block, the coded video bitstream includes signaling indicating that the prediction direction is aligned with the arrow (102), i.e., the sample is predicted from the sample 45 degrees above and to the right from the horizontal. Then, samples S41, S32, S23, and S14 are predicted from the same reference sample R05. And sample S44 is predicted from reference sample R08.

[0014] In certain cases, especially when the orientation is not evenly divisible by 45 degrees, the values ​​of several reference samples may be combined, for example by interpolation, to calculate the reference sample.

[0015] As video coding techniques develop, the number of possible directions increases. In H.264 (2003), nine different directions could be represented. This increased to 33 in H.265 (2013). Currently, JEM / VVC / BMS can support up to 65 directions. Experiments are performed to identify the most likely directions, and specific techniques in entropy coding are used to represent them with fewer bits, accepting penalties for less likely directions. Furthermore, the directions themselves may be predicted from neighboring directions used in neighboring already decoded blocks.

[0016] FIG. 1B shows a schematic diagram (110) depicting 65 intra prediction directions according to JEM to illustrate the gradually increasing number of prediction directions.

[0017] The mapping of intra-prediction direction bits to represent directions in an encoded video bitstream may vary from one video coding technique to another. Such mappings may range, for example, from simple direct mappings to complex adaptation schemes including codewords, most probable modes, and similar techniques. However, in most cases, there may be certain directions in the video content that are statistically less likely to occur than other certain directions. Since the goal of video compression is to reduce redundancy, a well-performing video coding technique will result in those less likely directions being represented with a larger number of bits than the more likely directions.

[0018] Inter-picture prediction with motion compensation may be used to perform image and / or video encoding and decoding. Motion compensation may be a lossy compression technique and may refer to a technique used for predicting a newly reconstructed picture or picture portion after blocks of sample data from a previously reconstructed picture or part thereof (reference picture) are spatially shifted in a direction pointed to by a motion vector (hereinafter MV). In some cases, the reference picture may be the same as the picture currently being reconstructed. The MV may have two dimensions X and Y, or may have three dimensions with the third being an indication of the reference picture to be used (the latter may be indirectly referred to as a temporal dimension).

[0019] In some video compression techniques, the MV applicable to a particular region of sample data can be predicted from another MV, e.g., from a MV preceding the MV in decoding order and relating to another region of sample data that is spatially adjacent to the region being reconstructed. Doing so can significantly reduce the amount of data required to code the MV, thereby removing redundancy and increasing compression. MV prediction can work effectively because, for example, when coding an input video signal (known as natural video) originating from a camera, there is a statistical likelihood that regions larger than the region to which a single MV is applicable move in a similar direction and can therefore be predicted using similar motion vectors, possibly derived from MVs of neighboring regions. This results in the MV found for a given region being similar or the same as the MV predicted from the surrounding MVs, and thus can be represented after entropy coding with fewer bits than would be used if coding the MV directly. In some cases, MV prediction can be an example of lossless compression of a signal (i.e., an MV) derived from an original signal (i.e., a sample stream). In other cases, the MV prediction itself may be non-reversible, for example due to rounding errors in computing the predictor from several surrounding MVs.

[0020] Various MV prediction mechanisms are described in H.265 / HEVC (ITU-T Recommendation H.265, “High Efficiency Video Coding”, December 2016). Among the many MV prediction mechanisms provided by H.265, a technique called “spatial merging” will be described below with reference to FIG.

[0021] Referring to Figure 2, a current block (201) has samples that the encoder found during the motion search process to be predictable from a spatially shifted previous block of the same size. Instead of coding the MV directly, the MV can be derived from metadata associated with one or more reference pictures, such as from the immediately preceding reference picture (in decoding order), using the MV associated with any one of five surrounding samples, denoted as A0, A1, and B0, B1, B2 (202 to 206, respectively). In H.265, MV prediction can use predictors from the same reference picture as the neighboring blocks use. Summary of the Invention

[0022] Aspects of the present disclosure provide a method and apparatus for video encoding / decoding. In some examples, the apparatus for video decoding includes a processing circuit. In one embodiment, the processing circuit receives displacement vector (DV) offset information of a current block in a current picture from an encoded video bitstream. The current block includes a plurality of sub-blocks to be reconstructed using a sub-block-based temporal motion vector prediction (SbTMVP) mode. An updated DV of the current block can be determined based on the DV of the current block and the DV offset of the current block. The DV offset is indicated by the DV offset information. The updated DV of the current block points to a co-located block in a co-located reference picture. The co-located block is co-located with the current block. The processing circuit determines motion information of a sub-block of the plurality of sub-blocks based on motion information of a corresponding sub-block in the co-located block, and reconstructs the sub-block of the plurality of sub-blocks based on the motion information of the sub-block of the plurality of sub-blocks.

[0023] In one embodiment, the processing circuitry determines the updated DV to be the vector sum of the DV and the MV offset.

[0024] In one embodiment, the DV offset information includes a DV offset signaled in the encoded video bitstream.

[0025] In one embodiment, the DV offset information indicates at least one index indicating the magnitude of the DV offset and the direction of the DV offset.

[0026] In one example, the at least one index includes a distance index indicating a magnitude of the DV offset, which is one of a set of predetermined distances, and a direction index indicating a direction of the DV offset, which is one of a set of predetermined directions.

[0027] In one example, the set of predetermined distances and the set of predetermined directions are used in a merged motion vector difference (MMVD) mode.

[0028] In one example, the updated DV is constrained such that the co-located block is within a constrained area in the co-located reference picture, the constrained area including the co-located area corresponding to the current CTU in the current picture, and the current CTU including the current block.

[0029] In one embodiment, a coded video bitstream having a current picture is received. The current picture includes a current block. The current block includes a plurality of sub-blocks. A processing circuit determines, based on a syntax element in the coded video bitstream, that a current block including the plurality of sub-blocks is coded in SbTMVP mode. The processing circuit obtains motion vector offset (MVO) information of the current block indicating a motion vector offset (MVO). The MVO indicates a motion offset of a displacement vector (DV) used to adjust a position of the co-located block in a co-located reference picture. The processing circuit determines an updated DV of the current block based on the DV and MVO of the current block. The updated DV indicates an adjusted position of the co-located block in the co-located reference picture. The processing circuit derives SbTMVP information of each sub-block of the plurality of sub-blocks based on at least motion information of a corresponding sub-block in the co-located block indicated by the updated DV, and reconstructs the plurality of sub-blocks in SbTMVP mode based on the SbTMVP information of the plurality of sub-blocks.

[0030] In one embodiment, the processing circuit receives motion vector (MV) offset information of a current block in a current picture from the coded video bitstream. The current block includes a plurality of sub-blocks to be reconstructed using a sub-block-based temporal motion vector prediction (SbTMVP) mode. The processing circuit determines a displacement vector (DV) of the current block indicating a co-located block in a co-located reference picture that is co-located with the current block. The processing circuit determines motion information of a sub-block of the plurality of sub-blocks based on motion information of a corresponding sub-block in the co-located block, and determines updated motion information of the sub-block of the plurality of sub-blocks based on the motion information of the sub-block of the plurality of sub-blocks and the MV offset of the current block indicated by the MV offset information. The processing circuit reconstructs the sub-block of the plurality of sub-blocks based on the updated motion information.

[0031] In one example, the MV offset information includes an MV offset signaled in the encoded video bitstream.

[0032] In one example, the MV offset information indicates at least one index indicating the magnitude of the MV offset and the direction of the MV offset.

[0033] In one example, the at least one index includes a distance index indicating a magnitude of the MV offset, which is one of a set of predetermined distances, and a direction index indicating a direction of the MV offset, which is one of a set of predetermined directions.

[0034] A set of predefined distances and a set of predefined directions are used in the merged motion vector difference (MMVD) mode.

[0035] In one example, the motion information of the subblock of the plurality of subblocks includes a first motion vector (MV) associated with a first reference picture from a first reference picture list L0. The processing circuit determines an updated first MV, which is a vector sum of the first MV and an MV offset, and the updated motion information includes the updated first MV.

[0036] In one example, the motion information of the subblock of the plurality of subblocks includes a second MV associated with a second reference picture from a second reference picture list L1. The processing circuit determines an updated second MV, which is one of (i) a vector difference between the second MV and an MV offset, or (ii) a vector sum between the second MV and a scaled MV offset. The scaled MV offset is based on the MV offset, a picture order count (POC) of the current picture, the POC of the first reference picture, and the POC of the second reference picture.

[0037] Aspects of the present disclosure also provide a non-transitory computer-readable medium storing instructions that, when executed by a computer for video decoding, cause the computer to perform a method for video decoding. [Brief description of the drawings]

[0038] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings. [Figure 1A] 1 illustrates a schematic diagram of an exemplary subset of intra-prediction modes. [Figure 1B] 1 illustrates an exemplary intra-prediction direction. [Diagram 2] An example of a current block (201) and surrounding samples is shown. [Diagram 3] 1 illustrates generally an example block diagram of a communication system (300). [Figure 4] 4 illustrates an example block diagram of a communication system (400). [Diagram 5] 2 illustrates an exemplary block diagram of a decoder; [Figure 6] 1 illustrates an exemplary block diagram of an encoder; [Figure 7] 1 shows a block diagram of an exemplary encoder. [Figure 8] 1 shows a block diagram of an exemplary decoder. [Figure 9] 4 illustrates the locations of spatial merge candidates according to one embodiment of the present invention. [Figure 10] 4 illustrates candidate pairs considered for redundancy checking of spatial merge candidates according to one embodiment of the present invention; [Figure 11] 1 illustrates an example motion vector scaling for temporal merge candidates. [Figure 12] 1 illustrates example candidate positions for temporal merge candidates for a current coding unit. [Figure 13]13-14 show an example of the search process in merged motion vector difference (MMVD) mode. [Figure 14] 13-14 show an example of the search process in merged motion vector difference (MMVD) mode. [Figure 15] 13 shows additional refinement positions along multiple oblique angles in MMVD mode. [Figure 16] 16-17 show an exemplary sub-block-based temporal motion vector prediction (SbTMVP) process used in SbTMVP mode. [Figure 17] 16-17 show an exemplary sub-block-based temporal motion vector prediction (SbTMVP) process used in SbTMVP mode. [Figure 18] 1 shows a flowchart outlining an encoding process according to some embodiments of the present disclosure. [Figure 19A] 1 shows a flowchart outlining a decoding process according to some embodiments of the present disclosure. [Figure 19B] 1 shows a flowchart outlining a decoding process according to some embodiments of the present disclosure. [Figure 20] 1 shows a flowchart outlining an encoding process according to some embodiments of the present disclosure. [Figure 21] 1 shows a flowchart outlining a decoding process according to some embodiments of the present disclosure. [Figure 22] FIG. 1 is a schematic diagram of a computer system according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0039] FIG. 3 illustrates an example block diagram of a communication system (300). The communication system (300) includes a plurality of terminal devices that can communicate with each other, for example, via a network (350). For example, the communication system (300) includes a first pair of terminal devices (310) and (320) interconnected via the network (350). In the example of FIG. 3, the first pair of terminal devices (310) and (320) perform a unidirectional transmission of data. For example, the terminal device (310) may code video data (e.g., a stream of video pictures captured by the terminal device (310)) for transmission to another terminal device (320) via the network (350). The encoded video data may be transmitted in the form of one or more encoded video bitstreams. The terminal device (320) may receive the encoded video data from the network (350), decode the encoded video data to reconstruct the video pictures, and display the video pictures according to the reconstructed video data. One-way data transmission may be common in media serving applications and the like.

[0040] In another example, the communication system (300) includes a second pair of terminal devices (330) and (340) that engage in bidirectional transmission of encoded video data, such as during a video conference. In a bidirectional transmission of data, in one example, each of the terminal devices (330) and (340) may code video data (e.g., a stream of video pictures captured by that terminal device) for transmission over the network (350) to the other of the terminal devices (330) and (340). Each of the terminal devices (330) and (340) may also receive encoded video data transmitted by the other of the terminal devices (330) and (340), decode the encoded video data to recover the video pictures, and display the video pictures on an accessible display device in accordance with the recovered video data.

[0041] In the example of FIG. 3, terminal devices (310), (320), (330), and (340) are shown as a server, a personal computer, and a smartphone, respectively, although the principles of the present disclosure may not be so limited. Embodiments of the present disclosure find application in laptop computers, tablet computers, media players, and / or dedicated videoconferencing equipment. Network (350) represents any number of networks that convey encoded video data between terminal devices (310), (320), (330), and (340), including, for example, wired (cable) and / or wireless communication networks. Communication network (350) may exchange data over circuit-switched and / or packet-switched channels. Exemplary networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For purposes of this description, the architecture and topology of network (350) may not be important to the operation of the present disclosure, unless otherwise described below.

[0042] 4 shows an arrangement of video encoders and video decoders in a streaming environment as an example of an application of the disclosed subject matter, which may be equally applied to other video-enabled applications including, for example, videoconferencing, digital TV, streaming services, and storage of compressed video on digital media including CDs, DVDs, memory sticks, and the like.

[0043] The streaming system may include a capture subsystem (413), which may include a video source (401), such as a digital camera, that produces a stream of uncompressed video pictures (402). In one example, the stream of video pictures (402) includes samples captured by a digital camera. The stream of video pictures (402) is depicted as a thick line to emphasize its high data volume compared to the encoded video data (404) (or encoded video bitstream), which may be processed by an electronics device (420) that includes a video encoder (403) coupled to the video source (401). The video encoder (403) may include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter, which are described in more detail below. The encoded video data (404) (or encoded video bitstream), which is depicted as a thin line to emphasize its low data volume compared to the stream of video pictures (402), may be stored on a streaming server (405) for later use. One or more streaming client subsystems, such as the client subsystems (406) and (408) of FIG. 4, can access the streaming server (405) to retrieve copies (407) and (409) of the encoded video data (404). The client subsystem (406) can include a video decoder (410), for example in an electronics device (430). The video decoder (410) can decode the incoming copy of the encoded video data (407) and produce an outgoing stream of video pictures (411), which can be rendered on a display (412) (e.g., a display screen) or other rendering device (not shown). In some streaming systems, the encoded video data (404), (407), and (409) (e.g., a video bitstream) can be encoded according to a particular video coding / compression standard.Examples of such standards include ITU-T Recommendation H.265. In one example, one video coding standard under development is informally known as Versatile Video Coding (VVC). The subject matter disclosed herein may be used in the context of VVC.

[0044] It should be noted that electronics devices 420 and 430 may include other components (not shown). For example, electronics device 420 may include a video decoder (not shown), and electronics device 430 may also include a video encoder (not shown).

[0045] 5 shows an example block diagram of a video decoder (510). The video decoder (510) may be included in an electronics device (530). The electronics device (530) may include a receiver (531) (e.g., a receiving circuit). The video decoder (510) may be used in place of the video decoder (410) in the example of FIG. 4.

[0046] A receiver (531) may receive one or more coded video sequences to be decoded by the video decoder (510). In one embodiment, one coded video sequence is received at a time, and the decoding of each coded video sequence is independent of the decoding of the other coded video sequences. The coded video sequences may be received from a channel (501), which may be a hardware / software link to a storage device that stores the coded video data. The receiver (531) may receive the coded video data along with other data, such as coded audio data and / or auxiliary data streams, which may be forwarded to their respective using entities (not shown). The receiver (531) may separate the coded video sequences from the other data. To combat network jitter, a buffer memory (515) may be coupled between the receiver (531) and the entropy decoder / parser 520 (hereinafter, "parser (520)"). In certain applications, the buffer memory (515) is part of the video decoder (510). In others, it may be external to the video decoder (510) (not shown). In still others, there may be a buffer memory (not shown) external to the video decoder (510), e.g., to combat network jitter, and yet another buffer memory (515) internal to the video decoder (510), e.g., to handle playback timing. When the receiver (531) is receiving data from a store / forward device with sufficient bandwidth and controllability, or from an equally synchronous network, the buffer memory (515) may not be needed or may be small. For use over a best-effort packet network, such as the Internet, the buffer memory (515) may be needed and may be relatively large and advantageously of adaptive size, and may be implemented at least in part in an operating system or similar element (not shown) external to the video decoder (510).

[0047] The video decoder (510) may include a parser (520) for reconstructing symbols (521) from the coded video sequence. The categories of symbols may include information used to manage the operation of the video decoder (510) and possibly information for controlling a rendering device, such as a render device (512) (e.g., a display screen) that is not an integral part of the electronics device (530) but may be coupled to the electronics device (530) as shown in FIG. 5. The control information for the rendering device(s) may be in the form of a Supplementary Enhancement Information (SEI) message or a Video Usability Information (VUI) parameter set fragment (not shown). The parser (520) may parse / entropy decode the received coded video sequence. The coding of the coded video sequence may be according to a video coding technique or standard and may follow various principles, including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser (520) may extract from the coded video sequence a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder based on at least one parameter corresponding to the group. The subgroups may include a group of pictures (GOP), a picture, a tile, a slice, a macroblock, a coding unit (CU), a block, a transform unit (TU), a prediction unit (PU), etc. The parser (520) may also extract information from the coded video sequence information, such as transform coefficients, quantization parameter values, motion vectors, etc.

[0048] The parser (520) may perform an entropy decoding / parsing process on the video sequence received from the buffer memory (515) to produce symbols (521).

[0049] The reconstruction of the symbols (521) may involve several different units, depending on the type of coded video picture or portion thereof and other factors (e.g., inter-picture and intra-picture, inter-block and intra-block, etc.). Which units are involved and how they are involved can be controlled by subgroup control information parsed from the coded video sequence by the parser (520). The flow of such subgroup control information between the parser (520) and the following units is not shown for the sake of clarity.

[0050] Beyond the functional blocks already described, the video decoder (510) may be conceptually subdivided into a number of functional units, as described below. In a practical implementation operating within commercial constraints, many of these units may interact closely with each other and may be at least partially integrated with each other. However, for purposes of illustrating the subject matter disclosed herein, the following conceptual subdivision into functional units is adequate:

[0051] The first unit is a scalar / inverse transform unit (551), which receives the quantized transform coefficients as symbol(s) (521) from the parser (520), along with control information including which transform to use, block size, quantization factor, quantization scaling matrix, etc. The scalar / inverse transform unit (551) can output blocks with sample values ​​that can be input to an aggregator (555).

[0052] In some cases, the output samples of the scaler / inverse transform unit (551) may relate to intra-coded blocks, which are blocks that do not use prediction information from a previously reconstructed picture, but can use prediction information from a previously reconstructed portion of the current picture. Such prediction information can be provided by an intra-picture prediction unit (552). In some cases, the intra-picture prediction unit (552) generates a block of the same size and shape as the block being reconstructed using surrounding already reconstructed information fetched from a current picture buffer (558). The current picture buffer (558) buffers, for example, a partially reconstructed current picture and / or a fully reconstructed current picture. The aggregator (555) adds, in some cases, on a sample-by-sample basis, the prediction information generated by the intra-prediction unit (552) to the output sample information provided by the scaler / inverse transform unit (551).

[0053] In other cases, the output samples of the scaler / inverse transform unit (551) may relate to a block that may be inter-coded and motion compensated. In such cases, the motion compensated prediction unit (553) may access the reference picture memory (557) to fetch samples used for prediction. After motion compensating the fetched samples according to the symbols (521) related to the block, these samples may be added by the aggregator (555) to the output of the scaler / inverse transform unit (551) (in this case referred to as residual samples or residual signals) to generate output sample information. The addresses in the reference picture memory (557) from which the motion compensated prediction unit (553) fetches the prediction samples may be controlled by a motion vector and are available to the motion compensated prediction unit (553) in the form of symbols (521) that may have, for example, X, Y, and reference picture components. Motion compensation may also include interpolation of sample values ​​fetched from the reference picture memory (557) when sub-sample accurate motion vectors are used, as well as motion vector prediction mechanisms.

[0054] The output samples of the aggregator (555) may be subjected to various loop filtering techniques in a loop filter unit (556). Video compression techniques may include in-loop filter techniques that are controlled by parameters included in the coded video sequence (also referred to as a coded video bitstream) and made available to the loop filter unit (556) as symbols (521) from the parser (520). Video compression may also be responsive to meta-information obtained during decoding of previous portions of the coded pictures or coded video sequences (in decoding order), as well as to previously reconstructed and loop filtered sample values.

[0055] The output of the loop filter unit (556) can be a sample stream that can be output to a render device (512), which can also be stored in a reference picture memory (557) for use in future inter-picture prediction.

[0056] Once a particular coded picture is fully reconstructed, it can be used as a reference picture for future prediction. For example, once a coded picture corresponding to a current picture is fully reconstructed and that coded picture is identified (e.g., by the parser (520)) as a reference picture, the current picture buffer (558) can become part of the reference picture memory (557), and a new current picture buffer can be reallocated before beginning reconstruction of the next coded picture.

[0057] The video decoder (510) may perform the decoding process according to a given video compression technique or standard, such as ITU-T Recommendation H.265. The encoded video sequence may conform to the syntax defined by the video compression technique or standard used, in the sense of adhering to both the syntax of the video compression technique or standard and the profile documented in the video compression technique or standard. In particular, a profile may select a particular tool from all tools available in the video compression technique or standard, such that it is the only tool available for use under that profile. Compliance also requires that the complexity of the encoded video sequence is within a range defined by the level of the video compression technique or standard. In some cases, the level constrains the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. The limits set by the level may be further constrained in some cases through a Hypothetical Reference Decoder (HRD) specification and metadata for HRD buffer management signaled in the encoded video sequence.

[0058] In one embodiment, the receiver (531) may receive additional (redundant) data along with the encoded video. The additional data may be included as part of the encoded video sequence(s). The additional data may be used by the video decoder (510) to properly decode the data and / or to more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0059] 6 shows an example block diagram of a video encoder (603). The video encoder (603) is included in an electronics device (620). For example, the electronics device (620) includes a transmitter (640) (e.g., a transmission circuit). The video encoder (603) can be used in place of the video encoder (403) in the example of FIG. 4.

[0060] The video encoder (603) may receive video samples from a video source (601) (which in the example of FIG. 6 is not part of the electronics device (620)) that may capture the video image(s) to be encoded by the encoder (603). In another example, the video source (601) is part of the electronics device (620).

[0061] The video source (601) may provide a source video sequence to be encoded by the video encoder (603) in the form of a digital video sample stream that may be of any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 Y CrCB, RGB, ...), and any suitable sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media service provision system, the video source (601) may be a storage device that stores pre-prepared video. In a video conferencing system, the video source (601) may be a camera that captures local image information as a video sequence. The video data may be provided as a number of individual pictures that convey motion when viewed in sequence. The pictures themselves may be organized as a spatial array of pixels, each of which may have one or more samples, depending on the sampling structure, color space, etc. used. Those skilled in the art will readily appreciate the relationship between pixels and samples. The following description focuses on samples.

[0062] According to one embodiment, the video encoder (603) may code and compress pictures of a source video sequence into an encoded video sequence (643) in real-time or under other required time constraints. Enforcing an appropriate coding rate is one function of the controller (650). In some embodiments, the controller (650) controls and is operatively coupled to other functional units, such as those described below, which couplings are not shown for clarity. Parameters set by the controller (650) may include rate control related parameters (picture skip, quantizer, lambda value for rate distortion optimization techniques, ...), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (650) may be configured with other suitable functions associated with the video encoder (603) that are optimized for a particular system design.

[0063] In some embodiments, the video encoder (603) is configured to operate in a coding loop. As an oversimplified explanation, in one example, the coding loop can include a source coder (630) (e.g., responsible for creating symbols, e.g., a symbol stream, based on an input picture to be coded and reference picture(s)) and a (local) decoder (633) embedded in the video encoder (603). The decoder (633) reconstructs the symbols to generate sample data, in a similar manner that the (remote) decoder also creates. The reconstructed sample stream (sample data) is input to a reference picture memory (634). Since the decoding of the symbol stream results in bit-accurate results that are independent of the decoder location (local or remote), the content in the reference picture memory (634) is also bit-accurate between the local and remote encoders. In other words, the predictive part of the encoder "sees" exactly the same sample values ​​as the reference picture samples that the decoder "sees" when using prediction during decoding. This basic principle of reference picture synchrony (and the resulting drift when synchrony cannot be maintained, for example due to channel errors) is also used in some related art.

[0064] The operation of the "local" decoder (633) may be the same as that of a "remote" decoder, such as the video decoder (510), which has already been described in detail above in relation to Figure 5. However, briefly referring also to Figure 5, because symbols are available and the encoding / decoding of the symbols into an encoded video sequence by the entropy coder (645) and parser (520) may be lossless, the entropy decoding portion of the video decoder (510), including the buffer memory (515) and the parser (520), may not be fully implemented in the local decoder (633).

[0065] In one embodiment, the decoder techniques, except for parsing / entropy decoding, present in the decoder are present in the corresponding encoder in the same or substantially the same functional form. Therefore, the subject matter disclosed focuses on the decoder operation. The description of the encoder techniques can be omitted since they are the reverse of the decoder techniques described in detail. In certain specific areas, more detailed descriptions are provided below.

[0066] In operation, in some examples, the source coder (630) may perform motion-compensated predictive coding, which predictively codes an input picture relative to one or more previously coded pictures from a video sequence designated as “reference pictures.” Thus, the coding engine (632) codes differences between pixel blocks of the input picture and pixel blocks of the reference picture(s) that may be selected as prediction reference(s) for the input picture.

[0067] The local video decoder (633) may decode the encoded video data of pictures that may be designated as reference pictures based on symbols created by the source coder (630). The operation of the coding engine (632) may advantageously be a lossy process. When the encoded video data may be decoded by a video decoder (not shown in FIG. 6), the reconstructed video sequence may typically be a replica of the source video sequence with some errors. The local video decoder (633) may replicate the decoding process that may be performed by a video decoder on the reference pictures, causing the reconstructed reference pictures to be stored in the reference picture memory (634). Thus, the video encoder (603) may locally store copies of reconstructed reference pictures that have common content with the reconstructed reference pictures that will be obtained by the far-end video decoder.

[0068] The predictor (635) may perform a predictive search for the coding engine (632). That is, for a new picture to be coded, the predictor (636) may search the reference picture memory (634) for sample data (as candidate reference pixel blocks) or specific metadata, such as reference picture motion vectors or block shapes, that may serve as suitable prediction references for the new picture. The predictor (635) may operate on a pixel block by pixel block basis to find a suitable prediction reference. In some cases, as determined by the search results obtained by the predictor (635), the input picture may have prediction references derived from multiple reference pictures stored in the reference picture memory (634).

[0069] The controller (650) may manage the coding process of the source coder (630), including, for example, setting the parameters and subgroup parameters used to encode the video data.

[0070] The outputs of all the aforementioned functional units may be subjected to entropy coding in an entropy coder (645), which converts the symbols produced by the various functional units into a coded video sequence by applying lossless compression to the symbols according to techniques such as Huffman coding, variable length coding, arithmetic coding, etc.

[0071] A transmitter (640) may buffer the encoded video sequence(s) generated by the entropy coder (645) and prepare them for transmission over a communication channel (660), which may be a hardware / software link to a storage device that stores the encoded video data. The transmitter (640) may merge the encoded video data from the video encoder (603) with other data to be transmitted, such as encoded audio data and / or auxiliary data streams (sources not shown).

[0072] The controller (650) may manage the operation of the video encoder (603). In coding, the controller (650) may assign each coded picture a particular coded picture type, which may affect the coding technique that may be applied to the respective picture. For example, pictures may often be assigned one of the following picture types:

[0073] An Intra picture (I-picture) may be one that can be coded and decoded without using any other picture in a sequence as a source of prediction. Some video codecs allow several different types of Intra pictures, including, for example, Independent Decoder Refresh (IDR) pictures. Those skilled in the art are aware of these variants of I-pictures, as well as their respective uses and characteristics.

[0074] A predictive picture (P-picture) may be encoded and decoded using intra- or inter-prediction, using at most one motion vector and reference index to predict the sample values ​​of each block.

[0075] Bidirectionally predicted pictures (B-pictures) may be encoded and decoded using intra- or inter-prediction, using at most two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, multi-predicted pictures may use more than two reference pictures and associated metadata for the reconstruction of a single block.

[0076] A source picture is generally spatially subdivided into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples each) and may be coded block by block. Blocks may be predictively coded with reference to other (already coded) blocks as determined by the coding assignment applied to their respective pictures. For example, blocks of I-pictures may be coded non-predictively or they may be predictively coded with reference to already coded blocks of the same picture (spatial or intra prediction). Pixel blocks of P-pictures may be coded non-predictively or via spatial or temporal prediction with reference to one previously coded reference picture. Blocks of B-pictures may be coded non-predictively or via spatial or temporal prediction with reference to one or two previously coded reference pictures.

[0077] The video encoder (603) may perform a coding process according to a given video coding technique or standard, such as ITU-T Recommendation H.265. In operation, the video encoder (603) may perform various compression processes, including predictive coding processes that exploit temporal and spatial redundancy in the input video sequence. The coded video data may therefore conform to a syntax defined by the video coding technique or standard being used.

[0078] In one embodiment, the transmitter (640) may transmit additional data along with the encoded video. The source coder (630) may include such data as part of the encoded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

[0079] A video may be captured as multiple source pictures (video pictures) in a time sequence. Intra-picture prediction (often abbreviated as intra-prediction) exploits spatial correlation within a given picture, while inter-picture prediction exploits correlation (temporal or other) between pictures. In one example, a particular picture being coded / decoded, called a current picture, is divided into multiple blocks. When a block in the current picture is similar to a reference block in a reference picture previously coded and still buffered in the video, the block in the current picture can be coded by a vector called a motion vector. A motion vector points to a reference block in a reference picture, and can have a third dimension that identifies the reference picture if multiple reference pictures are used.

[0080] In some embodiments, bi-prediction techniques can be used in inter-picture prediction. According to bi-prediction techniques, two reference pictures are used, such as a first reference picture and a second reference picture, both of which are earlier in the video in decoding order (but may be earlier and later, respectively, in display order) relative to the current picture. A block in the current picture can be coded by a first motion vector that points to a first reference block in the first reference picture and a second motion vector that points to a second reference block in the second reference picture. The block can be predicted by a combination of the first and second reference blocks.

[0081] Furthermore, to improve coding efficiency in inter-picture prediction, merge mode techniques can be used.

[0082] According to some embodiments of the present disclosure, prediction, such as inter-picture prediction and intra-picture prediction, is performed on a block-by-block basis. For example, according to the HEVC standard, a picture in a sequence of video pictures is divided into multiple coding tree units (CTUs) for compression, and the CTUs in a picture have the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. In general, a CTU includes three coding tree blocks (CTBs), one luma CTB and two chroma CTBs. Each CTU may be recursively quadtree partitioned into one or more coding units (CUs). For example, a CTU of 64×64 pixels may be partitioned into one CU of 64×64 pixels, or four CUs of 32×32 pixels, or 16 CUs of 16×16 pixels. In one example, each CU is analyzed to determine a prediction type for that CU, such as an inter prediction type or an intra prediction type. A CU is divided into one or more prediction units (PUs) depending on temporal and / or spatial predictability. In general, each PU includes a luma prediction block (PB) and two chroma PBs. In one embodiment, prediction operations during coding are performed in units of prediction blocks. Taking a luma prediction block as an example of a prediction block, the prediction block includes a matrix of pixel values ​​(e.g., luma values), such as 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, and the like.

[0083] 7 shows an example diagram of a video encoder (703). The video encoder (703) is configured to receive sample values ​​of a processing block (e.g., a predictive block) in a current video picture in a sequence of video pictures and encode the processing block into a coded picture that is part of a coded video sequence. In one example, the video encoder (703) is used in place of the video encoder (403) in the example of FIG. 4.

[0084] In an HEVC example, the video encoder (703) receives a matrix of sample values ​​for a processing block, e.g., 8x8 samples of a predictive block. The video encoder (703) determines, e.g., using rate-distortion optimization, whether the processing block is best coded using intra mode, inter mode, or bi-predictive mode. If the processing block is coded in intra mode, the video encoder (703) can code the processing block into a coded picture using intra prediction techniques, and if the processing block is coded in inter mode or bi-predictive mode, the video encoder (703) can code the processing block into a coded picture using inter prediction techniques or bi-predictive techniques, respectively. In certain video coding techniques, the merge mode may be an inter-picture prediction sub-mode in which motion vectors are derived from one or more motion vector predictors without the benefit of coded motion vector components outside the predictors. In certain other video coding techniques, there may be motion vector components applicable to the current block. In one example, the video encoder (703) includes other components, such as a mode decision module (not shown) for determining a mode of a processing block.

[0085] In the example of FIG. 7, the video encoder (703) includes an inter-encoder (730), an intra-encoder (722), a residual calculator (723), a switch (726), a residual encoder (724), a general controller (721), and an entropy encoder (725), coupled together as shown in FIG.

[0086] The inter-encoder (730) is configured to receive samples of a current block (e.g., a processing block), compare the block to one or more reference blocks in a reference picture (e.g., blocks in previous and subsequent pictures), generate inter-prediction information (e.g., a description of redundant information according to an inter-coding technique, motion vectors, merge mode information), and calculate an inter-prediction result (e.g., a prediction block) based on the inter-prediction information using any suitable technique. In some examples, the reference picture is a decoded reference picture based on the coded video information.

[0087] The intra encoder (722) receives samples of a current block (e.g., a processing block), in some cases compares the block to previously coded blocks in the same picture, generates transformed quantized coefficients, and in some cases also generates intra prediction information (e.g., intra prediction direction information according to one or more intra coding techniques). In one example, the intra encoder (722) also calculates intra prediction results (e.g., prediction blocks) based on the intra prediction information and reference blocks in the same picture.

[0088] The general controller (721) is configured to determine general control data and control other components of the video encoder (703) based on the general control data. In one example, the general controller (721) determines the mode of the block and provides a control signal to the switch (726) based on the mode. For example, when the mode is an intra mode, the general controller (721) controls the switch (726) to select an intra mode result for use by the residual calculator (723) and controls the entropy encoder (725) to select intra prediction information and include the intra prediction information in the bitstream, and when the mode is an inter mode, the general controller (721) controls the switch (726) to select an inter prediction result for use by the residual calculator (723) and controls the entropy encoder (725) to select inter prediction information and include the inter prediction information in the bitstream.

[0089] The residual calculator (723) is configured to calculate a difference (residual data) between a received block and a prediction result selected from the intra-encoder (722) or the inter-encoder (730). The residual encoder (724) operates on the residual data and is configured to encode the residual data to generate transform coefficients. In one example, the residual encoder (724) is configured to transform the residual data from a spatial domain to a frequency domain to generate transform coefficients. The transform coefficients are then subjected to a quantization process to obtain quantized transform coefficients. In various embodiments, the video encoder (703) also includes a residual decoder (728). The residual decoder (728) is configured to perform an inverse transform and generate decoded residual data. The decoded residual data can be suitably used by the intra-encoder (722) and the inter-encoder (730). For example, the inter-encoder (730) may generate decoded blocks based on the decoded residual data and the inter-prediction information, and the intra-encoder (722) may generate decoded blocks based on the decoded residual data and the intra-prediction information. The decoded blocks are suitably processed to generate decoded pictures, which may be buffered in a memory circuit (not shown) and may be used as a reference picture in some examples.

[0090] The entropy encoder (725) is configured to format a bitstream to include the encoded block. The entropy encoder (725) is configured to include various information in the bitstream according to a suitable standard, such as the HEVC standard. In one example, the entropy encoder (725) is configured to include general control data, selected prediction information (e.g., intra-prediction information or inter-prediction information), residual information, and other suitable information in the bitstream. Note that, in accordance with the subject matter of this disclosure, no residual information is present when coding a block in a merged sub-mode of either the inter mode or the bi-prediction mode.

[0091] 8 shows an example diagram of a video decoder (810). The video decoder (810) is configured to receive coded pictures that are part of a coded video sequence and decode the coded pictures to generate reconstructed pictures. In one example, the video decoder (810) is used in place of the video decoder (410) in the example of FIG. 4.

[0092] In the example of FIG. 8, the video decoder (810) includes an entropy decoder (871), an inter decoder (880), a residual decoder (873), a reconstruction module (874), and an intra decoder (872), coupled together as shown in FIG.

[0093] The entropy decoder (871) may be configured to reconstruct from the coded picture certain symbols that represent syntax elements that make up the coded picture. Such symbols may include, for example, prediction information (e.g., intra- or inter-prediction information) that may identify the mode in which the block is coded (e.g., intra- or inter-prediction mode, the latter two in merged or other submodes, etc.) as well as certain samples or metadata used for prediction by the intra- or inter-decoder (872) or inter-decoder (880), respectively. The symbols may also include residual information, for example in the form of quantized transform coefficients, and the like. In one example, when the prediction mode is an inter- or bi-prediction mode, the inter-prediction information is provided to the inter-decoder (880), and when the prediction type is an intra-prediction type, the intra-prediction information is provided to the intra-decoder (872). The residual information may be subjected to inverse quantization and provided to the residual decoder (873).

[0094] The inter decoder (880) is configured to receive the inter prediction information and to generate inter prediction results based on the inter prediction information.

[0095] The intra decoder (872) is configured to receive the intra prediction information and to generate a prediction result based on the intra prediction information.

[0096] The residual decoder (873) is configured to perform inverse quantization to extract dequantized transform coefficients, and then process the dequantized transform coefficients to transform the residual information from the frequency domain to the spatial domain. The residual decoder (873) may also require certain control information (including quantizer parameters (QP)), which may be provided by the entropy decoder (871) (data path not shown as this may be only low-volume control information).

[0097] The reconstruction module (874) is configured to combine the residual information output by the residual decoder (873) and the prediction result (output by the inter- or intra-prediction module, as the case may be) in the spatial domain to form a reconstructed block. The reconstructed block may be part of a reconstructed picture, which in turn may be part of a reconstructed video. However, other suitable processes may be performed to improve visual quality, such as deblocking processes and the like.

[0098] It should be noted that the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) may be implemented using any suitable technology. In one embodiment, the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) may be implemented using one or more integrated circuits. In another embodiment, the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) may be implemented using one or more processors executing software instructions.

[0099] In VVC, various inter prediction modes can be used. For an inter predicted CU, the motion parameters can include MV(s), one or more reference picture indexes, a reference picture list usage index, and additional information about the specific coding features used for sample generation by inter prediction. The motion parameters can be explicitly or implicitly signaled. When a CU is coded in skip mode, the CU can be associated with a PU and may not have significant residual coefficients, coded motion vector deltas or MV differences (e.g., MVDs), or reference picture indexes. A merge mode can be specified, where the motion parameters for the current CU are obtained from neighboring CU(s), including spatial and / or temporal candidates, and optionally additional information, such as introduced in VVC. The merge mode can be applied to inter predicted CUs as well as skip mode. In one example, an alternative to merge mode is explicit transmission of motion parameters, where (one or more) MVs, corresponding reference picture indexes for each reference picture list, reference picture list usage flags, and other information are explicitly signaled for each CU.

[0100] In one embodiment, e.g., in VVC, the VVC Test Model (VTM) reference software supports the following motion vector prediction modes: extended merge prediction, merge motion vector difference (MMVD) mode, adaptive motion vector prediction (AMVP) mode with symmetric MVD signaling, affine motion compensation prediction, subblock-based temporal motion vector prediction (SbTMVP), adaptive motion vector resolution (AMVR), motion field storage (1 / 16 luma sample MV storage and 8x8 motion field compression), bi-prediction with CU-level weights (BCW), bi-directional optical flow (BDOF), prediction refinement using optical flow (PROF), decoder side motion vector refinement (DMVR), combined inter and intra prediction (CIIP), geometric partitioning mode, and affine motion compensation prediction (AFP). The present invention includes one or more refined inter-prediction coding tools, including GPM (Grammar Partitioning Mode), and the like. Inter-prediction and related methods are described in more detail below.

[0101] Enhanced merge prediction may be used in some examples. In one example, such as in VTM4, a merge candidate list is constructed by including, in order, the following five types of candidates: spatial motion vector predictor (MVP)(s) from spatially neighboring CU(s), temporal MVP(s) from co-located CU(s), history-based MVP(s) (HMVP)(s) from a first-in-first-out (FIFO) table, pairwise average MVP(s), and zero MV(s).

[0102] The size of the merge candidate list can be signaled in the slice header. In one example, the maximum allowed size of the merge candidate list is 6 in VTM4. For each CU coded in merge mode, the index of the best merge candidate (e.g., merge index) can be coded using truncated unary binarization (TU). The first bin of the merge index can be coded with context (e.g., context-adaptive binary arithmetic coding (CABAC)) and bypass coding can be used for the other bins.

[0103] Some examples of the generation process for each category of merge candidates are provided below. In one embodiment, spatial candidate(s) are derived as follows: The derivation of spatial merge candidates in VVC may be the same as that in HEVC. In one example, up to four merge candidates are selected from the candidates in the positions shown in FIG. 9. FIG. 9 illustrates the positions of spatial merge candidates according to one embodiment of the present disclosure. With reference to FIG. 9, the derivation order is B1, A1, B0, A0, and B2. Position B2 is considered only if any CU in positions A0, B0, B1, and A1 is unavailable (e.g., because it belongs to another slice or another tile) or is intra-coded. After the candidate in position A1 is added, the addition of the remaining candidates is subjected to a redundancy check that ensures that candidates with the same motion information are removed from the candidate list so that coding efficiency is improved.

[0104] In order to reduce computational complexity, the above redundancy check does not consider all possible candidate pairs. Instead, only pairs connected by arrows in FIG. 10 are considered, and a candidate is added to the candidate list only if the corresponding candidates used for the redundancy check do not have the same motion information. FIG. 10 illustrates candidate pairs considered for redundancy check of spatial merge candidates according to one embodiment of the present disclosure. With reference to FIG. 10, the respective arrow-connected pairs are A1 and B1, A1 and A0, A1 and B2, B1 and B0, and B1 and B2. Thus, candidates at positions B1, A0, and / or B2 can be compared with candidates at positions A1, and candidates at positions B0 and / or B2 can be compared with candidates at positions B1.

[0105] In one embodiment, the (one or more) temporal candidates are derived as follows: In one example, only one temporal merge candidate is added to the candidate list. FIG. 11 illustrates an example motion vector scaling for temporal merge candidates. To derive a temporal merge candidate for a current CU (1111) in a current picture (1101), a scaled MV (1121) (e.g., shown by a dotted line in FIG. 11) may be derived based on a co-located CU (1112) belonging to a co-located reference picture (1104). In one example, a co-located reference picture (also referred to as a co-located picture) is a particular reference picture used, for example, for temporal motion vector prediction. A co-located reference picture used for temporal motion vector prediction may be indicated by a reference index in a syntax, for example, a high-level syntax (e.g., a picture header, a slice header).

[0106] The reference picture list used to derive the co-located CU (1112) may be explicitly signaled in the slice header. A scaled MV (1121) for the temporal merge candidate may be obtained as shown by the dotted line in FIG. 11. The scaled MV (1121) may be scaled from the MV of the co-located CU (1112) using picture order count (POC) distances tb and td. The POC distance tb may be defined to be the POC difference between the current reference picture (1102) of the current picture (1101) and the current picture (1101). The POC distance td may be defined to be the POC difference between the co-located reference picture (1104) of the co-located reference picture (1103) and the co-located reference picture (1103). The reference picture index of the temporal merge candidate may be set to zero.

[0107] FIG. 12 shows exemplary candidate positions (e.g., C0 and C1) for a temporal merge candidate of a current CU. A temporal merge candidate position can be selected from candidate positions C0 and C1. Candidate position C0 is located at the bottom right corner of the current CU's colocated CU (1210). Candidate position C1 is located at the center of the current CU's colocated CU (1210). If the CU at candidate position C0 is not available, is intra-coded, or is outside the current row of the CTU, candidate position C1 is used to derive the temporal merge candidate. Otherwise, for example, if the CU at candidate position C0 is available, is intra-coded, and is within the current row of the CTU, candidate position C0 is used to derive the temporal merge candidate.

[0108] The merge motion vector difference (MMVD) mode can be used in merge mode or skip mode with a motion vector representation method. For example, the merge candidate (one or more) as used in VVC can be reused in the MMVD mode. A candidate can be selected from among the merge candidates as a starting point (e.g., MV predictor (MVP)) and can be further extended by the MMVD mode. The MMVD mode can provide a new motion vector representation with simplified signaling. The motion vector representation method includes a starting point and an MV difference (MVD). In one example, the MVD is indicated by the magnitude of the MVD (or the motion magnitude) and the direction of the MVD (e.g., the motion direction).

[0109] The MMVD mode can use a merge candidate list, such as that used in VVC. In one embodiment, only candidates (one or more) that are of the default merge type (e.g., MRG_TYPE_DEFAULT_N) are considered in the MMVD mode. The starting point can be indicated or defined by a base candidate index (IDX). The base candidate index can indicate a candidate (e.g., the best candidate) among multiple candidates (e.g., multiple base candidates) in the merge candidate list. Table 1 shows an example relationship between the base candidate index and the corresponding starting point. A base candidate index of 0, 1, 2, or 3 indicates that the corresponding starting point is the first MVP, the second MVP, the third MVP, or the fourth MVP. In one example, if the number of base candidates is equal to 1, the base candidate IDX is not signaled. [Table 1]

[0110] The distance index may indicate motion magnitude information of the MVD, such as the magnitude of the MVD. For example, the distance index indicates a distance (e.g., a predetermined distance) from a starting point (e.g., the MVP indicated by the base candidate index). In one example, the distance is one of a plurality of predetermined distances, such as shown in Table 2. Table 2 shows an example relationship between the distance index and the corresponding distance (in units of samples or pixels). One pel in Table 2 means one sample or one pixel. For example, a distance index of 1 indicates that the distance is 1 / 2 pel, i.e., 1 / 2 sample. [Table 2]

[0111] The direction index can represent the direction of the MVD relative to the starting point. The direction index can represent one of multiple directions, such as the four directions shown in Table 3. For example, a direction index of 00 indicates that the direction of the MVD is along the positive x-axis. [Table 3]

[0112] The MMVD flag may be signaled after sending the skip and merge flags. If the skip and merge flags are true, the MMVD flag may be parsed. In one example, if the MMVD flag is equal to 1, the MMVD syntax (e.g., including distance index and / or direction index) may be parsed. If the MMVD flag is not equal to 1, the AFFINE flag may be parsed. If the AFFINE flag is equal to 1, the AFFINE mode is used to code the current block. If the AFFINE flag is not equal to 1, the skip / merge index may be parsed for the skip / merge mode, e.g., as used in VTM.

[0113] 13-14 show an example of a search process in MMVD mode, which can be performed to determine indexes including a base candidate index, a direction index, and / or a distance index for a current block (1300) in a current picture (or referred to as a current frame) (1301).

[0114] A first motion vector (MV) (1311) and a second MV (1321) belonging to a first merging candidate are shown. The first merging candidate may be a merging candidate in a merging candidate list constructed for the current block (1300). The first and second MVs (1311) and (1321) may be associated with two reference pictures (1302) and (1303) in the reference picture lists L0 and L1, respectively. Thus, two starting points (1411) and (1421) in Figures 13-14 may be determined in the reference pictures (1302) and (1303), respectively.

[0115] In one example, based on the starting points (1411) and (1421), a number of predefined points extending vertically (represented by +Y or -Y) or horizontally (represented by +X and -X) from the starting points (1411) and (1421) within the reference pictures (1302) and (1303) may be evaluated. In one example, pairs of points that are mirror images of each other with respect to the respective starting points (1411) or (1421), such as the pair of points (1414) and (1424) or the pair of points (1415) and (1425), may be used to determine a pair of MVs (1314) and (1324) or a pair of MVs (1315) and (1325) that may form an MV predictor (MVP) candidate for the current block (1300). MVP candidates determined based on predefined points surrounding the starting points (1411) and / or (1421) may be evaluated. Referring to Figure 13, the MVD (1312) between the first MV (1311) and MV (1314) has a magnitude of 1S. The MVD (1322) between the second MV (1321) and MV (1324) has a magnitude of 1S. Similarly, the MVD between the first MV (1311) and MV (1315) has a magnitude of 2S. The MVD between the second MV (1321) and MV (1325) has a magnitude of 2S.

[0116] In addition to the first merging candidate, other available or valid merging candidates in the merging candidate list of the current block (1300) may be evaluated as well. In one example, for uni-predictive merging candidates, only one prediction direction associated with one of the two reference picture lists is evaluated.

[0117] In one example, the best MVP candidate can be determined based on the evaluation. Thus, from the merge list, an optimal merge candidate corresponding to the optimal MVP candidate can be selected, and the motion direction and motion distance can also be determined. For example, a base candidate index can be determined based on the selected merge candidate and Table 1. Based on the selected MVP, such as the one corresponding to the predetermined point (1415) (or (1425)), the direction and distance (e.g., 2S) of the point (1415) relative to the start point (1411) can be determined. The direction index and distance index can be determined accordingly according to Tables 2 and 3.

[0118] As described above, the MVD in the MMVD mode can be indicated using two indexes, e.g., a distance index and a direction index. Alternatively, the MVD in the MMVD mode can be indicated using a single index, e.g., using a table that pairs a single index with the MVD.

[0119] In some prediction modes, such as MMVD and affine MMVD modes, template matching (TM) based candidate sorting may be used. In one embodiment, the MMVD offset is extended for MMVD and affine MMVD modes. FIG. 15 shows additional refinement positions along multiple diagonal angles, such as k×π / 8 diagonal angles, where k is an integer between 0 and 15. The additional refinement positions along multiple diagonal angles can increase the number of directions, such as from 4 directions (e.g., +X, −X, +Y, and −Y) to 16 directions (e.g., k=0, 1, 2, ..., 15). In one example, each of these 16 directions is represented by an angle between the +X direction and a direction indicated by the center point (1500) and one of points 1 to 16. For example, point 1 indicates the +X direction with an angle of 0 (ie, k=0), point 2 indicates a direction along an angle of 1×π / 8 (ie, k=1), and so on.

[0120] TM may be performed in MMVD mode. In one example, for each MMVD refinement position, a TM cost may be determined based on the current template of the current block and one or more reference templates. The TM cost may be determined using any method, such as sum of absolute differences (SAD) (e.g., SAD cost), sum of absolute transformed differences (SATD), sum of squared errors (SSE), mean removed SAD / SATD / SSE, variance, partial SAD, partial SSE, partial SATD, or the like.

[0121] The current template of the current block may include any suitable samples, such as samples in a row above the current block and / or samples in a column to the left of the current block. The MMVD refinement positions, such as all possible MMVD refinement positions (e.g., 16×6 representing 16 directions and 6 magnitudes) for each base candidate (e.g., MVP), may be sorted based on the TM cost (e.g., SAD cost) between the current template and the corresponding reference template for the refinement positions. In one example, the top MMVD refinement positions with the smallest TM cost (e.g., smallest SAD cost) are retained as the MMVD refinement positions available for MMVD index coding. For example, a subset (e.g., 8) of the MMVD refinement positions with the smallest TM cost are used for MMVD index coding. For example, the MMVD index indicates which one of the subset of the MMVD refinement positions with the smallest TM cost is selected to code the current block. In one example, an MMVD index of 0 indicates that the MVD corresponding to the smallest TM cost (e.g., the MMVD refinement position) is used to code the current block. The MMVD index may be binarized, for example, by a Rice code with a parameter equal to 2.

[0122] In one embodiment, in addition to the MMVD offset extension described above, e.g., in Fig. 15, the affine MMVD reordering is extended to add further refinement positions along a diagonal angle of kxπ / 4. After reordering, the top ½ refinement positions with the smallest TM cost (e.g., SAD cost) are kept for coding the current block.

[0123] To improve coding efficiency and reduce transmission overhead of MV(s), subblock-level MV refinement may be applied to extend CU-level temporal motion vector prediction (TMVP). In one example, subblock-based TMVP (SbTMVP) mode allows inheriting subblock-level motion information from a co-located reference picture. As mentioned above, the co-located reference picture may be indicated by a reference index in a syntax, such as a high-level syntax (e.g., picture header, slice header). Each sub-block of a current CU (e.g., a current CU with a large size) in a current picture may have its own motion information without explicitly transmitting a block partition structure or its own motion information. In SbTMVP mode, the motion information of each sub-block may be obtained, for example, in three steps as follows: In the first step, a displacement vector (DV) of the current CU may be derived. The DV may indicate a block in a co-located reference picture, for example, the DV points from a current block in a current picture to a block in a co-located reference picture. Therefore, the block indicated by DV is considered to be collocated with the current block and is called the colocated block of the current block. In a second step, the availability of SbTMVP candidates can be checked, and the central motion (e.g., the central motion of the current CU) can be derived. In a third step, sub-block motion information can be derived from the corresponding sub-block in the colocated block using DV. These three steps may be combined into one or two steps, and / or the order of these three steps may be adjusted.

[0124] Unlike TMVP candidate derivation, which derives temporal MVs from co-located blocks in a reference frame or picture, in SbTMVP mode, for each sub-block in the current CU in the current picture, a DV (e.g., a DV derived from the MV of the current CU's left neighboring CU) can be applied to locate the corresponding sub-block in the co-located reference picture. If the corresponding sub-block is not inter-coded, the motion information of the current sub-block can be set to the central motion of the co-located block.

[0125] The SbTMVP mode may be supported by various video coding standards, including, for example, VVC. Similar to the TMVP mode, for example, in HEVC, the SbTMVP mode may use a motion field (also referred to as a motion information field or MV field) in a co-located reference picture to improve the MV prediction and merge mode for a CU in a current picture. In one example, the same co-located reference picture used by the TMVP mode is used in the SbTMVP mode. In one example, the SbTMVP mode differs from the TMVP mode in the following respects: (i) the TMVP mode predicts motion information at a CU level, whereas the SbTMVP mode predicts motion information at a sub-CU level; and (ii) the TMVP mode fetches temporal MV from a co-located block in the co-located reference picture (e.g., the co-located block is a bottom-right or center block relative to the current CU), whereas the SbTMVP mode may apply a motion shift before fetching temporal motion information from the co-located reference picture. In one example, the motion shift used in SbTMVP mode is obtained from the MV of one of the spatially neighboring blocks of the current CU.

[0126] 16-17 show an exemplary SbTMVP process used in the SbTMVP mode. The SbTMVP process can predict the MV of a sub-CU (e.g., sub-block) in a current CU (e.g., current block) (1601) of a current picture (1711), for example, in two steps. In a first step, a spatial neighbor (e.g., A1) of the current block (1601) in FIG. 16-17 is examined. If the spatial neighbor (e.g., A1) has an MV (1721) that uses a co-located reference picture (1712) as the reference picture of the spatial neighbor (e.g., A1), the MV (1721) can be selected to be the motion shift (or DV) to be applied to the current block (1601). If no such MV (e.g., an MV that uses the co-located reference picture (1712) as a reference picture) is identified, the motion shift or DV may be set to a zero MV (e.g., (0,0)). In some examples, if no such MV is identified for spatial neighbor A1, the MV(s) of further spatial neighbors, such as A0, B0, and B1, are examined.

[0127] In a second step, the motion shift or DV (1721) identified in the first step may be applied to the current block (1601) (e.g., adding the DV (1721) to the coordinates of the current block) to obtain sub-CU level motion information (e.g., including MV and reference index) from the co-located reference picture (1712). In the example shown in FIG. 17, the motion shift or DV (1721) is set to be the MV of the spatial neighbor A1 (e.g., block A1) of the current block (1601). For each sub-CU or sub-block (1731) in the current block (1601), the motion information of the sub-CU or sub-block (1731) may be derived using the motion information of the corresponding co-located block (1701) in the co-located reference picture (1712) (e.g., the motion information of the minimum motion grid covering the center sample of the co-located block (1701)). After the motion information of the co-located sub-CU (1732) in the co-located block (1701) is identified, the motion information of the co-located sub-CU (1732) can be converted into motion information (e.g., an MV and one or more reference indexes) of the current sub-CU (1731) using a scaling method, such as in a manner similar to the TMVP process used in HEVC, in which temporal motion scaling is applied to align a temporal MV reference picture to a reference picture of the current CU.

[0128] The motion field of the current block (1601), derived based on the DV (1721), may include motion information for each sub-block (1731) in the current block (1601), such as the MV(s) and one or more associated reference indexes. The motion field of the current block (1601), which may also be referred to as an SbTMVP candidate, corresponds to the DV (1721).

[0129] 17 shows an example of a motion field or SbTMVP candidate of a current block 1601. For example, the motion information of a bi-predicted sub-block (1731(1)) includes a first MV and a first index indicating a first reference picture in reference picture list 0 (L0), a second MV and a second index indicating a second reference picture in reference picture list 1 (L1). In one example, the motion information of a uni-predicted sub-block (1731(2)) includes an MV and an index indicating a reference picture in L0 or L1.

[0130] In one example, the DV (1721) is applied to the center position of the current block (1601) to locate the displaced center position in the co-located reference picture (1712). If the block containing the displaced center position is not inter-coded, the SbTMVP candidate is considered not available. Otherwise, if the block containing the displaced center position (e.g., the co-located block (1701)) is inter-coded, the motion information of the center position of the current block (1601), referred to as the central motion of the current block (1601), may be derived from the motion information of the block containing the displaced center position in the co-located reference picture (1712). In one example, a scaling process may be used to derive the central motion of the current block (1601) from the motion information of the block containing the displaced center position in the co-located reference picture (1712). When SbTMVP candidates are available, for each sub-block (1732) of the current block (1601), the DV (1721) can be applied to find the corresponding sub-block (1731) in the co-located reference picture (1712). The motion information of the corresponding sub-block (1732) can be used to derive motion information for the sub-block (1731) in the current block (1601), such as in the same manner as used to derive the central motion of the current block (1601). In one example, if the corresponding sub-block (1732) is not inter-coded, the motion information of the current sub-block (1731) is set to be the central motion of the current block (1601).

[0131] In some examples, such as in VVC, a combined sub-block based merge list containing SbTMVP candidates and (one or more) affine merge candidates is used in signaling the sub-block based merge mode. The SbTMVP mode may be enabled or disabled by a sequence parameter set (SPS) flag. When the SbTMVP mode is enabled, the SbTMVP candidate (or SbTMVP predictor) is added as the first entry of the sub-block based merge list containing the sub-block based merge candidates, which may be followed by (one or more) affine merge candidates. The size of the sub-block based merge list may be signaled in the SPS. In one example, the maximum allowed size of the sub-block based merge list is 5 in VVC. In one example, multiple SbTMVP candidates are included in the sub-block based merge list.

[0132] In some examples, such as in VVC, the sub-CU size used in SbTMVP mode is fixed at 8×8, as used in affine merge mode. In one example, SbTMVP mode is only applicable to CUs with both width and height equal to or greater than 8. The sub-block size (e.g., 8×8) may be configurable to other sizes, such as 4×4 in ECM software model usage for VVC and beyond exploration. In one example, multiple collocated reference pictures, such as two collocated frames, are utilized to provide temporal motion information for SbTMVP and / or TMVP in AMVP mode.

[0133] In some examples of SbTMVP modes, such as in VVC and ECM, the DV of the current CU (e.g., DV (1721) in FIG. 17) is derived only from the MVs of the neighboring CUs of the current CU. However, the SbTMVP candidates derived using the DVs may not be an exact match.

[0134] A DV offset (DVO) can be used in SbTMVP mode. In one example, to obtain a more accurate match, the DV (e.g., the original DV) can be modified by the DV offset to determine an updated DV'. In one example, the updated DV' is a vector sum of the DV (e.g., the original DV) and the DVO. The original DV can be determined using any method, such as those described in Figures 16-17. For example, the original DV is determined based on the MVs of neighboring blocks of the current block. For the current block, the DVO can be signaled and analyzed to indicate additional motion offsets of the original DV. The DVO can be indicated, for example, by signaling an index that points to the DVO from multiple DVO candidates. In one example, the DVO is signaled. In one example, the MMVD mode is used to indicate the DVO, and the DVO is an MVD indicated by a direction index and / or a distance index, such as those described in Tables 2-3. By using DVO, the position of the co-located CU (or co-located block) in the co-located reference picture can be adjusted, and thus the MV field of the co-located CU (or co-located block) can change based on DVO. When DVO is not 0, the updated DV' can be used as a displacement vector indicating the position of the co-located CU (or co-located block) to perform the SbTMVP process. Referring to FIG. 17, instead of using the original DV (e.g., DV(1721)), which is the MV of the spatial neighbor A1, the updated DV' can be used to determine the co-located block for the current block. The updated DV' (e.g., the vector sum of the original DV and DVO) can be used to derive the SbTMVP candidates of the current block.

[0135] In one embodiment, DVO is signaled directly using any signaling method used to signal MVD, such as in AMVP mode, AMVR mode, and / or the like. In AMVR mode, the MVD of a block can be signaled at different resolutions, such as ¼, ½, 1, or 4 luma sample resolution. DVO can be signaled at different resolutions using AMVR mode.

[0136] A given DVO list may include multiple DVO candidates (e.g., DVOs that are possible for use by the current block). One or more indexes may be signaled to indicate which of the DVO candidates is to be selected as the DVO.

[0137] In one example, the DVO is signaled using the MMVD mode, for example, as described in Tables 2-3, two indexes are signaled to indicate the DVO candidate, including a first index (e.g., a distance index or a step index) indicating the size of the DVO candidate and a second index (e.g., a direction index) indicating the direction of the DVO candidate.

[0138] Referring again to FIG. 14, the distance index (or step index) and the direction index may be predefined as described above with reference to the MMVD mode. The distance index indicates motion magnitude information, such as the magnitude of the DVO. For example, the distance index indicates a certain predefined distance from a starting point (e.g., the initial DV). In one example, available predefined distances are shown in Table 2. The direction index represents the direction of the DVO relative to the starting point (e.g., the initial DV). The direction index may indicate one of a number of directions, such as the four directions shown in Table 3.

[0139] In one example, a co-located CTU in a co-located reference picture is co-located with a current CTU that includes the current block. The current CTU is located within the current picture. In one embodiment, the location of the co-located block corresponding to the updated DV' is constrained to be within a first area in the co-located reference picture. In one example, the first area in the co-located reference picture includes the co-located CTU. In one example, the first area in the co-located reference picture includes the co-located CTU plus one column of 4×4 blocks at the right boundary of the co-located CTU. The updated DV' may be constrained such that the co-located block corresponding to the updated DV' is within the first area in the co-located reference picture. In one example, the DVO (e.g., the horizontal component DVO x and / or the vertical component of DVO y ) is constrained to ensure that the updated DV′ satisfies the collocated block position constraints mentioned above.

[0140] In one example, the maximum vertical component of the updated DV′ is H. y ) is set to be less than or equal to H minus the vertical component of DV, DVy, e.g., DVO y ≦H-DV y The maximum horizontal component of the updated DV' is W, and DVO is the horizontal component of DV from W to DV x For example, DVO x ≦W-DV x It is.

[0141] As described in Figures 16-17, a co-located block (e.g., (1701)) in a co-located reference picture (e.g., (1712)) may be determined based on the DV (e.g., (1721)) of a current block (e.g., (1601)) in a current picture (e.g., (1711)). Thus, the motion information (e.g., TMVP) of each sub-block in the current block may be based on the motion information of the corresponding sub-block in the co-located block. According to one embodiment of the present disclosure, updated motion information (e.g., updated TMVP) of each sub-block in the current block may be determined based on the motion information (e.g., TMVP) of the sub-block in the current block and the motion vector offset (MVO) of the current block.

[0142] In one embodiment, the MVO is added to each derived sub-block level TMVP of each sub-block in the current block to generate an updated sub-block level TMVP. The MVO can be signaled and analyzed to indicate the additional motion offset of each sub-block-based TMVP of each sub-block in the current block that is determined using the SbTMVP mode.

[0143] In one embodiment, the MVO is signaled directly using any signaling method used to signal the MVD, such as in AMVP mode, AMVR mode, and / or the like. In AMVR mode, the MVD of a block can be signaled at different resolutions, such as ¼, ½, 1, or 4 luma sample resolutions. The MVO can be signaled at different resolutions using AMVR mode.

[0144] A given MVO list may include multiple MVO candidates (e.g., MVOs that are possible for use by the current block). One or more indexes may be signaled to indicate which MVO candidates among the DVO candidates in a given MVO list may be selected as an MVO.

[0145] In one example, the MVO is signaled using the MMVD mode, for example, as described in Tables 2-3, two indexes are signaled to indicate the MVO candidate, including a first index (e.g., a distance index or a step index) indicating the size of the MVO candidate and a second index (e.g., a direction index) indicating the direction of the MVO candidate.

[0146] Referring again to FIG. 14, the distance index (or step index) and direction index of the MVO may be predefined as described above with reference to the MMVD mode. The distance index indicates motion magnitude information, such as the magnitude of the MVO, and may indicate, for example, a certain predefined distance from a starting point (e.g., the DV used to determine the position of the co-located block in the co-located reference picture). In one example, available predefined distances are shown in Table 2. The direction index represents the direction of the MVO relative to the starting point (e.g., the DV used to determine the position of the co-located block in the co-located reference picture). The direction index may indicate one of a number of directions, such as the four directions shown in Table 3.

[0147] In one example, a subblock in a current block is bi-predicted. Referring to FIG. 17, a subblock (1731(1)) is bi-predicted and has a first MV associated with a first reference picture in reference list L0 and a second MV associated with a second reference picture in reference list L1. An MVO may be applied to the first MV associated with reference list L0. The updated first MV may be a vector sum of the first MV and the MVO. The following embodiments may be applied to the second MV associated with reference list L1.

[0148] In one example, the MVO is not applied to the second MV associated with reference list L1. For example, no MVO is applied to the second MV associated with reference list L1. Thus, the updated motion information of subblock (1731(1)) includes the updated first MV and the second MV.

[0149] In one example, a mirror MVO of the MVO is applied to the second MV associated with the reference list L1. The mirror MVO and the MVO can have the same magnitude and opposite orientation. The horizontal and vertical components of the MVO (e.g., signaled) are multiplied by −1 to obtain the horizontal and vertical components of the mirror MVO, respectively. The updated second MV can be a vector sum of the second MV and the mirror MVO, or a vector difference between the second MV and the MVO. Thus, the updated motion information of the sub-block (1731(1)) includes an updated first MV (e.g., the first MV+MVO) and an updated second MV (e.g., the second MV-MVO).

[0150] In one example, a scaled MVO (MVO') may be applied to a second MV associated with reference list L1. The value of each component (e.g., horizontal and vertical components) of the MVO may be scaled based on the first and second POC differences as shown in Equation 1. Equation 1 may be applied to vectors MVO' and MVO. Equation 1 may be applied to each component of vectors MVO' and MVO. The first POC difference is the POC of the current picture (POC curr ) and the POC of the first reference picture in the reference picture list L0 (POC L0 ) The second POC difference is the difference between the POC of the current picture (POC curr ) and the POC of the second reference picture in the reference picture list L1 (POC L1 ) is the difference between MVO' = MVO × (POC L1 -POC curr ) / (POC L0 -POC curr ) formula 1

[0151] In one example, the scaled MVO (MVO') is added to a second MV associated with reference list L1. In one example, a mirror image MVO' of the scaled MVO (MVO') is added to a second MV associated with reference list L1.

[0152] As described above, in one example of the SbTMVP mode (e.g., the variation on the SbTMVP mode described in Figures 16-17), applying DVO to DV can adjust the position of the co-located block in the co-located reference picture, which can then affect the motion information of the sub-block in the current block. In another example of the SbTMVP mode (e.g., the variation on the SbTMVP mode described in Figures 16-17), applying MVO can directly adjust the motion information of the sub-block in the current block. The DVO and MVO can be signaled using the same method. For example, a predefined MVO list is the same as a predefined DVO list, and the DVO and MVO can use the same predefined MVO list.

[0153] In some examples, applying DVO to DV may allow the position of the co-located block in the co-located reference picture to be adjusted. After obtaining the motion information of the sub-block in the current block based on the motion information of the corresponding sub-block in the co-located reference picture determined based on the updated DV' (e.g., DV+DVO), MVO may be applied to further adjust the motion information of the sub-block in the current block. DVO may be the same as or different from MVO.

[0154] FIG. 18 shows a flow chart outlining an encoding process (1800) according to one embodiment of the disclosure. The process (1800) may be used in a video encoder. The process (1800) may be performed by an apparatus for video coding, which may include processing circuitry. In various embodiments, the process (1800) is performed by a processing circuit, such as the processing circuitry of the terminal devices (310), (320), (330), and (340), the processing circuitry performing the functions of a video encoder (e.g., (403), (603), (703)), and the like. In some embodiments, the process (1800) is implemented in software instructions, such that the processing circuitry performs the process (1800) when the processing circuitry executes the software instructions. The process begins at (S1801) and proceeds to (S1810).

[0155] At (S1810), an updated DV of the current block in the current picture may be determined based on the DV of the current block and the DV offset of the current block (also referred to as MV offset (MVO)). The DV may be determined as described above, such as in Figures 16-17. The updated DV of the current block refers to a co-located block in the co-located picture. The co-located block is co-located with the current block.

[0156] The current block includes multiple sub-blocks that are coded using a sub-block-based temporal motion vector prediction (SbTMVP) mode.

[0157] In one example, the DV offset (or MVO) is determined from the DV offset candidates using any suitable method.

[0158] In one example, the updated DV is determined to be the vector sum of the DV and the DV offset.

[0159] In one example, the updated DV is constrained such that the co-located block is within a constrained area in the co-located reference picture, the constrained area including the co-located area corresponding to the current CTU in the current picture, and the current CTU including the current block.

[0160] At (S1820), motion information of a sub-block of the plurality of sub-blocks may be determined based on motion information of a corresponding sub-block in a co-located block.

[0161] At (S1830), DV offset information (also referred to as MVO information) indicating a DV offset may be coded. The sub-block among the plurality of sub-blocks may be coded based on motion information of the sub-block among the plurality of sub-blocks.

[0162] In one embodiment, the DV offset information indicates a magnitude of the DV offset and at least one index indicating a direction of the DV offset, in one example, the at least one index includes a distance index indicating the magnitude of the DV offset, which is one of a set of predetermined distances, and a direction index indicating the direction of the DV offset, which is one of a set of predetermined directions.

[0163] In one example, the set of predetermined distances and the set of predetermined directions are used in a merged motion vector difference (MMVD) mode.

[0164] At (S1840), the encoded DV offset information may be included in the bitstream and signaled to the decoder.

[0165] In one example, the DV offset information includes a DV offset that is encoded and signaled in the bitstream.

[0166] Then, the process (1800) proceeds to (S1899) and ends.

[0167] Process (1800) may be suitably adapted to various scenarios, and the steps of process (1800) may be adjusted accordingly. One or more of the steps of process (1800) may be adapted, omitted, repeated, and / or combined. Any suitable order may be used to perform process (1800). Further step(s) may be added.

[0168] In one embodiment, an updated displacement vector (DV) of a current block in a current picture is determined based on the DV and MVO (also referred to as DV offset) of the current block. The MVO indicates a motion offset of the DV used to adjust the position of the co-located block in the co-located reference picture. The updated DV indicates the adjusted position of the co-located block in the co-located reference picture. The current block is coded in SbTMVP mode.

[0169] SbTMVP information (e.g., motion information) of each sub-block of the plurality of sub-blocks may be derived based on at least the motion information of a corresponding sub-block in a co-located block indicated by the updated DV. The plurality of sub-blocks may be coded in SbTMVP mode based on the SbTMVP information of the plurality of sub-blocks.

[0170] FIG. 19A shows a flow chart outlining a decoding process (1900A) according to one embodiment of the present disclosure. The process (1900A) can be used in a video decoder. The process (1900A) can be performed by an apparatus for video coding, which can include receiving circuitry and processing circuitry. In various embodiments, the process (1900A) is performed by a processing circuit, such as the processing circuits of the terminal devices (310), (320), (330), and (340), the processing circuits performing the functions of the video encoder (403), the processing circuits performing the functions of the video decoder (410), the processing circuits performing the functions of the video decoder (510), the processing circuits performing the functions of the video encoder (603), and the like. In some embodiments, the process (1900A) is implemented in software instructions, and thus the processing circuit performs the process (1900A) when the processing circuit executes the software instructions. The process starts at (S1901) and proceeds to (S1910).

[0171] At (S1910), displacement vector (DV) offset (also referred to as MV offset (MVO)) information of a current block in a current picture may be received from an encoded video bitstream. The current block includes multiple sub-blocks that are reconstructed using a sub-block-based temporal motion vector prediction (SbTMVP) mode. The DV offset (or MVO) information may indicate a motion offset relative to the DV that is used to adjust a position of the co-located block in a co-located reference picture. In one example, the position of the co-located block in the co-located reference picture is adjusted by the DV offset.

[0172] In one example, the DV offset information includes a DV offset (or MVO) signaled in the encoded video bitstream.

[0173] In one embodiment, the DV offset information indicates a magnitude of the DV offset and at least one index indicating a direction of the DV offset, in one example, the at least one index includes a distance index indicating the magnitude of the DV offset, which is one of a set of predetermined distances, and a direction index indicating the direction of the DV offset, which is one of a set of predetermined directions.

[0174] In one example, the set of predetermined distances and the set of predetermined directions are used in a merged motion vector difference (MMVD) mode.

[0175] At (S1920), an updated DV of the current block may be determined based on the DV of the current block and the DV offset of the current block. The DV offset is indicated by the DV offset information. The updated DV of the current block points to a block in the co-located reference picture. The block is considered to be co-located with the current block and is referred to as the co-located block of the current block.

[0176] In one example, the updated DV is determined to be the vector sum of the DV and the DV offset.

[0177] In one example, the updated DV is constrained such that the co-located block is within a constrained area in the co-located reference picture, the constrained area including the co-located area corresponding to the current CTU in the current picture, and the current CTU including the current block.

[0178] At (S1930), motion information of a sub-block of the plurality of sub-blocks may be determined based on motion information of a corresponding sub-block in a co-located block.

[0179] At (S1940), the sub-block of the plurality of sub-blocks may be reconstructed based on motion information of the sub-block of the plurality of sub-blocks.

[0180] The process (1900A) proceeds to (S1999) and ends.

[0181] The process (1900A) may be suitably adapted to various scenarios, and the steps of the process (1900A) may be adjusted accordingly. One or more of the steps of the process (1900A) may be adapted, omitted, repeated, and / or combined. Any suitable order may be used to perform the process (1900A). Further step(s) may be added.

[0182] FIG. 19B shows a flow chart outlining a decoding process (1900B) according to one embodiment of the present disclosure. Process (1900B) is a variation of decoding process (1900A). Process (1900B) can be used in a video decoder. Process (1900B) can be performed by an apparatus for video coding, which can include receiving circuitry and processing circuitry. In various embodiments, process (1900A) is performed by a processing circuit, such as the processing circuitry of terminal devices (310), (320), (330), and (340), a processing circuitry performing the functions of a video encoder (403), a processing circuitry performing the functions of a video decoder (410), a processing circuitry performing the functions of a video decoder (510), a processing circuitry performing the functions of a video encoder (603), and the like. In some embodiments, process (1900B) is implemented in software instructions, such that the processing circuitry performs process (1900B) when the processing circuitry executes the software instructions. The process starts at (S1902) and proceeds to (S1912).

[0183] At (S1912), a coded video bitstream having a current picture is received, the current picture including a current block, the current block including a plurality of sub-blocks.

[0184] At (S1922), it is determined based on a syntax element in the coded video bitstream that the current block including the plurality of sub-blocks is coded in SbTMVP mode.

[0185] At (S1932), motion vector offset (MVO) information of the current block is obtained. The MVO indicates the motion offset of a displacement vector (DV) used to adjust the position of the co-located block in the co-located reference picture.

[0186] At (S1942), an updated DV of the current block is determined based on the DV and MVO of the current block. The updated DV indicates an adjusted position of the co-located block in the co-located reference picture.

[0187] At (S1952), SbTMVP information (eg, motion information) of each sub-block of the plurality of sub-blocks is derived based at least on the motion information of a corresponding sub-block in the co-located block indicated by the updated DV.

[0188] In (S1962), the plurality of sub-blocks are reconstructed in the SbTMVP mode based on the SbTMVP information of the plurality of sub-blocks.

[0189] The process (1900B) proceeds to (S1992) and ends.

[0190] Process (1900B) may be suitably adapted to various scenarios, and steps of process (1900B) may be adjusted accordingly. One or more of the steps of process (1900B) may be adapted, omitted, repeated, and / or combined. Any suitable order may be used to perform process (1900B). Further step(s) may be added.

[0191] The embodiments of the present disclosure may be used separately or in any order, and each of the methods (or embodiments), the encoder, and the decoder may be implemented by processing circuitry (e.g., one or more processors, or one or more integrated circuits). In one example, the one or more processors execute a program stored on a non-transitory computer-readable medium.

[0192] FIG. 20 shows a flow chart outlining an encoding process (2000) according to one embodiment of the present disclosure. The process (2000) may be used in a video encoder. The process (2000) may be performed by an apparatus for video coding, which may include a processing circuit. In various embodiments, the process (2000) is performed by a processing circuit, such as the processing circuit of the terminal devices (310), (320), (330), and (340), the processing circuit that performs the functions of a video encoder (e.g., (403), (603), (703)), and the like. In some embodiments, the process (2000) is implemented in software instructions, and thus the processing circuit performs the process (2000) when the processing circuit executes the software instructions. The process starts at (S2001) and proceeds to (S2010).

[0193] At (S2010), a displacement vector (DV) of a current block in a current picture may be determined, for example as described in FIG. 16-FIG. 17. The current block includes multiple sub-blocks that are coded using a sub-block-based temporal motion vector prediction (SbTMVP) mode. The DV indicates a co-located block in a co-located reference picture that is co-located with the current block.

[0194] At (S2020), motion information of a sub-block among the plurality of sub-blocks may be determined based on motion information of a corresponding sub-block in a co-located block, for example, as described in FIG. 16-FIG.

[0195] At (S2030), updated motion information of the subblock of the plurality of subblocks may be determined based on the motion information of the subblock of the plurality of subblocks and a motion vector (MV) offset of a current block.

[0196] In one example, the motion information of the subblock of the plurality of subblocks includes a first motion vector (MV) associated with a first reference picture from a first reference picture list L0. An updated first MV may be determined to be a vector sum of the first MV and an MV offset. The updated motion information includes the updated first MV.

[0197] In one example, the motion information of the subblock of the plurality of subblocks includes a second MV associated with a second reference picture from a second reference picture list L1. The updated second MV may be determined to be one of (i) a vector difference between the second MV and an MV offset, or (ii) a vector sum between the second MV and a scaled MV offset. The scaled MV offset may be based on the MV offset, a picture order count (POC) of the current picture, a POC of the first reference picture, and a POC of the second reference picture. The updated motion information includes the updated second MV.

[0198] At (S2040), MV offset information indicating an MV offset may be encoded. The sub-block of the plurality of sub-blocks may be encoded based on the updated motion information. The MV offset information may be included in a bitstream.

[0199] In one example, the MV offset information includes an MV offset.

[0200] In one example, the MV offset information indicates at least one index indicating a magnitude of the MV offset and a direction of the MV offset, the at least one index including a distance index indicating the magnitude of the MV offset, the distance index being one of a set of predetermined distances, and a direction index indicating the direction of the MV offset, the direction index being one of a set of predetermined directions, the set of predetermined distances and the set of predetermined directions being used in a merged motion vector difference (MMVD) mode.

[0201] Then, the process (2000) proceeds to (S2099) and ends.

[0202] The process (2000) may be suitably adapted to various scenarios and the steps of the process (2000) may be adjusted accordingly. One or more of the steps of the process (2000) may be adapted, omitted, repeated, and / or combined. Any suitable order may be used to perform the process (2000). Further step(s) may be added.

[0203] FIG. 21 shows a flow chart outlining a decoding process (2100) according to one embodiment of the present disclosure. The process (2100) can be used in a video decoder. The process (2100) can be performed by an apparatus for video coding, which can include receiving circuitry and processing circuitry. In various embodiments, the process (2100) is performed by a processing circuit, such as the processing circuitry of the terminal devices (310), (320), (330), and (340), a processing circuitry performing the functions of a video encoder (403), a processing circuitry performing the functions of a video decoder (410), a processing circuitry performing the functions of a video decoder (510), a processing circuitry performing the functions of a video encoder (603), and the like. In some embodiments, the process (2100) is implemented in software instructions, and thus the processing circuitry performs the process (2100) when the processing circuitry executes the software instructions. The process starts at (S2101) and proceeds to (S2110).

[0204] At (S2110), motion vector (MV) offset information for a current block in a current picture may be received from a coded video bitstream. The current block includes a plurality of sub-blocks that are reconstructed using a sub-block-based temporal motion vector prediction (SbTMVP) mode.

[0205] At (S2120), a displacement vector (DV) of the current block may be determined, for example as described in FIG. 16-FIG. 17. The DV may indicate a block in a collocated reference picture that is collocated with the current block. The block may be referred to as the collocated block of the current block.

[0206] At (S2130), motion information of a sub-block of the plurality of sub-blocks may be determined based on motion information of a corresponding sub-block in a co-located block, for example as described in FIG. 16-FIG.

[0207] At (S2140), updated motion information of the sub-block among the plurality of sub-blocks may be determined based on the motion information of the sub-block among the plurality of sub-blocks and the MV offset of the current block indicated by the MV offset information.

[0208] In one example, the MV offset information includes an MV offset signaled in the encoded video bitstream.

[0209] In one example, the MV offset information indicates at least one index indicating a magnitude of the MV offset and a direction of the MV offset, the at least one index including a distance index indicating the magnitude of the MV offset, the distance index being one of a set of predetermined distances, and a direction index indicating the direction of the MV offset, the direction index being one of a set of predetermined directions, the set of predetermined distances and the set of predetermined directions being used in a merged motion vector difference (MMVD) mode.

[0210] In one example, the motion information of the subblock of the plurality of subblocks includes a first motion vector (MV) associated with a first reference picture from a first reference picture list L0. An updated first MV may be determined to be a vector sum of the first MV and an MV offset. The updated motion information includes the updated first MV.

[0211] In one example, the motion information of the subblock of the plurality of subblocks includes a second MV associated with a second reference picture from a second reference picture list L1. The updated second MV may be determined to be one of (i) a vector difference between the second MV and an MV offset, or (ii) a vector sum between the second MV and a scaled MV offset. The scaled MV offset may be based on the MV offset, a picture order count (POC) of the current picture, a POC of the first reference picture, and a POC of the second reference picture. The updated motion information includes the updated second MV.

[0212] At (S2150), the sub-blocks of the plurality of sub-blocks may be reconstructed based on updated motion information.

[0213] Then, the process (2100) proceeds to (S2199) and ends.

[0214] The process (2100) may be suitably adapted to various scenarios, and the steps of the process (2100) may be adjusted accordingly. One or more of the steps of the process (2100) may be adapted, omitted, repeated, and / or combined. Any suitable order may be used to perform the process (2100). Further step(s) may be added.

[0215] The techniques described above can be implemented as computer software with computer readable instructions physically stored on one or more computer readable media. For example, Figure 22 illustrates a computer system (2200) suitable for implementing certain embodiments of the disclosed subject matter.

[0216] The computer software may be coded using any suitable machine code or computer language that can be assembled, compiled, linked, or similarly processed to produce code having instructions that can be executed by one or more computer central processing units (CPUs), graphics processing units (GPUs), and the like, either directly or via interpretation, microcode execution, and the like.

[0217] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smart phones, gaming devices, Internet of Things devices, and the like.

[0218] 22 with respect to the computer system (2200) are exemplary in nature and are not intended to suggest any limitation as to the scope of use or functionality of the computer software for implementing embodiments of the present disclosure, nor should the arrangement of components be interpreted as having any dependency or requirement related to any one or combination of components illustrated in this exemplary embodiment of the computer system (2200).

[0219] The computer system (2200) may include certain human interface input devices. Such human interface input devices may be responsive to input by one or more human users, for example, via tactile input (e.g., keystrokes, swipes, moving a data glove, etc.), audio input (e.g., voice, clapping, etc.), visual input (e.g., gestures, etc.), olfactory input (not shown). Human interface devices may also be used to capture certain media that are not necessarily directly related to conscious human input, such as audio (e.g., speech, music, ambient sounds, etc.), images (e.g., scanned images, photographic images obtained from a still camera, etc.), video (e.g., two-dimensional video, three-dimensional video including stereoscopic video, etc.).

[0220] The input human interface devices may include one or more of a keyboard (2201), a mouse (2202), a trackpad (2203), a touch screen (2210), a data glove (not shown), a joystick (2205), a microphone (2206), a scanner (2207), and a camera (2208) (only one of each shown).

[0221] The computer system (2200) may also include certain human interface output devices. Such human interface output devices may stimulate one or more of the senses of a human user, for example, through haptic output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touch screen (2210), data gloves (not shown), or joystick (2205), although there may also be haptic feedback devices that do not function as input devices), audio output devices (e.g., speakers (2209), headphones (not shown), etc.), visual output devices (e.g., screens (2210) including CRT screens, LCD screens, plasma screens, OLED screens (each with or without touch screen input capability, each with or without haptic feedback capability, some of which may be capable of outputting two-dimensional visual output, or output in four or more dimensions through means such as stereoscopic output), virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown), etc.), and printers (not shown).

[0222] The computer system (2200) may also include human-accessible storage devices and their associated media, such as optical media including, for example, a CD / DVD ROM / RW (2220) with CD / DVD or similar media (2221), thumb drives (2222), removable hard drives or solid state drives (2223), legacy magnetic media such as tape and floppy disks (registered trademark, not shown), specialized ROM / ASIC / PLD based devices (not shown) such as security dongles, and the like.

[0223] Those skilled in the art will also appreciate that the term "computer-readable medium" as used in connection with the subject matter disclosed herein does not include transmission media, carrier waves, or other transitory signals.

[0224] The computer system (2200) may also include an interface (2254) to one or more communication networks (2255). The networks may be, for example, wireless, wired, or optical. The networks may further be local, wide area, metropolitan, vehicular, and industrial, real-time, delay tolerant, and the like. Examples of networks include local area networks, such as Ethernet, wireless LANs, cellular networks, including GSM, 3G, 4G, 5G, LTE, and the like, TV wired or wireless wide area digital networks, including cable TV, satellite TV, and terrestrial broadcast TV, vehicular and industrial, including CANBus, and the like. Certain networks typically require an external network interface adapter that is attached to a particular general-purpose data port or peripheral bus (2249) (e.g., a USB port of the computer system (2200)), while others are typically integrated into the core of the computer system (2200) by attachment to a system bus, as described below (e.g., an Ethernet interface to a PC computer system, or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system (2200) can communicate with other entities. Such communications may be one-way receive only (e.g., broadcast TV), one-way transmit only (e.g., CANbus to a particular CANbus device), or two-way, for example, to other computer systems using local or wide area digital networks. Specific protocols and protocol stacks may be used on each of the networks and network interfaces, as described above.

[0225] The aforementioned human interface devices, human accessible storage devices, and network interfaces may be attached to a core (2240) of the computer system (2200).

[0226] The cores (2240) may include one or more central processing units (CPUs) (2241), graphics processing units (GPUs) (2242), specialized programmable processing units in the form of field programmable gate arrays (FPGAs) (2243), hardware accelerators for specific tasks (2244), graphics adapters (2250), etc. These devices may be connected via a system bus (2248) along with read-only memory (ROM) (2245), random access memory (2246), internal mass storage (2247), such as an internal non-user-accessible hard drive, SSD, and the like (2247). In some computer systems, the system bus (2248) may be made accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, and the like. Peripheral devices may be attached either directly to the core's system bus (2248) or via a peripheral bus (2249). In one example, the screen 2210 can be connected to a graphics adapter 2250. Peripheral bus architectures include PCI, USB, and the like.

[0227] The CPU (2241), GPU (2242), FPGA (2243), and accelerator (2244) may execute certain instructions that may combine to constitute the aforementioned computer code. The computer code may be stored in ROM (2245) or RAM (2246). Transient data may also be stored in RAM (2246), while permanent data may be stored, for example, in internal mass storage (2247). Rapid storage and retrieval from any of the memory devices may be enabled by the use of cache memory that may be associated with one or more of the CPU (2241), GPU (2242), mass storage (2247), ROM (2245), RAM (2246), and the like.

[0228] The computer-readable media can have computer code thereon for performing various computer-implemented processes. The media and computer code may be those specially designed and constructed for the purposes of the present disclosure, or they may be of the kind well known and available to those skilled in the computer software arts.

[0229] As an example, but not by way of limitation, a computer system having the architecture (2200), and in particular the core (2240), can provide functionality as a result of a processor (or processors) (including CPUs, GPUs, FPGAs, accelerators, and the like) executing software embodied in one or more tangible computer-readable media. Such computer-readable media can be media associated with the specific storage of the core (2240) that is non-transitory in nature, such as the core's internal mass storage (2247) or ROM (2245), and with user-accessible mass storage as introduced above. Software implementing various embodiments of the present disclosure can be stored in such devices and executed by the core (2240). The computer-readable media can include one or more memory devices or chips according to specific needs. The software may cause the core (2240) and in particular the processors therein (including CPUs, GPUs, FPGAs, and the like) to perform certain processes or certain portions of certain processes described herein, including by defining data structures stored in RAM (2246) and modifying such data structures according to processes defined by the software. Additionally or alternatively, the computer system may provide functionality as a result of logic hardwired or otherwise embodied in circuitry (e.g., accelerators (2244)) that may operate in place of or in conjunction with software to perform certain processes or certain portions of certain processes described herein. Reference to software includes logic, and vice versa, where appropriate. Reference to a computer-readable medium may include circuitry (e.g., an integrated circuit (IC) or the like) that stores software for execution, circuitry that embodies logic for execution, or both, where appropriate. The present disclosure includes any suitable combination of hardware and software. Appendix A: Acronyms JEM: joint exploration model VVC: versatile video coding BMS:benchmark set MV: Motion Vector HEVC: High Efficiency Video Coding SEI: Supplementary Enhancement Information VUI: Video Usability Information GOPs: Groups of Pictures TUs: Transform Units, PU: Prediction Units CTUs: Coding Tree Units CTBs: Coding Tree Blocks PBs: Prediction Blocks HRD: Hypothetical Reference Decoder SNR: Signal Noise Ratio CPUs: Central Processing Units GPUs: Graphics Processing Units CRT:Cathode Ray Tube LCD: Liquid-Crystal Display OLED: Organic Light-Emitting Diode CD: Compact Disc DVD: Digital Video Disc ROM: Read-Only Memory RAM: Random Access Memory ASIC: Application-Specific Integrated Circuit PLD: Programmable Logic Device LAN: Local Area Network GSM: Global System for Mobile communications LTE: Long-Term Evolution CANBus: Controller Area Network Bus USB: Universal Serial Bus PCI: Peripheral Component Interconnect FPGA: Field Programmable Gate Areas SSD: solid-state drive IC: Integrated Circuit CU: Coding Unit

[0230] While this disclosure describes several exemplary embodiments, there are alterations, permutations, and various equivalent alternatives that fall within the scope of the disclosure. It will thus be understood that those skilled in the art will be able to devise numerous systems and methods that, although not explicitly shown or described herein, embody the principles of the disclosure and are therefore within its spirit and scope.

Claims

1. 1. A method of video decoding performed by a decoder, comprising: receiving a coded video bitstream having a current picture, the current picture including a current block, the current block including a plurality of sub-blocks; determining, based on a syntax element in the coded video bitstream, that the current block including the plurality of sub-blocks is coded in a sub-block-based temporal motion vector prediction (SbTMVP) mode; obtaining motion vector offset (MVO) information of the current block, the MVO indicating a motion offset of a displacement vector (DV) used to adjust a position of the co-located block in a co-located reference picture; determining an updated DV of the current block based on the DV and the MVO of the current block, the updated DV indicating an adjusted position of the co-located block within the co-located reference picture; deriving SbTMVP information for each sub-block of the plurality of sub-blocks based at least on motion information of a corresponding sub-block in the co-located block indicated by the updated DV; reconfiguring the plurality of sub-blocks in the SbTMVP mode based on the SbTMVP information of the sub-blocks of the plurality of sub-blocks; The method according to claim 1,

2. The step of determining the updated DV comprises: determining the updated DV to be a vector sum of the DV and the MVO; 2. The method of claim 1, comprising:

3. the MVO information includes the MVO signaled in the encoded video bitstream; The method of claim 1.

4. The MVO information indicates at least one index indicating the size of the MVO and the direction of the MVO. The method of claim 1.

5. the at least one index includes a distance index indicating the magnitude of the MVO, the magnitude being one of a set of predetermined distances, and a direction index indicating the direction of the MVO, the direction being one of a set of predetermined directions. The method according to claim 4.

6. the set of predetermined distances and the set of predetermined directions are used in a merged motion vector difference (MMVD) mode. The method according to claim 5.

7. The updated DV is constrained such that the co-located block is within a constrained area in the co-located reference picture, the constrained area including a co-located area corresponding to a current CTU in the current picture, the current CTU including the current block. The method of claim 1.

8. 1. A method of video decoding performed by a decoder, comprising: receiving motion vector (MV) offset information for a current block in a current picture from a coded video bitstream, the current block including a plurality of sub-blocks to be reconstructed using a sub-block-based temporal motion vector prediction (SbTMVP) mode; determining a displacement vector (DV) of the current block that points to a collocated block in a collocated reference picture that is collocated with the current block; determining motion information of a sub-block of the plurality of sub-blocks based on motion information of a corresponding sub-block in the co-located block; determining updated motion information of the sub-block of the plurality of sub-blocks based on the motion information of the sub-block of the plurality of sub-blocks and an MV offset of the current block indicated by the MV offset information; reconstructing the sub-blocks of the plurality of sub-blocks based on the updated motion information; The method according to claim 1,

9. the MV offset information includes the MV offset signaled in the encoded video bitstream; The method according to claim 8.

10. The MV offset information indicates at least one index indicating a magnitude of the MV offset and a direction of the MV offset. The method according to claim 8.

11. the at least one index includes a distance index indicating the magnitude of the MV offset being one of a set of predetermined distances, and a direction index indicating the direction of the MV offset being one of a set of predetermined directions. The method of claim 10.

12. the set of predetermined distances and the set of predetermined directions are used in a merged motion vector difference (MMVD) mode. The method of claim 11.

13. the motion information for the sub-block of the plurality of sub-blocks includes a first motion vector (MV) associated with a first reference picture from a first reference picture list L0; the determining the updated motion information step includes determining an updated first MV that is a vector sum of the first MV and the MV offset, the updated motion information including the updated first MV; The method according to claim 8.

14. the motion information of the sub-block of the plurality of sub-blocks includes a second MV associated with a second reference picture from a second reference picture list L1; The step of determining the updated motion information further comprises: (i) the vector difference between the second MV and the MV offset; or (ii) a vector sum of the second MV and a scaled MV offset, the scaled MV offset being based on the MV offset, a Picture Order Count (POC) of the current picture, a POC of the first reference picture, and a POC of the second reference picture; determining an updated second MV, the second MV being one of: The method of claim 13.

15. 1. A method of video encoding performed by an encoder, comprising: determining an updated displacement vector (DV) of a current block in a current picture based on a DV of the current block and a DV offset of the current block, the updated DV indicating a co-located block in a co-located reference picture that is co-located with the current block, the current block including a plurality of sub-blocks that are coded using a sub-block-based temporal motion vector prediction (SbTMVP) mode; determining motion information of a sub-block of the plurality of sub-blocks based on motion information of a corresponding sub-block in the co-located block; encoding DV offset information indicating the DV offset; encoding the sub-block of the plurality of sub-blocks based on motion information of the sub-block of the plurality of sub-blocks; The method according to claim 1,

16. 1. A method of video encoding performed by an encoder, comprising: determining a displacement vector (DV) for a current block in a current picture, the current block including a plurality of sub-blocks coded using a sub-block-based temporal motion vector prediction (SbTMVP) mode, the DV indicating a co-located block in a co-located reference picture that is co-located with the current block; determining motion information of a sub-block of the plurality of sub-blocks based on motion information of a corresponding sub-block in the co-located block; determining updated motion information of the sub-block of the plurality of sub-blocks based on the motion information of the sub-block of the plurality of sub-blocks and a motion vector (MV) offset of the current block; encoding MV offset information indicating the MV offset; encoding the sub-block of the plurality of sub-blocks based on the updated motion information; The method according to claim 1,

17. one or more processors; one or more memories storing a computer program; having The computer program causes the one or more processors to carry out a method according to any one of claims 1 to 14. Device.

18. one or more processors; one or more memories storing a computer program; having The computer program causes the one or more processors to perform the method according to claim 15 or 16. Device.

19. A computer program causing a computer to carry out the method according to any one of claims 1 to 14.

20. A computer program causing a computer to carry out the method according to claim 15 or 16.

Images