Method, apparatus, and computer program for video coding

Abstract

Aspects of the present disclosure provide a method and an apparatus including a processing circuit, and determine, based on a syntax element in an encoded video bitstream, that a current block including a plurality of sub-blocks is coded in a sub-block-based temporal motion vector prediction (SbTMVP) mode. Motion vector offset (MVO) information indicating a MVO is received. The MVO indicates a motion offset of a displacement vector (DV) used to adjust a position of the co-located block in a co-located reference picture. An updated DV of the current block is determined based on the DV and the MVO. SbTMVP information of each sub-block of the plurality of sub-blocks is derived based on motion information of a corresponding sub-block in the co-located block indicated by the updated DV. The plurality of sub-blocks in the SbTMVP mode are reconstructed based on the SbTMVP information of the plurality of sub-blocks.

Description

[Technical Field]

[0001] This application claims the benefit of priority to U.S. Provisional Application No. 63 / 332,131, entitled "SUBBLOCK BASED MOTION VECTOR PREDICTOR WITH MOTION VECTOR OFFSET," filed April 18, 2022, which claims the benefit of priority to U.S. Patent Application No. 17 / 984,123, entitled "A SUB-BLOCK BASED TEMPORAL MOTION VECTOR PREDICTOR WITH AN MOTION VECTOR OFFSET," filed November 9, 2022. The disclosures of these prior applications are incorporated herein by reference in their entireties.

[0002] This disclosure describes embodiments generally related to video coding. [Background technology]

[0003] The background discussion provided herein is intended to provide a general overview of the context for the disclosure. To the extent described in this background section, the work of the named inventors, and aspects of the disclosure that may not otherwise qualify as prior art at the time of filing, are not admitted, explicitly or implicitly, as prior art to the present disclosure.

[0004] An uncompressed digital image and / or video includes a series of pictures, each having spatial dimensions of, for example, 1920 x 1080 luminance samples and associated chrominance samples. The series of pictures can have a fixed or variable picture rate (informally known as frame rate), for example, 60 pictures per second, i.e., a 60 Hz picture rate. Uncompressed images and / or video have specific bitrate requirements. For example, 1080p60 4:2:0 video (1920 x 1080 luminance sample resolution at a 60 Hz frame rate) with 8 bits per sample requires a bandwidth approaching 1.5 Gbit / s. One hour of such video requires over 600 GBytes of storage space.

[0005] One goal of image and / or video encoding and decoding may be to reduce redundancy in the input image and / or video signal through compression. Compression can help reduce the aforementioned bandwidth and / or storage space requirements, potentially by more than two orders of magnitude. While the description herein uses video encoding / decoding as an illustrative example, the same techniques can be applied to image encoding / decoding in a similar manner without departing from the spirit of this disclosure. Both lossless and lossy compression, as well as combinations thereof, can be used. Lossless compression refers to techniques that allow an exact replica of the original signal to be reconstructed from a compressed original signal. When lossy compression is used, the reconstructed signal may not be identical to the original signal, but the distortion between the original signal and the reconstructed signal is small enough to make the reconstructed signal useful for the intended application. For video, lossy compression is widely used. The amount of acceptable distortion depends on the application; for example, users of certain consumer streaming applications may tolerate higher distortion than users of television distribution applications. The achievable compression ratio reflects this, and higher acceptable / tolerable distortion can result in a higher compression ratio.

[0006] Video encoders and decoders can utilize techniques from several broad categories, including, for example, motion compensation, transform processing, quantization, and entropy coding.

[0007] Video codec technology can include a technique known as intra-coding. In intra-coding, sample values ​​are represented without reference to samples or other data from a previously reconstructed reference picture. In some video codecs, a picture is spatially subdivided into blocks of samples. If all the samples of a block are coded in intra mode, the picture can be an intra-picture. Intra-pictures and their derivatives, such as independent decoder refresh pictures, can be used to reset the decoder state and can therefore be used as the first picture in a coded video bitstream and video session or as a still image. Samples of an intra-block can be subjected to a transform, and the transform coefficients can be quantized before entropy coding. Intra-prediction can be a technique that minimizes sample values ​​in the pre-transform domain. In some cases, the smaller the DC value and the smaller the AC coefficients after the transform, the fewer bits are required for a given quantization step size to represent the block after entropy coding.

[0008] Traditional intra-coding, such as that used in MPEG-2 generation coding techniques, does not use intra-prediction. However, some newer video compression techniques include techniques that attempt to make predictions based on, for example, surrounding sample data and / or metadata obtained during the encoding and / or decoding of a block of data. Such techniques are hereinafter referred to as "intra-prediction" techniques. Note that, at least in some cases, intra-prediction uses only reference data from the current picture being reconstructed, and not from reference pictures.

[0009] Many different forms of intra-prediction may exist. If more than one such technique may be used in a given video coding technique, the particular technique used may be coded as a particular intra-prediction mode that uses that particular technique. In certain cases, an intra-prediction mode may have sub-modes and / or parameters that may be coded separately or included in a mode codeword that defines the prediction mode used. The codeword used for a given mode, sub-mode, and / or parameter combination may affect the coding efficiency gain through intra-prediction, as may the entropy coding technique used to convert the codeword into a bitstream.

[0010] A specific mode of intra prediction was introduced in H.264, improved in H.265, and further improved in newer coding techniques, such as the Joint Exploration Model (JEM), Versatile Video Coding (VVC), and Benchmark Set (BMS). A predictor block may be formed using neighboring sample values ​​of already available samples. The sample values ​​of the neighboring samples are copied to the predictor block according to a direction. The reference direction to use may be coded into the bitstream or may be predicted itself.

[0011] Referring to FIG. 1A, a subset of nine known predictor directions from the 33 possible predictor directions defined in H.265 (corresponding to the 33 angle modes among the 35 intra modes) is depicted in the lower right. The confluence of arrows (101) represents the sample being predicted. The arrow indicates the direction from which the sample is predicted. For example, arrow (102) indicates that sample (101) is predicted from one or more samples in the upper right corner at an angle of 45 degrees from horizontal. Similarly, arrow (103) indicates that sample (101) is predicted from one or more samples in the lower left corner at an angle of 22.5 degrees from horizontal.

[0012] Still referring to FIG. 1A , a square block (104) of 4×4 samples (indicated by a thick dashed line) is depicted at the upper left. The square block (104) contains 16 samples, each labeled with an “S” and its position in the Y dimension (e.g., row index) and its position in the X dimension (e.g., column index). For example, sample S21 is the second sample (from the top) in the Y dimension and the first sample (from the left) in the X dimension. Similarly, sample S44 is the fourth sample in both the Y and X dimensions within the block (104). Because the block is 4×4 samples in size, S44 is located at the lower right. Also shown are reference samples following a similar numbering scheme. The reference samples are labeled with an R and their Y position (e.g., row index) and X position (column index) relative to the block (104). In both H.264 and H.265, the prediction samples are adjacent to the block being reconstructed, and therefore there is no need to use negative values.

[0013] Intra-picture prediction works by copying reference sample values ​​from neighboring samples indicated by the signaled prediction direction. For example, assume that for this block, the coded video bitstream includes signaling indicating that the prediction direction is aligned with arrow (102), i.e., that the sample is predicted from the sample 45 degrees above and to the right of horizontal. In that case, samples S41, S32, S23, and S14 are predicted from the same reference sample R05. And sample S44 is predicted from reference sample R08.

[0014] In certain cases, especially when the direction is not evenly divisible by 45 degrees, the values ​​of several reference samples may be combined, for example by interpolation, to calculate the reference sample.

[0015] As video coding technology evolves, the number of possible directions increases. H.264 (2003) could represent nine different directions. This increased to 33 in H.265 (2013). Currently, JEM / VVC / BMS can support up to 65 directions. Experiments are performed to identify the most likely directions, and specific techniques in entropy coding are used to represent these with fewer bits, at the expense of penalizing less likely directions. Furthermore, these directions themselves can be predicted from neighboring directions used in adjacent, already decoded blocks.

[0016] FIG. 1B shows a schematic diagram (110) depicting 65 intra-prediction directions according to JEM to illustrate the increasing number of prediction directions.

[0017] The mapping of intra-prediction direction bits to represent directions within an encoded video bitstream may vary from one video coding technique to another. Such mappings may range, for example, from simple direct mappings to complex adaptive schemes including codewords, most probable modes, and similar techniques. However, in most cases, there may be certain directions in the video content that are statistically less likely to occur than other certain directions. Because the goal of video compression is to reduce redundancy, a well-performing video coding technique will represent these less likely directions with more bits than more likely directions.

[0018] Inter-picture prediction with motion compensation can be used to encode and decode images and / or videos. Motion compensation can be a lossy compression technique and can refer to a technique used to predict a newly reconstructed picture or picture portion after blocks of sample data from a previously reconstructed picture or portion thereof (reference picture) are spatially shifted in a direction indicated by a motion vector (hereinafter, MV). In some cases, the reference picture can be the same as the picture currently being reconstructed. The MV can have two dimensions, X and Y, or three dimensions, with the third being an indication of the reference picture to be used (the latter can indirectly be referred to as the temporal dimension).

[0019] In some video compression techniques, the MV applicable to a particular region of sample data can be predicted from another MV, such as from an MV that precedes the MV in decoding order and that relates to another region of sample data that is spatially adjacent to the region being reconstructed. Doing so can significantly reduce the amount of data required to code the MV, thereby removing redundancy and increasing compression. MV prediction can work effectively because, for example, when coding an input video signal derived from a camera (known as natural video), there is a statistical likelihood that regions larger than the region to which a single MV is applicable move in a similar direction and therefore can be predicted using similar motion vectors, possibly derived from MVs of neighboring regions. This results in the MV found for a given region being similar or the same as the MV predicted from surrounding MVs and thus, after entropy coding, can be represented with fewer bits than would be used to code the MV directly. In some cases, MV prediction can be an example of lossless compression of a signal (i.e., an MV) derived from the original signal (i.e., a sample stream). In other cases, the MV prediction itself may be lossy, for example due to rounding errors in computing the predictor from some surrounding MVs.

[0020] Various MV prediction mechanisms are described in H.265 / HEVC (ITU-T Recommendation H.265, “High Efficiency Video Coding”, December 2016). Among the many MV prediction mechanisms provided by H.265, a technique called “spatial merging” will be described below with reference to FIG. 2.

[0021] Referring to Figure 2, a current block (201) has samples that the encoder found during the motion search process to be predictable from a spatially shifted previous block of the same size. Instead of coding the MV directly, the MV can be derived from metadata associated with one or more reference pictures, such as from the immediately preceding reference picture (in decoding order), using the MV associated with any one of five surrounding samples, denoted A0, A1, and B0, B1, B2 (202 to 206, respectively). In H.265, MV prediction can use predictors from the same reference picture as neighboring blocks. Summary of the Invention

[0022] Aspects of the present disclosure provide methods and apparatuses for video encoding / decoding. In some examples, the apparatus for video decoding includes a processing circuit. In one embodiment, the processing circuit receives displacement vector (DV) offset information of a current block in a current picture from an encoded video bitstream. The current block includes multiple sub-blocks to be reconstructed using a sub-block-based temporal motion vector prediction (SbTMVP) mode. An updated DV of the current block can be determined based on the DV of the current block and the DV offset of the current block. The DV offset is indicated by the DV offset information. The updated DV of the current block points to a co-located block in a co-located reference picture. The co-located block is co-located with the current block. The processing circuit determines motion information of a sub-block of the multiple sub-blocks based on motion information of a corresponding sub-block in the co-located block, and reconstructs the sub-block of the multiple sub-blocks based on the motion information of the sub-block of the multiple sub-blocks.

[0023] In one embodiment, the processing circuitry determines the updated DV to be the vector sum of the DV and the MV offset.

[0024] In one embodiment, the DV offset information includes a DV offset signaled in the coded video bitstream.

[0025] In one embodiment, the DV offset information indicates at least one index indicating the magnitude of the DV offset and the direction of the DV offset.

[0026] In one example, the at least one index includes a distance index indicating a magnitude of the DV offset, which is one of a set of predetermined distances, and a direction index indicating a direction of the DV offset, which is one of a set of predetermined directions.

[0027] In one example, the set of predetermined distances and the set of predetermined directions are used in a merged motion vector difference (MMVD) mode.

[0028] In one example, the updated DV is constrained so that the co-located block is within a constrained area in the co-located reference picture, where the constrained area includes the co-located area corresponding to the current CTU in the current picture, and the current CTU includes the current block.

[0029] In one embodiment, a coded video bitstream having a current picture is received. The current picture includes a current block. The current block includes multiple sub-blocks. A processing circuit determines, based on a syntax element in the coded video bitstream, that the current block including the multiple sub-blocks is coded in SbTMVP mode. The processing circuit obtains motion vector offset (MVO) information for the current block, the MVO indicating a motion vector offset. The MVO indicates a motion offset of a displacement vector (DV) used to adjust a position of the co-located block in a co-located reference picture. The processing circuit determines an updated DV for the current block based on the DV and MVO of the current block. The updated DV indicates an adjusted position of the co-located block in the co-located reference picture. The processing circuit derives SbTMVP information for each sub-block of the multiple sub-blocks based on at least motion information of a corresponding sub-block in the co-located block indicated by the updated DV, and reconstructs the multiple sub-blocks in SbTMVP mode based on the SbTMVP information for the multiple sub-blocks.

[0030] In one embodiment, a processing circuit receives motion vector (MV) offset information for a current block in a current picture from a coded video bitstream. The current block includes a plurality of sub-blocks to be reconstructed using a sub-block-based temporal motion vector prediction (SbTMVP) mode. The processing circuit determines a displacement vector (DV) for the current block that indicates a co-located block in a co-located reference picture that is co-located with the current block. The processing circuit determines motion information for a sub-block of the plurality of sub-blocks based on motion information for a corresponding sub-block in the co-located block, and determines updated motion information for the sub-block of the plurality of sub-blocks based on the motion information for the sub-block of the plurality of sub-blocks and the MV offset of the current block indicated by the MV offset information. The processing circuit reconstructs the sub-block of the plurality of sub-blocks based on the updated motion information.

[0031] In one example, the MV offset information includes MV offsets signaled in the coded video bitstream.

[0032] In one example, the MV offset information indicates at least one index indicating the magnitude of the MV offset and the direction of the MV offset.

[0033] In one example, the at least one index includes a distance index indicating a magnitude of the MV offset, which is one of a set of predetermined distances, and a direction index indicating a direction of the MV offset, which is one of a set of predetermined directions.

[0034] A set of predetermined distances and a set of predetermined directions are used in merged motion vector difference (MMVD) mode.

[0035] In one example, the motion information of the sub-block of the plurality of sub-blocks includes a first motion vector (MV) associated with a first reference picture from a first reference picture list L0, and the processing circuit determines an updated first MV that is a vector sum of the first MV and an MV offset, and the updated motion information includes the updated first MV.

[0036] In one example, the motion information of the sub-block of the plurality of sub-blocks includes a second MV associated with a second reference picture from a second reference picture list L1. The processing circuit determines an updated second MV, which is one of (i) a vector difference between the second MV and an MV offset, or (ii) a vector sum of the second MV and a scaled MV offset. The scaled MV offset is based on the MV offset, a picture order count (POC) of the current picture, the POC of the first reference picture, and the POC of the second reference picture.

[0037] Aspects of the present disclosure also provide a non-transitory computer-readable medium storing instructions that, when executed by a computer for video decoding, cause the computer to perform a method for video decoding. [Brief explanation of the drawings]

[0038] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings. [Figure 1A] 10 illustrates schematically an exemplary subset of intra-prediction modes. [Figure 1B] 1 illustrates an exemplary intra-prediction direction. [Figure 2] An example of a current block (201) and surrounding samples is shown. [Figure 3] 1 illustrates an exemplary block diagram of a communication system (300). [Figure 4] 4 illustrates an exemplary block diagram of a communication system (400). [Figure 5] 1 shows an exemplary block diagram of a decoder. [Figure 6] 1 illustrates an exemplary block diagram of an encoder. [Figure 7] 1 shows a block diagram of an exemplary encoder. [Figure 8] 1 shows a block diagram of an exemplary decoder. [Figure 9] 1 illustrates the location of spatial merge candidates according to one embodiment of the present invention. [Figure 10] 1 illustrates candidate pairs considered for redundancy checking of spatial merge candidates according to one embodiment of the present invention. [Figure 11] 10 illustrates an example motion vector scaling for temporal merge candidates. [Figure 12] 10 illustrates exemplary candidate positions for temporal merge candidates for the current coding unit. [Figure 13]13-14 show an example of the search process in merged motion vector difference (MMVD) mode. [Figure 14] 13-14 show an example of the search process in merged motion vector difference (MMVD) mode. [Figure 15] 10 shows additional refinement positions along multiple oblique angles in MMVD mode. [Figure 16] 16-17 illustrate an exemplary sub-block-based temporal motion vector prediction (SbTMVP) process used in SbTMVP mode. [Figure 17] 16-17 illustrate an exemplary sub-block-based temporal motion vector prediction (SbTMVP) process used in SbTMVP mode. [Figure 18] 1 shows a flowchart outlining an encoding process according to some embodiments of the present disclosure. [Figure 19A] 1 shows a flowchart outlining a decoding process according to some embodiments of the present disclosure. [Figure 19B] 1 shows a flowchart outlining a decoding process according to some embodiments of the present disclosure. [Figure 20] 1 shows a flowchart outlining an encoding process according to some embodiments of the present disclosure. [Figure 21] 1 shows a flowchart outlining a decoding process according to some embodiments of the present disclosure. [Figure 22] FIG. 1 is a schematic diagram of a computer system according to one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0039] Figure 3 shows an exemplary block diagram of a communication system (300). The communication system (300) includes multiple terminal devices that can communicate with each other, for example, via a network (350). For example, the communication system (300) includes a first pair of terminal devices (310) and (320) interconnected via the network (350). In the example of Figure 3, the first pair of terminal devices (310) and (320) perform unidirectional data transmission. For example, the terminal device (310) may code video data (e.g., a stream of video pictures captured by the terminal device (310)) for transmission to another terminal device (320) via the network (350). The encoded video data may be transmitted in the form of one or more coded video bitstreams. The terminal device (320) may receive the coded video data from the network (350), decode the coded video data to reconstruct the video pictures, and display the video pictures according to the reconstructed video data. One-way data transmission may be common in media serving applications and the like.

[0040] In another example, the communication system 300 includes a second pair of terminal devices 330 and 340 that engage in bidirectional transmission of coded video data, such as during a video conference. In one example, in the bidirectional transmission of data, each of the terminal devices 330 and 340 may code video data (e.g., a stream of video pictures captured by that terminal device) for transmission to the other of the terminal devices 330 and 340 over the network 350. Each of the terminal devices 330 and 340 may also receive coded video data transmitted by the other of the terminal devices 330 and 340, decode the coded video data to recover the video pictures, and display the video pictures on an accessible display device in accordance with the recovered video data.

[0041] In the example of FIG. 3 , terminal devices 310, 320, 330, and 340 are shown as a server, a personal computer, and a smartphone, respectively, although the principles of the present disclosure may not be so limited. Embodiments of the present disclosure find application in laptop computers, tablet computers, media players, and / or dedicated videoconferencing equipment. Network 350 represents any number of networks, including, for example, wired and / or wireless communication networks, that convey encoded video data between terminal devices 310, 320, 330, and 340. Communication network 350 may exchange data over circuit-switched and / or packet-switched channels. Exemplary networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For purposes of this description, the architecture and topology of network 350 may not be important to the operation of the present disclosure, unless otherwise described below.

[0042] 4 shows the arrangement of a video encoder and a video decoder in a streaming environment as an example of an application of the disclosed subject matter, which can be equally applied to other video-enabled applications, including, for example, video conferencing, digital TV, streaming services, and storage of compressed video on digital media including CDs, DVDs, memory sticks, and the like.

[0043] The streaming system may include a capture subsystem (413), which may include a video source (401), such as a digital camera, that produces a stream of uncompressed video pictures (402). In one example, the stream of video pictures (402) includes samples captured by the digital camera. The stream of video pictures (402), depicted as a thick line to emphasize its high data volume compared to the encoded video data (404) (or encoded video bitstream), may be processed by an electronics device (420) that includes a video encoder (403) coupled to the video source (401). The video encoder (403) may include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter, as described in more detail below. The encoded video data (404) (or encoded video bitstream), depicted as a thin line to emphasize its low data volume compared to the stream of video pictures (402), may be stored on a streaming server (405) for later use. One or more streaming client subsystems, such as the client subsystems (406) and (408) of FIG. 4, can access the streaming server (405) to retrieve copies (407) and (409) of the encoded video data (404). The client subsystem (406) can include a video decoder (410), for example, within an electronics device (430). The video decoder (410) can decode the incoming copy of the encoded video data (407) and produce an outgoing stream of video pictures (411), which can be rendered on a display (412) (e.g., a display screen) or other rendering device (not shown). In some streaming systems, the encoded video data (404), (407), and (409) (e.g., a video bitstream) can be encoded according to a particular video coding / compression standard.Examples of such standards include ITU-T Recommendation H.265. In one example, a video coding standard under development is informally known as Versatile Video Coding (VVC). The subject matter disclosed herein may be used in the context of VVC.

[0044] It should be noted that electronics devices 420 and 430 may include other components (not shown). For example, electronics device 420 may include a video decoder (not shown), and electronics device 430 may also include a video encoder (not shown).

[0045] 5 shows an example block diagram of a video decoder (510). The video decoder (510) may be included in an electronics device (530). The electronics device (530) may include a receiver (531) (e.g., receiving circuitry). The video decoder (510) may be used in place of the video decoder (410) in the example of FIG. 4.

[0046] The receiver (531) can receive one or more coded video sequences to be decoded by the video decoder (510). In one embodiment, one coded video sequence is received at a time, and the decoding of each coded video sequence is independent of the decoding of other coded video sequences. The coded video sequences can be received from a channel (501), which can be a hardware / software link to a storage device that stores the coded video data. The receiver (531) can also receive the coded video data along with other data, such as coded audio data and / or auxiliary data streams, which can be forwarded to their respective using entities (not shown). The receiver (531) can separate the coded video sequences from other data. To combat network jitter, a buffer memory (515) can be coupled between the receiver (531) and the entropy decoder / parser 520 (hereinafter, "parser (520)"). In certain applications, the buffer memory (515) is part of the video decoder (510). In others, it may be external to the video decoder (510) (not shown). In still others, there may be a buffer memory (not shown) external to the video decoder (510), e.g., to combat network jitter, and there may be another buffer memory (515) internal to the video decoder (510), e.g., to handle playback timing. When the receiver (531) is receiving data from a store-and-forward device with sufficient bandwidth and controllability or from an isosynchronous network, the buffer memory (515) may not be needed or may be small. For use over best-effort packet networks, such as the Internet, the buffer memory (515) may be needed and may be relatively large and advantageously sized adaptively, and may be implemented, at least in part, in an operating system or similar element (not shown) external to the video decoder (510).

[0047] The video decoder (510) may include a parser (520) for reconstructing symbols (521) from the coded video sequence. These symbol categories include information used to manage the operation of the video decoder (510) and possibly information for controlling a rendering device, such as a render device (512) (e.g., a display screen) that is not an integral part of the electronics device (530) but can be coupled to the electronics device (530), as shown in FIG. 5. The control information for the rendering device(s) may be in the form of a Supplementary Enhancement Information (SEI) message or a Video Usability Information (VUI) parameter set fragment (not shown). The parser (520) may parse / entropy decode the received coded video sequence. The coding of the coded video sequence may be according to a video coding technique or standard and may follow various principles, including variable-length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser (520) may extract from the coded video sequence a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder based on at least one parameter corresponding to the group. The subgroup may include a group of pictures (GOP), a picture, a tile, a slice, a macroblock, a coding unit (CU), a block, a transform unit (TU), a prediction unit (PU), etc. The parser (520) may also extract information from the coded video sequence information, such as transform coefficients, quantization parameter values, motion vectors, etc.

[0048] The parser (520) may perform an entropy decoding / parsing process on the video sequence received from the buffer memory (515) to produce symbols (521).

[0049] The reconstruction of the symbols (521) may involve several different units, depending on the type of coded video picture or portion thereof and other factors (e.g., inter-picture and intra-picture, inter-block and intra-block, etc.). Which units are involved and how they are involved can be controlled by subgroup control information parsed from the coded video sequence by the parser (520). The flow of such subgroup control information between the parser (520) and the following units is not shown for clarity.

[0050] Beyond the functional blocks already described, the video decoder (510) may be conceptually subdivided into a number of functional units, as described below. In practical implementations operating within commercial constraints, many of these units may interact closely with each other and may be at least partially integrated with each other. However, for purposes of describing the subject matter of this disclosure, the following conceptual division into functional units is appropriate:

[0051] The first unit is a scalar / inverse transform unit (551), which receives quantized transform coefficients as symbol(s) (521) from the parser (520), along with control information including which transform to use, block size, quantization coefficients, quantization scaling matrices, etc. The scalar / inverse transform unit (551) can output blocks of sample values ​​that can be input to an aggregator (555).

[0052] In some cases, the output samples of the scaler / inverse transform unit (551) may relate to intra-coded blocks. Intra-coded blocks are blocks that do not use prediction information from a previously reconstructed picture but can use prediction information from a previously reconstructed portion of the current picture. Such prediction information can be provided by the intra-picture prediction unit (552). In some cases, the intra-picture prediction unit (552) generates a block of the same size and shape as the block being reconstructed using surrounding, already reconstructed information fetched from the current picture buffer (558). The current picture buffer (558), for example, buffers partially reconstructed and / or fully reconstructed current pictures. The aggregator (555), in some cases, adds, on a sample-by-sample basis, the prediction information generated by the intra-prediction unit (552) to the output sample information provided by the scaler / inverse transform unit (551).

[0053] In other cases, the output samples of the scalar / inverse transform unit (551) may relate to a block that may be inter-coded and motion-compensated. In such cases, the motion-compensated prediction unit (553) may access the reference picture memory (557) to fetch samples used for prediction. After motion-compensating the fetched samples according to the symbols (521) related to the block, these samples may be added by the aggregator (555) to the output of the scalar / inverse transform unit (551) (in this case, referred to as residual samples or residual signals) to generate output sample information. The addresses in the reference picture memory (557) from which the motion-compensated prediction unit (553) fetches prediction samples may be controlled by a motion vector and are available to the motion-compensated prediction unit (553) in the form of symbols (521), which may have, for example, X, Y, and reference picture components. Motion compensation can also include interpolation of sample values ​​fetched from the reference picture memory (557) when sub-sample accurate motion vectors are used, motion vector prediction mechanisms, and the like.

[0054] The output samples of the aggregator (555) may be subjected to various loop filtering techniques in a loop filter unit (556). Video compression techniques may include in-loop filtering techniques controlled by parameters included in the coded video sequence (also referred to as a coded video bitstream) and made available to the loop filter unit (556) as symbols (521) from the parser (520). Video compression may also be responsive to meta-information obtained during decoding of previous portions (in decoding order) of the coded picture or coded video sequence, as well as to previously reconstructed and loop-filtered sample values.

[0055] The output of the loop filter unit (556) can be a sample stream that can be output to a render device (512), which can also be stored in a reference picture memory (557) for use in future inter-picture prediction.

[0056] Once a particular coded picture is fully reconstructed, it can be used as a reference picture for future prediction. For example, once the coded picture corresponding to the current picture is fully reconstructed and identified as a reference picture (e.g., by the parser (520)), the current picture buffer (558) can become part of the reference picture memory (557), and a new current picture buffer can be reallocated before beginning reconstruction of the next coded picture.

[0057] The video decoder (510) may perform decoding according to a given video compression technology or standard, such as ITU-T Recommendation H.265. The coded video sequence may conform to the syntax specified by the video compression technology or standard used, in the sense of adhering to both the syntax of the video compression technology or standard and the profile documented in the video compression technology or standard. Specifically, a profile may select specific tools from all tools available in the video compression technology or standard, such that only those tools are available for use under that profile. Compliance also requires that the complexity of the coded video sequence be within a range specified by the level of the video compression technology or standard. In some cases, the level constrains the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. The limits set by the level may be further constrained, in some cases, through a Hypothetical Reference Decoder (HRD) specification and metadata for HRD buffer management signaled in the coded video sequence.

[0058] In one embodiment, the receiver (531) may receive additional (redundant) data along with the encoded video. The additional data may be included as part of one or more encoded video sequences. The additional data may be used by the video decoder (510) to properly decode the data and / or to more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0059] 6 shows an example block diagram of a video encoder (603). The video encoder (603) is included in an electronics device (620). For example, the electronics device (620) includes a transmitter (640) (e.g., a transmitting circuit). The video encoder (603) can be used in place of the video encoder (403) in the example of FIG. 4.

[0060] The video encoder (603) may receive video samples from a video source (601) (which in the example of FIG. 6 is not part of the electronics device (620)) that may capture video image(s) to be encoded by the encoder (603). In another example, the video source (601) is part of the electronics device (620).

[0061] The video source (601) may provide a source video sequence to be encoded by the video encoder (603) in the form of a digital video sample stream that may be of any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 Y CrCB, RGB, ...), and any suitable sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media service provision system, the video source (601) may be a storage device containing pre-prepared video. In a video conferencing system, the video source (601) may be a camera capturing local image information as a video sequence. The video data may be provided as multiple individual pictures that convey motion when viewed in sequence. The pictures themselves may be organized as a spatial array of pixels, each of which may have one or more samples, depending on the sampling structure, color space, etc. used. Those skilled in the art will readily understand the relationship between pixels and samples. The following description focuses on samples.

[0062] According to one embodiment, the video encoder (603) may code and compress pictures of a source video sequence into an encoded video sequence (643) in real time or under other required time constraints. Enforcing an appropriate coding rate is one function of the controller (650). In some embodiments, the controller (650) controls and is operatively coupled to other functional units, such as those described below, which are not shown for clarity. Parameters set by the controller (650) may include rate control-related parameters (picture skip, quantizer, lambda value for rate-distortion optimization techniques, etc.), picture size, group-of-picture (GOP) layout, maximum motion vector search range, etc. The controller (650) may be configured with other suitable functions associated with the video encoder (603) that are optimized for a particular system design.

[0063] In some embodiments, the video encoder (603) is configured to operate in a coding loop. As an overly simplified explanation, in one example, the coding loop can include a source coder (630) (e.g., responsible for creating symbols, e.g., a symbol stream, based on an input picture to be coded and one or more reference pictures) and a (local) decoder (633) embedded in the video encoder (603). The decoder (633) reconstructs the symbols to generate sample data, in a manner similar to that used by a (remote) decoder. The reconstructed sample stream (sample data) is input to a reference picture memory (634). Because decoding of the symbol stream yields bit-accurate results independent of the decoder location (local or remote), the contents of the reference picture memory (634) are also bit-accurate between the local and remote encoders. In other words, the predictive portion of the encoder "sees" the exact same sample values ​​as the decoder "sees" when using prediction during decoding. This basic principle of reference picture synchronism (and the resulting drift when synchronism cannot be maintained, for example due to channel errors) is also used in some related art.

[0064] The operation of the "local" decoder (633) may be the same as that of a "remote" decoder, such as the video decoder (510), which has already been described in detail above in connection with Figure 5. However, briefly referring also to Figure 5, because symbols are available and the encoding / decoding of the symbols into a coded video sequence by the entropy coder (645) and parser (520) may be lossless, the entropy decoding portion of the video decoder (510), including the buffer memory (515) and parser (520), may not be fully implemented in the local decoder (633).

[0065] In one embodiment, decoder technology, excluding parsing / entropy decoding, present in the decoder is present in the corresponding encoder in the same or substantially the same functional form. Therefore, the subject matter of this disclosure focuses on decoder operation. A description of the encoder technology can be omitted, as it is the reverse of the decoder technology that has been thoroughly described. In certain areas, more detailed descriptions are provided below.

[0066] In operation, in some examples, the source coder (630) may perform motion-compensated predictive coding, which predictively codes an input picture relative to one or more previously coded pictures from a video sequence designated as “reference pictures.” Thus, the coding engine (632) codes differences between pixel blocks of the input picture and pixel blocks of one or more reference pictures that may be selected as prediction reference(s) for the input picture.

[0067] The local video decoder (633) may decode coded video data for pictures that may be designated as reference pictures based on symbols created by the source coder (630). The operation of the coding engine (632) may advantageously be a lossy process. When the coded video data is decoded by a video decoder (not shown in FIG. 6), the reconstructed video sequence may typically be a replica of the source video sequence, with some error. The local video decoder (633) may replicate the decoding process that may be performed by a video decoder on the reference pictures, causing the reconstructed reference pictures to be stored in the reference picture memory (634). In this way, the video encoder (603) may locally store copies of reconstructed reference pictures that have content in common with reconstructed reference pictures that will be obtained by a far-end video decoder.

[0068] The predictor (635) may perform a predictive search for the coding engine (632). That is, for a new picture to be coded, the predictor (636) may search the reference picture memory (634) for sample data (as candidate reference pixel blocks) or specific metadata, such as reference picture motion vectors or block shapes, that can serve as an appropriate prediction reference for the new picture. The predictor (635) may operate on a pixel block-by-pixel block basis to find an appropriate prediction reference. In some cases, as determined by the search results obtained by the predictor (635), the input picture may have prediction references drawn from multiple reference pictures stored in the reference picture memory (634).

[0069] The controller (650) may manage the coding process of the source coder (630), including, for example, setting the parameters and subgroup parameters used to encode the video data.

[0070] The outputs of all the aforementioned functional units may be subjected to entropy coding in an entropy coder (645), which converts the symbols produced by the various functional units into a coded video sequence by applying lossless compression to the symbols according to techniques such as Huffman coding, variable length coding, or arithmetic coding.

[0071] A transmitter (640) may buffer the encoded video sequence(s) produced by the entropy coder (645) and prepare them for transmission over a communication channel (660), which may be a hardware or software link to a storage device that stores the encoded video data. The transmitter (640) may merge the encoded video data from the video encoder (603) with other data to be transmitted, such as encoded audio data and / or auxiliary data streams (sources not shown).

[0072] The controller (650) may manage the operation of the video encoder (603). In coding, the controller (650) may assign each coded picture a particular coded picture type, which may affect the coding technique that may be applied to the respective picture. For example, pictures may often be assigned one of the following picture types:

[0073] An intra picture (I-picture) may be one that can be coded and decoded without using any other picture in a sequence as a source of prediction. Some video codecs allow several different types of intra pictures, including, for example, Independent Decoder Refresh (IDR) pictures. Those skilled in the art are aware of these variants of I-pictures and their respective uses and characteristics.

[0074] A predicted picture (P picture) may be encoded and decoded using intra- or inter-prediction, using at most one motion vector and reference index to predict the sample values ​​of each block.

[0075] Bidirectionally predicted pictures (B-pictures) may be coded and decoded using intra- or inter-prediction, using at most two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, multi-predicted pictures may use more than two reference pictures and associated metadata for the reconstruction of a single block.

[0076] A source picture is generally spatially subdivided into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples each) and may be coded block by block. Blocks may be predictively coded with reference to other (already coded) blocks as determined by the coding assignment applied to their respective pictures. For example, blocks of an I-picture may be coded non-predictively, or they may be predictively coded with reference to already coded blocks of the same picture (spatial prediction or intra-prediction). Pixel blocks of a P-picture may be coded non-predictively or via spatial or temporal prediction with reference to one previously coded reference picture. Blocks of a B-picture may be coded non-predictively or via spatial or temporal prediction with reference to one or two previously coded reference pictures.

[0077] The video encoder (603) may perform coding operations according to a predetermined video coding technique or standard, such as ITU-T Recommendation H.265. In operation, the video encoder (603) may perform various compression operations, including predictive coding operations that exploit temporal and spatial redundancy in the input video sequence. The coded video data may therefore conform to a syntax defined by the video coding technique or standard being used.

[0078] In one embodiment, the transmitter (640) may transmit additional data along with the encoded video. The source coder (630) may include such data as part of the encoded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

[0079] Video may be captured as multiple source pictures (video pictures) in a time sequence. Intra-picture prediction (often abbreviated as intra-prediction) exploits spatial correlation within a given picture, while inter-picture prediction exploits correlation (temporal or otherwise) between pictures. In one example, a particular picture being coded / decoded, called the current picture, is divided into multiple blocks. When a block in the current picture is similar to a reference block in a previously coded and still buffered reference picture in the video, the block in the current picture can be coded by a vector called a motion vector. A motion vector points to a reference block in the reference picture and may have a third dimension that identifies the reference picture if multiple reference pictures are used.

[0080] In some embodiments, bi-prediction techniques can be used in inter-picture prediction. Bi-prediction techniques use two reference pictures, such as a first reference picture and a second reference picture, both of which are earlier in a video than the current picture in decoding order (but may be earlier and later, respectively, in display order). A block in the current picture can be coded with a first motion vector pointing to a first reference block in the first reference picture and a second motion vector pointing to a second reference block in the second reference picture. The block can be predicted by a combination of the first and second reference blocks.

[0081] Furthermore, merge mode techniques can be used to improve coding efficiency in inter-picture prediction.

[0082] According to some embodiments of the present disclosure, prediction, such as inter-picture prediction and intra-picture prediction, is performed on a block-by-block basis. For example, according to the HEVC standard, a picture in a sequence of video pictures is divided into multiple coding tree units (CTUs) for compression, and the CTUs within a picture have the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. Generally, a CTU includes three coding tree blocks (CTBs), one luma CTB and two chroma CTBs. Each CTU may be recursively quadtree-decomposed into one or more coding units (CUs). For example, a 64×64 pixel CTU can be divided into one CU of 64×64 pixels, four CUs of 32×32 pixels, or 16 CUs of 16×16 pixels. In one example, each CU is analyzed to determine the prediction type of that CU, such as an inter-prediction type or an intra-prediction type. A CU is divided into one or more prediction units (PUs) depending on temporal and / or spatial predictability. Generally, each PU includes a luma prediction block (PB) and two chroma PBs. In one embodiment, prediction operations during coding (encoding / decoding) are performed in units of prediction blocks. Taking a luma prediction block as an example of a prediction block, the prediction block includes a matrix of pixel values ​​(e.g., luma values), such as 8x8 pixels, 16x16 pixels, 8x16 pixels, 16x8 pixels, and the like.

[0083] 7 shows an example diagram of a video encoder (703). The video encoder (703) is configured to receive sample values ​​of a processing block (e.g., a predictive block) in a current video picture in a sequence of video pictures and encode the processing block into a coded picture that is part of the coded video sequence. In one example, the video encoder (703) is used in place of the video encoder (403) in the example of FIG. 4.

[0084] In an HEVC example, the video encoder (703) receives a matrix of sample values ​​for a processing block, such as 8x8 samples of a predictive block. The video encoder (703) determines, for example, using rate-distortion optimization, whether the processing block is best coded using intra-mode, inter-mode, or bi-predictive mode. If the processing block is coded in intra-mode, the video encoder (703) can encode the processing block into a coded picture using intra-prediction techniques; if the processing block is coded in inter-mode or bi-predictive mode, the video encoder (703) can encode the processing block into a coded picture using inter-prediction techniques or bi-prediction techniques, respectively. In certain video coding techniques, the merge mode may be an inter-picture prediction sub-mode in which a motion vector is derived from one or more motion vector predictors without the benefit of coded motion vector components outside the predictors. In certain other video coding techniques, there may be motion vector components applicable to the current block. In one example, the video encoder (703) includes other components, such as a mode decision module (not shown) for determining the mode of a processing block.

[0085] In the example of Figure 7, the video encoder (703) includes an inter-encoder (730), an intra-encoder (722), a residual calculator (723), a switch (726), a residual encoder (724), a general controller (721), and an entropy encoder (725), coupled together as shown in Figure 7.

[0086] The inter-encoder (730) is configured to receive samples of a current block (e.g., a processing block), compare the block to one or more reference blocks in a reference picture (e.g., blocks in a previous picture and a later picture), generate inter-prediction information (e.g., a description of redundant information according to an inter-coding technique, motion vectors, merge mode information), and calculate an inter-prediction result (e.g., a predicted block) based on the inter-prediction information using any suitable technique. In some examples, the reference picture is a decoded reference picture based on coded video information.

[0087] The intra encoder (722) receives samples of a current block (e.g., a processing block), in some cases compares the block with previously coded blocks in the same picture, generates transformed quantized coefficients, and in some cases also generates intra prediction information (e.g., intra prediction direction information according to one or more intra coding techniques). In one example, the intra encoder (722) also calculates intra prediction results (e.g., predicted blocks) based on the intra prediction information and reference blocks in the same picture.

[0088] The general controller (721) is configured to determine general control data and control other components of the video encoder (703) based on the general control data. In one example, the general controller (721) determines the mode of the block and provides a control signal to the switch (726) based on the mode. For example, when the mode is intra mode, the general controller (721) controls the switch (726) to select intra mode results for use by the residual calculator (723) and controls the entropy encoder (725) to select intra prediction information and include the intra prediction information in the bitstream. When the mode is inter mode, the general controller (721) controls the switch (726) to select inter prediction results for use by the residual calculator (723) and controls the entropy encoder (725) to select inter prediction information and include the inter prediction information in the bitstream.

[0089] The residual calculator (723) is configured to calculate the difference (residual data) between the received block and a prediction result selected from the intra-encoder (722) or inter-encoder (730). The residual encoder (724) operates based on the residual data and is configured to encode the residual data to generate transform coefficients. In one example, the residual encoder (724) is configured to transform the residual data from the spatial domain to the frequency domain to generate transform coefficients. The transform coefficients are then subjected to a quantization process to obtain quantized transform coefficients. In various embodiments, the video encoder (703) also includes a residual decoder (728). The residual decoder (728) is configured to perform an inverse transform and generate decoded residual data. The decoded residual data can be suitably used by the intra-encoder (722) and the inter-encoder (730). For example, the inter-encoder (730) can generate decoded blocks based on the decoded residual data and inter-prediction information, and the intra-encoder (722) can generate decoded blocks based on the decoded residual data and intra-prediction information. The decoded blocks are suitably processed to generate decoded pictures, which can be buffered in a memory circuit (not shown) and, in some examples, used as reference pictures.

[0090] The entropy encoder (725) is configured to format a bitstream to include the encoded blocks. The entropy encoder (725) is configured to include various information in the bitstream according to a suitable standard, such as the HEVC standard. In one example, the entropy encoder (725) is configured to include general control data, selected prediction information (e.g., intra-prediction information or inter-prediction information), residual information, and other suitable information in the bitstream. Note that, according to the subject matter of this disclosure, residual information is not present when coding blocks in a merged sub-mode of either inter mode or bi-prediction mode.

[0091] 8 shows an example diagram of a video decoder (810). The video decoder (810) is configured to receive coded pictures that are part of a coded video sequence and decode the coded pictures to generate reconstructed pictures. In one example, the video decoder (810) is used in place of the video decoder (410) in the example of FIG. 4.

[0092] In the example of Figure 8, the video decoder (810) includes an entropy decoder (871), an inter-decoder (880), a residual decoder (873), a reconstruction module (874), and an intra-decoder (872), coupled together as shown in Figure 8.

[0093] The entropy decoder (871) may be configured to reconstruct, from the coded picture, specific symbols representing the syntax elements that make up the coded picture. Such symbols may include, for example, prediction information (e.g., intra-prediction information or inter-prediction information) that may identify the mode in which a block is coded (e.g., intra-mode, inter-mode, bi-prediction mode, merged sub-mode, or the latter two in other sub-modes) as well as specific samples or metadata used for prediction by the intra-decoder (872) or inter-decoder (880), respectively. The symbols may also include residual information, for example, in the form of quantized transform coefficients, and the like. In one example, when the prediction mode is an inter-mode or bi-prediction mode, the inter-prediction information is provided to the inter-decoder (880), and when the prediction type is an intra-prediction type, the intra-prediction information is provided to the intra-decoder (872). The residual information may be subjected to inverse quantization and provided to the residual decoder (873).

[0094] The inter decoder (880) is configured to receive the inter prediction information and generate an inter prediction result based on the inter prediction information.

[0095] The intra decoder (872) is configured to receive intra prediction information and generate a prediction result based on the intra prediction information.

[0096] The residual decoder (873) is configured to perform inverse quantization to extract dequantized transform coefficients, and then process the dequantized transform coefficients to transform the residual information from the frequency domain to the spatial domain. The residual decoder (873) may also require certain control information (including quantizer parameters (QP)), which may be provided by the entropy decoder (871) (data path not shown as this may be only low-volume control information).

[0097] The reconstruction module (874) is configured to combine the residual information output by the residual decoder (873) and the prediction results (output by the inter- or intra-prediction module, as the case may be) in the spatial domain to form reconstructed blocks. The reconstructed blocks may be part of reconstructed pictures, which in turn may be part of reconstructed video. Note that other suitable processes, such as deblocking and the like, may be performed to improve visual quality.

[0098] It should be noted that the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) may be implemented using any suitable technology. In one embodiment, the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) may be implemented using one or more integrated circuits. In another embodiment, the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) may be implemented using one or more processors executing software instructions.

[0099] VVC can use various inter-prediction modes. For an inter-predicted CU, motion parameters can include MV(s), one or more reference picture indexes, a reference picture list usage index, and additional information about specific coding features used for sample generation by inter prediction. Motion parameters can be signaled explicitly or implicitly. When a CU is coded in skip mode, the CU can be associated with a PU and may not have significant residual coefficients, coded motion vector deltas or MV differences (e.g., MVDs), or reference picture indexes. A merge mode can be specified, and motion parameters for the current CU are obtained from neighboring CU(s), including spatial and / or temporal candidates, and optionally additional information, such as introduced in VVC. The merge mode can be applied not only to skip mode but also to inter-predicted CUs. In one example, an alternative to merge mode is explicit transmission of motion parameters, where (one or more) MVs, corresponding reference picture indexes for each reference picture list, reference picture list usage flags, and other information are explicitly signaled for each CU.

[0100] In one embodiment, e.g., in VVC, the VVC Test Model (VTM) reference software supports enhanced merge prediction, merge motion vector difference (MMVD) mode, adaptive motion vector prediction (AMVP) mode with symmetric MVD signaling, affine motion compensation prediction, subblock-based temporal motion vector prediction (SbTMVP), adaptive motion vector resolution (AMVR), motion field storage (1 / 16 luma sample MV storage and 8x8 motion field compression), bi-prediction with CU-level weights (BCW), bi-directional optical flow (BDOF), prediction refinement using optical flow (PROF), decoder side motion vector refinement (DMVR), combined inter and intra prediction (CIIP), geometric partitioning mode, and more. Inter-prediction and related methods are described in more detail below.

[0101] Enhanced merge prediction may be used in some examples. In one example, such as in VTM4, a merge candidate list is constructed by including, in order, the following five types of candidates: spatial motion vector predictor (MVP)(s) from spatially neighboring CU(s), temporal MVP(s) from co-located CU(s), history-based MVP(s) (HMVP(s)) from a first-in-first-out (FIFO) table, pairwise average MVP(s), and zero MV(s).

[0102] The size of the merge candidate list can be signaled in the slice header. In one example, the maximum allowed size of the merge candidate list is 6 in VTM4. For each CU coded in merge mode, the index of the best merge candidate (e.g., merge index) may be coded using truncated unary binarization (TU). The first bin of the merge index may be coded using context (e.g., context-adaptive binary arithmetic coding (CABAC)), and bypass coding may be used for the other bins.

[0103] Some examples of the generation process for each category of merge candidates are provided below. In one embodiment, spatial candidate(s) are derived as follows: The derivation of spatial merge candidates in VVC may be the same as that in HEVC. In one example, up to four merge candidates are selected from the candidates at the positions shown in FIG. 9. FIG. 9 illustrates the positions of spatial merge candidates according to one embodiment of the present disclosure. Referring to FIG. 9, the derivation order is B1, A1, B0, A0, and B2. Position B2 is considered only if any CU at positions A0, B0, B1, and A1 is unavailable (e.g., because the CU belongs to another slice or another tile) or is intra-coded. After the candidate at position A1 is added, the addition of the remaining candidates is subjected to a redundancy check that ensures that candidates with the same motion information are excluded from the candidate list, so that coding efficiency is improved.

[0104] To reduce computational complexity, the aforementioned redundancy check does not consider all possible candidate pairs. Instead, only pairs connected by arrows in FIG. 10 are considered, and a candidate is added to a candidate list only if the corresponding candidates used in the redundancy check do not have the same motion information. FIG. 10 illustrates candidate pairs considered for a spatial merge candidate redundancy check according to one embodiment of the present disclosure. Referring to FIG. 10, the respective arrow-connected pairs are A1 and B1, A1 and A0, A1 and B2, B1 and B0, and B1 and B2. Thus, candidates at positions B1, A0, and / or B2 can be compared with the candidate at position A1, and candidates at positions B0 and / or B2 can be compared with the candidate at position B1.

[0105] In one embodiment, the temporal candidate(s) are derived as follows: In one example, only one temporal merge candidate is added to the candidate list. Figure 11 shows exemplary motion vector scaling for temporal merge candidates. To derive a temporal merge candidate for a current CU (1111) in a current picture (1101), a scaled MV (1121) (e.g., shown by a dotted line in Figure 11) may be derived based on a co-located CU (1112) belonging to a co-located reference picture (1104). In one example, a co-located reference picture (also referred to as a co-located picture) is a specific reference picture used, for example, for temporal motion vector prediction. The co-located reference picture used for temporal motion vector prediction can be indicated by a reference index in syntax, for example, high-level syntax (e.g., picture header, slice header).

[0106] The reference picture list used to derive the co-located CU (1112) may be explicitly signaled in the slice header. A scaled MV (1121) for the temporal merge candidate may be obtained, as shown by the dotted line in Figure 11. The scaled MV (1121) may be scaled from the MV of the co-located CU (1112) using picture order count (POC) distances tb and td. The POC distance tb may be defined as the POC difference between the current reference picture (1102) of the current picture (1101) and the current picture (1101). The POC distance td may be defined as the POC difference between the co-located reference picture (1104) of the co-located reference picture (1103) and the co-located reference picture (1103). The reference picture index of the temporal merge candidate may be set to zero.

[0107] FIG. 12 shows exemplary candidate positions (e.g., C0 and C1) for temporal merge candidates for the current CU. A temporal merge candidate position can be selected from candidate positions C0 and C1. Candidate position C0 is located at the bottom right corner of the current CU's co-located CU (1210). Candidate position C1 is located at the center of the current CU's co-located CU (1210). If the CU at candidate position C0 is unavailable, intra-coded, or outside the current row of the CTU, candidate position C1 is used to derive the temporal merge candidate. Otherwise, for example, if the CU at candidate position C0 is available, intra-coded, and within the current row of the CTU, candidate position C0 is used to derive the temporal merge candidate.

[0108] The merge motion vector difference (MMVD) mode can be used in merge mode or skip mode with a motion vector representation method. For example, one or more merge candidates, such as those used in VVC, can be reused in the MMVD mode. A candidate can be selected from among the merge candidates as a starting point (e.g., MV predictor (MVP)) and can be further extended by the MMVD mode. The MMVD mode can provide a new motion vector representation using simplified signaling. The motion vector representation method includes a starting point and an MV difference (MVD). In one example, the MVD is indicated by the magnitude (or motion magnitude) of the MVD and the direction (e.g., motion direction) of the MVD.

[0109] The MMVD mode can use a merge candidate list, such as that used in VVC. In one embodiment, only candidate(s) of the default merge type (e.g., MRG_TYPE_DEFAULT_N) are considered in the MMVD mode. The starting point can be indicated or defined by a base candidate index (IDX). The base candidate index can indicate a candidate (e.g., the best candidate) among multiple candidates (e.g., multiple base candidates) in the merge candidate list. Table 1 shows an example relationship between the base candidate index and the corresponding starting point. A base candidate index of 0, 1, 2, or 3 indicates that the corresponding starting point is the first MVP, the second MVP, the third MVP, or the fourth MVP. In one example, if the number of base candidates is equal to 1, the base candidate IDX is not signaled. [Table 1]

[0110] The distance index may indicate motion magnitude information of the MVD, such as the magnitude of the MVD. For example, the distance index indicates a distance (e.g., a predetermined distance) from a starting point (e.g., an MVP indicated by a base candidate index). In one example, the distance is one of a plurality of predetermined distances, such as those shown in Table 2. Table 2 shows an exemplary relationship between the distance index and the corresponding distance (in units of samples or pixels). 1 pel in Table 2 means 1 sample or 1 pixel. For example, a distance index of 1 indicates that the distance is 1 / 2 pel, i.e., 1 / 2 sample. [Table 2]

[0111] The direction index can represent the direction of the MVD relative to the starting point. The direction index can represent one of multiple directions, such as the four directions shown in Table 3. For example, a direction index of 00 indicates that the direction of the MVD is along the positive x-axis. [Table 3]

[0112] The MMVD flag can be signaled after sending the skip and merge flags. If the skip and merge flags are true, the MMVD flag can be parsed. In one example, if the MMVD flag is equal to 1, the MMVD syntax (e.g., including distance indexes and / or direction indexes) can be parsed. If the MMVD flag is not equal to 1, the AFFINE flag can be parsed. If the AFFINE flag is equal to 1, the AFFINE mode is used to code the current block. If the AFFINE flag is not equal to 1, the skip / merge index can be parsed for the skip / merge mode, for example, as used in VTM.

[0113] 13-14 show an example of a search process in MMVD mode, which can be performed to determine indices, including a base candidate index, a direction index, and / or a distance index, for a current block (1300) in a current picture (or current frame) (1301).

[0114] A first motion vector (MV) (1311) and a second MV (1321) belonging to a first merging candidate are shown. The first merging candidate may be a merging candidate in the merging candidate list constructed for the current block (1300). The first and second MVs (1311) and (1321) may be associated with two reference pictures (1302) and (1303) in the reference picture lists L0 and L1, respectively. Thus, two starting points (1411) and (1421) in Figures 13-14 may be determined within the reference pictures (1302) and (1303), respectively.

[0115] In one example, based on the starting points 1411 and 1421, multiple predetermined points extending vertically (represented by +Y or −Y) or horizontally (represented by +X and −X) from the starting points 1411 and 1421 within the reference pictures 1302 and 1303 may be evaluated. In one example, pairs of points that are mirror images of each other with respect to the respective starting points 1411 and 1421, such as the pair of points 1414 and 1424 or the pair of points 1415 and 1425, may be used to determine a pair of motion vectors 1314 and 1324 or a pair of motion vectors 1315 and 1325 that may form candidate motion vector predictors (MVPs) for the current block 1300. MVP candidates determined based on the predetermined points surrounding the starting points 1411 and / or 1421 may be evaluated. Referring to Figure 13, the MVD (1312) between the first MV (1311) and MV (1314) has a magnitude of 1S. The MVD (1322) between the second MV (1321) and MV (1324) has a magnitude of 1S. Similarly, the MVD between the first MV (1311) and MV (1315) has a magnitude of 2S. The MVD between the second MV (1321) and MV (1325) has a magnitude of 2S.

[0116] In addition to the first merge candidate, other available or valid merge candidates in the merge candidate list of the current block (1300) may also be evaluated. In one example, for unipredictive merge candidates, only one prediction direction associated with one of the two reference picture lists is evaluated.

[0117] In one example, the best MVP candidate can be determined based on the evaluation. Therefore, an optimal merge candidate corresponding to the optimal MVP candidate can be selected from the merge list, and the motion direction and motion distance can also be determined. For example, a base candidate index can be determined based on the selected merge candidate and Table 1. Based on the selected MVP, such as the one corresponding to a predetermined point 1415 (or 1425), the direction and distance (e.g., 2S) of point 1415 relative to the starting point 1411 can be determined. The direction index and distance index can be determined accordingly according to Tables 2 and 3.

[0118] As described above, two indexes, such as a distance index and a direction index, can be used to indicate the MVD in the MMVD mode. Alternatively, a single index can be used to indicate the MVD in the MMVD mode, for example, using a table that pairs a single index with the MVD.

[0119] In some prediction modes, such as the MMVD mode and the affine MMVD mode, template matching (TM)-based candidate sorting may be used. In one embodiment, the MMVD offset is extended for the MMVD mode and the affine MMVD mode. FIG. 15 illustrates additional refinement positions along multiple diagonal angles, such as k×π / 8 diagonal angles, where k is an integer from 0 to 15. The additional refinement positions along multiple diagonal angles can increase the number of directions, such as from four directions (e.g., +X, −X, +Y, and −Y) to 16 directions (e.g., k=0, 1, 2, ..., 15). In one example, each of these 16 directions is represented by an angle between the +X direction and a direction indicated by the center point (1500) and one of points 1 to 16. For example, point 1 indicates the +X direction with an angle of 0 (ie, k=0), point 2 indicates a direction along an angle of 1×π / 8 (ie, k=1), and so on.

[0120] The TM may be performed in MMVD mode. In one example, for each MMVD refinement position, a TM cost may be determined based on the current template of the current block and one or more reference templates. The TM cost may be determined using any method, such as sum of absolute differences (SAD) (e.g., SAD cost), sum of absolute translation differences (SATD), sum of squared errors (SSE), mean-removed SAD / SATD / SSE, variance, partial SAD, partial SSE, partial SATD, or the like.

[0121] The current template of the current block may include any suitable samples, such as samples in a row above the current block and / or samples in a column to the left of the current block. Based on the TM cost (e.g., SAD cost) between the current template and the corresponding reference template for the refinement position, the MMVD refinement positions, such as all possible MMVD refinement positions (e.g., 16×6 positions representing 16 directions and 6 magnitudes) for each base candidate (e.g., MVP), may be sorted. In one example, the top MMVD refinement positions with the smallest TM costs (e.g., smallest SAD costs) are retained as MMVD refinement positions available for MMVD index coding. For example, a subset (e.g., 8) of the MMVD refinement positions with the smallest TM costs are used for MMVD index coding. For example, an MMVD index indicates which one of the subset of MMVD refinement positions with the smallest TM costs is selected to code the current block. In one example, an MMVD index of 0 indicates that the MVD corresponding to the smallest TM cost (e.g., the MMVD refinement position) is used to code the current block. The MMVD index may be binarized, for example, by a Rice code with a parameter equal to 2.

[0122] In one embodiment, in addition to the MMVD offset extension described above, e.g., in Figure 15, the affine MMVD reordering is extended to add further refinement positions along a k x π / 4 diagonal angle. After reordering, the top half of refinement positions with the smallest TM cost (e.g., SAD cost) are kept for coding the current block.

[0123] To improve coding efficiency and reduce the transmission overhead of one or more MVs, sub-block-level MV refinement may be applied to extend CU-level temporal motion vector prediction (TMVP). In one example, sub-block-based TMVP (SbTMVP) mode enables inheriting sub-block-level motion information from a co-located reference picture. As described above, the co-located reference picture may be indicated by a reference index in a syntax, such as a high-level syntax (e.g., a picture header, a slice header). Each sub-block of a current CU (e.g., a current CU with a large size) in a current picture may have its own motion information without explicitly transmitting a block partition structure or its own motion information. In SbTMVP mode, the motion information of each sub-block may be obtained, for example, in three steps as follows: In the first step, a displacement vector (DV) of the current CU may be derived. The DV may indicate a block in a co-located reference picture, for example, the DV points from a current block in a current picture to a block in a co-located reference picture. Therefore, the block indicated by DV is considered to be co-located with the current block and is called the co-located block of the current block. In the second step, the availability of SbTMVP candidates can be checked, and central motion (e.g., central motion of the current CU) can be derived. In the third step, sub-block motion information can be derived from corresponding sub-blocks in the co-located block using DV. These three steps may be combined into one or two steps, and / or the order of these three steps may be adjusted.

[0124] Unlike TMVP candidate derivation, which derives temporal MVs from co-located blocks in a reference frame or reference picture, in SbTMVP mode, a DV (e.g., a DV derived from the MV of the current CU's left neighboring CU) can be applied to locate the corresponding sub-block in the co-located reference picture for each sub-block in the current CU in the current picture. If the corresponding sub-block is not inter-coded, the motion information of the current sub-block can be set to the central motion of the co-located block.

[0125] The SbTMVP mode may be supported by various video coding standards, including, for example, VVC. Similar to the TMVP mode, for example, in HEVC, the SbTMVP mode may use a motion field (also referred to as a motion information field or MV field) in a co-located reference picture to improve MV prediction and merging for a CU in a current picture. In one example, the same co-located reference picture used by the TMVP mode is used in the SbTMVP mode. In one example, the SbTMVP mode differs from the TMVP mode in the following respects: (i) the TMVP mode predicts motion information at the CU level, whereas the SbTMVP mode predicts motion information at the sub-CU level; and (ii) the TMVP mode fetches temporal MV from a co-located block in the co-located reference picture (e.g., the co-located block is the bottom-right or center block with respect to the current CU), whereas the SbTMVP mode may apply a motion shift before fetching temporal motion information from the co-located reference picture. In one example, the motion shift used in SbTMVP mode is obtained from the MV of one of the spatially neighboring blocks of the current CU.

[0126] Figures 16-17 show an exemplary SbTMVP process used in SbTMVP mode. The SbTMVP process can predict the motion vector (MV) of a sub-CU (e.g., a sub-block) in a current CU (e.g., a current block) (1601) of a current picture (1711), for example, in two steps. In the first step, a spatial neighbor (e.g., A1) of the current block (1601) in Figures 16-17 is examined. If the spatial neighbor (e.g., A1) has a motion vector (MV) (1721) that uses a co-located reference picture (1712) as its reference picture, the motion vector (MV) (1721) can be selected as the motion shift (or DV) to be applied to the current block (1601). If no such MV (e.g., an MV that uses the co-located reference picture (1712) as a reference picture) is identified, the motion shift or DV may be set to a zero MV (e.g., (0,0)). In some examples, if no such MV is identified for spatial neighbor A1, the MV(s) of further spatial neighbors, such as A0, B0, and B1, are examined.

[0127] In a second step, the motion shift or DV (1721) identified in the first step may be applied to the current block (1601) (e.g., by adding the DV (1721) to the coordinates of the current block) to obtain sub-CU level motion information (e.g., including an MV and a reference index) from the co-located reference picture (1712). In the example shown in FIG. 17, the motion shift or DV (1721) is set to be the MV of the spatial neighbor A1 (e.g., block A1) of the current block (1601). For each sub-CU or sub-block (1731) in the current block (1601), the motion information of the sub-CU or sub-block (1731) may be derived using the motion information of the corresponding co-located block (1701) in the co-located reference picture (1712) (e.g., the motion information of the minimum motion grid covering the center sample of the co-located block (1701)). After the motion information of the co-located sub-CU (1732) in the co-located block (1701) is identified, the motion information of the co-located sub-CU (1732) can be converted into motion information (e.g., MV and one or more reference indices) of the current sub-CU (1731) using a scaling method, such as in a manner similar to the TMVP process used in HEVC, in which temporal motion scaling is applied to align the reference picture of the temporal MV to the reference picture of the current CU.

[0128] The motion field of the current block (1601), derived based on the DV (1721), can include motion information for each sub-block (1731) in the current block (1601), such as one or more MVs and one or more associated reference indices. The motion field of the current block (1601) can also be referred to as an SbTMVP candidate and corresponds to the DV (1721).

[0129] 17 shows an example of a motion field or SbTMVP candidate for the current block 1601. For example, the motion information of a bi-predicted sub-block (1731(1)) includes a first motion vector (MV) and a first index indicating a first reference picture in reference picture list 0 (L0), a second motion vector (MV) and a second index indicating a second reference picture in reference picture list 1 (L1). In one example, the motion information of a uni-predicted sub-block (1731(2)) includes a motion vector and an index indicating a reference picture in L0 or L1.

[0130] In one example, the DV (1721) is applied to the center position of the current block (1601) to locate the displaced center position in the co-located reference picture (1712). If the block including the displaced center position is not inter-coded, the SbTMVP candidate is considered unavailable. Otherwise, if the block including the displaced center position (e.g., the co-located block (1701)) is inter-coded, motion information of the center position of the current block (1601), referred to as the central motion of the current block (1601), may be derived from motion information of the block including the displaced center position in the co-located reference picture (1712). In one example, a scaling process may be used to derive the central motion of the current block (1601) from motion information of the block including the displaced center position in the co-located reference picture (1712). When SbTMVP candidates are available, for each sub-block (1732) of the current block (1601), the DV (1721) can be applied to find the corresponding sub-block (1731) in the co-located reference picture (1712). The motion information of the corresponding sub-block (1732) can be used to derive motion information for the sub-block (1731) in the current block (1601), for example, in the same manner as used to derive the central motion of the current block (1601). In one example, if the corresponding sub-block (1732) is not inter-coded, the motion information of the current sub-block (1731) is set to be the central motion of the current block (1601).

[0131] In some examples, such as in VVC, a combined sub-block-based merge list containing an SbTMVP candidate and one or more affine merge candidates is used to signal the sub-block-based merge mode. The SbTMVP mode can be enabled or disabled by a sequence parameter set (SPS) flag. When the SbTMVP mode is enabled, the SbTMVP candidate (or SbTMVP predictor) is added as the first entry of the sub-block-based merge list containing the sub-block-based merge candidate, which can be followed by one or more affine merge candidates. The size of the sub-block-based merge list can be signaled within the SPS. In one example, the maximum allowed size of the sub-block-based merge list is 5 in VVC. In one example, multiple SbTMVP candidates are included in the sub-block-based merge list.

[0132] In some examples, such as in VVC, the sub-CU size used in SbTMVP mode is fixed at 8x8, as used in affine merge mode, for example. In one example, SbTMVP mode is only applicable to CUs whose width and height are both 8 or greater. The sub-block size (e.g., 8x8) may be configurable to other sizes, such as 4x4 in ECM software model applications for VVC and later exploration. In one example, multiple co-located reference pictures, such as two co-located frames, are used to provide temporal motion information for SbTMVP and / or TMVP in AMVP mode.

[0133] In some examples of SbTMVP modes, such as in VVC and ECM, the DV of the current CU (e.g., DV (1721) in FIG. 17) is derived only from the MVs of the current CU's neighboring CUs. However, the SbTMVP candidates derived using the DVs may not be an exact match.

[0134] A DV offset (DVO) can be used in SbTMVP mode. In one example, to obtain a more accurate match, the DV (e.g., the initial DV) can be modified by the DV offset to determine an updated DV'. In one example, the updated DV' is the vector sum of the DV (e.g., the initial DV) and the DVO. The initial DV can be determined using any method, such as those described in Figures 16 and 17. For example, the initial DV can be determined based on the MVs of neighboring blocks of the current block. For the current block, the DVO can be signaled and analyzed to indicate an additional motion offset of the initial DV. The DVO can be indicated, for example, by signaling an index that indicates the DVO from multiple DVO candidates. In one example, the DVO is signaled. In one example, the MMVD mode is used to indicate the DVO, and for example, the DVO is an MVD indicated by a direction index and / or a distance index, such as those listed in Tables 2 and 3. By using DV_O, the position of the co-located CU (or co-located block) in the co-located reference picture can be adjusted, and therefore, the MV field of the co-located CU (or co-located block) can change based on DV_O. When DV_O is not 0, the updated DV' can be used as a displacement vector indicating the position of the co-located CU (or co-located block) to perform the SbTMVP process. Referring to Figure 17, instead of using the original DV (e.g., DV(1721)), which is the MV of the spatial neighbor A1, the updated DV' can be used to determine the co-located block for the current block. The updated DV' (e.g., the vector sum of the original DV and DV_O) can be used to derive SbTMVP candidates for the current block.

[0135] In one embodiment, DVO is signaled directly using any signaling method used to signal MVD, such as in AMVP mode, AMVR mode, and / or the like. In AMVR mode, the MVD of a block can be signaled at different resolutions, such as 1 / 4, 1 / 2, 1, or 4 luma sample resolution. DVO can be signaled at different resolutions using AMVR mode.

[0136] A given DVO list may include multiple DVO candidates (e.g., DVOs that are possible to be used by the current block), and one or more indexes may be signaled to indicate which of the DVO candidates is selected as the DVO.

[0137] In one example, the DVO is signaled using the MMVD mode. For example, as shown in Table 2-Table 3, two indexes are signaled to indicate the DVO candidate, including a first index (e.g., a distance index or a step index) indicating the size of the DVO candidate and a second index (e.g., a direction index) indicating the direction of the DVO candidate.

[0138] Referring again to FIG. 14, the distance index (or step index) and direction index may be predetermined as described above with reference to the MMVD mode. The distance index indicates motion magnitude information, such as the magnitude of the DVO. For example, the distance index indicates a predetermined distance from the starting point (e.g., the initial DV). In one example, available predetermined distances are shown in Table 2. The direction index represents the direction of the DVO relative to the starting point (e.g., the initial DV). The direction index may indicate one of multiple directions, such as the four directions shown in Table 3.

[0139] In one example, a co-located CTU in a co-located reference picture is co-located with the current CTU that includes the current block. The current CTU is located within the current picture. In one embodiment, the location of the co-located block corresponding to the updated DV′ is constrained to be within a first area in the co-located reference picture. In one example, the first area in the co-located reference picture includes the co-located CTU. In one example, the first area in the co-located reference picture includes the co-located CTU plus one column of 4×4 blocks at the right boundary of the co-located CTU. The updated DV′ may be constrained such that the co-located block corresponding to the updated DV′ is within the first area in the co-located reference picture. In one example, the DVO (e.g., the horizontal component DVO x and / or the vertical component of DVO DVO y ) is constrained to ensure that the updated DV′ satisfies the co-located block position constraints mentioned above.

[0140] In one example, the maximum vertical component of the updated DV′ is H. y ) is less than or equal to H minus the vertical component of DV, DVy, e.g., DVO y ≦HDV y The maximum horizontal component of the updated DV' is W, and DVO is the horizontal component of DV from W to DV. x For example, DVO x ≦W-DV x is.

[0141] As described in Figures 16-17, a co-located block (e.g., (1701)) in a co-located reference picture (e.g., (1712)) can be determined based on the DV (e.g., (1721)) of a current block (e.g., (1601)) in a current picture (e.g., (1711)). Thus, motion information (e.g., TMVP) of each sub-block in the current block may be based on motion information of a corresponding sub-block in the co-located block. According to one embodiment of the present disclosure, updated motion information (e.g., updated TMVP) of each sub-block in the current block may be determined based on the motion information (e.g., TMVP) of the sub-block in the current block and the motion vector offset (MVO) of the current block.

[0142] In one embodiment, the MVO is added to each derived sub-block-level TMVP of each sub-block in the current block to generate an updated sub-block-level TMVP. The MVO can be signaled and analyzed to indicate the additional motion offset of each sub-block-based TMVP of each sub-block in the current block determined using the SbTMVP mode.

[0143] In one embodiment, the MVO is signaled directly using any signaling method used to signal the MVD, such as in AMVP mode, AMVR mode, and / or the like. In AMVR mode, the MVD of a block can be signaled at different resolutions, such as 1 / 4, 1 / 2, 1, or 4 luma sample resolution. The MVO can be signaled at different resolutions using AMVR mode.

[0144] A given MVO list may include multiple MVO candidates (e.g., MVOs that are possible for use by the current block). One or more indexes may be signaled to indicate which MVO candidates among the DVO candidates in a given MVO list may be selected as an MVO.

[0145] In one example, the MVO is signaled using the MMVD mode. For example, as shown in Tables 2-3, two indexes are signaled to indicate the MVO candidate, including a first index (e.g., a distance index or a step index) indicating the size of the MVO candidate and a second index (e.g., a direction index) indicating the direction of the MVO candidate.

[0146] Referring again to Figure 14, the distance index (or step index) and direction index of the MVO may be predetermined as described above with reference to the MMVD mode. The distance index indicates motion magnitude information, such as the size of the MVO, and may indicate, for example, a certain predetermined distance from a starting point (e.g., a DV used to determine the position of a co-located block in a co-located reference picture). In one example, available predetermined distances are shown in Table 2. The direction index represents the direction of the MVO relative to the starting point (e.g., a DV used to determine the position of a co-located block in a co-located reference picture). The direction index may indicate one of multiple directions, such as the four directions shown in Table 3.

[0147] In one example, a sub-block in a current block is bi-predicted. Referring to FIG. 17, sub-block (1731(1)) is bi-predicted and has a first MV associated with a first reference picture in reference list L0 and a second MV associated with a second reference picture in reference list L1. An MVO may be applied to the first MV associated with reference list L0. The updated first MV may be a vector sum of the first MV and the MVO. The following embodiments may be applied to the second MV associated with reference list L1.

[0148] In one example, the MVO is not applied to the second MV associated with reference list L1. For example, no MVO is applied to the second MV associated with reference list L1. Therefore, the updated motion information of sub-block (1731(1)) includes the updated first MV and the second MV.

[0149] In one example, a mirror MVO of the MVO is applied to a second MV associated with reference list L1. The mirror MVO and the MVO may have the same magnitude and opposite orientation. The horizontal and vertical components of the MVO (e.g., signaled) are multiplied by −1 to obtain the horizontal and vertical components of the mirror MVO, respectively. The updated second MV may be a vector sum of the second MV and the mirror MVO, or a vector difference between the second MV and the MVO. Thus, the updated motion information of sub-block (1731(1)) includes an updated first MV (e.g., first MV+MVO) and an updated second MV (e.g., second MV−MVO).

[0150] In one example, a scaled MVO (MVO') may be applied to a second MV associated with reference list L1. The value of each component (e.g., horizontal and vertical components) of the MVO may be scaled based on the first POC difference and the second POC difference, as shown in Equation 1. Equation 1 may be applied to vectors MVO' and MVO. Equation 1 may be applied to each component of vectors MVO' and MVO. The first POC difference is calculated based on the POC of the current picture (POC curr ) and the POC of the first reference picture in the reference picture list L0 (POC L0 ) The second POC difference is the difference between the POC of the current picture (POC curr ) and the POC of the second reference picture in the reference picture list L1 (POC L1 ) is the difference between MVO' = MVO × (POC L1 -POC curr ) / (POC L0 -POC curr ) formula 1

[0151] In one example, the scaled MVO (MVO') is added to a second MV associated with reference list L1. In one example, a mirror image MVO' of the scaled MVO (MVO') is added to a second MV associated with reference list L1.

[0152] As described above, in one example of the SbTMVP mode (e.g., a variation on the SbTMVP mode described in Figures 16-17), applying DVO to DV can adjust the position of a co-located block in a co-located reference picture, thereby affecting the motion information of a sub-block within the current block. In another example of the SbTMVP mode (e.g., another variation on the SbTMVP mode described in Figures 16-17), applying MVO can directly adjust the motion information of a sub-block within the current block. DVO and MVO can be signaled using the same method. For example, a predetermined MVO list is the same as a predetermined DVO list, and DVO and MVO can use the same predetermined MVO list.

[0153] In some examples, applying DVO to DV may allow the position of the co-located block in the co-located reference picture to be adjusted. After obtaining the motion information of the sub-block in the current block based on the motion information of the corresponding sub-block in the co-located reference picture determined based on the updated DV' (e.g., DV+DVO), MVO may be applied to further adjust the motion information of the sub-block in the current block. DVO may be the same as or different from MVO.

[0154] FIG. 18 shows a flowchart outlining an encoding process (1800) according to one embodiment of the present disclosure. The process (1800) may be used in a video encoder. The process (1800) may be performed by an apparatus for video coding, which may include processing circuitry. In various embodiments, the process (1800) is performed by processing circuitry, such as the processing circuitry of terminal devices (310), (320), (330), and (340), processing circuitry performing the functions of a video encoder (e.g., (403), (603), (703)), and the like. In some embodiments, the process (1800) is implemented in software instructions, and thus the processing circuitry performs the process (1800) when the processing circuitry executes the software instructions. The process begins at (S1801) and proceeds to (S1810).

[0155] At (S1810), an updated DV of the current block in the current picture may be determined based on the DV of the current block and the DV offset of the current block (also referred to as MV offset (MVO)). The DV may be determined as described above, for example, in Figures 16-17. The updated DV of the current block indicates a co-located block in the co-located picture. The co-located block is co-located with the current block.

[0156] The current block includes multiple sub-blocks that are coded using the sub-block-based temporal motion vector prediction (SbTMVP) mode.

[0157] In one example, the DV offset (or MVO) is determined from the DV offset candidates using any suitable method.

[0158] In one example, the updated DV is determined to be the vector sum of the DV and the DV offset.

[0159] In one example, the updated DV is constrained so that the co-located block is within a constrained area in the co-located reference picture, where the constrained area includes the co-located area corresponding to the current CTU in the current picture, and the current CTU includes the current block.

[0160] At (S1820), motion information of a sub-block of the plurality of sub-blocks may be determined based on motion information of a corresponding sub-block in a co-located block.

[0161] At (S1830), DV offset information (also referred to as MVO information) indicating a DV offset may be coded. The sub-block among the plurality of sub-blocks may be coded based on motion information of the sub-block among the plurality of sub-blocks.

[0162] In one embodiment, the DV offset information indicates at least one index indicating a magnitude of the DV offset and a direction of the DV offset, in one example, the at least one index includes a distance index indicating a magnitude of the DV offset, the magnitude being one of a set of predetermined distances, and a direction index indicating a direction of the DV offset, the direction being one of a set of predetermined directions.

[0163] In one example, the set of predetermined distances and the set of predetermined directions are used in a merged motion vector difference (MMVD) mode.

[0164] At (S1840), the encoded DV offset information may be included in the bitstream and signaled to the decoder.

[0165] In one example, the DV offset information includes a DV offset that is encoded and signaled in the bitstream.

[0166] Then, the process (1800) proceeds to (S1899) and ends.

[0167] Process 1800 can be suitably adapted to various scenarios, and the steps of process 1800 can be adjusted accordingly. One or more of the steps of process 1800 can be adapted, omitted, repeated, and / or combined. Process 1800 can be performed using any suitable order. Additional step(s) can be added.

[0168] In one embodiment, an updated displacement vector (DV) of a current block in a current picture is determined based on the DV and MVO (also referred to as DV offset) of the current block. The MVO indicates a motion offset of the DV used to adjust the position of the co-located block in the co-located reference picture. The updated DV indicates the adjusted position of the co-located block in the co-located reference picture. The current block is coded in SbTMVP mode.

[0169] SbTMVP information (e.g., motion information) of each sub-block of the plurality of sub-blocks may be derived based on at least the motion information of a corresponding sub-block in a co-located block indicated by the updated DV. The plurality of sub-blocks may be coded in SbTMVP mode based on the SbTMVP information of the plurality of sub-blocks.

[0170] FIG. 19A shows a flowchart outlining a decoding process (1900A) according to one embodiment of the present disclosure. The process (1900A) can be used in a video decoder. The process (1900A) can be performed by an apparatus for video coding, which can include receiving circuitry and processing circuitry. In various embodiments, the process (1900A) is performed by a processing circuit, such as the processing circuitry of the terminal devices (310), (320), (330), and (340), a processing circuit that performs the functions of the video encoder (403), a processing circuit that performs the functions of the video decoder (410), a processing circuit that performs the functions of the video decoder (510), a processing circuit that performs the functions of the video encoder (603), and the like. In some embodiments, the process (1900A) is implemented in software instructions, and thus the processing circuit performs the process (1900A) when the processing circuit executes the software instructions. The process begins at (S1901) and proceeds to (S1910).

[0171] At (S1910), displacement vector (DV) offset (also referred to as MV offset (MVO)) information of a current block in a current picture may be received from a coded video bitstream. The current block includes multiple sub-blocks that are reconstructed using a sub-block-based temporal motion vector prediction (SbTMVP) mode. The DV offset (or MVO) information may indicate a motion offset relative to the DV that is used to adjust the position of the co-located block in the co-located reference picture. In one example, the position of the co-located block in the co-located reference picture is adjusted by the DV offset.

[0172] In one example, the DV offset information includes a DV offset (or MVO) signaled in the coded video bitstream.

[0173] In one embodiment, the DV offset information indicates at least one index indicating a magnitude of the DV offset and a direction of the DV offset, in one example, the at least one index includes a distance index indicating a magnitude of the DV offset, the magnitude being one of a set of predetermined distances, and a direction index indicating a direction of the DV offset, the direction being one of a set of predetermined directions.

[0174] In one example, the set of predetermined distances and the set of predetermined directions are used in a merged motion vector difference (MMVD) mode.

[0175] At (S1920), an updated DV of the current block can be determined based on the DV of the current block and the DV offset of the current block. The DV offset is indicated by the DV offset information. The updated DV of the current block points to a block in the co-located reference picture. This block is considered to be co-located with the current block and is called the co-located block of the current block.

[0176] In one example, the updated DV is determined to be the vector sum of the DV and the DV offset.

[0177] In one example, the updated DV is constrained so that the co-located block is within a constrained area in the co-located reference picture, where the constrained area includes the co-located area corresponding to the current CTU in the current picture, and the current CTU includes the current block.

[0178] At (S1930), motion information of a sub-block of the plurality of sub-blocks may be determined based on motion information of a corresponding sub-block in a co-located block.

[0179] At (S1940), the sub-block of the plurality of sub-blocks may be reconstructed based on motion information of the sub-block of the plurality of sub-blocks.

[0180] The process (1900A) proceeds to (S1999) and ends.

[0181] Process 1900A can be suitably adapted to various scenarios, and the steps of process 1900A can be adjusted accordingly. One or more of the steps of process 1900A can be adapted, omitted, repeated, and / or combined. Process 1900A can be performed using any suitable order. Additional step(s) can be added.

[0182] FIG. 19B shows a flowchart outlining a decoding process (1900B) according to one embodiment of the present disclosure. Process (1900B) is a variation of decoding process (1900A). Process (1900B) can be used in a video decoder. Process (1900B) can be performed by an apparatus for video coding, which can include receiving circuitry and processing circuitry. In various embodiments, process (1900A) is performed by a processing circuit, such as the processing circuitry of terminal devices (310), (320), (330), and (340), a processing circuit that performs the functions of a video encoder (403), a processing circuit that performs the functions of a video decoder (410), a processing circuit that performs the functions of a video decoder (510), a processing circuit that performs the functions of a video encoder (603), and the like. In some embodiments, process (1900B) is implemented in software instructions, and thus, the processing circuit performs process (1900B) when the processing circuit executes the software instructions. The process starts at (S1902) and proceeds to (S1912).

[0183] At (S1912), a coded video bitstream having a current picture is received, the current picture including a current block, the current block including a plurality of sub-blocks.

[0184] At (S1922), it is determined based on syntax elements in the coded video bitstream that the current block including the plurality of sub-blocks is coded in SbTMVP mode.

[0185] At (S1932), motion vector offset (MVO) information of the current block is obtained, where MVO indicates the motion offset of a displacement vector (DV) used to adjust the position of the co-located block in the co-located reference picture.

[0186] At (S1942), an updated DV of the current block is determined based on the DV and MVO of the current block. The updated DV indicates the adjusted position of the co-located block in the co-located reference picture.

[0187] At (S1952), SbTMVP information (eg, motion information) of each sub-block of the plurality of sub-blocks is derived based at least on the motion information of the corresponding sub-block in the co-located block indicated by the updated DV.

[0188] In (S1962), the plurality of sub-blocks are reconstructed in the SbTMVP mode based on the SbTMVP information of the plurality of sub-blocks.

[0189] The process (1900B) proceeds to (S1992) and ends.

[0190] Process 1900B can be suitably adapted to various scenarios, and the steps of process 1900B can be adjusted accordingly. One or more of the steps of process 1900B can be adapted, omitted, repeated, and / or combined. Process 1900B can be performed using any suitable order. Additional step(s) can be added.

[0191] The embodiments of the present disclosure may be used separately or in combination in any order. Also, each of these methods (or embodiments), encoders, and decoders may be implemented by processing circuitry (e.g., one or more processors, or one or more integrated circuits). In one example, the one or more processors execute a program stored on a non-transitory computer-readable medium.

[0192] FIG. 20 shows a flowchart outlining an encoding process (2000) according to one embodiment of the present disclosure. The process (2000) may be used in a video encoder. The process (2000) may be performed by an apparatus for video coding, which may include processing circuitry. In various embodiments, the process (2000) is performed by processing circuitry, such as the processing circuitry of terminal devices (310), (320), (330), and (340), processing circuitry performing the functions of a video encoder (e.g., (403), (603), (703)), and the like. In some embodiments, the process (2000) is implemented in software instructions, and thus the processing circuitry performs the process (2000) when the processing circuitry executes the software instructions. The process begins at (S2001) and proceeds to (S2010).

[0193] At (S2010), a displacement vector (DV) of a current block in a current picture may be determined, for example, as described in Figures 16-17. The current block includes multiple sub-blocks coded using sub-block-based temporal motion vector prediction (SbTMVP) mode. The DV indicates a co-located block in a co-located reference picture that is co-located with the current block.

[0194] In (S2020), for example, as described in FIGS. 16-17, motion information of a sub-block among the plurality of sub-blocks may be determined based on motion information of a corresponding sub-block in a co-located block.

[0195] At (S2030), updated motion information of the sub-block among the plurality of sub-blocks may be determined based on the motion information of the sub-block among the plurality of sub-blocks and a motion vector (MV) offset of the current block.

[0196] In one example, the motion information of the sub-block of the plurality of sub-blocks includes a first motion vector (MV) associated with a first reference picture from a first reference picture list L0. An updated first MV may be determined to be a vector sum of the first MV and an MV offset. The updated motion information includes the updated first MV.

[0197] In one example, the motion information of the sub-block of the plurality of sub-blocks includes a second MV associated with a second reference picture from a second reference picture list L1. The updated second MV may be determined to be one of (i) a vector difference between the second MV and an MV offset, or (ii) a vector sum of the second MV and a scaled MV offset. The scaled MV offset may be based on the MV offset, a picture order count (POC) of the current picture, a POC of the first reference picture, and a POC of the second reference picture. The updated motion information includes the updated second MV.

[0198] At (S2040), MV offset information indicating an MV offset may be coded. The sub-block of the plurality of sub-blocks may be coded based on updated motion information. The MV offset information may be included in a bitstream.

[0199] In one example, the MV offset information includes an MV offset.

[0200] In one example, the MV offset information indicates at least one index indicating a magnitude of the MV offset and a direction of the MV offset. The at least one index includes a distance index indicating the magnitude of the MV offset, which is one of a set of predetermined distances, and a direction index indicating the direction of the MV offset, which is one of a set of predetermined directions. The set of predetermined distances and the set of predetermined directions are used in merged motion vector difference (MMVD) mode.

[0201] Then, the process (2000) proceeds to (S2099) and ends.

[0202] The process 2000 can be suitably adapted to various scenarios, and the steps of the process 2000 can be adjusted accordingly. One or more of the steps of the process 2000 can be adapted, omitted, repeated, and / or combined. Any suitable order can be used to perform the process 2000. Additional step(s) can be added.

[0203] FIG. 21 shows a flowchart outlining a decoding process (2100) according to one embodiment of the present disclosure. The process (2100) can be used in a video decoder. The process (2100) can be performed by an apparatus for video coding, which can include receiving circuitry and processing circuitry. In various embodiments, the process (2100) is performed by a processing circuit, such as the processing circuitry of the terminal devices (310), (320), (330), and (340), a processing circuit that performs the functions of the video encoder (403), a processing circuit that performs the functions of the video decoder (410), a processing circuit that performs the functions of the video decoder (510), a processing circuit that performs the functions of the video encoder (603), and the like. In some embodiments, the process (2100) is implemented in software instructions, and thus the processing circuit performs the process (2100) when the processing circuit executes the software instructions. The process begins at (S2101) and proceeds to (S2110).

[0204] At (S2110), motion vector (MV) offset information for a current block in a current picture may be received from a coded video bitstream, where the current block includes multiple sub-blocks that are reconstructed using a sub-block-based temporal motion vector prediction (SbTMVP) mode.

[0205] At (S2120), a displacement vector (DV) of the current block may be determined, for example, as described in Figures 16-17. The DV may indicate a block in a co-located reference picture that is co-located with the current block. This block may be referred to as the co-located block of the current block.

[0206] At (S2130), for example, as described in FIGS. 16-17, motion information of a sub-block among the plurality of sub-blocks may be determined based on motion information of a corresponding sub-block in a co-located block.

[0207] At (S2140), updated motion information of the sub-block among the plurality of sub-blocks may be determined based on the motion information of the sub-block among the plurality of sub-blocks and the MV offset of the current block indicated by the MV offset information.

[0208] In one example, the MV offset information includes MV offsets signaled in the coded video bitstream.

[0209] In one example, the MV offset information indicates at least one index indicating a magnitude of the MV offset and a direction of the MV offset. The at least one index includes a distance index indicating the magnitude of the MV offset, which is one of a set of predetermined distances, and a direction index indicating the direction of the MV offset, which is one of a set of predetermined directions. The set of predetermined distances and the set of predetermined directions are used in merged motion vector difference (MMVD) mode.

[0210] In one example, the motion information of the sub-block of the plurality of sub-blocks includes a first motion vector (MV) associated with a first reference picture from a first reference picture list L0. An updated first MV may be determined to be a vector sum of the first MV and an MV offset. The updated motion information includes the updated first MV.

[0211] In one example, the motion information of the sub-block of the plurality of sub-blocks includes a second MV associated with a second reference picture from a second reference picture list L1. The updated second MV may be determined to be one of (i) a vector difference between the second MV and an MV offset, or (ii) a vector sum of the second MV and a scaled MV offset. The scaled MV offset may be based on the MV offset, a picture order count (POC) of the current picture, a POC of the first reference picture, and a POC of the second reference picture. The updated motion information includes the updated second MV.

[0212] At (S2150), the sub-blocks of the plurality of sub-blocks may be reconstructed based on updated motion information.

[0213] Then, the process (2100) proceeds to (S2199) and ends.

[0214] Process 2100 can be suitably adapted to various scenarios, and the steps of process 2100 can be adjusted accordingly. One or more of the steps of process 2100 can be adapted, omitted, repeated, and / or combined. Any suitable order can be used to perform process 2100. Additional step(s) can be added.

[0215] The techniques described above can be implemented as computer software with computer-readable instructions physically stored on one or more computer-readable media. For example, Figure 22 illustrates a computer system (2200) suitable for implementing certain embodiments of the disclosed subject matter.

[0216] Computer software may be coded using any suitable machine code or computer language that can be assembled, compiled, linked, or similarly subjected to mechanisms to produce code having instructions that can be executed by one or more computer central processing units (CPUs), graphics processing units (GPUs), and the like, either directly or via interpretation, microcode execution, and the like.

[0217] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet of Things devices, and the like.

[0218] 22 with respect to computer system (2200) are exemplary in nature and are not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing embodiments of the present disclosure, nor should the arrangement of components be interpreted as having any dependency or requirement related to any one or combination of components illustrated in this exemplary embodiment of computer system (2200).

[0219] The computer system 2200 may include certain human interface input devices. Such human interface input devices may respond to input by one or more human users via, for example, tactile input (e.g., keystrokes, swipes, moving a data glove, etc.), audio input (e.g., voice, clapping, etc.), visual input (e.g., gestures, etc.), or olfactory input (not shown). The human interface devices may also be used to capture certain media that are not necessarily directly related to conscious human input, such as audio (e.g., speech, music, ambient sounds, etc.), images (e.g., scanned images, photographic images obtained from a still camera, etc.), or video (e.g., two-dimensional video, three-dimensional video including stereoscopic video, etc.).

[0220] The input human interface devices may include one or more of a keyboard (2201), a mouse (2202), a trackpad (2203), a touchscreen (2210), a data glove (not shown), a joystick (2205), a microphone (2206), a scanner (2207), and a camera (2208) (only one of each is shown).

[0221] The computer system 2200 may also include certain human interface output devices. Such human interface output devices may stimulate one or more of the human user's senses, for example, through tactile output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touchscreen 2210, data gloves (not shown), or joystick 2205, although some haptic feedback devices may not function as input devices), audio output devices (e.g., speakers 2209, headphones (not shown), etc.), visual output devices (e.g., screens 2210, including CRT screens, LCD screens, plasma screens, and OLED screens (each with or without touchscreen input capability, each with or without haptic feedback capability, some of which may be capable of outputting two-dimensional visual output or four or more dimensional output through means such as stereoscopic output), virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown), etc.), and printers (not shown).

[0222] The computer system (2200) may also include human-accessible storage devices and their associated media, such as optical media including, for example, a CD / DVD ROM / RW (2220) with a CD / DVD or similar media (2221), a thumb drive (2222), a removable hard drive or solid state drive (2223), legacy magnetic media such as tape and floppy disks (registered trademark, not shown), specialized ROM / ASIC / PLD-based devices (not shown) such as security dongles, and the like.

[0223] Those skilled in the art will also appreciate that the term "computer-readable medium" as used in connection with the subject matter disclosed herein does not include transmission media, carrier waves, or other transitory signals.

[0224] The computer system 2200 may also include an interface 2254 to one or more communications networks 2255. Networks may be, for example, wireless, wired, or optical. Networks may further be local, wide-area, metropolitan, vehicular, and industrial, real-time, latency-tolerant, and the like. Examples of networks include local area networks such as Ethernet, wireless LANs, cellular networks including GSM, 3G, 4G, 5G, LTE, and the like, TV wired or wireless wide-area digital networks including cable TV, satellite TV, and terrestrial broadcast TV, and vehicular and industrial networks including CANbus. Certain networks typically require an external network interface adapter that attaches to a particular general-purpose data port or peripheral bus 2249 (e.g., a USB port on the computer system 2200), while others are typically integrated into the core of the computer system 2200 by attachment to a system bus (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system 2200 can communicate with other entities. Such communication may be one-way receive only (e.g., broadcast TV), one-way transmit only (e.g., CANbus to a particular CANbus device), or two-way, for example, to other computer systems using local or wide-area digital networks. Specific protocols and protocol stacks may be used on each network and network interface, as described above.

[0225] The aforementioned human interface devices, human-accessible storage devices, and network interfaces can be attached to the core (2240) of the computer system (2200).

[0226] The cores (2240) may include one or more central processing units (CPUs) (2241), graphics processing units (GPUs) (2242), specialized programmable processing units in the form of field programmable gate arrays (FPGAs) (2243), task-specific hardware accelerators (2244), graphics adapters (2250), etc. These devices may be connected via a system bus (2248), along with read-only memory (ROM) (2245), random access memory (2246), internal mass storage (2247), such as an internal non-user-accessible hard drive, SSD, and the like. In some computer systems, the system bus (2248) may be made accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, and the like. Peripheral devices may be attached either directly to the core's system bus (2248) or via a peripheral bus (2249). In one example, the screen 2210 can be connected to a graphics adapter 2250. Peripheral bus architectures include PCI, USB, and the like.

[0227] The CPU (2241), GPU (2242), FPGA (2243), and accelerator (2244) may execute specific instructions that, in combination, may constitute the aforementioned computer code. The computer code may be stored in ROM (2245) or RAM (2246). Transient data may also be stored in RAM (2246), while permanent data may be stored, for example, in internal mass storage (2247). Rapid storage and retrieval from any of the memory devices may be enabled through the use of cache memory, which may be associated with one or more of the CPU (2241), GPU (2242), mass storage (2247), ROM (2245), RAM (2246), and the like.

[0228] The computer-readable media may have computer code thereon for performing various computer-implemented processes. The media and computer code may be those specially designed and constructed for the purposes of the present disclosure, or they may be of the kind well known and available to those skilled in the computer software arts.

[0229] By way of example, and not limitation, a computer system having the architecture (2200), and in particular the core (2240), can provide functionality as a result of the execution by one or more processors (including CPUs, GPUs, FPGAs, accelerators, and the like) of software embodied in one or more tangible computer-readable media. Such computer-readable media can be specific storage of the core (2240) that is non-transitory in nature, such as the core's internal mass storage (2247) or ROM (2245), and media associated with user-accessible mass storage as introduced above. Software implementing various embodiments of the present disclosure can be stored on such devices and executed by the core (2240). The computer-readable media can include one or more memory devices or chips, depending on specific needs. Software may cause the cores (2240) and particularly the processors therein (including CPUs, GPUs, FPGAs, and the like) to perform particular processes or portions of particular processes described herein, including by defining data structures stored in RAM (2246) and modifying such data structures according to processes defined by the software. Additionally, or alternatively, the computer system may provide functionality as a result of logic hardwired or otherwise embodied in circuitry (e.g., accelerators (2244)) that can operate in place of or in conjunction with software to perform particular processes or portions of particular processes described herein. References to software include logic, and vice versa, where appropriate. References to computer-readable media may include circuitry (e.g., integrated circuits (ICs)) that store software for execution, circuitry that embodies logic for execution, or both, where appropriate. The present disclosure includes any suitable combination of hardware and software. Appendix A: Acronyms JEM: joint exploration model VVC: versatile video coding BMS:benchmark set MV: Motion Vector HEVC: High Efficiency Video Coding SEI: Supplementary Enhancement Information VUI: Video Usability Information GOPs: Groups of Pictures TUs: Transform Units PU: Prediction Units CTUs: Coding Tree Units CTBs: Coding Tree Blocks PBs: Prediction Blocks HRD: Hypothetical Reference Decoder SNR: Signal-to-Noise Ratio CPUs: Central Processing Units GPUs: Graphics Processing Units CRT:Cathode Ray Tube LCD: Liquid-Crystal Display OLED: Organic Light-Emitting Diode CD: Compact Disc DVD: Digital Video Disc ROM: Read-Only Memory RAM: Random Access Memory ASIC: Application-Specific Integrated Circuit PLD: Programmable Logic Device LAN: Local Area Network GSM: Global System for Mobile communications LTE: Long-Term Evolution CANBus: Controller Area Network Bus USB: Universal Serial Bus PCI: Peripheral Component Interconnect FPGA: Field Programmable Gate Array SSD: solid-state drive IC: Integrated Circuit CU: Coding Unit

[0230] While this disclosure describes several exemplary embodiments, there are alterations, permutations, and various equivalent alternatives that fall within the scope of the disclosure. It will thus be appreciated that those skilled in the art will be able to devise numerous systems and methods that, although not explicitly shown or described herein, embody the principles of the disclosure and are therefore within its spirit and scope.

Claims

1. 1. A method of video decoding performed by a decoder, comprising: receiving a coded video bitstream having a current picture, the current picture including a current block, the current block including a plurality of sub-blocks; determining, based on a syntax element in the coded video bitstream, that the current block including the plurality of sub-blocks is coded in a sub-block-based temporal motion vector prediction (SbTMVP) mode; obtaining, from the coded video bitstream, motion vector offset (MVO) information of the current block, indicating a motion vector offset (MVO), the MVO indicating a motion offset of a displacement vector (DV) used to adjust a position of a co-located block in a co-located reference picture; determining an updated DV of the current block based on the DV and the MVO of the current block, where the updated DV indicates an adjusted position of the co-located block within the co-located reference picture; deriving SbTMVP information for each sub-block of the plurality of sub-blocks based at least on motion information of a corresponding sub-block within the co-located block indicated by the updated DV; reconfiguring the plurality of sub-blocks in the SbTMVP mode based on the SbTMVP information of the sub-blocks of the plurality of sub-blocks; A method having the following.

2. The step of determining the updated DV comprises: determining the updated DV to be the vector sum of the DV and the MVO; 2. The method of claim 1, comprising:

3. the MVO information includes the MVO signaled in the coded video bitstream; The method of claim 1.

4. The MVO information indicates at least one index indicating the magnitude and direction of the MVO. The method of claim 1.

5. the at least one index includes a distance index indicating the magnitude of the MVO, the distance index being one of a set of predetermined distances, and a direction index indicating the direction of the MVO, the direction being one of a set of predetermined directions. The method of claim 4.

6. the set of predetermined distances and the set of predetermined directions are used in a merge motion vector difference (MMVD) mode; The method of claim 5.

7. the updated DV is constrained such that the co-located block is within a constrained area in the co-located reference picture, the constrained area including a co-located area corresponding to a current CTU in the current picture, and the current CTU including the current block; The method of claim 1.

8. 1. A method of video decoding performed by a decoder, comprising: receiving motion vector (MV) offset information for a current block in a current picture from a coded video bitstream, the current block including a plurality of sub-blocks to be reconstructed using a sub-block-based temporal motion vector prediction (SbTMVP) mode; determining a displacement vector (DV) of the current block that indicates a co-located block in a co-located reference picture that is co-located with the current block; determining motion information of a sub-block among the plurality of sub-blocks based on motion information of a corresponding sub-block in the co-located block; determining updated motion information of the sub-block among the plurality of sub-blocks based on the motion information of the sub-block among the plurality of sub-blocks and the motion vector offset of the current block indicated by the motion vector offset information; reconstructing the sub-blocks of the plurality of sub-blocks based on the updated motion information; A method having the following.

9. the MV offset information includes the MV offset signaled in the coded video bitstream; The method of claim 8.

10. The MV offset information indicates at least one index indicating the magnitude and direction of the MV offset. The method of claim 8.

11. the at least one index includes a distance index indicating the magnitude of the MV offset, the distance being one of a set of predetermined distances, and a direction index indicating the direction of the MV offset, the direction being one of a set of predetermined directions; The method of claim 10.

12. the set of predetermined distances and the set of predetermined directions are used in a merge motion vector difference (MMVD) mode; The method of claim 11.

13. the motion information for the sub-block of the plurality of sub-blocks includes a first motion vector (MV) associated with a first reference picture from a first reference picture list L0; the determining the updated motion information includes determining an updated first MV that is a vector sum of the first MV and the MV offset, and the updated motion information includes the updated first MV. The method of claim 8.

14. the motion information of the sub-block of the plurality of sub-blocks includes a second MV associated with a second reference picture from a second reference picture list L1; The step of determining the updated motion information comprises: (i) the vector difference between the second MV and the MV offset; or (ii) a vector sum of the second MV and a scaled MV offset, the scaled MV offset being based on the MV offset, a Picture Order Count (POC) of the current picture, a POC of the first reference picture, and a POC of the second reference picture; determining an updated second MV, the second MV being one of: The method of claim 13.

15. one or more processors; one or more memories storing a computer program; and The computer program causes the one or more processors to perform the method of any one of claims 1 to 14. Device.

16. A computer program causing a computer to carry out the method according to any one of claims 1 to 14.

Images