Method, apparatus, and program for video decoding, and method for video encoding

Abstract

This provides methods for video encoding / decoding, etc. [Solution] A video bitstream is received. The video bitstream includes a current block having multiple subblocks and a template region of the current block having multiple template subblocks adjacent to at least one of the upper and left sides of the current block. A motion vector (MV) located at the center of the current block is determined. The MV is determined based on at least one MV of the multiple subblocks of the current block. The MV of each of the multiple template subblocks is determined based on the MV located at the center of the current block and the MV of each corresponding subblock among the multiple subblocks adjacent to the respective template subblock. The current block is reconstructed based on the determined MVs of the multiple template subblocks.

Description

[Technical Field]

[0001] This disclosure describes embodiments that generally relate to video coding. [Background technology]

[0002] The background information provided herein is for general purposes only. The research of the currently designated inventors is not considered prior art to this disclosure, either explicitly or implicitly, to the extent described in the background section, and any aspects of the description that would otherwise not qualify as prior art at the time of filing are not considered prior art to this disclosure.

[0003] Image / video compression can help transmit image / video data between different devices, storage, and networks with minimal quality degradation. In some cases, video codec technology can compress video based on spatial and temporal redundancy. In one example, a video codec can use a technique called intra-prediction, which can compress images based on spatial redundancy. For example, intra-prediction can use reference data from the currently reconstructed picture for sample prediction. In another example, a video codec can use a technique called inter-prediction, which can compress images based on temporal redundancy. For example, inter-prediction can predict samples in the current picture from a previously reconstructed picture using motion compensation. Motion compensation can be indicated by motion vectors (MV). [Overview of the project]

[0004] Aspects of this disclosure include methods and apparatus for video coding / decoding. In some examples, the apparatus for video decoding includes processing circuits.

[0005] A method for video decoding performed by a video decoder is provided according to an aspect of this disclosure. In the method, a video bitstream is received. The video bitstream includes a current block having a plurality of subblocks and a template region of the current block having a plurality of template subblocks adjacent to at least one of the upper and left sides of the current block. A motion vector (MV) located at the center of the current block is determined. The MV is determined based on at least one MV of the plurality of subblocks of the current block. The MV of each of the plurality of template subblocks is determined based on the MV located at the center of the current block and the MV of each of the corresponding subblocks among the plurality of subblocks adjacent to the respective template subblock. The current block is reconstructed based on the determined MVs of the plurality of template subblocks.

[0006] In the example, the MV currently located at the center of the block is determined to be an MV originating from one of the top-left subblocks, bottom-left subblocks, top-right subblocks, and bottom-right subblocks of the current block.

[0007] In the example, the MV currently located at the center of the block is determined to be the MV from one of several subblocks, and one of the several subblocks is selected based on either the prediction mode or the median sample value of that particular subblock.

[0008] In the example, the MV currently located at the center of a block is determined as the average of a subset of the MVs of multiple subblocks.

[0009] In one aspect, the MV of each of the multiple template subblocks is determined as a single-prediction MV based on the fact that the MV currently located at the center of the block is a single-prediction MV. In another aspect, the MV of each of the multiple template subblocks is determined as a bi-prediction MV based on the fact that the MV currently located at the center of the block is a bi-prediction MV.

[0010] In the example, based on (i) the MV of the first subblock among multiple subblocks is a one-predicted MV in the first reference list, (ii) the MV of the first template subblock among multiple template subblocks adjacent to the first subblock is a one-predicted MV in the second reference list, and (iii) the MV currently located at the center of the block is a one-predicted MV in the second reference list, the MV of the first template subblock is determined to be the MV currently located at the center of the block.

[0011] In the example, based on (i) the MV of the first subblock among multiple subblocks is a one-predictive MV in the first reference list, (ii) the MV of the first template subblock among multiple template subblocks adjacent to the first subblock is a two-predictive MV, and (iii) the MV currently located at the center of the block is a two-predictive MV containing the first component in the first reference list and the second component in the second reference list, it is determined that the MV of the first template subblock contains the MV of the first subblock in the first reference list and the second component of the MV currently located at the center of the block in the second reference list.

[0012] In the example, based on the fact that the MV currently located at the center of the block is a one-sided predicted MV, the MV of each of the multiple template subblocks is determined as the MV of the subblock that is adjacent to that template subblock.

[0013] In the example, based on the fact that the MV currently located at the center of the block is a bipredicted MV, the MV of each of the multiple template subblocks is determined as the MV of the subblock that is adjacent to that template subblock.

[0014] In the example, based on (i) the MV currently located at the center of the block is a one-predicted MV in the first reference list, and (ii) the MV of the first subblock among the multiple subblocks adjacent to the first template subblock is a one-predicted MV in the second reference list, the MV of the first template subblock is determined to be a two-predicted MV that includes the MV currently located at the center of the block in the first reference list and the MV of the first subblock in the second reference list.

[0015] In the example, to reconstruct the current block, the referenced block of the current block is determined based on the difference between the template region of the referenced block and the template region of the current block, and the template region of the referenced block is indicated by the MV of multiple template subblocks. Each of the multiple subblocks is further reconstructed based on the respective subblocks of the referenced block.

[0016] Apparatus is provided according to other aspects of this disclosure. Apparatus includes processing circuitry. Processing circuitry may be configured to perform any of the described methods of video decoding / encoding.

[0017] Aspects of this disclosure also provide non-temporary computer-readable media that, when executed by a computer, stores instructions causing a computer to perform a video decoding / encoding method.

[0018] Further features, properties, and various advantages of the disclosed subject matter will become clearer from the detailed description below and the accompanying drawings. [Brief explanation of the drawing]

[0019] [Figure 1] This is a schematic diagram of an exemplary block diagram of a video processing system (100). [Figure 2] This is a schematic diagram of an exemplary block diagram of a decoder. [Figure 3] This is a schematic diagram of an example block diagram of an encoder. [Figure 4] An exemplary spatial proximity block used for temporal motion vector prediction (TMVP) is shown. [Figure 5] It is a schematic diagram of a sub-block based TMVP (SbTMVP) process. [Figure 6] An exemplary block coded by SbTMVP is shown. [Figure 7] According to some embodiments of the present disclosure, a first example of a sub-block based template matching process for SbTMVP is shown. [Figure 8] According to some embodiments of the present disclosure, a second example of a sub-block based template matching process for SbTMVP is shown. [Figure 9] According to some embodiments of the present disclosure, a third example of a sub-block based template matching process for SbTMVP is shown. [Figure 10] According to some embodiments of the present disclosure, a fourth example of a sub-block based template matching process for SbTMVP is shown. [Figure 11] According to some embodiments of the present disclosure, a fifth example of a sub-block based template matching process for SbTMVP is shown. [Figure 12] A flowchart explaining a decoding process according to some embodiments of the present disclosure is shown. [Figure 13] A flowchart explaining an encoding process according to some embodiments of the present disclosure is shown. [Figure 14] It is a schematic diagram of an exemplary computer system according to an embodiment.

BEST MODE FOR CARRYING OUT THE INVENTION

[0020] Figure 1 shows a block diagram of a video processing system (100) in several examples. The video processing system (100) is an example of the application of the disclosed subject, namely a video encoder and video decoder in a streaming environment. The disclosed subject can similarly be applied to other video-enabled applications, including, for example, video conferencing, digital TV, streaming services, and storage of compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0021] The video processing system (100) includes a capture subsystem (113) which may include a video source (101), such as a digital camera, that generates a stream (102) of uncompressed video pictures. For example, the stream (102) of video pictures includes a sample captured by the digital camera. The stream (102) of video pictures is represented by a thick line to highlight its higher data volume compared to encoded video data (104) (or encoded video bitstream) and may be processed by an electronic device (120) which includes a video encoder (103) coupled to the video source (101). The video encoder (103) may include hardware, software, or a combination thereof to enable or implement the aspects of the subject disclosed, as will be described in more detail below. The encoded video data (104) (or encoded video bitstream) is represented by a thin line to highlight its lower data volume compared to the stream (102) of video pictures and may be stored in a streaming server (105) for future use. One or more streaming client subsystems, such as client subsystems (106) and (108) in Figure 1, can access a streaming server (105) to read copies (107) and (109) of the encoded video data (104). Client subsystem (106) may include a video decoder (110), for example, in an electronic device (130). The video decoder (110) decodes the incoming copy (107) of the encoded video data and generates an output stream (111) of a video picture that can be rendered on a display (112) (e.g., a display screen) or other rendering device (not shown). In some streaming systems, the encoded video data (104), (107), and (109) (e.g., video bitstreams) may be encoded according to a specific video coding / compression standard. An example of such a standard is ITU-T Recommendation H.265.For example, a video coding standard under development is commonly known as Versatile Video Coding (VVC). The disclosed material may be used in connection with VVC.

[0022] Electronic devices (120) and (130) may include other components (not shown). For example, electronic device (120) may include a video decoder (not shown), and electronic device (130) may similarly include a video encoder (not shown).

[0023] Figure 2 shows an exemplary block diagram of a video decoder (210). The video decoder (210) may be contained within an electronic device (230). The electronic device (230) may include a receiver (231) (e.g., a receiving circuit). The video decoder (210) may be used in place of the video decoder (110) in the example of Figure 1.

[0024] The receiver (231) may receive one or more coded video sequences, for example, contained in a bitstream, to be decoded by the video decoder (210). In embodiments, one coded video sequence may be received at a time, and the decoding of each coded video sequence is independent of the decoding of other coded video sequences. The coded video sequences may be received from a channel (201), which may be a hardware / software link to a storage device storing coded video data. The receiver (231) may receive coded video data together with other data, such as coded audio data and / or auxiliary data streams, which may be forwarded to their respective usage entities (not shown). The receiver (231) may isolate the coded video sequences from other data. To combat network jitter, a buffer memory (215) may be coupled between the receiver (231) and the entropy decoder / parser (220) (hereinafter "parser (220)"). In certain applications, the buffer memory (215) is part of the video decoder (210). Otherwise, it can be outside the video decoder (210) (not shown). Further variations may include a buffer memory outside the video decoder (210) (not shown), for example, to counter network jitter, and another buffer memory (215) within the video decoder (210), for example, to manipulate playback timing. When the receiver (231) is receiving data from a sufficiently bandwidth and controllable storage / transfer device or from an isosynchronous network, the buffer memory (215) may not be necessary or may be small. For use in best-effort packet networks such as the Internet, the buffer memory (215) may be necessary, may be relatively large, and advantageously, may be adaptively sized, and may be implemented at least partially outside the video decoder (210) in an operating system or similar element (not shown).

[0025] The video decoder (210) may include a parser (220) for reconstructing symbols (221) from the coded video sequence. The categories of these symbols include information used to manage the operation of the video decoder (210) and information for controlling rendering devices, such as a render device (212) (e.g., a display screen), which are not essential parts of the electronic device (230) but can be coupled to the electronic device (230), as shown in Figure 2. The control information for rendering devices may take the form of Supplemental Enhancement Information (SEI) messages or Video Usability Information (VUI) parameter set fragments (not shown). The parser (220) can parse / entropy decode the received coded video sequence. The coding of the coded video sequence may follow video coding techniques or standards and may follow various principles, including variable-length coding, Huffman coding, context-dependent or independent arithmetic coding, etc. The parser(220) may extract from the coded video sequence a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder, based on at least one parameter corresponding to that group. Subgroups may include groups of pictures (GOP), pictures, tiles, slices, macroblocks, coding units (CU), blocks, transform units (TU), predictive units (PU), and so on. The parser(220) may also extract information from the coded video sequence such as transform coefficients, quantization parameter values, and motion vectors.

[0026] The parser (220) may perform an entropy decoding / parsing operation on the video sequence received from the buffer memory (215) in order to generate symbols (221).

[0027] The reconstruction of symbol (221) can have a number of different units depending on the type of coded video picture or part thereof (e.g., inter and intra pictures, inter and intra blocks) and other factors. How each unit is included may be controlled by subgroup control information parsed by parser (220) from the coded video sequence. The flow of such subgroup control information between parser (220) and the following multiple units is not shown for clarity.

[0028] Beyond the functional blocks already described, the video decoder (210) can be conceptually subdivided into numerous functional units, as described below. In actual implementations operating under commercial constraints, many of these units may interact closely with each other and be at least partially integrated with one another. However, for the purpose of illustrating the subject matter disclosed, the conceptual subdivision into functional units described below is appropriate.

[0029] The first unit is the scaler / inverse unit (251). The scaler / inverse unit (251) receives control information from the parser (220) as symbols (221), along with the quantized transformation coefficients, including which transformation to use, block size, quantization coefficients, and quantization scaling matrix. The scaler / inverse unit (251) can output a block containing sample values ​​that can be input to the aggregator (255).

[0030] In some cases, the output samples of the scaler / inverse converter (251) may relate to intracoded blocks. Intracoded blocks are blocks that do not use predictive information from previously reconstructed pictures but can use predictive information from portions reconstructed before the current picture. Such predictive information may be supplied by an intrapicture predictive unit (252). In some cases, the intrapicture predictive unit (252) generates a block of the same size and shape as the block being reconstructed, using surrounding already reconstructed information fetched from the current picture buffer (258). The current picture buffer (258) buffers, for example, partially reconstructed current pictures and / or fully reconstructed current pictures. In some cases, the aggregator (255) adds the predictive information generated by the intra predictive unit (252) to the output sample information supplied by the scaler / inverse converter (251), sample by sample.

[0031] In other cases, the output samples of the scaler / inverse unit (251) may relate to an interconnected and potentially motion-compensated block. In such cases, the motion-compensated prediction unit (253) may access the reference picture memory (257) to fetch samples to be used for prediction. After the fetched samples have been motion-compensated according to the symbols (221) relating to the block, those samples may be added by the aggregator (255) to the output of the scaler / inverse unit (251) (referred to in this case to residual samples or residual signal) to generate output sample information. The address in the reference picture memory (257) from which the motion-compensated prediction unit (253) fetches the predicted samples may be controlled by a motion vector that can be made available to the motion-compensated prediction unit (253) in the form of a symbol (221) which may have, for example, X, Y and reference picture components. Motion compensation may also include interpolation of sample values ​​fetched from reference picture memory (257) when the precise motion vectors of subsamples are used, as well as motion vector prediction mechanisms.

[0032] The output samples of the aggregator (255) can undergo various loop filtering techniques in the loop filter unit (256). Video compression techniques may include in-loop filtering techniques, which are included in the coded video sequence (also called the coded video bitstream) and controlled by parameters made available to the loop filter unit (256) as symbols (221) from the parser (220). Video compression may also respond to metadata obtained during the decoding of previous parts (in the decoding order) of the coded picture or coded video sequence, and further, to previously configured loop-filtered sample values.

[0033] The output of the loop filter unit (256) can be a sample stream that is output to the render device (212) and can also be stored in reference picture memory (257) for use in future interpicture prediction.

[0034] A particular coded picture, once fully reconfigured, may be used as a reference picture for future predictions. For example, once the coded picture corresponding to the current picture is fully reconfigured and the coded picture is identified as a reference picture (e.g., by the parser (220)), the current picture buffer (258) may become part of the reference picture memory (257), and any unused current picture buffer may be reallocated before the reconfiguration of subsequent coded pictures begins.

[0035] The video decoder (210) may perform decoding operations in accordance with a specified video compression technique or standard, such as the ITU-T recommended H.265. The coded video sequence may conform to the syntax defined by the video compression technique or standard in use, in the sense that the coded video sequence conforms to both the syntax of the video compression technique or standard and the profile documented in the video compression technique or standard. Specifically, a profile may select a particular tool from all the tools available in the video compression technique or standard as the only tool available for use under that profile. Furthermore, the complexity of the coded video sequence must be within the boundaries defined by the level of the video compression technique or standard in order to comply. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples / second), maximum reference picture size, etc. The limits set by the level may, in some cases, be further limited through the Hypothetical Reference Decoder (HRD) specification and metadata for HRD buffer management notified in the coded video sequence.

[0036] In an embodiment, the receiver (231) may receive additional (redundant) data along with the encoded video. The additional data may also be included as parts of the coded video sequence. The additional data may be used by the video decoder (210) to properly decode the data and / or to more accurately reconstruct the original video data. The additional data may take the form of, for example, a time, space, or signal-to-noise ratio (SNR) enhancement layer, redundant slices, redundant pictures, forward error correction codes, etc.

[0037] Figure 3 shows an exemplary block diagram of a video encoder (303). The video encoder (303) is contained within an electronic device (320). The electronic device (320) includes a transmitter (340) (e.g., a transmitting circuit). The video encoder (303) can be used as a replacement for the video encoder (303) in the example of Figure 1.

[0038] The video encoder (303) may receive video samples from a video source (301) (not part of the electronic device (320) in the example in Figure 3) that can capture video images to be coded by the video encoder (303). In other examples, the video source (301) is part of the electronic device (320).

[0039] The video source (301) may supply a source video sequence to be coded by the video encoder (303) in the form of a digital video sample stream, which can have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, etc.), any color space (e.g., BT.601 YCrCB, RGB, etc.), and any suitable sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media serving system, the video source (301) may be a storage device storing pre-prepared video. In a video conferencing system, the video source (301) may be a camera that captures local image information as a video sequence. The video data may be supplied as a series of individual pictures that give motion when viewed sequentially. The picture itself may be organized as a spatial array of pixels, each pixel may have one or more samples depending on the sampling structure, color space, etc., in use. This specification will focus on samples below.

[0040] According to the embodiment, the video encoder (303) can encode and compress pictures of a source video sequence into a coded video sequence (343) in real time or under any other time constraints required. Enforcing an appropriate coding speed is a function of the controller (350). In some embodiments, the controller (350) controls and is functionally coupled to other functional units, such as those described below. Couplings are not shown for clarity. Parameters set by the controller (350) may include parameters related to rate control (e.g., picture skip, quantizer, lambda value of rate distortion optimization technique), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (350) may be configured to have other appropriate functions related to the video encoder (303) optimized for a particular system design.

[0041] In some embodiments, the video encoder (303) is configured to operate in a coding loop. In an overly simplified description, in the example, the coding loop may include a source coder (330) (which, for example, is involved in generating symbols, such as a symbol stream, based on an input picture to be coded and a reference picture) and a (local) decoder (333) embedded in the video encoder (303). The decoder (333) reconstructs the symbols to generate sample data in a similar manner to how a (remote) decoder would also generate. The reconstructed sample stream (sample data) is fed into a reference picture memory (334). Since the decoding of the symbol stream yields a bit-exact result independent of the decoder's location (local or remote), the contents within the reference picture memory (334) are also bit-exact between the local and remote encoders. In other words, the predictive portion of the encoder "sees" the same sample values ​​as reference picture samples that the decoder would "see" when using predictions during decoding. This fundamental principle of reference picture synchronization (and the resulting drift when synchronization cannot be maintained, for example, due to channel errors) is also used in several related technologies.

[0042] The operation of the “local” decoder (333) can be the same as that of the “remote” decoder, such as the video decoder (210), which has already been described in detail above with reference to Figure 2. Referring temporarily to Figure 2, however, the entropy decoding portion of the video decoder (210), including the buffer memory (215) and parser (220), does not have to be fully implemented in the local decoder (233), provided that symbols are available and the encoding / decoding of symbols to the coded video sequence by the entropy coder (345) and parser (220) can be reversible.

[0043] In embodiments, decoder techniques, excluding the parsing / entropy decoding present in the decoder, exist in the corresponding encoder in the same or substantially the same functional form. Therefore, the disclosed subject matter focuses on the operation of the decoder. Descriptions of encoder techniques can be omitted, as they are the reverse of the decoder techniques, which are comprehensively described. In certain scopes, more detailed descriptions are given below.

[0044] During operation, in some examples, the source coder (330) may perform motion-compensated predictive coding. This predictively codes the input picture by referencing one or more previously coded pictures from a video sequence designated as “reference pictures”. In this way, the coding engine (332) codes the difference between the pixel blocks of the reference picture that may be selected as predictive references for the input picture and the pixel blocks of the input picture.

[0045] A local video decoder (333) can decode coded video data of a picture that may be designated as a reference picture, based on symbols generated by the source coder (330). The operation of the coding engine (332) can, advantageously, be an irreversible process. When coded video data can be decoded by a video decoder (not shown in Figure 3), the reconstructed video sequence is typically a copy of the source video sequence with some errors. The local video decoder (333) may replicate the decoding process that the video decoder may perform on the reference picture, so that the reconstructed reference picture is stored in the reference picture cache (334). In this way, the video encoder (303) can locally store a copy of the reconstructed reference picture that has content in common with the reconstructed reference picture that will be obtained by the far-end video decoder (without transmission errors).

[0046] The predictor (335) may perform a predictive search for the coding engine (332). That is, for a new picture to be coded, the predictor (335) may search the reference picture memory (334) for specific metadata or sample data (as candidate reference pixel blocks), such as reference picture motion vectors, block shapes, etc., which may serve as appropriate predictive criteria for the new picture. The predictor (335) may operate on a sample block-by-pixel block basis to find appropriate predictive criteria. In some cases, the input picture may have predictive criteria drawn from multiple reference pictures stored in the reference picture memory (334), as determined by the search results obtained by the predictor (335).

[0047] The controller (350) may manage the coding operation of the source coder (330), including, for example, setting parameters and subgroup parameters used to encode video data.

[0048] The outputs of all the above functional units can undergo entropic coding in the entropicorder (345). The entropicorder (345) converts the symbols generated by the various functional units into coded video sequences by lossless compression of the symbols according to techniques such as Huffman coding, variable-length coding, and arithmetic coding.

[0049] The transmitter (340) may buffer the coded video sequence generated by the entropicorder (345) to prepare it for transmission over the communication channel (360). The communication channel (360) may be a hardware / software link to a storage device that stores the coded video data. The transmitter (340) may merge the coded video data from the videocoder (303) with other data to be transmitted, such as coded audio data and / or auxiliary data streams (source not shown).

[0050] The controller (350) may manage the operation of the video encoder (303). During coding, the controller (350) may assign each coded picture to a specific coding picture type that may affect the coding techniques that may be applied to each picture. For example, a picture may often be assigned as one of the following picture types:

[0051] An intra-picture (I-picture) can be encoded and decoded without using any other pictures in the sequence as a source for prediction. Some video codecs allow various types of intra-pictures, including, for example, independent decoder refresh (IDR) pictures.

[0052] A predictive picture (P-picture) can be encoded and decoded by intra-prediction or inter-prediction, which uses motion vectors and reference indices to predict the sample values ​​of each block.

[0053] A bidirectionally predictive picture (B-picture) can be encoded and decoded using intra-prediction or inter-prediction, which uses two motion vectors and a reference index to predict the sample values ​​for each block. Similarly, multiple-predictive picture(s) can use more than two reference pictures and associated metadata for the reconstruction of a single block.

[0054] A source picture can generally be spatially subdivided into multiple sample blocks (e.g., 4x4, 8x8, 4x8, or 16x16 sample blocks, respectively), and each block can be coded. Blocks can be predictively coded by referencing other (already coded) blocks, determined by the coding assignment applied to each picture in the block. For example, blocks in picture I may be coded non-predictively, or they may be coded predictively by referencing already coded blocks of the same picture (spatial prediction or intra-prediction). Pixel blocks in picture P may be coded predictively by spatial prediction or temporal prediction by referencing one previously coded reference picture. Blocks in picture B may be coded predictively by spatial prediction or temporal prediction by referencing one or two previously coded reference pictures.

[0055] The video encoder (303) may perform coding operations in accordance with a predetermined video coding technique or standard, such as ITU-T Recommended H.265. During these operations, the video encoder (303) may perform various compression operations, including predictive coding operations that utilize temporal and spatial redundancy in the input video sequence. Thus, the coded video data may conform to the syntax defined by the video coding technique or standard being used.

[0056] In an embodiment, the transmitter (340) may transmit additional data along with the encoded video. The source coder (330) may include such data as part of the coded video sequence. The additional data may include time / space / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, SEI messages, VUI parameter set fragments, and the like.

[0057] Video can be captured as multiple source pictures (video pictures) in a time sequence. Intra-picture prediction (often abbreviated as intra-prediction) utilizes spatial correlations within a given picture, while inter-picture prediction utilizes (temporal or other) correlations between pictures. For example, a particular picture being encoded / decoded, called the current picture, is partitioned into blocks. If a block within the current picture is similar to a reference block in a previously coded, still-buffering reference picture within the video, that block in the current picture may be coded by a vector called a motion vector. The motion vector points to a reference block in the reference picture and may have a third dimension to identify the reference picture if multiple reference pictures are used.

[0058] In some embodiments, a dual prediction technique may be used in interpicture prediction. According to the dual prediction technique, two reference pictures are used, for example, a first reference picture and a second reference picture, both of which precede the current picture in the video in the decoding order (but may be past and future, respectively, in the display order). A block in the current picture may be coded by a first motion vector pointing to a first reference block in the first reference picture and a second motion vector pointing to a second reference block in the second reference picture. That block is predictable by a combination of the first and second reference blocks.

[0059] Furthermore, merge mode techniques can be used in interpicture prediction to improve coding efficiency.

[0060] In some embodiments of this disclosure, predictions such as interpicture prediction and intrapicture prediction are performed in units of blocks. For example, according to the HEVC standard, pictures in a sequence of video pictures are partitioned into coding tree units (CTUs) for compression, and the CTUs in a picture have the same size, such as 64x64 pixels, 32x32 pixels, or 16x16 pixels. Generally, a CTU contains three coding tree blocks (CTBs), which are one lumen CTB and two chroma CTBs. Each CTU can be recursively quadtree-partitioned into one or more coding units (CUs). For example, a 64x64 pixel CTU can be divided into one 64x64 pixel CU, or four 32x32 pixel CUs, or sixteen 16x16 pixel CUs. As an example, each CU is parsed to determine a prediction type for the CU, such as an inter-prediction type or an intra-prediction type. A CU is divided into one or more prediction units (PUs) depending on its temporal and / or spatial predictability. Generally, each PU contains one lumar prediction block (PB) and two chroma PBs. In embodiments, the prediction operation in coding (encoding / decoding) is performed in units of prediction blocks. Using a lumar prediction block as an example of a prediction block, the prediction block contains a matrix of pixel values ​​(e.g., lumar values), such as 8x8 pixels, 16x16 pixels, 8x16 pixels, 16x8 pixels, etc.

[0061] The video encoders (103) and (303) and the video decoders (110) and (210) can be implemented by any suitable technology. In one embodiment, the video encoders (103) and (203) and the video decoders (110) and (210) can be implemented using one or more integrated circuits. In another embodiment, the video encoders (103) and (203) and the video decoders (110) and (210) can be implemented using one or more processors that execute software instructions.

[0062] This disclosure includes aspects relating to motion vector (MV) derivation, such as the derivation of MVs of subblock templates in the current template based on subblock-based template matching in subblock-based temporal motion vector prediction (SbTMVP) mode.

[0063] To improve coding efficiency and reduce motion vector transmission overhead, subblock-level motion vector refinement can be applied to extend CU-level temporal motion vector prediction (TMVP). Subblock-based TMVP (SbTMVP) can enable the inheritance of subblock-level motion information from collocated reference pictures. Each subblock of a large CU can have its own motion information without explicitly transmitting block partition structure or motion information. In an example, SbTMVP can obtain motion information for each subblock in three steps. In the first step, the displacement vector (DV) of the current CU may be derived. In the second step, the availability of SbTMVP candidates may be checked and the motion of the center may be derived. In the third step, subblock motion information may be derived from the corresponding subblock by the DV. Unlike TMVP candidate derivation, which can derive temporal motion vectors from juxtaposed blocks within a reference frame, SbTMVP can apply a DV, which can be derived from the MV of the left-adjacent CU of the current CU, to find the corresponding subblock in the juxtaposed picture for each subblock of the current CU. If the corresponding subblock is not encoded, the motion information of the current subblock may be set as the central motion.

[0064] SbTMVP can be supported by codecs such as VVC. Similar to TMVP provided by HEVC, SbTMVP can apply a playground to juxtaposed pictures to improve the merge mode and motion vector prediction of CUs within the picture. The juxtaposed pictures used by TMVP can also be used with SbTMVP. SbTMVP may differ from TMVP in the following two main aspects: (1) TMVP predicts motion at the CU level, while SbTMVP predicts motion at the sub-CU level; (2) TMVP can fetch temporal motion vectors from the juxtaposed blocks in the juxtaposed picture (for example, the juxtaposed block may be the lower right block or the center block relative to the current CU). SbTMVP can apply motion shifts before the temporal motion information is fetched from the juxtaposed picture, and motion shifts may be obtained from motion vectors from one of the spatially adjacent blocks of the current CU.

[0065] An example SbTMVP process may be shown in Figures 4 and 5. Figure 4 shows exemplary spatial neighbor blocks (e.g., A0, A1, B0, and B1) of the current block (402) used in TMVP. Figure 5 shows an exemplary SbTMVP process (500). As shown in Figure 5, the current CU (502) may be contained in the current picture (504). The current CU (502) contains multiple sub-CUs (e.g., (506)). The current picture (504) may correspond to a juxtaposed picture (508). In the example, SbTMVP can predict the motion vectors of sub-CUs (e.g., (506)) within the current CU (502) in two steps. In the first step, spatial neighbors of the current CU (502), such as A1, may be tested. Exemplary candidate spatial neighbors applied to the SbTMVP process (500) may be shown in Figure 4. If a spatial neighbor such as A1 has a motion vector (510) that uses the juxtaposed picture (508) as a reference picture, the motion vector (510) may be selected as the motion shift (or displacement vector) for the SbTMVP process (500). If no such motion vector is specified, the motion shift may be set to (0,0).

[0066] In the second step, the motion shift identified in the first step (e.g., (510)) is applied and, for example, added to the coordinates of the current CU (502) to obtain sub-CU level motion information (e.g., motion vector and reference index) from the juxtaposed picture (508). As shown in Figure 5, a reference block A1' in the juxtaposed picture (508) can be identified according to the motion shift derived based on the motion vector (510) of the spatial neighbor A1. The reference block A1' can correspond to a reference CU (512) in the juxtaposed picture (508). Thus, for each sub-CU (e.g., (506)) of the current CU (502), the motion information of the corresponding block (or corresponding sub-CU (e.g., (514))) in the reference CU (512) of the juxtaposed picture (508) can be used to derive the motion information of the sub-CU (e.g., (506)). After the motion information of the juxtaposed subCU (e.g., (514)) is identified, the motion information can be converted to the motion vector and reference index of the current subCU (e.g., (506)) in a manner similar to the HEVC TMVP process. At this point, temporal motion scaling may be applied to align the temporal motion vector of the reference picture (508) with the temporal motion vector of the current CU (502).

[0067] A composite subblock-based merge list can include both SbTMVP candidates and affine merge candidates and can be used in subblock-based merge mode. SbTMVP can be enabled / disabled by the Sequence Parameter Set (SPS) flag. When SbTMVP mode is enabled, an SbTMVP predictor is added as the first entry in the list of subblock-based merge candidates, followed by affine merge candidates. The size of the subblock-based merge list can be signaled by the SPS, and the maximum allowable size of a subblock-based merge list can be 5, for example in VVC.

[0068] As in VVC, the sub-CU size used in SbTMVP may be fixed at 8x8. Similar to affine merge mode, SbTMVP mode can be applied to CUs where both width and height are 8 or greater. Sub-block (or sub-CU) sizes may be explored beyond VVC. For example, in ECM, the sub-CU size can be set to other sizes such as 4x4. Two juxtaposed pictures, or frames, may be used to provide temporal motion information for SbTMVP and TMVP in AMVP mode.

[0069] To obtain a better (or improved) matching, a signaled extra motion vector offset (MVO) may be added to the displacement motion vector (DV). MVO(x o ,y o The position of the MV field of the juxtaposed CU can be adjusted by using MVO(x o ,y o When ) is not a zero motion offset, DV', which can be the sum of DV and MVO, can be used as a displacement vector to indicate the position of the juxtaposed CU in order to derive SbTMVP.

[0070] In a related example, DV can be used as the motion vector for template matching for SbTMVP. However, DV for SbTMVP is configured to point to the position of the playground in the juxtaposed reference picture. Therefore, using DV in template matching can be less reliable because DV may not be used as the motion vector for the current CU in SbTMVP with SbTMVP and MMVD.

[0071] In a related example, multiple juxtaposed pictures may be used in SbTMVP. However, different derivation methods from multiple juxtaposed reference pictures may have different coding performance.

[0072] In related examples, either the central MV or the adjacent subblock MV from SbTMVP may be used to point to a reference template or a subblock reference template. However, neither the central MV nor the adjacent subblock MV may be combined to derive the MV for each subblock template.

[0073] This disclosure provides the derivation of the MV of a subblock template within a template. The MV of a subblock template can be derived based on subblock-based template matching in SbTMVP mode. In the example, the central MV (e.g., the MV located at the center of the current block) is used instead of the MV at (0,0) to prevent random initial values ​​that would cause a mismatch between the encoder and decoder. The accuracy and predictability of the coding process can be improved.

[0074] In one embodiment, when the current CU is coded in SbTMVP mode, the DV is used to indicate the center position of the current CU's MV field in a juxtaposed reference picture. In that juxtaposed reference picture, the MV data can be obtained from the center position of the corresponding MV field within the juxtaposed reference picture, which may be called the “center MV” in SbTMVP. For example, in Figure 6, the MV data at position (2,2) in the MV field may be called the “center MV” in SbTMVP. In the example, the template is divided into subblock templates, and the motion vector of each subblock template is derived not only by using the MV from the adjacent subblock within the current coding block, but also by using the center MV described above.

[0075] In an embodiment, when the current CU is coded in SbTMVP mode, the DV may be used to point from the reference position of the current CU to the reference position of the reference CU in the juxtaposed reference picture. The reference position is, in the example, the center position. In the juxtaposed reference picture, the MV data can be obtained from the center position of the corresponding MV field in the juxtaposed reference picture, and the MV data at the center position may be called the “center MV” in SbTMVP. For example, as shown in Figure 6, the MV data at position (2,2) in the MV field (600) may be called the “center MV” in SbTMVP.

[0076] In this disclosure, a template (or template region) may be divided into multiple subblock templates (or template subblocks). The motion vector of each subblock template may be derived not only by using the motion vectors from adjacent subblocks of the current coding block, but also by using the central motion vector associated with the current coding block.

[0077] In one embodiment, the central MV may be derived from any subblock MV of SbTMVP, for example, the central MV is the MV from the top-left, bottom-left, top-right, or bottom-right subblock of SbTMVP.

[0078] In this embodiment, the central MV associated with the current block coded in SbTMVP mode can be derived from any suitable subblock MV of the SbTMVP. For example, the central MV can be determined as the MV from one of the top-left subblock, bottom-left subblock, top-right subblock, and bottom-right subblock of the SbTMVP.

[0079] In one embodiment, the central MV can be derived from filtering the subblock MV of SbTMVP. The filters used may be, but are not limited to, median filters, mode filters, weighted average filters, etc.

[0080] In some embodiments, the central MV can be derived by filtering the subblock MVs of the SbTMVP. The filters used may be, but are not limited to, median filters, modal filters, or weighted average filters. For example, based on a median filter, the central MV may be determined as the MV from the selected current block's subblocks, based on the median sample value of the selected subblock. For example, based on a modal filter, the central MV may be determined as the MV from the selected current block's subblocks, based on the prediction mode of the selected subblock. For example, based on a weighted average filter, the central MV may be determined as the average (or weighted combination) of a subset of the MVs of multiple subblocks.

[0081] In one embodiment, the interpretation direction from L0, L1, or both of the subblock template is determined by the interpretation direction of the central MV of the SbTMVP.

[0082] In the embodiment, the interpretation direction can be a first prediction associated with a first reference list (e.g., L0), a second prediction associated with a second reference list (e.g., L1), or a bipretation direction associated with both reference lists (e.g., L0 and L1). The interpretation direction of a subblock template may be determined by the interpretation direction of the central MV of the SbTMVP. For example, if the interpretation direction of the central MV is a bipretation direction, then the interpretation direction of the subblock template is also a bipretation direction.

[0083] In one embodiment, if the reference index in reference list x of an adjacent subblock MV of SbTMVP is not valid, but the reference index in reference list x of the central MV of SbTMVP is valid, the central MV and the reference index on reference list x may be used as the MV on reference list x for the subblock template on reference list x. Examples can be shown in Figures 7 and 8.

[0084] In one embodiment, if the reference index in the reference list x of an adjacent subblock MV of SbTMVP is not valid, but the reference index in the reference list x of the central MV of SbTMVP is valid, then the central MV and the reference index in the reference list x of the central MV can be used as the MV of reference list x for the subblock templates of reference list x.

[0085] In the example, as shown in Figure 7, the current block (700) may contain multiple subblocks, such as subblocks (702) and (704). The current block (700) may also contain a template region above and to the left of the current block (700). The template region may contain multiple template subblocks (or subblock templates), such as template subblocks (706) and (708). The MV of the first template subblock (706) may be determined as the central MV (710) based on the following: (i) the MV of the first subblock (e.g., (704)) among the multiple subblocks of the current block (700) is a one-sided predicted MV in the first reference list (e.g., L0); (ii) the MV of the first template subblock (706) among the multiple template subblocks adjacent to the first subblock (e.g., (704)) is a one-sided predicted MV in the second reference list (e.g., L1); and (iii) the central MV (710) is a one-sided predicted MV in the second reference list.

[0086] In other examples, as shown in Figure 8, the current block (800) may contain multiple subblocks such as subblocks (802) and (804). The current block (800) may contain a template region containing multiple template subblocks such as template subblocks (806), (808), (810), and (812). (i) The MV of the first subblock of the multiple subblocks (e.g., (804)) is a one-predictive MV in the first reference list (e.g., L0), (ii) The MV of the first template subblock (e.g., (810)) of the multiple template subblocks adjacent to the first subblock (e.g., (804)) is a two-predictive MV, and (iii) The central MV (814) is the first component (e.g., MV) in the first reference list (e.g., L0). P-L0 ) and the second component in the second reference list (e.g., L1) (e.g., MV P-L1 Based on the fact that it is a bipredictive MV including ), the MV of the first template subblock (e.g., (810)) is the MV of the first subblock (e.g., (804)) in the first reference list (e.g., L0) (e.g., MV C-L0 ) and the second component of the central MV(814) in the second reference list (e.g., L1) (e.g., MV P-L1 ) can be included.

[0087] In one embodiment, when the central MV is bipredictive, the process in Figure 8 may be applied. Otherwise, when the central MV is unpredictive, the subblock template region uses only its adjacent subblock MVs of the SbTMVP block, as shown in Figure 9.

[0088] In an embodiment, when the central MV is bipredictive, the derivation of the MV for the template subblock shown in Figure 8 may be applied. When the central MV is unpredictive, the derivation of the MV for the template subblock may use the MV from the adjacent subblock of the SbTMVP block. For example, as shown in Figure 9, the current block (900) may contain multiple subblocks such as subblocks (902) and (904). The current block (900) may contain multiple template subblocks such as template subblocks (906) and (908). Based on the central MV (910) being unpredictive, the MV of each of the multiple template subblocks may be determined as the MV of that subblock among the multiple subblocks adjacent to each template subblock. For example, the MV of template subblock (906) may be determined as the MV of subblock (902), and the MV of template subblock (908) may be determined as the MV of subblock (904).

[0089] In one embodiment, when the central MV is a single prediction, the process shown in Figure 7 may be applied. Otherwise, when the central MV is a double prediction, the subblock template region uses only its adjacent subblock MVs of the SbTMVP block, as shown in Figure 10.

[0090] In an embodiment, when the central MV is a uniprediction, the derivation of the MV for the template subblock shown in Figure 7 may be applied. When the central MV is a biprediction, the derivation of the MV for the template subblock may use the MV from the adjacent subblock of the SbTMVP block. For example, as shown in Figure 10, the current block (1000) may contain multiple subblocks such as subblocks (1002) and (1004). The current block (1000) may contain multiple template subblocks such as template subblocks (1006) and (1008). Based on the central MV (1010) being a biprediction MV, the MV of each of the multiple template subblocks may be determined as the MV of that subblock among the multiple subblocks adjacent to each template subblock. For example, the MV of template subblock (1006) can be determined as the MV of subblock (1002), and the MV of template subblock (1008) can be determined as the MV of subblock (1004).

[0091] In one embodiment, as shown in Figure 11, when the central MV is a single prediction with MVs on one reference list (e.g., L0) and the adjacent subblocks within the SbTMVP block are single predictions with MVs on another reference list (e.g., L1), the template subblock region uses a double prediction joined by MVs from the central MV and the adjacent subblock MVs.

[0092] In an embodiment, when the central MV is a one-sided prediction with MVs on one reference list (e.g., L0), and the adjacent sub-blocks within the SbTMVP block are one-sided predictions with MVs on another reference list (e.g., L1), the template sub-block area can use a two-sided prediction combined by the MVs from the central MV and the adjacent sub-block MVs. For example, as shown in FIG. 11, the current block (or SbTMVP block) (1100) can include a plurality of sub-blocks such as sub-block (1102) and sub-block (1104). The current block (1100) can include a plurality of template sub-blocks such as template sub-blocks (1106) and (1108). (i) The central MV (1110) is a one-sided prediction MV in the first reference list (e.g., L0), and (ii) the MV of the first sub-block (e.g., (1104)) among the plurality of sub-blocks adjacent to the first template sub-block (e.g., (1108)) among the plurality of template sub-blocks (e.g., MV C-L1 ) is a one-sided prediction MV in the second reference list (e.g., L1), based on which the MV of the first template sub-block (e.g., (1108)) can be determined as a two-sided prediction MV including the central MV (1110) in the first reference list (e.g., L0) and the MV of the first sub-block (e.g., (1104)) (e.g., MV C-L1 ) in the second reference list (e.g., L1). Similarly, the MV of the template sub-block (1106) can include the MV of the sub-block (1102) (e.g., MV E-L1 ) and the central MV (e.g., MV P-L0 ).

[0093] Figure 12 shows a flowchart illustrating a process (1200) according to an embodiment of the present disclosure. Process (1200) may be used in a video decoder. In various embodiments, process (1200) is performed by processing circuits such as a processing circuit that performs the functions of a video decoder (110), a processing circuit that performs the functions of a video decoder (210), and so on. In some embodiments, process (1200) is performed by software instructions, so that when a processing circuit executes its software instructions, the processing circuit executes process (1200). Process (1200) begins at (S1201) and proceeds to (S1210).

[0094] In (S1210), the video bitstream is received. The video bitstream includes a current block having multiple subblocks and a template region of the current block having multiple template subblocks adjacent to at least one of the upper and left sides of the current block. For example, as shown in one of Figures 7 to 1, the current block includes multiple subblocks and the template region includes multiple template subblocks.

[0095] In (S1220), the motion vector (MV) at the current center position of the block is determined. The MV is determined based on the MV of at least one of the subblocks of the current block. For example, in Figure 6, the MV data at position (2,2) in the MV field may be used as the central MV.

[0096] In (S1230), the MV for each of the multiple template subblocks is determined based on the MV currently located at the center of the block and the MVs for the corresponding subblocks among the multiple subblocks adjacent to each template subblock. For example, as shown in Figures 7 and 8, if the reference index of the reference list x of the adjacent subblock MV of SbTMVP is not valid, but the reference index of the reference list x of the central MV of SbTMVP is valid, then the central MV and the reference index of the reference list x of the central MV may be used as the MV of the reference list x for the subblock templates in the reference list x.

[0097] In (S1240), the current block is reconfigured based on the determined MV of multiple template subblocks.

[0098] In this example, the MV currently located at the center of the block is determined to originate from one of the following subblocks: the top-left subblock, the bottom-left subblock, the top-right subblock, and the bottom-right subblock of the current block.

[0099] In the example, the MV currently located at the center of the block is determined to be the MV from one of several subblocks, and that one of the several subblocks is selected based on either the prediction mode or the median sample value of that one subblock.

[0100] In the example, the MV currently located at the center of a block is determined as the average of a subset of the MVs of multiple subblocks.

[0101] In one embodiment, the MV of each of the multiple template subblocks is determined as a single-prediction MV based on the fact that the MV currently located at the center of the block is a single-prediction MV. In another embodiment, the MV of each of the multiple template subblocks is determined as a double-prediction MV based on the fact that the MV currently located at the center of the block is a double-prediction MV.

[0102] In the example, the MV of the first subblock is determined to be the MV currently located at the center of the block, based on the following: (i) the MV of the first subblock among multiple subblocks is a one-predicted MV in the first reference list; (ii) the MV of the first template subblock among multiple template subblocks adjacent to the first subblock is a one-predicted MV in the second reference list; and (iii) the MV currently located at the center of the block is a one-predicted MV in the second reference list.

[0103] In the example, based on (i) the MV of the first subblock among multiple subblocks is a one-predictive MV in the first reference list, (ii) the MV of the first template subblock among multiple template subblocks adjacent to the first subblock is a two-predictive MV, and (iii) the MV currently located at the center of the block is a two-predictive MV containing the first component in the first reference list and the second component in the second reference list, it is determined that the MV of the first template subblock contains the MV of the first subblock in the first reference list and the second component of the MV currently located at the center of the block in the second reference list.

[0104] In the example, based on the fact that the MV currently located at the center position of the block is a single predicted MV, the MV of each of the multiple template subblocks is determined to be the MV of the subblock among the multiple subblocks adjacent to each template subblock.

[0105] In the example, based on the fact that the MV currently located at the center of the block is a bipredicted MV, the MV of each of the multiple template subblocks is determined as the MV of the subblock that is adjacent to that template subblock.

[0106] In the example, based on (i) the MV currently located at the center of the block is a one-predicted MV in the first reference list, and (ii) the MV of the first subblock among the multiple subblocks adjacent to the first template subblock is a one-predicted MV in the second reference list, the MV of the first template subblock is determined to be a two-predicted MV that includes the MV currently located at the center of the block in the first reference list and the MV of the first subblock in the second reference list.

[0107] In the example, to reconstruct the current block, the referenced block of the current block is determined based on the difference between the template region of the referenced block and the template region of the current block, and the template region of the referenced block is indicated by the MV of multiple template subblocks. Each of the multiple subblocks is further reconstructed based on each of the subblocks of the referenced block.

[0108] Then the process proceeds to (S1299) and terminates.

[0109] Process (1200) may be appropriately adapted. The steps of Process (1200) may be changed and / or omitted. Additional steps may be added. Any appropriate order of implementation may be used.

[0110] Figure 13 shows a flowchart illustrating a process (1300) according to an embodiment of the present disclosure. Process (1300) may be used in a video encoder. In various embodiments, process (1300) is performed by processing circuits such as a processing circuit that performs the functions of a video encoder (103), a processing circuit that performs the functions of a video encoder (303), and so on. In some embodiments, process (1300) is performed by software instructions, so that when a processing circuit executes its software instructions, the processing circuit executes process (1300). Process (1300) begins at (S1301) and proceeds to (S1310).

[0111] In (S1310), the MV located at the center of the current block is determined. The current block includes multiple subblocks, and the MV located at the center of the current block is determined based on at least one MV of the multiple subblocks of the current block. For example, in Figure 6, the MV data at position (2,2) in the MV field may be used as the central MV of the current block (or the MV located at the center of the current block).

[0112] In (S1320), the MV of each of the multiple template subblocks is determined based on the MV located at the center of the current block and the MV of one of the multiple subblocks adjacent to each template subblock. The multiple template subblocks are adjacent to at least one of the above or left sides of the current block. For example, as shown in Figures 7 and 8, if the reference index of the reference list x of the adjacent subblock MV of SbTMVP is not valid, but the reference index of the reference list x of the central MV of SbTMVP is valid, then the central MV and the reference index of the reference list x of the central MV may be used as the MV of the reference list x for the subblock templates in the reference list x.

[0113] In (S1330), the sample of the current block is encoded based on the determined MV of multiple template subblocks.

[0114] Then the process proceeds to (S1399) and terminates.

[0115] Process (1300) may be appropriately adapted. The steps of Process (1300) may be changed and / or omitted. Additional steps may be added. Any appropriate order of implementation may be used.

[0116] The above technology can be implemented as computer software that uses computer-readable instructions and is physically stored on one or more computer-readable media. For example, Figure 14 shows a computer system (1400) suitable for implementing a particular embodiment of the subject of disclosure.

[0117] Computer software can be coded in any suitable machine code or computer language that can follow mechanisms such as assembly, compilation, and linking to generate code that includes instructions that can be executed directly or through interpretation, microcode execution, etc., by one or more central processing units (CPUs), graphics processing units (GPUs), etc.

[0118] The instructions can be executed on various types of computers or their components, including, for example, personal computers, tablet computers, servers, smartphones, game consoles, and Internet of Things devices.

[0119] The components shown in Figure 14 with respect to the computer system (1400) are illustrative in nature and are not intended to imply any limitation on the scope of use or functionality of computer software implementing embodiments of the present disclosure. The configuration of the components should not be construed as having any dependence or requirement on any one or combination of components described in the exemplary embodiments of the computer system (1400).

[0120] The computer system (1400) may include certain human interface input devices. Such human interface input devices may respond to input from one or more users, for example, through haptic input (e.g., keyboard, swipe, dataglobe motion), voice input (e.g., voice, clapping), visual input (e.g., gestures), or olfactory input (not shown). The human interface devices may also be used to capture certain media that are not necessarily directly related to conscious human input, such as sound (e.g., speech, music, ambient sounds), images (e.g., scanned images, photographic images taken from a still camera), or video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0121] The input human interface device may include one or more of the following: keyboard (1401), mouse (1402), trackpad (1403), touchscreen (1410), data glove (not shown), joystick (1405), microphone (1406), scanner (1407), and camera (1408) (only one of each is shown).

[0122] The computer system (1400) may also include certain human interface output devices that can stimulate the senses of one or more users, for example, through tactile output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touchscreen (1410), data glove (not shown), or joystick (1405), although haptic feedback devices that do not function as input devices may also exist), audio output devices (e.g., speakers (1409), headphones (not shown)), visual output devices (e.g., CRT screens, LCD screens, plasma screens, OLED screens, each with or without touchscreen input functionality and each with or without haptic feedback functionality, some of which are screens (1410) capable of outputting two-dimensional visual output or output in more than three dimensions by means such as stereoscopic output, virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).

[0123] The computer system (1400) may also include human-accessible storage devices and their associated media, such as CD / DVD ROM / RW (1420) including CD / DVD or similar media (1421), thumb drives (1422), removable hard disks or solid-state drives (1423), legacy magnetic media, such as tapes and floppy disks (not shown), dedicated ROM / ASIC / PLD-based devices, such as security dongles (not shown), and the like.

[0124] Those skilled in the art will understand that the term “computer-readable medium” as used in relation to the subject matter currently disclosed does not include transmission media, carrier waves, or other transient signals.

[0125] A computer system (1400) may also include an interface (1454) to one or more communication networks (1455). These networks may be, for example, wireless, wireline, or optical. They may also be local, wide-area, metropolitan, vehicle and industrial, real-time, latency-tolerant, etc. Examples of networks include local area networks such as Ethernet®, cellular networks including wireless LAN, GSM, 3G, 4G, 5G, LTE, etc., TV wireline or wireless wide-area digital networks including cable TV, satellite TV, and terrestrial TV, and vehicle and factory networks including CAN bus. Certain networks generally require an external network interface adapter attached to a specific general-purpose digital port or peripheral bus (1449) (e.g., a USB port on a computer system (1400)). Others are generally integrated into the core of the computer system (1400) by attachment to a system bus as described below (e.g., an Ethernet network to a PC computer system, or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system (1400) can communicate with other entities. Such communication can be unidirectional and receivable only (e.g., broadcast TV) or unidirectional and receivable only (e.g., a CAN bus to a specific CAN bus device), or it can be bidirectional to other computer systems using, for example, a local or wide-area digital network. Specific protocols or protocol stacks are available for use with each of the networks and network interfaces described above.

[0126] The above-mentioned human interface devices, human-accessible memory devices, and network interfaces may be attached to the core (1440) of the computer system (1400).

[0127] The core (1440) may include one or more central processing units (CPUs) (1441), graphics processing units (GPUs) (1442), dedicated programmable processing units in the form of field-programmable gate areas (FPGAs) (1443), hardware accelerators for specific tasks (1444), graphics adapters (1450), etc. These devices may be connected via a system bus (1448) along with read-only memory (ROM) (1445), random access memory (RAM) (1446), internal mass storage devices such as internal user-inaccessible hard drives, SSDs, etc. (1447). In some computer systems, the system bus (1448) may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripherals may be attached directly to the core's system bus (1448) or via a peripheral bus (1449). In the example, the display (1410) may be connected to the graphics adapter (1450). Architectures for peripheral buses include PCI and USB.

[0128] The CPU (1441), GPU (1442), FPGA (1443), and accelerator (1444) are capable of executing specific instructions that can be combined to constitute the computer code described above. This computer code can be stored in ROM (1445) or RAM (1446). Temporary data can also be stored in RAM (1446), while persistent data can be stored, for example, in a built-in mass storage device (1447). High-speed storage and retrieval to any of the memory devices can be made possible by the use of cache memory. Cache memory may be closely associated with one or more CPUs (1441), GPUs (1442), mass storage devices (1447), ROMs (1445), RAM (1446), etc.

[0129] Computer-readable media may contain computer code for performing various computer implementation operations. The media and computer code may be specifically designed and configured for the purposes of this disclosure, or they may be of a type that is well known and available to those skilled in the art in computer software technology.

[0130] For example, and not as an limitation, a computer system having architecture (1400), specifically a core (1440), can provide functionality as a result of a processor (including CPUs, GPUs, FPGAs, accelerators, etc.) that runs software embodied in one or more tangible computer-readable media. Such computer-readable media can be media related to user-accessible mass storage devices introduced earlier, in addition to specific storage devices of the core (1440) that have a non-transient nature, such as core-integrated mass storage (1447) or ROM (1445). Software implementing various embodiments of the present disclosure is stored in such devices and is executable by the core (1440). The computer-readable media may include one or more memory devices or chips, depending on the specific needs. The software may cause the core (1440) and, specifically, the processor within it (including CPUs, GPUs, FPGAs, etc.) to execute a particular process or a particular part of a particular process as described herein, including defining data structures stored in RAM (1446) and modifying such data structures according to a process defined by the software. Additionally, or alternatively, a computer system may provide functionality as a result of logic (e.g., accelerators (1444)) hardwired or otherwise embodied in the circuitry, which can operate in place of or with the software to execute a particular process or a particular part of a particular process as described herein. References to software may, as necessary, include logic, and vice versa. References to computer-readable media may, as necessary, include circuitry storing software for execution (e.g., integrated circuits (ICs)), circuitry embodying logic for execution, or both. This disclosure also encompasses any suitable combination of hardware and software.

[0131] The use of “at least one of…” or “one of…” in this disclosure is intended to include any one or combination of the elements described. For example, references to at least one of A, B, or C, at least one of A, B, and C, at least one of A, B, and / or C, and at least one of A through C are intended to include A only, B only, C only, or any combination thereof. References to one of A or B, and one of A and B are intended to include A or B or (A and B). The use of “one of…” does not exclude any combination of the elements described where applicable, such as when the elements are not mutually exclusive.

[0132] While this disclosure has described several exemplary embodiments, alternatives, substitutions, and various substitute equivalents exist and are included within the scope of this disclosure. Therefore, it will be apparent to those skilled in the art that numerous systems and methods embodying the principles of this disclosure, and thus falling within its spirit and scope, can be conceived, even if not explicitly illustrated or described herein.

[0133] This application claims priority to U.S. Patent Application No. 18 / 241084, filed on 31 August 2023, titled "MOTION VECTOR DERIVATION OF SUBBLOCK-BASED TEMPLATE-MATCHING FOR SUBBLOCK BASED MOTION VECTOR PREDICTOR," which is titled "MOTION VECTOR DERIVATION OF SUBBLOCK-BASED TEMPLATE-MATCHING FOR SUBBLOCK BASED MOTION VECTOR PREDICTOR," and." The disclosures of the prior applications are incorporated herein by reference in their entirety.

Claims

[Claim 1] A method of video decoding performed by a video decoder, The steps include receiving a video bitstream that includes a current block having multiple subblocks and a template region of the current block having multiple template subblocks adjacent to at least one of the upper and left sides of the current block, The step of determining the motion vector (MV) located at the center position of the current block, wherein the MV is determined based on the MV of at least one of the plurality of subblocks of the current block. The steps include determining the respective MV of each of the plurality of template subblocks based on the MV located at the central position of the current block and the respective MVs of the corresponding subblocks among the plurality of subblocks adjacent to each of the template subblocks, The steps include: reconstructing the current block based on the determined MV of the plurality of template subblocks; A method of having.

Images