Sub-block temporal motion vector prediction for video coding

Abstract

A computing device performs a method for decoding video data comprising: determining a co-located picture of a current coding unit; locating a spatially adjacent block of the current coding unit corresponding to the co-located picture; determining a motion shift vector for the current coding unit from one or more motion vectors associated with the spatially adjacent block according to a predetermined fixed order; and reconstructing a sub-block-based temporal motion vector for a plurality of sub-blocks of the current coding unit from a corresponding sub-block within the co-located picture based on the motion shift.

Description

Technology Field

[0001] The present invention generally relates to video data encoding and decoding, and specifically to a method and system for predicting sub-block motion vectors during video data encoding and decoding. Background Technology

[0002] Digital video can be supported by various electronic devices such as digital TVs, laptops or desktop computers, tablet computers, digital cameras, digital recording devices, digital media players, video game consoles, smartphones, video conferencing devices, and video streaming devices. Electronic devices can transmit, receive, encode, decode, and / or store digital video data by implementing video compression / decompression standards defined by MPEG-4, ITU-T H.263, ITU-T H.264 / MPEG-4, Part 10, Advanced Video Coding (AVC), High Efficiency Video Coding (HEVC), and Versatile Video Coding (VVC). Video compression may generally involve performing spatial (intra-frame) prediction and / or temporal (inter-frame) prediction to reduce or eliminate redundancy inherent in video data. In the case of block-based video coding, a video frame is divided into one or more slices, and each slice may contain multiple video blocks, also called coding tree units (CTUs). Each CTU may contain one coding unit (CU) or may be recursively divided into smaller CUs until a predetermined minimum CU size is reached. Each CU (also called a "leaf CU") may contain one or more transformation units (TU), and each CU may contain one or more prediction units (PU). Each CU may be coded in intra, inter, or IBC mode. Video blocks within an intra-coded (I) slice of a video frame may be encoded using spatial predictions for reference samples of adjacent blocks within the same video frame.For video blocks within an intercoded (P or B) slice of a video frame, spatial predictions for reference samples of adjacent blocks within the same video frame or temporal predictions for reference samples of other previous and / or future reference video frames may be used.

[0003] For example, spatial or temporal predictions based on previously encoded reference blocks, such as adjacent blocks, can generate prediction blocks for the current video block to be coded. The process of finding the reference block can be performed by a block matching algorithm. Residual data representing the pixel difference between the current block to be coded and the prediction block can be referred to as residual blocks or prediction errors. Inter-coded blocks can be encoded according to residual blocks and motion vectors pointing to reference blocks within the reference frame that constitute the prediction blocks. Generally, the process of determining motion vectors is called motion estimation. Intra-coded blocks can be encoded according to intra-prediction modes and residual blocks. For additional compression, residual blocks are transformed from the pixel domain to a transformation domain, such as the frequency domain, and as a result, residual transformation coefficients that are subsequently quantized can be generated. Quantized transformation coefficients, initially arranged in a 2D matrix, are scanned to generate a 1D vector of transformation coefficients, which is then entropy-encoded into the next video bitstream to achieve further compression.

[0004] Next, the encoded video bitstream may be stored on a computer-readable storage medium (e.g., flash memory) and accessed by another electronic device with digital video capabilities, or transmitted directly to the electronic device via wired or wireless connection. Next, the electronic device may perform video restoration (a process opposite to the video compression described above) by, for example, parsing the encoded video bitstream to obtain systax elements from the bitstream, and reconstructing digital video data from the encoded video bitstream into its original format based on at least a portion of the systax elements obtained from the bitstream, and render the reconstructed digital video data on the display of the electronic device.

[0005] As digital video quality changes from high definition to 4K x 2K or 8K x 4K, the amount of video data to be encoded / decoded increases rapidly. Finding a way to encode / decode video data more efficiently while maintaining the image quality of the decoded video data is a constant challenge. The problem to be solved

[0006] This application describes implementation methods related to systems and methods for video data encoding and decoding, particularly sub-block motion vector prediction. means of solving the problem

[0007] According to a first aspect of the present application, a method for decoding a current coding unit within a current picture comprises: determining a co-located picture of the current picture; determining a motion shift vector for the current coding unit based on a motion vector of a spatially adjacent block of the current coding unit, wherein the motion shift vector indicates a spatial positional shift between a sub-block of a plurality of sub-blocks within the current coding unit within the current picture and a corresponding sub-block within the co-located picture; and reconstructing a sub-block-based temporal motion vector for the sub-block of the plurality of sub-blocks within the current coding unit from the corresponding sub-block within the co-located picture based on the motion shift vector.

[0008] According to a second aspect of the present application, a computing device comprises one or more processors, memory, and a plurality of programs stored in memory. When executed by one or more processors, the programs cause the computing device to perform the operations described above.

[0009] According to a third aspect of the present application, a non-transient computer-readable storage medium stores a plurality of programs to be executed by a computing device having one or more processors. The programs cause the computing device to perform the operations described above when executed by said one or more processors. Brief explanation of the drawing

[0010] The attached drawings, which are included to provide a further understanding of the implementation methods, are integrated into the text, and constitute part of the specification. They serve to illustrate the described implementation methods and describe the underlying principles along with the explanations. Identical reference numerals indicate corresponding parts. FIG. 1 is a block diagram illustrating an exemplary video encoding and decoding system according to some implementation methods of the present disclosure. FIG. 2 is a block diagram illustrating an exemplary video encoder according to some implementation methods of the present disclosure. FIG. 3 is a block diagram illustrating an exemplary video decoder according to some implementation methods of the present disclosure. FIGS. 4A to 4E are block diagrams illustrating that a frame according to some implementations of the present disclosure is recursively divided into a plurality of video blocks of different sizes and shapes. FIG. 5 is a block diagram illustrating spatially adjacent locations and temporally co-located block locations of the current CU to be encoded according to some implementations of the present disclosure. FIGS. 6A through 6D are block diagrams illustrating steps for deriving temporal motion vector predictors of a current block or sub-block temporal motion vector predictors of a sub-block within a current block according to some implementations of the present disclosure. FIG. 7 illustrates a block diagram for determining an effective region for deriving temporal motion vector predictors and sub-block temporal motion vector predictors according to some implementations of the present disclosure. FIGS. 8A and 8B illustrate a flowchart illustrating an exemplary process in which a video coder according to some implementations of the present invention implements a technique for deriving sub-block temporal motion vector predictors. Specific details for implementing the invention

[0011] Specific implementations illustrated in the accompanying drawings are described in detail below. In the following detailed description, a number of non-limiting specific details are described to aid in understanding the contents described herein. However, it is obvious to those skilled in the art that various modifications may be made without departing from the scope of the claims and that the subject matter may be practiced without these specific details. For example, it is obvious to those skilled in the art that the technical solutions described herein may be implemented in various types of electronic devices equipped with digital video functions.

[0012] FIG. 1 is a block diagram illustrating an exemplary system (10) for encoding and decoding video blocks in parallel according to some implementations of the present disclosure. As illustrated in FIG. 1, the system (10) includes a source device (12) that generates and encodes video data to be later decoded by a destination device (14). The source device (12) and the destination device (14) may include any one of various electronic devices such as a desktop or laptop computer, a tablet computer, a smartphone, a set-top box, a digital television, a camera, a display device, a digital media player, a video game console, or a video streaming device. In some implementations, the source device (12) and the destination device (14) may have wireless communication capabilities.

[0013] In some implementations, the destination device (14) may receive encoded video data to be decoded via a link (16). The link (16) may include any type of communication medium or device capable of moving the encoded video data from the source device (12) to the destination device (14). In one example, the link (16) may be a communication medium that enables the source device (12) to directly transmit the encoded video data to the destination device (14) in real time. The encoded video data may be modulated according to a communication standard, such as a wireless communication protocol, and transmitted to the destination device (14). The communication medium may include any wireless or wired communication medium, such as a radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form part of a packet-based network, such as a local area network or a wide-area network, or a global network, such as the Internet. The communication medium may include a router, switch, base station, or any other equipment that can be used to facilitate communication from a source device (12) to a destination device (14).

[0014] In some other implementations, the encoded video data may be transferred from the output interface (22) to the storage device (32). Next, the encoded video data within the storage device (32) can be accessed by the destination device (14) through the input interface (28). The storage device (32) may include any one of various distributed or local access data storage media (e.g., hard drive, Blu-ray disc, DVD, CD-ROM, flash memory, volatile or non-volatile memory), or any other digital storage media suitable for storing the encoded video data. In additional examples, the storage device (32) may correspond to a file server or other intermediate storage device capable of holding the encoded video data generated by the source device (12). The destination device (14) may access the stored video data from the storage device (32) via streaming or downloading. The file server may be any type of computer capable of storing the encoded video data and transferring the encoded video data to the destination device (14). An exemplary file server may include a web server (e.g., for a website), an FTP server, a network attached storage (NAS) device, or a local disk drive. The destination device (14) may access the encoded video data stored in the file server via any standard data connection, including a wireless channel (e.g., Wi-Fi connection), a wired connection (e.g., DSL, cable modem, etc.), or a combination thereof, suitable for accessing the encoded video data. The transmission of the encoded video data from the storage device (32) may be a streaming transmission, a download transmission, or a combination thereof.

[0015] As illustrated in FIG. 1, the source device (12) includes a video source (18), a video encoder (20), and an output interface (22). The video source (18) may include a video capture device (e.g., a video camera), a video archive containing previously captured video, a video feed interface for receiving video from a video content provider, and / or a computer graphics system for generating computer graphics data into a source video, or a combination thereof. For example, if the video source (18) is a video camera of a security surveillance system, the source device (12) and the destination device (14) may form a camera phone or a video phone. However, the implementations described in this application are generally applicable to video coding and may be applicable to wireless and / or wired applications.

[0016] A captured, previously captured, or computer-generated video may be encoded by a video encoder (20). The encoded video data may be transmitted directly to a destination device (14) through an output interface (22) of a source device (12). Additionally, the encoded video data may be stored (or alternatively) in a storage device (32) so that it can be accessed later by the destination device (14) or another device to be decoded and / or played back. The output interface (22) may further include a modem and / or transmitter.

[0017] The destination device (14) includes an input interface (28), a video decoder (30), and a display device (34). The input interface (28) may include a receiver and / or a modem and may receive encoded video data via the link (16). The encoded video data transmitted via the link (16) or provided on the storage device (32) may include various syntax elements generated by the video encoder (20) and used by the video decoder (30) to decode the video data. These syntax elements may be included in the encoded video data transmitted via a communication medium, stored on a storage medium, or stored on a file server.

[0018] In some implementations, the destination device (14) may include a display device (34) which may be an integrated display device and an external display device configured to communicate with the destination device (14). The display device (34) displays decoded video data to a user and may include any one of various display devices such as a liquid crystal display (LCD), a plasma display, an organic light-emitting diode (OLED) display, or other types of display devices.

[0019] The video encoder (20) and video decoder (30) may operate according to proprietary or industry standards such as VVC, HEVC, MPEG-4, Part 10, AVC (Advanced Video Coding), or extensions of these standards. It should be understood that this application is not limited to the specified video coding / decoding standards and may apply to other video coding / decoding standards. Generally, it should be considered that the video encoder (20) of the source device (12) may be configured to encode video data according to any one of these current or future standards. Similarly, it should be generally considered that the video decoder (30) of the destination device (14) may be configured to decode video data according to any one of these current or future standards.

[0020] Each of the video encoder (20) and video decoder (30) may be implemented in any one of various suitable encoder circuits, software, hardware, firmware, or combinations thereof, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), and discrete logic. In the case of partial software implementation, the electronic device may perform the video coding / decoding operations disclosed in this disclosure by storing instructions for the software on a suitable non-transient computer-readable medium and executing the instructions in hardware using one or more processors. Each of the video encoder (20) and video decoder (30) may be included in one or more encoders or decoders, and may be directly connected to each device as part of a combined encoder / decoder (CODEC).

[0021] FIG. 2 is a block diagram illustrating an exemplary video encoder (20) according to some implementation methods described in the present application. The video encoder (20) can perform intra and inter prediction coding of video blocks within a video frame. Intra prediction coding can reduce or eliminate spatial redundancy of video data within a given video frame or picture by relying on spatial prediction. Inter prediction coding can reduce or eliminate temporal redundancy of video data within adjacent video frames or pictures of a video sequence by relying on temporal prediction.

[0022] As illustrated in FIG. 2, the video encoder (20) includes a video data memory (40), a prediction processing unit (41), a decoding picture buffer (DPB, 64), an adder (50), a transformation processing unit (52), a quantization unit (54), and an entropy encoding unit (56). The prediction processing unit (41) includes a motion estimation unit (42), a motion compensation unit (44), a segmentation unit (45), an intra prediction processing unit (46), and an intra block copying (BC) unit (48). In some implementations, the video encoder (20) also includes an inverse quantization unit (58), an inverse transformation processing unit (60), and an adder (62) for video block reconstruction. A de-blocking filter (not shown) may be positioned between the adder (62) and the DPB (64) to filter block boundaries and remove blockiness artifacts from the reconstructed video. In addition to the de-blocking filter, an in-loop filter (not shown) may also be used to filter the output of the adder (62). The video encoder (20) may be configured in the form of a fixed or programmable hardware unit or divided into one or more of the illustrated fixed or programmable hardware devices.

[0023] The video data memory (40) can store video data to be encoded by the components of the video encoder (20). The video data in the video data memory (40) can be obtained, for example, from a video source (18). The DPB (64) is a buffer that stores reference video data used by the video encoder (20) to encode video data (for example, in intra or inter predictive coding mode). The video data memory (40) and the DPB (64) can be formed by any one of various memory devices. In various examples, the video data memory (40) can be on-chip with other components of the video encoder (20) or off-chip with respect to these components.

[0024] As illustrated in FIG. 2, after receiving video data, the partitioning unit (45) within the prediction processing unit (41) partitions the video data into video blocks. Meanwhile, this partitioning may include dividing the video frame into slices, tiles, or other larger coding units (CUs) according to a preset partitioning structure, such as a quad-tree structure associated with the video data. The video frame may be partitioned into a plurality of video blocks (or a set of video blocks called tiles). The prediction processing unit (41) may select one of a plurality of possible prediction coding modes, such as one of a plurality of intra prediction coding modes or one of a plurality of inter prediction coding modes for the current video block, based on error results (e.g., coding rate and distortion level). The prediction processing unit (41) provides the intra or inter prediction coded block, which is the processing result, to an adder (50) to generate a residual block, and provides it to an adder (62) to reconstruct the encoded block so that it can be used later as part of a reference frame. Meanwhile, the prediction processing unit (41) can provide syntax elements such as motion vectors, intra-mode indicators, partition information, and other such syntax information to the entropy encoding unit (56).

[0025] To select an appropriate intra-predictive coding mode for the current video block, the intra-predictive processing unit (46) within the prediction processing unit (41) may provide spatial prediction by performing intra-predictive coding of the current video block for one or more adjacent blocks within the same frame as the current block to be coded. The motion estimation unit (42) and motion compensation unit (44) within the prediction processing unit (41) may provide temporal prediction by performing inter-predictive coding of the current video block for one or more prediction blocks among one or more reference frames. The video encoder (20) may perform multiple coding passes to select, for example, an appropriate coding mode for each block of video data.

[0026] In some implementations, the motion estimation unit (42) determines the inter-prediction mode for the current video frame by generating a motion vector according to a predetermined pattern within the video frame sequence, said motion vector may indicate the displacement of the prediction unit (PU) of the video block in the current video frame relative to the prediction block in the reference video frame. The motion estimation performed by the motion estimation unit (42) is the process of generating motion vectors that estimate the motion of the video blocks. For example, the motion vector may indicate the displacement of the PU of the video block in the current video frame or picture relative to the prediction block in the reference frame (or other coded unit) relative to the current block being coded in the current frame (or other coded unit). The predetermined pattern may designate the video frame of the sequence as either a P frame or a B frame. The intra-BC unit (48) can determine vectors for intra-BC coding (e.g., block vectors) in a manner similar to the determination of motion vectors of the motion estimation unit (42) for inter-prediction, or can determine block vectors using the motion estimation unit (42).

[0027] The prediction block is a block of reference frames considered to be closest to the PU of the video block to be coded in terms of pixel difference, and the pixel difference can be determined by the sum of absolute differences (SAD), the sum of square differences (SSD), or other difference metrics. In some implementations, the video encoder (20) can calculate values ​​for sub-integer pixel locations of reference frames stored in the DPB (64). For example, the video encoder (20) can interpolate values ​​for 1 / 4 pixel locations, 1 / 8 pixel locations, or other fractional pixel locations of the reference frames. Thus, the motion estimator (42) can perform motion search for whole pixel locations and fractional pixel locations and output a motion vector with fractional pixel precision.

[0028] The motion estimation unit (42) calculates a motion vector for the PU of a video block within an inter-predicted coded frame by comparing the position of the prediction block of a reference frame with the position of the PU, wherein the reference frame may be selected from a first reference frame list (List 0) or a second reference frame list (List 1) that each identify one or more reference frames stored in the DPB (64). The motion estimation unit (42) transmits the calculated motion vector to the motion compensation unit (44) and then to the next entropy encoding unit (56).

[0029] Motion compensation performed by the motion compensation unit (44) may include fetching or generating a prediction block based on a motion vector determined by the motion estimation unit (42). Upon receiving a motion vector for the PU of the current video block, the motion compensation unit (44) may find the prediction block indicated by the motion vector in one of the reference frame lists, retrieve the prediction block from the DPB (64), and pass the prediction block to the adder (50). Next, the adder (50) may form a pixel difference value of the remaining video block by subtracting the pixel value of the prediction block provided by the motion compensation unit (44) from the pixel value of the current video block being coded. The pixel difference value forming the remaining video block may include a luminance or chroma difference component, or both. Additionally, the motion compensation unit (44) may generate syntax elements associated with the video block of the video frame that the video decoder (30) uses when decoding the video block of the video frame. Syntax elements may include, for example, syntax elements defining motion vectors used to identify a prediction block, any flag indicating a prediction mode, or any other syntax information described herein. The motion estimation unit (42) and the motion compensation unit (44) may be highly direct but are shown separately for conceptual purposes.

[0030] In some implementations, the intra-BC unit (48) may generate vectors and fetch prediction blocks in a manner similar to that described above in relation to the motion estimation unit (42) and motion compensation unit (44), but these prediction blocks are within the same frame as the current block being coded, and the vectors are referred to as block vectors rather than motion vectors. Specifically, the intra-BC unit (48) may determine the intra-prediction mode used to encode the current block. In some examples, the intra-BC unit (48) may encode the current block using various intra-prediction modes, for example, during each encoding pass, and may test their performance through rate-distortion analysis. Next, the intra-BC unit (48) may select and use an appropriate intra-prediction mode from among the various tested intra-prediction modes and generate an intra-mode identifier accordingly. For example, the intra-BC section (48) can calculate rate-distortion values ​​using rate-distortion analyses for various tested intra-prediction modes, and can select and use an intra-prediction mode having optimal rate-distortion characteristics among the tested modes as an appropriate intra-prediction mode. Generally, rate-distortion analysis determines the amount of distortion (or error) between the encoded block and the original unencoded block being encoded to generate the encoded block, and the bit rate (i.e., number of bits) for generating the encoded block. The intra-BC section (48) can determine which intra-prediction mode indicates the optimal rate-distortion value for the block by calculating a ratio from the distortion and rate for various encoded blocks.

[0031] In other examples, the intra-BC unit (48) may perform functions for intra-BC prediction according to the implementation methods described herein by using the motion estimation unit (42) and the motion compensation unit (44) wholly or partially. In both cases, for an intra-block copy, the prediction block may be a block considered to be closely matched to the block to be coded in terms of pixel difference determined by the sum of absolute differences (SAD), the sum of squared differences (SSD), or other difference metrics, and the identification of the prediction block may include the calculation of values ​​for sub-integer pixel locations.

[0032] Regardless of whether the prediction block is generated from the same frame according to intra prediction or from a different frame according to inter prediction, the video encoder (20) can form a residual video block by subtracting the pixel values ​​of the prediction block from the pixel values ​​of the current video block being coded to form pixel difference values. The pixel difference values ​​forming the residual video block may include both lumina and chroma component differences.

[0033] As an alternative to the inter-prediction performed by the motion estimation unit (42) and motion compensation unit (44) described above, or the intra-block copy prediction performed by the intra-BC unit (48), the intra-prediction processing unit (46) can intra-predict the current video block. Specifically, the intra-prediction processing unit (46) can determine the intra-prediction mode to use to encode the current block. To this end, the intra-prediction processing unit (46) can encode the current block using various intra-prediction modes, for example, during each encoding pass, and the intra-prediction processing unit (46) (or the mode selection unit in some examples) can select and use an appropriate intra-prediction mode from the tested intra-prediction modes. The intra-prediction processing unit (46) can provide information indicating the selected intra-prediction mode for the block to the entropy encoding unit (56). The entropy encoding unit (56) can encode the information indicating the selected intra-prediction mode into the bitstream.

[0034] After the prediction processing unit (41) determines a prediction block for the current video block through inter-prediction or intra-prediction, the adder (50) subtracts the prediction block from the current video block to form a residual video block. The residual video data within the residual block may be included in one or more transformation units (TU) and provided to the transformation processing unit (52). The transformation processing unit (52) converts the residual video data into residual transformation coefficients using a transformation such as the Discrete Cosine Transform (DCT) or a conceptually similar transformation.

[0035] The conversion processing unit (52) can transmit the obtained conversion coefficients to the quantization unit (54). The quantization unit (54) can quantize the conversion coefficients to further reduce the bit rate. Additionally, the quantization process can reduce the bit depth associated with some or all of the coefficients. The degree of quantization can be modified by adjusting the quantization parameters. In some examples, the quantization unit (54) can perform a scan of a matrix containing the quantized conversion coefficients. Alternatively, the entropy encoding unit (56) can perform the scan.

[0036] After quantization, the entropy encoding unit (56) entropies the quantized transform coefficients into a video bitstream using, for example, context adaptive variable length coding (CAVLC), context adaptive binary arithmetic coding (CABAC), syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or other entropy encoding methods or techniques. The encoded bitstream can be transmitted to a video decoder (30), stored in a storage device (32) and subsequently transmitted to a video decoder (30), or retrieved by a video decoder (30). The entropy encoding unit (56) can also entropy encode motion vectors and other syntax elements for the current video frame being encoded.

[0037] The inverse quantization unit (58) and the inverse transform processing unit (60) can reconstruct the remaining video blocks in the pixel domain by applying inverse quantization and inverse transform, respectively, to generate reference blocks for predicting other video blocks. As described above, the motion compensation unit (44) can generate motion-compensated prediction blocks from one or more reference blocks of frames stored in the DPB (64). The motion compensation unit (44) can also apply one or more interpolation filters to the prediction blocks to produce sub-integer pixel values ​​for use in motion estimation.

[0038] The adder (62) can generate a reference block to be stored in the DPB (64) by adding the reconstructed residual block and the motion-compensated prediction block generated by the motion compensation unit (44). Next, the reference block is used as a prediction block by the intra-BC unit (48), the motion estimation unit (42), and the motion compensation unit (44) to inter-predict other video blocks within a subsequent video frame.

[0039] FIG. 3 is a block diagram illustrating an exemplary video decoder (30) according to some implementation methods of the present application. The video decoder (30) includes a video data memory (79), an entropy decoding unit (80), a prediction processing unit (81), an inverse quantization unit (86), an inverse transform processing unit (88), an adder (90), and a DPB (92). The prediction processing unit (81) further includes a motion compensation unit (82), an intra prediction unit (84), and an intra BC unit (85). The video decoder (30) can generally perform a decoding process opposite to the encoding process described above for the video encoder (20) in relation to FIG. 2. For example, the motion compensation unit (82) can generate prediction data based on motion vectors received from the entropy decoding unit (80), and the intra prediction unit (84) can generate prediction data based on intra prediction mode indicators received from the entropy decoding unit (80).

[0040] In some examples, the components of the video decoder (30) may perform functions to implement the implementations of the present application. Additionally, in some examples, the components of the video decoder (30) may be divided into one or more parts to implement the implementations of the present disclosure. For example, the intra-BC unit (85) may perform the implementations of the present application alone or in combination with other parts of the video decoder (30), such as the motion compensation unit (82), the intra-prediction unit (84), and the entropy decoding unit (80). In some examples, the video decoder (30) may not include the intra-BC unit (85), and the function of the intra-BC unit (85) may be performed by other parts of the prediction processing unit (81), such as the motion compensation unit (82).

[0041] The video data memory (79) may store video data to be decoded by other parts of the video decoder (30), such as an encoded video bitstream. The video data stored in the video data memory (79) may be obtained, for example, from a storage device (32), obtained via wired or wireless network communication of the video data, or obtained from a local video source, such as a camera, by accessing a physical data storage medium (e.g., a flash drive or a hard disk). The video data memory (79) may include a coded picture buffer (CPB) that stores the encoded video data of the encoded video bitstream. The decoded picture buffer (DPB) (92) of the video decoder (30) may store reference video data for the video decoder (30) to use to decode the video data (e.g., in intra or inter predictive coding mode). The video data memory (79) and DPB (92) may be formed from any of the various memory devices, such as SDRAM, MRAM, RRAM, DRAM, or other types of memory devices. For illustrative purposes, the video data memory (79) and DPB (92) are shown in FIG. 3 as two separate parts of the video decoder (30). However, it is obvious to those skilled in the art that the video data memory (79) and DPB (92) may be provided by the same memory device or by separate memory devices. In some examples, the video data memory (79) may be on-chip with other parts of the video decoder (30) or off-chip with these parts.

[0042] During the decoding process, the video decoder (30) receives an encoded video bitstream pointing to video blocks of the encoded video frame and associated syntax elements. The video decoder (30) may receive syntax elements at the video frame level and / or video block level. The entropy decoding unit (80) of the video decoder (30) may entropy decode the bitstream to generate quantized coefficients, motion vectors, or intra-prediction mode indicators and other syntax elements. Next, the entropy decoding unit (80) transmits the motion vectors and other syntax elements to the prediction processing unit (81).

[0043] When a video frame is coded into an intra-predicted coding (I) frame or coded into an intra-coded prediction block in another type of frame, the intra-predicted unit (84) of the prediction processing unit (81) can generate prediction data for a video block of the current video frame based on the signaled intra-predicted mode and reference data from previously decoded blocks of the current frame.

[0044] When a video frame is coded as an inter-predicted coded (i.e., B or P) frame, the motion compensation unit (82) of the prediction processing unit (81) generates one or more prediction blocks for the video blocks of the current video frame based on motion vectors and other syntax elements received from the entropy decoding unit (80). Each prediction block may be generated from a reference frame within one of the reference frame lists. The video decoder (30) may generate reference frame lists (List0 and List1) using default configuration techniques based on the reference frames stored in the DPB (92).

[0045] In some examples, when a video block is coded according to the intra-BC mode described herein, the intra-BC section (85) of the prediction processing unit (81) generates prediction blocks for the current video block based on block vectors and other syntax elements received from the entropy decoding unit (80). The prediction blocks may be included within a reconstruction area of ​​the same picture as the current video block defined by the video encoder (20).

[0046] The motion compensation unit (82) and / or the intra-BC unit (85) parses motion vectors and other syntax elements to determine prediction information for video blocks of the current video frame, and then uses the prediction information to generate prediction blocks for the current video block to be decoded. For example, the motion compensation unit (82) may determine, using some of the received syntax elements, the prediction mode used for coding the video blocks of the video frame (e.g., intra or inter prediction), the inter prediction frame type (e.g., B or P), configuration information for one or more of the reference frame lists for the frame, motion vectors for each inter prediction coded video block of the frame, the inter prediction state for each inter prediction coded video block of the frame, and other information for decoding the video blocks of the current video frame.

[0047] Similarly, the intra-BC section (85) can use some of the received syntax elements (e.g., flags) to determine the construction information that the current video block was predicted using intra-BC mode, which video blocks of the frame are within the reconstructed area, which should be stored in the DPB (92), block vectors for each intra-BC predicted video block of the frame, the intra-BC predicted state for each intra-BC predicted video block of the frame, and other information for decoding the video blocks within the current video frame.

[0048] The motion compensation unit (82) may also perform interpolation using interpolation filters used by the video encoder (20) during the encoding of video blocks to calculate interpolation values ​​for sub-integer pixels of reference blocks. In this case, the motion compensation unit (82) may determine the interpolation filters used by the video encoder (20) from the received syntax elements and generate prediction blocks using the determined interpolation filters.

[0049] The inverse quantization unit (86) determines the degree of quantization by inversely quantizing the transform coefficients provided in the bitstream and entropy decoded by the entropy decoding unit (80), using the same quantization parameters calculated by the video encoder (20) for each video block of the video frame. The inverse transform processing unit (88) applies an inverse transform (e.g., inverse DCT, inverse integer transform, or a conceptually similar inverse transform process) to the transform coefficients to reconstruct the remaining blocks in the pixel domain.

[0050] After generating a prediction block for the current video block based on vectors and other syntax elements in the motion compensation unit (82) or the intra-BC unit (85), the adder (90) reconstructs the decoded video block for the current video block by adding the residual block provided by the inverse transformation processing unit (88) and the corresponding prediction block generated by the motion compensation unit (82) and the intra-BC unit (85). An in-loop filter (not shown) may be positioned between the adder (90) and the DPB (92) to further process the decoded video block. Decoded video blocks within a given frame are stored in the DPB (92), which stores reference frames used for subsequent motion compensation of the next video blocks. The DPB (92) or a memory device separate from the DPB (92) may also store the decoded video so that it can be displayed later on a display device such as the display device (34) of FIG. 1.

[0051] In a typical video coding process, a video sequence generally contains a set of frames or pictures in a predetermined order. Each frame may contain three sample matrices denoted as SL, SCb, and SCr. SL is a two-dimensional matrix of luminance samples. SCb is a two-dimensional matrix of Cb chroma samples. SCr is a two-dimensional matrix of Cr chroma samples. In other cases, since the frame may be monochrome, it may contain only one two-dimensional matrix of luminance samples.

[0052] As illustrated in FIG. 4A, a video encoder (20) (or more specifically, a splitting unit (45)) first generates an encoded representation of a frame by splitting the frame into a set of coding tree units (CTUs). A video frame may contain an integer number of CTUs arranged sequentially in raster scan order from left to right and from top to bottom. Each CTU is the largest logical coding unit, and the width and height of the CTU are signaled by the video encoder (20) as a sequence parameter set, so that all CTUs in the video sequence have the same size, which is one of 128×128, 64×64, 32×32, and 16×16. However, it should be noted that the present application is not necessarily limited to a specific size. As illustrated in FIG. 4B, each CTU may include one coding tree block (CTB) of luminance samples, two corresponding coding tree blocks of chroma samples, and syntax elements used to code the samples of the coding tree blocks. The syntax elements describe the characteristics of different types of units of the coded pixel blocks, including inter or intra prediction, intra prediction mode, motion vector, and other parameters, and how the video sequence can be reconstructed in the video decoder (30). In monochrome pictures or pictures having three distinct color planes, the CTU may include a single coding tree block and syntax elements used to code the samples of the coding tree block. The coding tree block may be an N×N block of samples.

[0053] To achieve better performance, the video encoder (20) may recursively perform tree partitioning, such as binary-tree partitioning, ternary-tree partitioning, quad-tree partitioning, or a combination thereof, on the coding tree blocks of the CTU and partition the CTU into smaller coding units (CUs). As illustrated in FIG. 4C, a 64×64 CTU (400) is first partitioned into four smaller CUs, each having a block size of 32×32. Of the four smaller CUs, CU (410) and CU (420) are each partitioned into four 16×16 CUs according to their block size. The two 16×16 CUs (430 and 440) are further partitioned into four 8×8 CUs according to their block size. FIG. 4D illustrates a quad-tree data structure representing the final result of the partitioning process of a CTU (400) as illustrated in FIG. 4C, where each leaf node of the quad tree corresponds to a single CU of a corresponding size ranging from 32×32 to 8×8. Similar to the CTU illustrated in FIG. 4B, each CU may include a coding block (CB) of luminance samples, two corresponding coding blocks of chroma samples of the same size frame, and syntax elements used to code the samples of the coding block. In monochrome pictures or pictures having three distinct color planes, the CU may include a single coding block and syntax structures used to code the samples of the coding block. It should be noted that the quad-tree partitioning illustrated in FIG. 4C and 4D is for illustrative purposes only, and that a single CTU may be partitioned into CUs to adapt to various local characteristics based on quad / ternary / binary-tree partitionings. In a multi-type tree structure, one CTU is divided into a quad-tree structure, and each quad-tree leaf CU can be further divided into binary and ternary-tree structures.As shown in FIG. 4E, for a coding block having width (W) and height (H), there are five possible partition types: quaternary partition, horizontal binary partition, vertical binary partition, horizontal ternary partition, and vertical ternary partition.

[0054] In some implementations, the video encoder (20) may further subdivide the coding block of the CU into one or more M×N prediction blocks (PB). A prediction block is a rectangular (square or non-square) block of samples to which the same prediction (inter or intra prediction) is applied. A prediction unit (PU) of the CU may include a prediction block of luminance samples, two corresponding prediction blocks of chroma samples, and syntax elements used to predict the prediction blocks. In monochrome pictures or pictures having three distinct color planes, the PU may include a single prediction block and syntax structures used to predict the prediction block. The video encoder (20) may generate a predicted luminance block, a predicted Cb block, and a predicted Cr block for each PU of the CU, a predicted Cb block, and a predicted Cr block.

[0055] The video encoder (20) may use intra prediction or inter prediction to generate prediction blocks for the PU. If the video encoder (20) uses intra prediction to generate prediction blocks for the PU, the video encoder (20) may generate prediction blocks for the PU based on decoded samples of frames associated with the PU. If the video encoder (20) uses inter prediction to generate prediction blocks for the PU, the video encoder (20) may generate prediction blocks for the PU based on decoded samples of one or more frames that are not associated with the PU.

[0056] After the video encoder (20) generates predicted luma blocks, Cb blocks, and Cr blocks of one or more PUs of the CU, the video encoder (20) may generate a luma residual block for the CU by subtracting the predicted luma blocks of the CU from the original luma coding block of the CU, so that each sample of the luma residual block of the CU represents the difference between one luma sample of the predicted luma blocks of the CU and the corresponding sample of the original luma coding block of the CU. Similarly, the video encoder (20) may generate a Cb residual block and a Cr residual block for the CU, respectively, so that each sample of the Cb residual block of the CU represents the difference between one Cb sample of the predicted Cb blocks of the CU and the corresponding sample of the original Cb coding block of the CU, and each sample of the Cr residual block of the CU represents the difference between one Cr sample of the predicted Cr blocks of the CU and the corresponding sample of the original Cr coding block of the CU.

[0057] Additionally, as illustrated in FIG. 4C, the video encoder (20) may use quad-tree partitioning to decompose the luminance, Cb, and Cr residual blocks of the CU into one or more luminance, Cb, and Cr transform blocks. A transform block is a rectangular (square or non-square) block of samples to which the same transform is applied. A transform unit (TU) of the CU may include a transform block of luminance samples, two corresponding transform blocks of chroma samples, and syntax elements used to transform the transform block samples. Thus, each TU of the CU may be associated with a luminance transform block, a Cb transform block, and a Cr transform block. In some examples, the luminance transform block associated with the TU may be a sub-block of the luminance residual block of the CU. The Cb transform block may be a sub-block of the Cb residual block of the CU. The Cr transform block may be a sub-block of the Cr residual block of the CU. In monochrome pictures or pictures having three distinct color planes, TU may include a single transform block and syntax structures used to predict samples of the transform block.

[0058] The video encoder (20) may apply one or more transformations to the luminance transformation block of the TU to generate a luminance coefficient block for the TU. The coefficient block may be a two-dimensional array of transformation coefficients. The transformation coefficients may be scalar quantities. The video encoder (20) may apply one or more transformations to the Cb transformation block of the TU to generate a Cb coefficient block for the TU. The video encoder (20) may apply one or more transformations to the Cr transformation block of the TU to generate a Cr coefficient block for the TU.

[0059] After generating a coefficient block (e.g., a luma coefficient block, a Cb coefficient block, or a Cr coefficient block), the video encoder (20) can quantize the coefficient block. Quantization generally refers to a process that provides additional compression by quantizing the transform coefficients to reduce as much as possible the amount of data used to represent the transform coefficients. After the video encoder (20) quantizes the coefficient block, the video encoder (20) can entropy encode the syntax elements representing the quantized transform coefficients. For example, the video encoder (20) can perform Context-Adaptive Binary Arithmetic Coding (CABAC) on the syntax elements representing the quantized transform coefficients. Finally, the video encoder (20) can output a bitstream containing a sequence of bits forming a representation of the coded frames and related data, which is stored in the storage device (32) or transmitted to the destination device (14).

[0060] After receiving a bitstream generated by a video encoder (20), a video decoder (30) can parse the bitstream to obtain syntax elements from the bitstream. The video decoder (30) can reconstruct frames of video data based at least partially on the syntax elements obtained from the bitstream. The process of reconstructing video data is generally opposite to the encoding process performed by the video encoder (20). For example, the video decoder (30) can reconstruct residual blocks associated with the TUs of the current CU by performing inverse transformation on the coefficient blocks associated with the TUs of the current CU. The video decoder (30) also reconstructs coding blocks of the current CU by adding samples of prediction blocks for the PUs of the current CU to the corresponding samples of transformation blocks for the TUs of the current CU. After reconstructing the coding blocks for each CU of the frame, the video decoder (30) can reconstruct the frame.

[0061] As previously mentioned, video coding achieves video compression primarily using two modes: intra-frame prediction (or intra-prediction) and inter-frame prediction (or inter-prediction). It should be noted that IBC can be considered as an intra-frame prediction or a third mode. Between the two modes, inter-frame prediction contributes more to coding efficiency than intra-frame prediction by using motion vectors to predict the current video block from a reference video block.

[0062] However, due to continuous improvements in video data capturing technology and finer video block sizes to preserve video data details, the amount of data required to represent motion vectors for the current frame has increased significantly. One way to overcome this problem is to take advantage of the fact that adjacent CU groups in both spatial and temporal domains not only have similar video data for prediction purposes but also have similar motion vectors between these adjacent CUs. Therefore, motion information from spatially adjacent CUs and / or temporally co-located CUs can be used as an approximation of the motion information (e.g., motion vector) of the current CU by exploring spatial and temporal correlations, also known as the "motion vector predictor (MVP)" of the current CU.

[0063] As described above in relation to FIG. 2, instead of encoding the actual motion vector of the current CU determined by the motion estimation unit (42) into the video bitstream, the motion vector predictor of the current CU is subtracted from the actual motion vector of the current CU to generate the motion vector difference (MVD) for the current CU. By doing so, there is no need to encode the motion vector determined by the motion estimation unit (42) into the video bitstream for each CU of the frame, and the amount of data used to represent motion information in the video bitstream can be significantly reduced.

[0064] Similar to the process of selecting a prediction block from a reference frame during inter-frame prediction of a code block, both the video encoder (20) and the video decoder (30) construct a motion vector candidate list (also called a “merge list”) for the current CU using potential candidate motion vectors associated with spatially adjacent CUs and / or temporally identical CUs of the current CU, and then a series of rules must be adopted by both the video encoder (20) and the video decoder (30) to select one member from the motion vector candidate list as a motion vector predictor for the current CU. By doing so, there is no need to transmit the motion vector candidate list itself between the video encoder (20) and the video decoder (30), and the index of the motion vector predictor selected within the motion vector candidate list is sufficient for the video encoder (20) and the video decoder (30) to use the same motion vector predictor within the motion vector candidate list to encode and decode the current CU.

[0065] In some implementations, each inter-prediction CU has three motion vector prediction modes, including inter (also referred to as "advanced motion vector prediction (AMVP)"), skip, and merge, to construct a list of motion vector candidates. In each mode, one or more motion vector candidates may be added to the list of motion vector candidates according to an algorithm described below. Ultimately, one of the candidate lists is used as the best motion vector predictor of the inter-prediction CU to be encoded into a video bitstream by a video encoder (20) or decoded from a video bitstream by a video decoder (30). To find the best motion vector predictor from the list of candidates, a motion vector competition (MVC) method is introduced to select a motion vector from a given set of candidates for motion vectors, namely the list of motion vector candidates, which includes spatial and temporal motion vector candidates.

[0066] In addition to deriving motion vector predictor candidates from spatially adjacent or temporally identical CUs, motion vector predictor candidates may also be derived from a so-called "history-based motion vector prediction (HMVP)" table. The HMVP table accommodates a predefined quantity of motion vector predictors, each used to encode / decode a specific CU of the CTUs in the same row (or sometimes the same CTU). Due to the spatial and temporal proximity of these CUs, it is highly likely that one of the motion vector predictors in the HMVP table will be reused to encode / decode other CUs within the same row of CTUs. Therefore, higher coding efficiency can be achieved by including the HMVP table in the process of constructing the motion vector candidate list.

[0067] In some implementations, the HMVP table has a fixed length (e.g., 5) and is managed in a quasi-First-In-First-Out (FIFO) manner. For example, when decoding a single inter-coded block of a CU, the motion vector for the CU is reconstructed. Since this motion vector can serve as a motion vector predictor for subsequent CUs, the HMVP table is immediately updated with the reconstructed motion vector. When updating the HMVP table, two scenarios exist: (i) the reconstructed motion vector is different from other existing motion vectors in the HMVP table, or (ii) the reconstructed motion vector is identical to one of the existing motion vectors in the HMVP table. In the first scenario, if the HMVP table is not full, the reconstructed motion vector is added to the HMVP table as the latest motion vector. If the HMVP table is already full, the oldest motion vector in the HMVP table must first be removed before the reconstructed motion vector is added as the latest motion vector. In other words, the HMVP table in this case is similar to a FIFO buffer, located at the front of the FIFO buffer, and motion information associated with other blocks that are previously inter-coded is shifted out of the buffer, and the reconstructed motion vector is added to the tail of the FIFO buffer as the latest member of the HMVP table. In the second scenario, before adding the reconstructed motion vector to the HMVP table as the latest motion vector, the existing motion vector in the HMVP table that is substantially identical to the reconstructed motion vector is removed from the HMVP table.If the HMVP table is also maintained in the form of a FIFO buffer, motion vector predictors following the same motion vector in the HMVP table are shifted forward by one element to occupy the space left by the removed motion vector, and the reconstructed motion vector is added to the tail of the FIFO buffer as the latest member of the HMVP table.

[0068] In different prediction modes such as AMVP, merge, and skip, motion vectors from the HMVP table can be added to the motion vector candidate lists. It can be found that motion information from previous inter-coded blocks in the HMVP table can be utilized for more efficient motion vector prediction, even if they are not adjacent to the current block.

[0069] After selecting one MVP candidate from a given set of candidate motion vectors for the current CU, the video encoder (20) may generate one or more syntax elements for the corresponding MVP candidate and encode them into a video bitstream so that the video decoder (30) can use the syntax elements to search for the MVP candidate from the video bitstream. Depending on the specific mode used to construct the set of motion vector candidates, different modes (e.g., AMVP, merge, skip, etc.) have different sets of syntax elements. In the case of AMVP mode, the syntax elements include inter-prediction indicators (list 0, list 1, or bi-directional prediction), reference indices, motion vector candidate indices, motion vector prediction residual signals, etc. In the case of skip mode and merge mode, only the merge indices are encoded into the bitstream because the current CU inherits other syntax elements, including inter-prediction indicators, reference indices, and motion vectors, from an adjacent CU referenced by the encoded merge index. In the case of skip-coded CUs, the residual signal for motion vector prediction is also omitted.

[0070] FIG. 5 is a block diagram illustrating spatially adjacent and temporally co-located block locations of the current CU to be encoded / decoded according to some implementations of the present disclosure. For a given mode, the availability of motion vectors associated with spatially left and upper adjacent block locations and the availability of motion vectors associated with temporally co-located block locations are checked first, and then motion vectors in the HMVP table are checked to construct a motion vector prediction (MVP) candidate list. In the process of constructing the MVP candidate list, some duplicate MVP candidates are removed from the candidate list, and if necessary, zero-value motion vectors are added so that the candidate list has a fixed length (the fixed length may differ for different modes). After constructing the MVP candidate list, the video encoder (20) can select the optimal motion vector predictor from the candidate list and encode the corresponding index pointing to the selected candidate into a video bitstream.

[0071] In some embodiments, the candidate list (also known as the merge candidate list) is composed of the following five types of candidates in order:

[0072] 1. Spatial MVP of spatially adjacent CUs (i.e., motion vector predictor)

[0073] 2. Temporary MVP of Same-Location CUs

[0074] 3. History-Based MVP of FIFO Tables

[0075] 4. Pairwise Averaging MVP

[0076] 5. Zero MVs

[0077] In some embodiments, the size of the candidate list is signaled in the slice header, and the maximum allowed size of the candidate list is 6 (e.g., in VVC). For each CU code of the merge mode, the optimal merge candidate index is encoded using truncated unary binarization (TU). The first bin of the merge index is coded as the context, and bypass coding is used for the other bins. In the following context of the present disclosure, this extended merge mode is also referred to as the regular merge mode because its concept is identical to the merge mode used in HEVC.

[0078] For example, as shown in Fig. 5, assuming the length of the candidate list is fixed at 2, the following steps can be performed in order in AMVP mode to construct a Motion Vector Predictor (MVP) candidate list for the current CU.

[0079] 1) Select MVP candidates from spatially adjacent CUs.

[0080] a) Deriv at most one non-scaled MVP candidate from one of the two left spatially adjacent CUs starting at A0 and ending at A1.

[0081] b) If no non-scaled MVP candidate is available from the left in the previous step, derive at most one scaled MVP candidate from one of the two left space adjacent CUs starting from A0 and ending at A1.

[0082] c) Deriving at most one non-scaled MVP candidate from one of the three spatially adjacent CUs on the upper side, starting from B0, going to the next B1, and ending at B2.

[0083] d) If A0 and A1 are unavailable or coded in intra-modes, derive at most one scaled MVP candidate from one of the three spatially adjacent CUs on the upper side, starting with B0, then B1, and ending with B2.

[0084] 2) If two MVP candidates are found in the previous step and they are identical, remove one of the two candidates from the MVP candidate list.

[0085] 3) Select MVP candidates from CUs that are in the same location in time.

[0086] a) If there are no 2 MVP candidates in the list of MVP candidates after the previous step, derive at most 1 MVP candidate from the CUs in the same temporal position (e.g., T0).

[0087] 4) Select MVP candidates from the HMVP table.

[0088] a) If the list of MVP candidates after the previous step does not contain 2 MVP candidates, derive up to 2 history-based MVPs from the HMVP table.

[0089] 5) If the MVP candidate list after the previous step does not contain 2 MVP candidates, add up to 2 0-value MVPs to the MVP candidate list.

[0090] Since there are only two candidates in the AMVP mode MVP candidate list configured as above, related syntax elements such as binary flags can be encoded into a bitstream to indicate which of the two MVP candidates in the candidate list is used for decoding the current CU.

[0091] In some implementations, the MVP candidate list for the current CU under skip or merge mode can be constructed by performing a similar set of steps in a similar order as above. The MVP candidate list for skip or merge mode also includes a special type of merge candidate called a "pair-wise merge candidate." The pair-wise merge candidate is generated by averaging the MVs of two previously derived merge mode motion vector candidates. The size of the merge MVP candidate list (e.g., from 1 to 6) is signaled in the slice header of the current CU. For each CU in merge mode, the index of the best merge candidate is encoded using truncated unary binarization (TU). The first bin of the merge index is coded as context, and bypass coding is used for the other bins.

[0092] As described above, history-based MVPs can be added to the AMVP-mode MVP candidate list or the merged MVP candidate list after the spatial MVP and temporal MVP. Motion information of previously inter-coded CUs is stored in the HMVP table and used as an MVP candidate for the current CU. The HMVP table is maintained during the encoding / decoding process. Whenever a non-sub-block inter-coded CU exists, the motion vector information stored in the first entry of the HMVP table is removed, and the associated motion vector information is added as a new candidate to the last entry of the HMVP table (if the HMVP table is already full and there is no duplicate of the associated motion vector information in the table). Alternatively, duplicates of the associated motion vector information are removed from the table before the associated motion vector information is added to the last entry of the HMVP table.

[0093] As mentioned above, an intra block copy (IBC) can significantly improve the coding efficiency of screen content materials. Since the IBC mode is implemented as a block-level coding mode, block matching (BM) is performed in the video encoder (20) to find the optimal block vector for each CU. Here, the block vector is used to indicate the displacement from the current block to a reference block that has already been reconstructed within the current picture. The IBC mode is treated as a third prediction mode other than intra or inter prediction modes.

[0094] At the CU level, the IBC mode can be signaled as IBC AMVP mode or IBC Skip / Merge mode as follows:

[0095] - IBC AMVP Mode: The block vector difference (BVD) between the actual block vector of the CU and the block vector predictor of the CU selected from the block vector candidates of the CU is encoded in the same way as the motion vector difference is encoded under the AMVP mode described above. The block vector prediction method uses two block vector candidates as predictors: one is the left adjacent block vector, and the other is the upper adjacent block vector (if IBC coded). If one of the adjacent block vectors is unavailable, the default block vector is used as the block vector predictor. A binary flag is signaled to indicate the block vector predictor index. The IBC AMVP candidate list consists of spatial and HMVP candidates.

[0096] - IBC Skip / Merge Mode: The merge candidate index is used to indicate which block vector candidates from the merge candidate list (also called the "merge list" or "candidate list") from adjacent IBC-coded blocks are used to predict the block vector for the current block. The IBC merge candidate list consists of spatial, HMVP, and pairwise candidates.

[0097] FIGS. 6A through 6D are block diagrams illustrating the step of deriving temporal motion vector predictors (TMVPs) of the current block or sub-block temporal motion vector predictors (SbTMVPS) of the sub-block according to some implementations of the present disclosure.

[0098] In some embodiments, as described in relation to FIG. 5, only one temporal motion vector predictor (TMVP) candidate is added to the merge candidate list. A first flag (sps_temporal_mvp_enabled_flag) is signaled in the sequence parameter set (SPS) of the picture, and a second flag (slice_temporal_mvp_enabled_flag) is signaled in the slice header to indicate whether the TMVP candidate is enabled or disabled. In particular, in the derivation of the temporal merge candidate, a motion vector scaled from the MVs of the same-position picture, which is the previously coded picture in the reference picture list, is derived. In the derivation of temporal motion candidates, an explicit flag (co-located_from_l0_flag) in the slice header is first sent to the decoder to indicate whether the co-located picture was selected from the first reference frame list (List 0) or the second reference frame list (List 1). A co-located reference index (co-located_ref_idx) is additionally sent to indicate which picture from the list used as the co-located picture for deriving temporal motion candidates was selected. The MVs of List 0 (also referred to as L0) and List 1 (also referred to as L1) of temporal motion candidates are independently derived from the co-located blocks of co-located pictures according to the pseudocode below, in a predefined order for the MVs of different lists.

[0099] Pseudocode for deriving temporal MV from a colocation block for TMVP When deriving the LX MV (where X can be 0 or 1) of the temporal motion candidate, the LY MV (where Y can be 0 or 1) of the same-location block is selected to derive the LX MV of the temporal motion candidate for the current block. The selected LY MV of the same-location block is scaled according to the POC distance as described in the following paragraph. If there is no inverse prediction for the current picture (i.e., there is no reference picture with a POC larger than the current picture), the LX MV of the same-location block is selected first. If the LX MV is unavailable, L(1-X) is selected. Otherwise (if there is an inverse prediction for the current picture), the LN MV of the same-location block is selected first. N is set to 1-the same-location picture list (0 or 1). If the LN MV is unavailable, L(1-N) is selected.

[0100] The scaled motion vector (602) for the temporal merge candidate is obtained by scaling from the selected motion vector of the same-position block using the POC distance tb (604) and POC distance td (606), as illustrated by the dashed line in FIG. 6A. Here, tb is defined as the POC difference between the reference picture of the current picture (e.g., current reference 608) and the current picture (e.g., current picture (610)), and td is defined as the POC difference between the reference picture of the same-position picture (same-position reference (614)) and the same-position picture (same-position picture (612)). The reference picture index of the temporal merge candidate is set to be equal to 0. The actual implementation of the scaling process is described in the HEVC specification. For B-slices, two motion vectors are obtained, one for reference picture list 0 and the other for reference picture list 1, and combined to form a bidirectional prediction merge candidate.

[0101] The scaled motion vector (602) for the temporal merge candidate is obtained by scaling from the selected motion vector of the same-position block using the POC distance tb (604) and POC distance td (606), as illustrated by the dashed line in FIG. 6A. Here, tb is defined as the POC difference between the reference picture of the current picture (e.g., current reference 608) and the current picture (e.g., current picture (610)), and td is defined as the POC difference between the reference picture of the same-position picture (same-position reference (614)) and the same-position picture (same-position picture (612)). The reference picture index of the temporal merge candidate is set to be equal to 0. The actual implementation of the scaling process is described in the HEVC specification. For B-slices, two motion vectors are obtained, one for reference picture list 0 and the other for reference picture list 1, and combined to form a bidirectional prediction merge candidate.

[0102] In a same-location block belonging to a reference frame (e.g., same-location block (620)), a location for a temporal candidate is selected between candidate C0 and C1 as illustrated in FIG. 6B. If the block of location C0 is unavailable, intra-coded, or is currently outside the CTU, location C1 is used. Otherwise, location C0 is used to derive the corresponding temporal merge candidate.

[0103] Some coding standards (e.g., VVC Test Model 1) support the Sub-Block-Based Temporal Motion Vector Prediction (SbTMVP) method. Similar to HEVC's Temporal Motion Vector Prediction (TMVP), SbTMVP improves motion vector prediction and merge modes for the current picture's CUs by using the motion field of a same-location picture. The same same-location picture used in TMVP is used in SbTMVP. SbTMVP differs from TMVP in the following two main aspects.

[0104] 1. TMVP predicts movement at the CU level, but SbTMVP predicts movement at the sub-CU level.

[0105] 2. TMVP selects temporal motion vectors from the same-position block of the same-position picture (the same-position block is the bottom-right or center block relative to the current CU), but SbTMVP applies a motion shift to the selected temporal motion information from the same-position picture. Here, the motion shift is obtained from a motion vector of one of the spatially adjacent blocks of the current CU.

[0106] The SbTMVP process is illustrated in FIGS. 6C-6D. The SbTMVP (SbTMVP (632) in FIG. 6D) predicts motion vectors of sub-CUs (e.g., sub-CUs (634)) within the current CU (current CU (636) in FIG. 6D) in two steps. In the first step, the spatially adjacent A1 in FIG. 6C (e.g., spatially adjacent (638)) is examined. If A1 has a motion vector using a same-position picture (e.g., same-position picture (612) in FIG. 6A) as a reference picture, that motion vector is selected as the motion shift to apply (e.g., motion shift (630) in FIG. 6D). If such a motion vector is not identified, the motion shift is set to a zero-value vector (0, 0). The first available motion vector among the List 0 and List 1 MVs of block A1 is set as the motion shift. In this way, in SbTMVP, the corresponding block can be identified more accurately by comparing it with TMVP, where the corresponding block (sometimes called the same-position block) can always be in the bottom-right or center position relative to the current CU. The pseudocode for determining the movement shift is as follows.

[0107] Pseudocode to confirm movement shifts for SbTMVP in VVC bool terminate = false;motion shift = 0;for (currRefListId = 0; currRefListId < (CurrentSliceType == B_SLICE ? 2 : 1) && !terminate; currRefListId++) { currRefPicList = RefPicList(LDC ? (ColFromL0Flag ? currRefListId : 1 - currRefListId) : currRefListId); if ((interDirA1 & (1 << currRefPicList)) && getRefPic(currRefPicList, refIdxA1[currRefListId]) == ColPic) { motion shift = mvA1[currRefListId]; terminate = true; break;}}

[0108] The variables and functions used in the table above are as follows.

[0109] - ColFromL0Flag: Syntax indicating whether the same-location picture came from the list of pictures referenced in List 0.

[0110] - LDC: Indicates whether all reference pictures have POC values ​​smaller than the current picture.

[0111] - CurrentSliceType: Type of the current slice (picture)

[0112] - count: Number of available merging candidates already derived

[0113] - interDirA1: The interDir of the Nth merging candidate (1:L0, 2:L1 or 3:Bi)

[0114] - refIdxA1[0]: L0 movement information of the Nth merging candidate (e.g., MV, reference index)

[0115] - refIdxA1[1]: L1 movement information of the Nth merging candidate (e.g., MV, reference index)

[0116] - getRefPic(M,I): A function that retrieves the reference picture with the same reference index as I from the reference picture list M.

[0117] In the second step, the motion shift identified in Step 1 is applied (i.e., added to the coordinates of the current block) to obtain sub-CU-level motion information (motion vectors and reference indices) from the same location picture as illustrated in FIG. 6D. In the example of FIG. 6D, it is assumed that the motion shift is set to the motion of block A1. In an actual implementation, the motion shift may be set to one of the motions of blocks A1, A2, B1, or B2.

[0118] First, a representative sub-CU is selected, and the motion information of the corresponding block of that representative sub-CU is used as default motion information. In the existing SbTMVP method, the sub-CU located at the bottom-right of the center position of the current CU is selected as the representative sub-CU. If valid motion information cannot be derived as default motion information from the corresponding block of the representative sub-CU, the SbTMVP candidate is considered unusable. If default motion information is available, the process moves to the next step to derive motion information for each sub-CU within the current CU. Whenever there is no available motion information for the corresponding block of any sub-CU, the default motion information is used as the temporal motion derived for that sub-CU.

[0119] Next, for each sub-CU, the motion information of the corresponding block (the smallest motion grid covering the central sample) within the same-position picture is used to derive motion information for the sub-CU. After identifying the motion information of the same-position sub-CU, it is converted into motion vectors and reference indices of the current sub-CU in a manner similar to the HEVC TMVP process, where temporal motion scaling is applied to align the reference pictures of the temporal motion vectors with the reference pictures of the current CU.

[0120] It should be noted that in the current design, only the motion field within the same-position CTU and the single column to the right of the same-position CTU within the same-position picture can be used to derive SbTMVP and TMVP for each CU. As illustrated in FIG. 7, only the motion information within the same-position CTU and the motion information in the single column to the right of the same-position CTU (CTU2 is the same-position CTU of the current CU in this example) are used to derive the temporal mv for SbTMVP and TMVP. For convenience of explanation, the said same-position CTU and single column are referred to as the "valid area" for deriving SbTMVP / TMVP. In this context, whenever a corresponding N×N block within the same-position picture of a sub-CU is located outside the valid area, the corresponding N×N block is replaced with a replacement block located within the same-position CTU. The location of the replacement N×N block is derived by clipping the original location of the corresponding N×N block to be within the valid area using the formula below. In the formula below (position clipping process for each sub-CU), CurPicWidthInSamplesY and CurPicHeightInSamplesY are the width and height of the coded picture, CTUWidthInSamplesX and CTUWidthInSamplesY are the width and height of the CTU, and xCtb and yCtb are the horizontal and vertical positions of the top-left sample of the same-position CTU. xColCtrCb and yColCtrCb are the horizontal and vertical positions of the representative sample of the sub-CU, and MotionShiftX and MotionShiftY are the x and y components of the motion shift, respectively. The Clip3(x,y,z) and Min(x,y) functions are defined as follows.

[0121] Clip3( x, y, z ) =

[0122] Min( x, y ) =

[0123] 동일 위치 픽처내 동일 위치 블록의 위치( xColCb, yColCb ) 다음과 같이 도출한다 xColCb = Clip3( xCtb, Min( CurPicWidthInSamplesY - 1, xCtb + CTUWidthInSamplesY + 3 ), xColCtrCb + MotionShiftX ) )yColCb = Clip3( yCtb, Min( CurPicHeightInSamplesY - 1, yCtb + CTUHeightInSamplesY - 1 ), yColCtrCb + MotionShiftY )

[0124] In VVC, a combined sub-block-based merge list containing both SbTMVP candidates and affine merge candidates is used for signaling the sub-block-based merge mode. The SbTMVP mode is enabled / disabled by the sequence parameter set (SPS) flag. When the SbTMVP mode is enabled, the SbTMVP predictor is added as the first entry to the sub-block-based merge candidate list, followed by the affine merge candidates. The size of the sub-block-based merge list is signaled by the SPS, and the maximum allowed size of the sub-block-based merge list in VVC is 5.

[0125] The sub-CU size used in SbTMVP is fixed at 8×8, and as with affine merge mode, SbTMVP mode is applicable only to CUs where both width and height are 8 or greater. Additionally, in the current VVC, field compression is performed in 8×8 units compared to HEVC's 16×16 units for storing temporal motion fields used by TMVP and SbTMVP.

[0126] In some embodiments, the movement shift for SbTMVP is always derived from the list 0 mv of the adjacent block, and if the list 0 mv is unavailable, the movement shift for SbTMVP is derived using the list 1 mv of the adjacent block. Pseudocode is described below.

[0127] Pseudocode to determine movement shift for SbTMVP bool terminate = false;for (currRefListId = 0; currRefListId < (CurrentSliceType == B_SLICE ? 2 : 1) && !terminate; currRefListId++) {currRefPicList = currRefListId; if ((interDirA1 & (1 << currRefPicList)) && getRefPic(currRefPicList, refIdxA1[currRefListId]) == ColPic) { motion shift = mvA1[currRefListId]; terminate = true; break;}}

[0128] In some embodiments, the movement shift for SbTMVP is always derived from List 1 mv of the adjacent block, and if List 1 mv is unavailable, List 0 mv of the adjacent block is used to derive the movement shift. Pseudocode is described below.

[0129] Pseudocode to determine movement shift for SbTMVP bool terminate = false;for (currRefListId = 0; currRefListId < (CurrentSliceType == B_SLICE ? 2 : 1) && !terminate; currRefListId++) {currRefPicList = 1-currRefListId; if ((interDirA1 & (1 << currRefPicList)) && getRefPic(currRefPicList, refIdxA1[currRefListId]) == ColPic) { motion shift = mvA1[currRefListId]; terminate = true; break;}}

[0130] In some embodiments, whenever there is a corresponding block of a sub-CU located outside the valid area, the zero vector is used as the motion shift vector to derive SbTMVP. By doing so, it is ensured that the corresponding blocks of all sub-CUs of the current CU are located within the valid area. Therefore, a position clipping process is not required for each sub-CU. There are various methods for determining whether the corresponding block of a sub-CU of the current CU is located outside the valid area. In one example, the corresponding block of the left-top N×N sub-CU and the corresponding block of the right-bottom N×N sub-CU are checked to determine whether the two corresponding blocks are within the valid area. If either one is located outside the valid area, the zero vector is used as the motion shift vector. Otherwise (if both corresponding blocks are located within the valid area), the derived motion shift is used in SbTMVP. In some embodiments, whenever there is a corresponding block of a sub-CU located outside the valid area, SbTMVP is considered unusable for the current CU.

[0131] In some embodiments, whenever there is a corresponding block of a sub-CU located outside the valid area, the motion shift is modified to ensure that the corresponding blocks of all sub-CUs are located within the valid area. Therefore, a position clipping process is not required for each sub-CU.

[0132] In some embodiments, a zero vector is always used for the motion shift to derive SbTMVP.

[0133] In some embodiments, it is proposed to use a default MV derived from a representative sub-CU as the MV of a sub-CU having a corresponding block located outside the valid area.

[0134] FIG. 7 illustrates a block diagram for determining a valid region for deriving TMVP and SbTMVP for a coding block (e.g., current CU (702)) within a current picture (e.g., current picture (704)) according to some implementations of the present disclosure. The valid region is a region within a same-location picture (e.g., same-location picture (704')) containing a CU (e.g., corresponding CU (702')) that corresponds to the current CU (e.g., current CU (702)) searching for TMVP or SbTMVP. In some implementations, the valid region is determined by a CTU (e.g., CTU2) and a single column (e.g., a single column TMV buffer (706)) to derive TMVP and SbTMVP. Limiting the valid region is a design for reducing memory usage. By limiting the valid area to the same-position CTU plus one column, only motion information within the valid area is stored in internal memory (e.g., cache), thereby reducing the average cost (time or energy) of accessing temporal motion data from external memory. Currently, the maximum CTU size in VVC is 128×128 (the maximum CTU size can be determined at a later stage for the VVC profile), and the CTU size can be set to less than 128×128 (e.g., 64×64 or 32×32). In one example, if the CTU size is set to 64×64, the valid area is limited to the same-position 64×64 block plus one column. Since the design of the temporal MV buffer for the maximum CTU already exists, using a valid area smaller than the maximum CTU size may not be wise from the perspective of coding efficiency. In some embodiments, the valid area is fixed to the maximum allowable CTU size plus one column, regardless of the CTU size in use.

[0135] In some embodiments, the valid region is modified to become just a same-location CTU.

[0136] In some embodiments, if the CTU size is equal to the maximum CTU size, the valid area is the same-position CTU plus one column. If the CTU size is smaller than the maximum CTU size, the valid area is modified to the same-position CTU with one column to the right of the same-position CTU and one row below the same-position CTU.

[0137] FIGS. 8A and 8B illustrate a flowchart illustrating an exemplary process (800) in which a video coder according to some implementations of the present invention implements a technique for deriving sub-block temporal motion vector predictors. The process (800) may be a decoding or encoding process, but for convenience, the process (800) is described as a decoding process performed by a video decoder (e.g., the video decoder (30) of FIG. 3).

[0138] As a first step, the decoder determines the same-position picture of the current coding unit (805) (e.g., receives a first syntax element from the bitstream indicating whether the same-position picture of the current frame is from the first list or the second list, and then receives a second syntax element from the bitstream indicating which frame of the selected list is used as the same-position frame). For example, referring to FIG. 6A, the current CU (601) in the current picture (610) corresponds to the same-position Cu (601') in the same-position picture (612).

[0139] Next, the decoder locates a spatially adjacent block of the current coding unit (810). For example, referring to FIG. 6D, the current coding unit (e.g., current CU (636)) has a spatially adjacent block (638, block A1). In some embodiments, the spatially adjacent block is a coding unit or a sub-block.

[0140] After locating spatially adjacent blocks, the decoder determines a motion shift vector for the current coding unit (815). The motion shift vector indicates a spatial position shift between the current coding unit (e.g., the current CU (636) in FIG. 6D) within the current picture (e.g., the current picture (610) in FIG. 6D)) and the corresponding same-position block (e.g., the spatially adjacent (638', block A1') in FIG. 6D) within the same-position picture (e.g., the same-position picture (612) in FIG. 6D).

[0141] To determine the motion shift vector, the decoder sequentially examines each of the motion vectors included in List 0 of the spatially adjacent blocks (820). As it is determined that the motion vector in List 0 uses a same-position picture as the reference picture for said motion vector (825), the decoder sets said motion vector in List 0 as the motion shift vector (e.g., motion shift vector (630)) (830) and abandons examining subsequent motion vectors in List 0 of the spatially adjacent blocks and motion vectors in List 1 (835). Consequently, the search for the motion vector ends and the first matching motion vector in List 0 is used as the motion shift vector. In other words, the decoder always checks the motion vectors included in List 0 of the spatially adjacent blocks first before checking List 1.

[0142] Meanwhile, as it is determined that there are no motion vectors in List 0 that use the same-position picture as a reference picture (840), the decoder sequentially examines each of the motion vectors included in List 1 of spatially adjacent blocks (845). That is, the decoder checks List 1 of spatially adjacent blocks of motion vectors only when the search for motion vectors in List 0 returns a negative result.

[0143] While searching for motion vectors in List 1 of spatially adjacent blocks, as it is determined that a motion vector in List 1 uses a same-position picture as a reference picture for said motion vector (850), the decoder sets said motion vector in List 1 as a motion shift vector (855) and abandons examining subsequent motion vectors in List 1 (860). That is, the first matching motion vector in List 1 is used as the motion shift vector. As it is determined that no motion vector in List 1 uses a same-position picture as a reference picture for said motion vector (865), the decoder sets the motion shift vector as a zero-value vector (870). As a result, the corresponding coding unit and the current coding unit are in the same relative position with respect to the same-position picture and the current picture (e.g., there is no shift of motion between the current coding unit and the corresponding coding unit).

[0144] Finally, the decoder reconstructs a sub-block-based temporal motion vector for each of the multiple sub-blocks within the current coding unit from the corresponding sub-block within the same-position picture based on the motion shift (875). For example, referring to FIG. 6D, the sub-block temporal motion vector predictor (632) is constructed by locating the corresponding sub-block temporal motion vector (631) using the motion shift vector (630) after scaling (e.g., the scaling process described in relation to FIG. 6A and the associated description). In some embodiments, the sub-block includes one or two temporal motion vectors from List 0 and List 1.

[0145] In some embodiments, the step of reconstructing a sub-block-based temporal motion vector for each sub-block of a plurality of sub-blocks of the current coding unit from a corresponding sub-block within a same-position picture based on a motion shift vector comprises predicting sub-block-based temporal motion vectors for each sub-block of a plurality of sub-blocks of the current coding unit, which includes: searching for a same-position sub-block corresponding to each sub-block within a predefined region (e.g., valid region) within the same-position picture based on a motion shift vector; The method includes the step of identifying one or two motion vectors of a same-position sub-block as it is determined that the same-position sub-block exists within a predefined area within the same-position picture, and setting a sub-block-based temporal motion vector for each sub-block to one or two motion vectors scaled based on a first POC (Picture Order count) distance between the current picture and the reference picture of the current picture (e.g., POC distance tb in FIG. 6A) and a second POC distance between the same-position picture and the reference picture of the same-position picture (e.g., POD distance td in FIG. 6A). In some embodiments, as it is determined that the same-position sub-block does not exist within a predefined area within the same-position picture, the sub-block-based temporal motion vectors for the corresponding sub-block are set to zero-value motion vectors. In some other embodiments, as it is determined that a same-location sub-block does not exist within a predefined area within a same-location picture, an alternative sub-block within a predefined area within a same-location picture is set as the corresponding sub-block. For example, the alternative sub-block is a boundary sub-block within a predefined area closest to the same-location sub-block.

[0146] In some embodiments, the predefined area has a size equal to the maximum allowable CTU size plus one column, regardless of the size of the CTU containing the same-location coding unit.

[0147] In some embodiments, the decoder first checks the motion vectors in List 1 of spatially adjacent blocks before checking List 0.

[0148] In one or more examples, the described functions may be implemented in hardware, software, firmware, or any combination thereof. When implemented in software, the functions may be stored on or transmitted through a computer-readable medium as one or more instructions or code and may be executed by a hardware-based processing unit. A computer-readable medium may include a computer-readable storage medium corresponding to a tangible medium such as a data storage medium, or a communication medium including any medium that facilitates the transmission of a computer program from one place to another (e.g., according to a communication protocol). In this way, a computer-readable medium may generally correspond to (1) a non-transient tangible computer-readable storage medium or (2) a communication medium such as a signal or a carrier wave. A data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, code and / or data structures for the implementation of the embodiment described in this application. A computer program product may include a computer-readable medium.

[0149] The terms used when describing the embodiments of this specification are used solely for the purpose of describing specific embodiments and are not intended to limit the claims. As used in the description of the embodiments and the appended claims, singular forms include plural forms unless the context clearly indicates a different meaning. The term "and / or" as used in this specification should be understood to refer to any or all possible combination of one or more related listed items. It will be understood that the terms "comprising" and / or "comprising" as used in this specification specify the presence of the mentioned features, elements, and / or components, but do not exclude the presence or addition of one or more other features, elements, components, and / or groups thereof.

[0150] It should be understood that while terms such as "first," "second," etc., may be used in this specification to describe various elements, these elements are not limited by such terms. These terms are used solely to distinguish one element from another. For example, without departing from the scope of the embodiments herein, the first electrode may be referred to as the second electrode, and similarly, the second electrode may also be referred to as the first electrode. Both the first electrode and the second electrode are electrodes, but they are not the same electrode.

[0151] The description of the contents of this application is provided for illustrative and illustrative purposes only and is not intended to encompass the description of the contents of this application or to limit the invention to the disclosed form. From the foregoing description and the accompanying drawings, those skilled in the art will readily understand various modifications, changes, and alternative embodiments. The embodiments are described to best utilize the basic principles and various embodiments of the invention so that those skilled in the art may understand the principles of the invention, practical applications, and the invention as implemented in various ways and modify the invention to suit a specific use. Accordingly, the scope of the claims is not limited to specific examples of the disclosed embodiments, and modifications and other embodiments should be interpreted as being included within the appended claims.

Claims

Claim 1 A step of dividing a current picture of a video into multiple coding units; a step of determining a co-located picture of the current picture; a step of determining a motion shift vector for the current coding unit based on a motion vector of a spatially adjacent block of the current coding unit, wherein the motion shift vector indicates a spatial positional shift between a sub-block of multiple sub-blocks within the current coding unit within the current picture and a corresponding sub-block within the co-located picture; and includes the step of reconstructing a sub-block-based temporal motion vector for the sub-blocks of the plurality of sub-blocks within the current coding unit from the corresponding sub-blocks within the same-position picture based on the motion shift vector, wherein the step of reconstructing a sub-block-based temporal motion vector for the sub-blocks of the plurality of sub-blocks within the current coding unit from the corresponding sub-blocks within the same-position picture based on the motion shift vector comprises the step of determining the same-position sub-blocks corresponding to each sub-block based on the motion shift vector within a predefined area within the same-position picture, wherein if the position determined according to the motion shift vector is located outside the predefined area, the position is clipped into the predefined area and determined as the position of the same-position sub-block; A video encoding method comprising the step of setting a sub-block-based temporal motion vector for each sub-block to one or two scaled motion vectors derived based on one or two motion vectors of the same-position sub-block. Claim 2 A video encoding method according to claim 1, wherein the step of setting a sub-block-based temporal motion vector for each sub-block to one or two scaled motion vectors derived based on one or two motion vectors of the same-position sub-block comprises the step of setting a sub-block-based temporal motion vector for each sub-block to one or two scaled motion vectors derived based on one or two motion vectors of the same-position sub-block, a first POC (Picture Order count) distance between the current picture and the reference picture of the current picture, and a second POC distance between the same-position picture and the reference picture of the same-position picture. Claim 3 A video encoding method according to claim 1, wherein the step of determining a motion shift vector for the current coding unit based on a motion vector of a spatially adjacent block of the current coding unit includes the step of setting the motion vector associated with the first reference picture list as the motion shift vector, as determined that the motion vector associated with the first reference picture list of the spatially adjacent block uses the same-position picture as the reference picture of the motion vector associated with the first reference picture list. Claim 4 A video encoding method according to claim 1 or 3, wherein the step of determining a motion shift vector for the current coding unit based on a motion vector of a spatially adjacent block of the current coding unit comprises: determining that the motion vector associated with the first reference picture list does not use the same-position picture as the reference picture of the motion vector associated with the first reference picture list; determining that the motion vector associated with the second reference picture list of the spatially adjacent block uses the same-position picture as the reference picture of the motion vector associated with the second reference picture list; or determining that the motion vector associated with the second reference picture list of the spatially adjacent block does not use the same-position picture as the reference picture of the motion vector associated with the second reference picture list; and setting the motion shift vector to a zero-value vector. Claim 5 A video encoding method according to claim 1, wherein the step of reconstructing a sub-block-based temporal motion vector for a plurality of sub-blocks within a current coding unit from a corresponding sub-block within the same-position picture based on the motion shift vector further comprises the step of determining whether the same-position sub-block exists within the predefined area within the same-position picture. Claim 6 In claim 5, the step of reconstructing a sub-block-based temporal motion vector for a plurality of sub-blocks within the current coding unit from a corresponding sub-block within the same-position picture based on the motion shift vector further comprises the step of setting the sub-block-based temporal motion vectors for the corresponding sub-blocks to zero-value motion vectors as determined that the same-position sub-block does not exist within the predefined region within the same-position picture. Claim 7 In claim 5, the step of reconstructing a sub-block-based temporal motion vector for a plurality of sub-blocks within a current coding unit from a corresponding sub-block within a same-position picture based on the motion shift vector further comprises the step of setting an alternative sub-block within the predefined area within the same-position picture as the corresponding sub-block, wherein the alternative sub-block is a boundary sub-block within the predefined area closest to the same-position sub-block. Claim 8 A video encoding method according to claim 1, wherein the spatially adjacent block of the current coding unit is a coding unit or a sub-block of a coding unit. Claim 9 A video encoding method according to claim 1, wherein the predefined region has a size equal to the maximum allowable CTU size plus one column, regardless of the size of the CTU containing the same-location sub-block, and the maximum allowable CTU size is 128×128. Claim 10 A step of acquiring a bit stream comprising a plurality of coding units, wherein the plurality of coding units divide the current picture of the video; a step of determining a co-located picture of the current picture; a step of determining a motion shift vector for the current coding unit based on a motion vector of a spatially adjacent block of the current coding unit, wherein the motion shift vector indicates a spatial position shift between a sub-block of a plurality of sub-blocks within the current coding unit within the current picture and a corresponding sub-block within the co-located picture; and includes the step of reconstructing a sub-block-based temporal motion vector for the sub-blocks of the plurality of sub-blocks within the current coding unit from the corresponding sub-blocks within the same-position picture based on the motion shift vector, wherein the step of reconstructing a sub-block-based temporal motion vector for the sub-blocks of the plurality of sub-blocks within the current coding unit from the corresponding sub-blocks within the same-position picture based on the motion shift vector comprises the step of determining the same-position sub-blocks corresponding to each sub-block based on the motion shift vector within a predefined area within the same-position picture, wherein if the position determined according to the motion shift vector is located outside the predefined area, the position is clipped into the predefined area and determined as the position of the same-position sub-block; A video decoding method comprising the step of setting a sub-block-based temporal motion vector for each sub-block to one or two scaled motion vectors derived based on one or two motion vectors of the same-position sub-block. Claim 11 A video decoding method according to claim 10, wherein the step of setting a sub-block-based temporal motion vector for each sub-block to one or two scaled motion vectors derived based on one or two motion vectors of the same-position sub-block comprises the step of setting a sub-block-based temporal motion vector for each sub-block to one or two scaled motion vectors derived based on one or two motion vectors of the same-position sub-block, a first POC (Picture Order count) distance between the current picture and the reference picture of the current picture, and a second POC distance between the same-position picture and the reference picture of the same-position picture. Claim 12 A video decoding method according to claim 10, wherein the step of determining a motion shift vector for the current coding unit based on a motion vector of a spatially adjacent block of the current coding unit comprises the step of setting the motion vector associated with the first reference picture list as the motion shift vector, as determined that the motion vector associated with the first reference picture list of the spatially adjacent block uses the same-position picture as the reference picture of the motion vector associated with the first reference picture list. Claim 13 A video decoding method comprising the step of determining a motion shift vector for a current coding unit based on a motion vector of a spatially adjacent block of the current coding unit, wherein, upon determining that the motion vector associated with the first reference picture list does not use the same-position picture as the reference picture of the motion vector associated with the first reference picture list; upon determining that the motion vector associated with the second reference picture list of the spatially adjacent block uses the same-position picture as the reference picture of the motion vector associated with the second reference picture list; and upon determining that the motion vector associated with the second reference picture list of the spatially adjacent block does not use the same-position picture as the reference picture of the motion vector associated with the second reference picture list, the motion shift vector associated with the second reference picture list is set as the motion shift vector; or upon determining that the motion vector associated with the second reference picture list of the spatially adjacent block does not use the same-position picture as the reference picture of the motion vector associated with the second reference picture list, the motion shift vector is set as a zero-value vector. Claim 14 A video decoding method according to claim 10, wherein the step of reconstructing a sub-block-based temporal motion vector for a plurality of sub-blocks within the current coding unit from a corresponding sub-block within the same-position picture based on the motion shift vector further comprises the step of determining whether the same-position sub-block exists within the predefined area within the same-position picture. Claim 15 A video decoding method according to claim 14, wherein the step of reconstructing a sub-block-based temporal motion vector for a plurality of sub-blocks within a current coding unit from a corresponding sub-block within a same-position picture based on the motion shift vector further comprises the step of setting the sub-block-based temporal motion vectors for the corresponding sub-blocks to zero-value motion vectors as determined that the same-position sub-block does not exist within the predefined region within the same-position picture. Claim 16 In claim 14, the step of reconstructing a sub-block-based temporal motion vector for a plurality of sub-blocks within a current coding unit from a corresponding sub-block within a same-position picture based on the motion shift vector further comprises the step of setting an alternative sub-block within the predefined area within the same-position picture as the corresponding sub-block, wherein the alternative sub-block is a boundary sub-block within the predefined area closest to the same-position sub-block. Claim 17 A video decoding method according to claim 10, wherein the spatially adjacent block of the current coding unit is a coding unit or a sub-block of a coding unit. Claim 18 A video decoding method according to claim 10, wherein the predefined region has a size equal to the maximum allowable CTU size plus one column, regardless of the size of the CTU containing the same-location sub-block, and the maximum allowable CTU size is 128×128. Claim 19 A computing device comprising: one or more processors; memory coupled to said one or more processors; and a plurality of programs stored in said memory, wherein when said plurality of programs are executed by said one or more processors, said computing device causes said computing device to perform a video decoding method of any one of claims 10 to 18. Claim 20 A non-transient computer-readable storage medium for storing a plurality of programs to be executed by a computing device having one or more processors, wherein, when the plurality of programs are executed by the one or more processors, the computing device is configured to perform a video encoding method of any one of claims 1 to 9 to generate a bitstream and transmit said bitstream, or receive a bitstream and perform a video decoding method of any one of claims 10 to 18 based on said bitstream. Claim 21 A bitstream storage method comprising: a step of generating the bitstream by performing a video encoding method according to any one of claims 1 to 9; and a step of storing the bitstream.

Images