Method and apparatus for encoding video data

Abstract

A method and apparatus for encoding video data. The method includes: determining whether a coding unit is in a sub-block-based temporal motion vector prediction (SbTMVP) mode; if the coding unit is in the SbTMVP mode, determining whether the coding unit and a spatially neighboring block used to determine the temporal vector of the coding unit are both within the same motion estimation region (MER); if the coding unit and the spatially neighboring block are both within the same MER, setting the temporal vector of the coding unit to zero; and sending a first syntax element and a second syntax element, wherein the first syntax element is used to define the MER and the second syntax element is used to indicate whether the coding unit is in the SbTMVP mode.

Description

[0001] This application is a divisional application of the invention patent application with the application date of December 23, 2020, application number "202080093766.2", and title "About Merging Candidate Motion Estimation Areas". Technical Field

[0002] The present application relates generally to video coding and compression, and more particularly to methods and apparatus for improving the construction of merge candidate lists. Background Art

[0003] Various electronic devices (such as digital televisions, laptop or desktop computers, tablet computers, digital cameras, digital recording devices, digital media players, video game consoles, smart phones, video teleconferencing devices, video streaming devices, etc.) support digital video. Electronic devices transmit, receive, encode, decode and / or store digital video data by implementing video compression / decompression standards defined by MPEG-4, ITU-T H.263, ITU-T H.264 / MPEG-4, Part 10, Advanced Video Codec (AVC), High Efficiency Video Codec (HEVC) and Versatile Video Codec (VVC) standards. Video compression typically includes performing spatial (intra-frame) prediction and / or temporal (inter-frame) prediction to reduce or remove redundancy inherent in video data. For block-based video codecs, a video frame is partitioned into one or more slices, each slice having multiple video blocks, which may also be referred to as coding tree units (CTUs). Each CTU may contain a coding unit (CU) or be recursively split into smaller CUs until a predefined minimum CU size is reached. Each CU (also called a leaf-CU) contains one or more transform units (TUs) and each CU also contains one or more prediction units (PUs). Each CU can be coded in intra, inter, or IBC mode. Video blocks in an intra-coded (I) slice of a video frame are coded using spatial prediction with respect to reference samples in neighboring blocks within the same video frame. Video blocks in an inter-coded (P or B) slice of a video frame can use spatial prediction with respect to reference samples in neighboring blocks within the same video frame or temporal prediction with respect to reference samples in other previous reference video frames and / or future reference video frames.

[0004] A prediction block for the current video block to be coded is derived based on spatial or temporal prediction of previously coded reference blocks (e.g., neighboring blocks). The process of finding the reference block can be accomplished using a block matching algorithm. The residual data representing the pixel differences between the current block to be coded and the prediction block is called a residual block or prediction error. Inter-coded blocks are encoded based on the residual block and motion vectors pointing to reference blocks in a reference frame that form the prediction block. The process of determining the motion vector is often referred to as motion estimation. Intra-coded blocks are encoded based on the intra-prediction mode and the residual block. For further compression, the residual block is transformed from the pixel domain to a transform domain (e.g., the frequency domain) to yield residual transform coefficients, which can then be quantized. The quantized transform coefficients, initially arranged in a two-dimensional array, can be scanned to generate a one-dimensional vector of transform coefficients, which can then be entropy coded into the video bitstream to achieve even greater compression.

[0005] The coded video bitstream is then stored in a computer-readable storage medium (e.g., a flash memory) for access by another electronic device with digital video capabilities or sent directly to the electronic device via wired or wireless communication. The electronic device then performs video decompression (which is the reverse process of the video compression described above), for example, by parsing the coded video bitstream to obtain syntax elements from the bitstream, and reconstructing digital video data from the coded video bitstream into its original format based at least in part on the syntax elements obtained from the bitstream, and the electronic device presents the reconstructed digital video data on a display of the electronic device.

[0006] As digital video quality evolves from HD to 4K×2K or even 8K×4K, the amount of video data to be encoded / decoded increases exponentially. Encoding / decoding video data more efficiently while maintaining the image quality of the decoded video data remains a long-standing challenge. Summary of the Invention

[0007] The present application describes embodiments related to video data encoding and decoding, and more particularly, describes embodiments related to a method and apparatus for constructing a merge candidate list.

[0008] According to a first aspect of the present application, a method for encoding video data includes: determining whether a coding unit is in a sub-block-based temporal motion vector prediction (SbTMVP) mode; when the coding unit is in the SbTMVP mode, determining whether the coding unit and a spatially neighboring block used to determine the temporal vector of the coding unit are both within the same motion estimation region (MER); when the coding unit and the spatially neighboring block are both within the same MER, setting the temporal vector of the coding unit to zero; sending a first syntax element and a second syntax element; wherein the first syntax element is used to define the MER and the second syntax element is used to indicate whether the coding unit is in the SbTMVP mode.

[0009] According to a second aspect of the present application, a method for encoding video data includes: determining whether a coding unit is in a sub-block-based temporal motion vector prediction SbTMVP mode; when the coding unit is in the SbTMVP mode, determining whether the coding unit and a first spatial neighboring block used to determine the temporal vector of the coding unit are both within the same motion estimation region MER; when the coding unit and the first spatial neighboring block are both within the same MER, selecting a second spatial neighboring block outside the MER as a replacement for the first spatial neighboring block to determine the temporal vector of the coding unit; sending a first syntax element and a second syntax element; wherein the first syntax element is used to define the MER and the second syntax element is used to indicate whether the coding unit is in the SbTMVP mode.

[0010] According to a third aspect of the present application, a method for encoding video data includes: determining whether a coding unit is in a merge mode; if the coding unit is in the merge mode, determining whether the coding unit and a first spatial neighboring block used to determine a first merge candidate are both within the same motion estimation region MER; if the coding unit and the first spatial neighboring block are both within the same MER, selecting a second merge candidate derived from a second spatial neighboring block as the first merge candidate; sending a first syntax element and a second syntax element; wherein the first syntax element is used to define the MER and the second syntax element is used to indicate whether the coding unit is in the merge mode

[0011] According to a fourth aspect of the present application, a method for encoding video data includes: determining whether a coding unit is in a merge mode; if the coding unit is in the merge mode, determining whether the coding unit and a first spatial neighboring block used to determine a merge candidate are both within the same motion estimation region MER; if the coding unit and the first spatial neighboring block are both within the same MER, selecting a second spatial neighboring block outside the MER to replace the first spatial neighboring block to determine the merge candidate; sending a first syntax element and a second syntax element; wherein the first syntax element is used to define the MER and the second syntax element is used to indicate whether the coding unit is in the merge mode.

[0012] In some embodiments, the merge mode is an intra block copy (IBC) merge.

[0013] In some embodiments, the merge mode is a combined inter and intra prediction (CIIP) merge.

[0014] According to a fifth aspect of the present application, an electronic device includes one or more processing units, a memory, and a plurality of programs stored in the memory. When the programs are executed by the one or more processing units, the electronic device performs the method for encoding video data as described above.

[0015] According to a sixth aspect of the present application, a non-volatile computer-readable storage medium stores a plurality of programs for execution by an electronic device having one or more processing units. When executed by the one or more processing units, the programs cause the electronic device to perform the method for encoding video data as described above.

[0016] According to the seventh aspect of the present application, a computer program product includes a non-transitory computer-readable storage medium storing a plurality of programs executed by a computing device having one or more processors, wherein the plurality of programs, when executed by the one or more processors, causes the computing device to perform the method of encoding video data as described above.

[0017] According to an eighth aspect of the present application, a non-volatile computer-readable storage medium stores a bit stream, wherein the bit stream is obtained by encoding by executing the method for encoding video data as described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings, which are included to provide a further understanding of the embodiments and are incorporated in and constitute a part of this specification, illustrate the described embodiments and together with the description serve to explain the basic principles. Like reference numerals designate corresponding parts.

[0019] Figure 1is a block diagram illustrating an exemplary video encoding and decoding system according to some embodiments of the present disclosure.

[0020] Figure 2 is a block diagram illustrating an exemplary video encoder according to some embodiments of the present disclosure.

[0021] Figure 3 is a block diagram illustrating an exemplary video decoder according to some embodiments of the present disclosure.

[0022] Figures 4A to 4E is a block diagram illustrating how a frame is recursively partitioned into multiple video blocks of different sizes and shapes according to some embodiments of the present disclosure.

[0023] Figure 5A and Figure 5B is a block diagram illustrating an exemplary sub-block based temporal motion vector prediction (SbTMVP) process in VVC according to some embodiments of the present disclosure.

[0024] Figure 6A and Figure 6B is a block diagram illustrating an exemplary control point-based affine motion model according to some embodiments of the present disclosure.

[0025] Figure 7 is a block diagram illustrating exemplary affine motion vector prediction (AMVP) for sub-blocks according to some embodiments of the present disclosure.

[0026] Figure 8 is a block diagram illustrating exemplary locations of inherited affine motion predictors according to some embodiments of the present disclosure.

[0027] Figure 9 is a block diagram illustrating exemplary control point motion vector inheritance according to some embodiments of the present disclosure.

[0028] Figure 10 is a block diagram illustrating exemplary positions of candidate positions for constructing an affine merge pattern according to some embodiments of the present disclosure.

[0029] Figure 11 is a block diagram illustrating exemplary locations of spatially neighboring blocks for deriving a temporal vector according to some embodiments of the present disclosure.

[0030] Figure 12 is a block diagram illustrating exemplary locations of IBC space candidates according to some embodiments of the present disclosure.

[0031] Figure 13 is a block diagram illustrating exemplary locations of IBC space candidates according to some embodiments of the present disclosure.

[0032] Figure 14 and Figure 15 is a block diagram illustrating exemplary locations of non-adjacent spatial candidates according to some embodiments of the present disclosure.

[0033] Figure 16 is a flowchart illustrating an exemplary process of determining a current merge candidate alternative when the current coding unit / block is in the same motion estimation region (MER) as a spatially neighboring block used to determine the current merge candidate according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0034] Reference will now be made in detail to the specific embodiments, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous non-limiting specific details are set forth to facilitate understanding of the subject matter presented herein. However, it will be apparent to those skilled in the art that various alternatives may be used without departing from the scope of the claims, and that the subject matter may be practiced without these specific details. For example, it will be apparent to those skilled in the art that the subject matter presented herein may be implemented on many types of electronic devices having digital video capabilities.

[0035] In some embodiments, several modifications to the newly derived codec tools are introduced to support parallel motion estimation or a more parallel-friendly design for parallel motion estimation of blocks within the same parallel motion estimation region (PMER). The size of the PMER can be implicitly predetermined at different levels for both the encoder and decoder. For example, the size of the PMER can be fixed to a predefined size (e.g., 32×32, 64×64), or the PMER can be set to the coding tree unit (CTU) size, the largest transform unit (TU) size, or the virtual pipe data unit (VPDU) size. Alternatively, additional syntax elements can be signaled at different levels (e.g., sequence level, picture level, slice level, tile level, CTU level, and / or block level) to indicate the size (e.g., width and height) of the PMER. Other syntax elements can also be signaled at different levels (e.g., sequence level, picture level, slice level, tile level, CTU level, and / or block level) to indicate whether the scheme is enabled.

[0036] Figure 1 FIG. 1 is a block diagram illustrating an exemplary system 10 for encoding and decoding video blocks in parallel according to some embodiments of the present disclosure. Figure 1As shown in FIG, system 10 includes a source device 12 that generates and encodes video data to be later decoded by a destination device 14. Source device 12 and destination device 14 may include any of a wide variety of electronic devices, including desktop or laptop computers, tablet computers, smartphones, set-top boxes, digital televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, etc. In some implementations, source device 12 and destination device 14 are equipped with wireless communication capabilities.

[0037] In some embodiments, target device 14 may receive the encoded video data to be decoded via link 16. Link 16 may include any type of communication medium or device capable of moving the encoded video data from source device 12 to target device 14. In one example, link 16 may include a communication medium that enables source device 12 to send the encoded video data directly to target device 14 in real time. The encoded video data may be modulated according to a communication standard (such as a wireless communication protocol) and sent to target device 14. The communication medium may include any wireless or wired communication medium, such as a radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form part of a packet-based network (e.g., a local area network, a wide area network, or a global network such as the Internet). The communication medium may include a router, a switch, a base station, or any other device that may be useful in facilitating communication from source device 12 to target device 14.

[0038] In some other embodiments, the encoded video data may be sent from the output interface 22 to a storage device 32. The encoded video data in the storage device 32 may then be accessed by the target device 14 via the input interface 28. The storage device 32 may include any of a variety of distributed or locally accessible data storage media, such as a hard drive, a Blu-ray disc, a DVD, a CD-ROM, a flash memory, a volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded video data. In another example, the storage device 32 may correspond to a file server or another intermediate storage device that can retain the encoded video data generated by the source device 12. The target device 14 may access the stored video data from the storage device 32 via streaming or downloading. The file server may be any type of computer capable of storing encoded video data and sending the encoded video data to the target device 14. Exemplary file servers include a web server (e.g., for a website), an FTP server, a network attached storage (NAS) device, or a local disk drive. Target device 14 may access the encoded video data via any standard data connection suitable for accessing encoded video data stored on a file server, including a wireless channel (e.g., a Wi-Fi connection), a wired connection (e.g., DSL, cable modem, etc.), or a combination of both. The transmission of the encoded video data from storage device 32 may be a streaming transmission, a download transmission, or a combination of both.

[0039] like Figure 1 As shown in , source device 12 includes a video source 18, a video encoder 20, and an output interface 22. Video source 18 may include a source such as a video capture device (e.g., a video camera), a video archive containing previously captured video, a video feed interface for receiving video from a video content provider, and / or a computer graphics system for generating computer graphics data as source video, or a combination of such sources. As an example, if video source 18 is a camera of a security monitoring system, source device 12 and target device 14 may form a camera phone or video phone. However, the embodiments described in this application are generally applicable to video encoding and decoding, and may be applied to wireless and / or wired applications.

[0040] Captured, pre-captured, or computer-generated video may be encoded by video encoder 20. The encoded video data may be sent directly to target device 14 via output interface 22 of source device 12. The encoded video data may also (or alternatively) be stored on storage device 32 for later access by target device 14 or other devices for decoding and / or playback. Output interface 22 may further include a modem and / or a transmitter.

[0041] Target device 14 includes an input interface 28, a video decoder 30, and a display device 34. Input interface 28 may include a receiver and / or a modem and receives encoded video data via link 16. The encoded video data communicated via link 16 or provided on storage device 32 may include various syntax elements generated by video encoder 20 for use by video decoder 30 in decoding the video data. Such syntax elements may be included within the encoded video data transmitted over a communication medium, stored on a storage medium, or stored on a file server.

[0042] In some implementations, target device 14 may include a display device 34, which may be an integrated display device or an external display device configured to communicate with target device 14. Display device 34 displays the decoded video data to a user and may include any of a variety of display devices, such as a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, or another type of display device.

[0043] The video encoder 20 and the video decoder 30 may operate according to proprietary standards or industry standards (e.g., VVC, HEVC, MPEG-4, Part 10, Advanced Video Codec (AVC)), or extensions of such standards. It should be understood that the present application is not limited to a particular video encoding / decoding standard and may be applicable to other video encoding / decoding standards. It is generally believed that the video encoder 20 of the source device 12 may be configured to encode video data according to any of these current or future standards. Similarly, it is also generally believed that the video decoder 30 of the target device 14 may be configured to decode video data according to any of these current or future standards.

[0044] The video encoder 20 and the video decoder 30 can each be implemented as any of a variety of suitable encoder circuits, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combination thereof. When implemented partially in software, the electronic device can store instructions for the software in a suitable non-volatile computer-readable medium and use one or more processors to execute the instructions in hardware to perform the video encoding / decoding operations disclosed in the present disclosure. Each of the video encoder 20 and the video decoder 30 can be included in one or more encoders or decoders, either of which can be integrated as part of a combined encoder / decoder (CODEC) in the corresponding device.

[0045] Figure 2is a block diagram illustrating an exemplary video encoder 20 according to some embodiments described herein. Video encoder 20 can perform intra-frame prediction and inter-frame prediction encoding of video blocks within a video frame. Intra-frame prediction encoding relies on spatial prediction to reduce or remove spatial redundancy in video data within a given video frame or picture. Inter-frame prediction encoding relies on temporal prediction to reduce or remove temporal redundancy in video data within adjacent video frames or pictures of a video sequence.

[0046] like Figure 2 As shown in FIG, video encoder 20 includes video data memory 40, prediction processing unit 41, decoded picture buffer (DPB) 64, adder 50, transform processing unit 52, quantization unit 54, and entropy coding unit 56. Prediction processing unit 41 further includes motion estimation unit 42, motion compensation unit 44, segmentation unit 45, intra-prediction processing unit 46, and intra-block copy (BC) unit 48. In some embodiments, video encoder 20 also includes inverse quantization unit 58, inverse transform processing unit 60, and adder 62 for video block reconstruction. A deblocking filter (not shown) may be located between adder 62 and DPB 64 to filter block boundaries to remove blocking artifacts from the reconstructed video. In addition to the deblocking filter, a loop filter (not shown) may also be used to filter the output of adder 62. Video encoder 20 may take the form of fixed or programmable hardware units, or may be distributed among one or more of the illustrated fixed or programmable hardware units.

[0047] Video data memory 40 may store video data to be encoded by components of video encoder 20. The video data in video data memory 40 may be obtained, for example, from video source 18. DPB 64 is a buffer that stores reference video data for use by video encoder 20 (e.g., in intra-frame or inter-frame prediction coding mode) when encoding the video data. Video data memory 40 and DPB 64 may be formed from any of a variety of memory devices. In various examples, video data memory 40 may be on-chip with other components of video encoder 20, or off-chip relative to those components.

[0048] like Figure 2As shown in , after receiving the video data, the segmentation unit 45 within the prediction processing unit 41 segments the video data into video blocks. This segmentation may also include segmenting the video frame into slices, tiles, or other larger coding units (CUs) according to a predefined splitting structure associated with the video data, such as a quadtree structure. The video frame may be divided into multiple video blocks (or sets of video blocks referred to as partitions). The prediction processing unit 41 may select one of multiple possible prediction coding modes for the current video block based on error results (e.g., codec rate and distortion level), such as one of one or more inter-frame prediction coding modes from multiple intra-frame prediction coding modes. The prediction processing unit 41 may provide the resulting intra-frame prediction coding block or inter-frame prediction coding block to the adder 50 to generate a residual block, and to the adder 62 to reconstruct the coding block for subsequent use as part of a reference frame. The prediction processing unit 41 also provides syntax elements (such as motion vectors, intra-frame mode indicators, segmentation information, and other such syntax information) to the entropy coding unit 56.

[0049] To select an appropriate intra-prediction coding mode for the current video block, intra-prediction processing unit 46 within prediction processing unit 41 may perform intra-prediction coding of the current video block with respect to one or more neighboring blocks in the same frame as the current block to be encoded to provide spatial prediction. Motion estimation unit 42 and motion compensation unit 44 within prediction processing unit 41 may perform inter-prediction coding of the current video block with respect to one or more prediction blocks in one or more reference frames to provide temporal prediction. Video encoder 20 may perform multiple encoding passes, for example, to select an appropriate coding mode for each block of video data.

[0050] In some embodiments, motion estimation unit 42 determines the inter-prediction mode for the current video frame by generating a motion vector according to a predetermined pattern within a sequence of video frames. The motion vector indicates the displacement of a prediction unit (PU) of a video block within the current video frame relative to a prediction block within a reference video frame. Motion estimation performed by motion estimation unit 42 is the process of generating a motion vector that estimates the motion of a video block. For example, a motion vector may indicate the displacement of a PU of a video block within the current video frame or picture relative to a prediction block (or other coding unit) within a reference frame associated with the current block (or other coding unit) being encoded within the current frame. The predetermined pattern may designate video frames in the sequence as P-frames or B-frames. Intra BC unit 48 may determine vectors (e.g., block vectors) for intra BC coding in a manner similar to the motion vectors determined by motion estimation unit 42 for inter prediction, or may utilize motion estimation unit 42 to determine the block vectors.

[0051] A prediction block is a block of a reference frame that is considered to closely match the PU of the video block to be encoded in terms of pixel differences, which may be determined by sum of absolute differences (SAD), sum of squared differences (SSD), or other difference metrics. In some embodiments, video encoder 20 may calculate values ​​for sub-integer pixel positions of the reference frame stored in DPB 64. For example, video encoder 20 may interpolate values ​​for quarter-pixel positions, eighth-pixel positions, or other fractional pixel positions of the reference frame. Thus, motion estimation unit 42 may perform a motion search relative to full-pixel positions and fractional pixel positions and output a motion vector with fractional pixel precision.

[0052] Motion estimation unit 42 calculates a motion vector for a PU of a video block in an inter-prediction coded frame by comparing the position of the PU with the position of a prediction block of a reference frame selected from a first reference frame list (List 0) or a second reference frame list (List 1), each of which identifies one or more reference frames stored in DPB 64. Motion estimation unit 42 sends the calculated motion vector to motion compensation unit 44 and then to entropy encoding unit 56.

[0053] Motion compensation performed by motion compensation unit 44 may involve extracting or generating a prediction block based on the motion vector determined by motion estimation unit 42. Upon receiving the motion vector for the PU of the current video block, motion compensation unit 44 may locate the prediction block pointed to by the motion vector in one of the reference frame lists, retrieve the prediction block from DPB 64, and forward the prediction block to adder 50. Adder 50 then forms a residual video block of pixel difference values ​​by subtracting the pixel values ​​of the prediction block provided by motion compensation unit 44 from the pixel values ​​of the current video block being encoded. The pixel difference values ​​forming the residual video block may include luma difference components, chroma difference components, or both. Motion compensation unit 44 may also generate syntax elements associated with the video block of the video frame for use by video decoder 30 when decoding the video block of the video frame. The syntax elements may include, for example, syntax elements defining a motion vector for identifying the prediction block, any flags indicating a prediction mode, or any other syntax information described herein. It should be noted that motion estimation unit 42 and motion compensation unit 44 may be highly integrated but are illustrated separately for conceptual purposes.

[0054] In some embodiments, intra BC unit 48 may generate vectors and extract prediction blocks in a manner similar to that described above in conjunction with motion estimation unit 42 and motion compensation unit 44, but these prediction blocks are in the same frame as the current block being encoded, and these vectors are referred to as block vectors rather than motion vectors. Specifically, intra BC unit 48 may determine the intra prediction mode to be used to encode the current block. In some examples, intra BC unit 48 may encode the current block using various intra prediction modes, for example during separate encoding passes, and test their performance using rate-distortion analysis. Intra BC unit 48 may then select an appropriate intra prediction mode to use from among the various tested intra prediction modes and generate an intra mode indicator accordingly. For example, intra BC unit 48 may calculate rate-distortion values ​​for the various tested intra prediction modes using rate-distortion analysis and select the intra prediction mode with the best rate-distortion characteristics among the tested modes as the appropriate intra prediction mode to use. Rate-distortion analysis generally determines the amount of distortion (or error) between a coded block and the original, uncoded block that was coded to generate the coded block, as well as the bit rate (i.e., the number of bits) used to generate the coded block. Intra BC unit 48 may calculate ratios from the distortion and rate for various coded blocks to determine which intra-prediction mode exhibits the best rate-distortion value for the block.

[0055] In other examples, intra BC unit 48 may use, in whole or in part, motion estimation unit 42 and motion compensation unit 44 to perform such functions for intra BC prediction in accordance with embodiments described herein. In either case, for intra block copying, the prediction block may be a block that is considered to closely match the block to be encoded in terms of pixel differences, which may be determined by sum of absolute differences (SAD), sum of squared differences (SSD), or other difference metrics, and identification of the prediction block may include calculating values ​​for sub-integer pixel positions.

[0056] Regardless of whether the prediction block is from the same frame according to intra-frame prediction or from a different frame according to inter-frame prediction, video encoder 20 can form pixel difference values ​​by subtracting the pixel values ​​of the prediction block from the pixel values ​​of the current video block being encoded, thereby forming a residual video block. The pixel difference values ​​forming the residual video block may include both luma component differences and chroma component differences.

[0057] As an alternative to the inter-frame prediction performed by motion estimation unit 42 and motion compensation unit 44 or the intra-frame block copy prediction performed by intra BC unit 48 as described above, intra-frame prediction processing unit 46 may perform intra-frame prediction on the current video block. Specifically, intra-frame prediction processing unit 46 may determine an intra-frame prediction mode to use for encoding the current block. To do so, intra-frame prediction processing unit 46 may use various intra-frame prediction modes to encode the current block, for example, during separate encoding passes, and intra-frame prediction processing unit 46 (or in some examples, mode selection unit) may select an appropriate intra-frame prediction mode to use from the tested intra-frame prediction modes. Intra-frame prediction processing unit 46 may provide information indicating the intra-frame prediction mode selected for the block to entropy encoding unit 56. Entropy encoding unit 56 may encode the information indicating the selected intra-frame prediction mode into the bitstream.

[0058] After prediction processing unit 41 determines a prediction block for the current video block via inter-prediction or intra-prediction, adder 50 forms a residual video block by subtracting the prediction block from the current video block. The residual video data in the residual block may be included in one or more transform units (TUs) and provided to transform processing unit 52. Transform processing unit 52 transforms the residual video data into residual transform coefficients using a transform, such as a discrete cosine transform (DCT) or a conceptually similar transform.

[0059] Transform processing unit 52 may send the resulting transform coefficients to quantization unit 54. Quantization unit 54 quantizes the transform coefficients to further reduce the bit rate. The quantization process may also reduce the bit depth associated with some or all of the coefficients. The degree of quantization may be modified by adjusting a quantization parameter. In some examples, quantization unit 54 may then perform a scan of the matrix comprising the quantized transform coefficients. Optionally, entropy encoding unit 56 may perform the scan.

[0060] After quantization, entropy encoding unit 56 entropy encodes the quantized transform coefficients into a video bitstream using, for example, context adaptive variable length coding (CAVLC), context adaptive binary arithmetic coding (CABAC), syntax-based context adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or another entropy coding method or technique. The encoded bitstream may then be sent to video decoder 30 or archived in storage device 32 for later transmission to or retrieval by video decoder 30. Entropy encoding unit 56 may also entropy encode motion vectors and other syntax elements for the current video frame being encoded.

[0061] Inverse quantization unit 58 and inverse transform processing unit 60 apply inverse quantization and inverse transform, respectively, to reconstruct the residual video block in the pixel domain for use in generating a reference block for predicting other video blocks. As noted above, motion compensation unit 44 may generate a motion compensated prediction block from one or more reference blocks of a frame stored in DPB 64. Motion compensation unit 44 may also apply one or more interpolation filters to the prediction block to calculate sub-integer pixel values ​​for use in motion estimation.

[0062] Adder 62 adds the reconstructed residual block to the motion compensated prediction block generated by motion compensation unit 44 to generate a reference block for storage in DPB 64. The reference block may then be used by intra BC unit 48, motion estimation unit 42, and motion compensation unit 44 as a prediction block to inter-predict another video block in a subsequent video frame.

[0063] Figure 3 3 is a block diagram illustrating an exemplary video decoder 30 according to some embodiments of the present application. The video decoder 30 includes a video data memory 79, an entropy decoding unit 80, a prediction processing unit 81, an inverse quantization unit 86, an inverse transform processing unit 88, an adder 90, and a DPB 92. The prediction processing unit 81 further includes a motion compensation unit 82, an intra-frame prediction unit 84, and an intra-frame BC unit 85. The video decoder 30 may perform the above-mentioned operations in combination with the above-mentioned operations. Figure 2 The decoding process is essentially the reciprocal of the encoding process described with respect to video encoder 20. For example, motion compensation unit 82 may generate prediction data based on motion vectors received from entropy decoding unit 80, and intra-prediction unit 84 may generate prediction data based on intra-prediction mode indicators received from entropy decoding unit 80.

[0064] In some examples, units of video decoder 30 may be tasked with performing embodiments of the present disclosure. Furthermore, in some examples, embodiments of the present disclosure may be dispersed across one or more of the multiple units of video decoder 30. For example, intra BC unit 85 may perform embodiments of the present disclosure alone or in combination with other units of video decoder 30, such as motion compensation unit 82, intra prediction unit 84, and entropy decoding unit 80. In some examples, video decoder 30 may not include intra BC unit 85, and the functionality of intra BC unit 85 may be performed by other components of prediction processing unit 81, such as motion compensation unit 82.

[0065] The video data memory 79 may store video data, such as an encoded video bitstream, to be decoded by other components of the video decoder 30. The video data stored in the video data memory 79 may be obtained, for example, from the storage device 32, from a local video source (such as a camera), via a wired or wireless network communication of video data, or by accessing a physical data storage medium (e.g., a flash drive or hard disk). The video data memory 79 may include a codec picture buffer (CPB) that stores encoded video data from the encoded video bitstream. A decoded picture buffer (DPB) 92 of the video decoder 30 stores reference video data for use by the video decoder 30 when decoding the video data (e.g., in intra-frame or inter-frame prediction codec modes). The video data memory 79 and the DPB 92 may be formed from any of a variety of memory devices, such as dynamic random access memory (DRAM) (including synchronous DRAM (SDRAM)), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. For illustrative purposes, the video data memory 79 and the DPB 92 are shown in FIG. Figure 3 92 as two distinct components of video decoder 30. However, it will be apparent to those skilled in the art that video data memory 79 and DPB 92 may be provided by the same memory device or by separate memory devices. In some examples, video data memory 79 may be on-chip with the other components of video decoder 30, or off-chip relative to those components.

[0066] During the decoding process, the video decoder 30 receives an encoded video bitstream representing video blocks of an encoded video frame and associated syntax elements. The video decoder 30 may receive syntax elements at the video frame level and / or the video block level. The entropy decoding unit 80 of the video decoder 30 entropy decodes the bitstream to generate quantization coefficients, motion vectors or intra-frame prediction mode indicators, and other syntax elements. The entropy decoding unit 80 then forwards the motion vectors and other syntax elements to the prediction processing unit 81.

[0067] When a video frame is encoded as an intra-frame prediction coded (I) frame or for intra-frame coded prediction blocks in other types of frames, intra-frame prediction unit 84 of prediction processing unit 81 may generate prediction data for a video block of the current video frame based on a signaled intra-frame prediction mode and reference data from a previously decoded block of the current frame.

[0068] When the video frame is encoded as an inter-frame prediction coded (i.e., B or P) frame, motion compensation unit 82 of prediction processing unit 81 generates one or more prediction blocks for a video block of the current video frame based on the motion vectors and other syntax elements received from entropy decoding unit 80. Each of the prediction blocks may be generated from a reference frame in one of the reference frame lists. Video decoder 30 may construct the reference frame lists, List 0 and List 1, using a default construction technique based on the reference frames stored in DPB 92.

[0069] In some examples, when a video block is encoded or decoded according to the intra BC mode described herein, intra BC unit 85 of prediction processing unit 81 generates a prediction block for the current video block based on the block vector and other syntax elements received from entropy decoding unit 80. The prediction block may be within a reconstructed region of the same picture as the current video block, as defined by video encoder 20.

[0070] The motion compensation unit 82 and / or the intra BC unit 85 determine prediction information for a video block of the current video frame by parsing the motion vectors and other syntax elements, and then uses the prediction information to generate a prediction block for the current video block being decoded. For example, the motion compensation unit 82 uses some of the received syntax elements to determine the prediction mode (e.g., intra prediction or inter prediction) used to encode or decode the video block of the video frame, the inter prediction frame type (e.g., B or P), construction information for one or more of the reference frame lists for the frame, the motion vector for each inter-prediction-encoded video block of the frame, the inter prediction state for each inter-prediction-encoded video block of the frame, and other information used to decode the video block in the current video frame.

[0071] Similarly, intra BC unit 85 may use some of the received syntax elements, such as flags, to determine whether the current video block is predicted using intra BC mode, construction information of which video blocks of the frame are within the reconstruction region and should be stored in DPB 92, block vectors for each intra BC predicted video block of the frame, intra BC prediction status for each intra BC predicted video block of the frame, and other information for decoding video blocks in the current video frame.

[0072] Motion compensation unit 82 may also perform interpolation using interpolation filters, as used during encoding of the video block by video encoder 20, to calculate interpolated values ​​for sub-integer pixels of a reference block. In this case, motion compensation unit 82 may determine the interpolation filters used by video encoder 20 from the received syntax elements and use these interpolation filters to generate the prediction block.

[0073] Inverse quantization unit 86 inverse quantizes the quantized transform coefficients provided in the bitstream and entropy decoded by entropy decoding unit 80, using the same quantization parameters that determine the degree of quantization calculated by video encoder 20 for each video block in the video frame. Inverse transform processing unit 88 applies an inverse transform (e.g., an inverse DCT, an inverse integer transform, or a conceptually similar inverse transform process) to the transform coefficients to reconstruct the residual block in the pixel domain.

[0074] After the motion compensation unit 82 or the intra BC unit 85 generates a prediction block for the current video block based on the vector and other syntax elements, the adder 90 reconstructs the decoded video block for the current video block by adding the residual block from the inverse transform processing unit 88 to the corresponding prediction block generated by the motion compensation unit 82 and the intra BC unit 85. A loop filter (not shown) may be located between the adder 90 and the DPB 92 to further process the decoded video block. The decoded video block in a given frame is then stored in the DPB 92, which stores reference frames for subsequent motion compensation of the next video block. The DPB 92, or a memory device separate from the DPB 92, may also store the decoded video for later presentation on a display device (e.g., Figure 1 on the display device 34).

[0075] In a typical video encoding and decoding process, a video sequence typically consists of an ordered set of frames or pictures. Each frame may include three sample arrays, denoted as SL, SCb, and SCr. SL is a two-dimensional array of luma samples. SCb is a two-dimensional array of Cb chroma samples. SCr is a two-dimensional array of Cr chroma samples. In other cases, a frame may be monochrome and therefore consist of only a two-dimensional array of luma samples.

[0076] like Figure 4A As shown in , the video encoder 20 (or more specifically, the segmentation unit 45) generates an encoded representation of a frame by first segmenting the frame into a set of coding tree units (CTUs). A video frame may include an integer number of CTUs ordered consecutively from left to right and from top to bottom in raster scan order. Each CTU is the largest logical coding unit, and the width and height of the CTU are signaled by the video encoder 20 in a sequence parameter set so that all CTUs in a video sequence have the same size, one of 128×128, 64×64, 32×32, and 16×16. However, it should be noted that the present application is not necessarily limited to a specific size. As Figure 4BAs shown in , each CTU may include one coding tree block (CTB) of luma samples, two corresponding coding tree blocks of chroma samples, and syntax elements for encoding and decoding the samples of the coding tree blocks. The syntax elements describe the properties of different types of units of coding pixel blocks and how the video sequence can be reconstructed at the video decoder 30, including inter-frame prediction or intra-frame prediction, intra-frame prediction mode, motion vectors, and other parameters. In a monochrome picture or a picture with three separate color planes, a CTU may include a single coding tree block and syntax elements for encoding and decoding the samples of the coding tree block. The coding tree block may be an N×N sample block.

[0077] To achieve better performance, the video encoder 20 may recursively perform tree partitioning, such as binary tree partitioning, ternary tree partitioning, quadtree partitioning, or a combination of both, on the coding tree block of the CTU and divide the CTU into smaller coding units (CUs). Figure 4C As depicted in FIG, a 64×64 CTU 400 is first divided into four smaller CUs, each having a block size of 32×32. Among the four smaller CUs, CU 410 and CU 420 are each divided into four CUs with a block size of 16×16. The two 16×16 CUs 430 and CU 440 are each further divided into four CUs with a block size of 8×8. Figure 4D Depicted is a diagram showing Figure 4C The quadtree data structure is the final result of the partitioning process of the CTU 400 depicted in FIG. , with each leaf node of the quadtree corresponding to a CU of various sizes ranging from 32×32 to 8×8. Figure 4B Each CU may include a coding block (CB) of luma samples and two corresponding coding blocks of chroma samples of the same size frame, and syntax elements for encoding and decoding the samples of the coding block. In a monochrome picture or a picture with three separate color planes, a CU may include a single coding block and syntax structures for encoding and decoding the samples of the coding block. It should be noted that Figure 4C and Figure 4D The quadtree partitioning depicted in FIG is for illustrative purposes only, and a CTU can be split into CUs based on quadtree / ternary tree / binary tree partitioning to adapt to varying local characteristics. In a multi-type tree structure, a CTU is partitioned by a quadtree structure, and each quadtree leaf CU can be further partitioned by a binary and ternary tree structure. Figure 4E As shown, there are five types of segmentation, namely, quaternary segmentation, horizontal binary segmentation, vertical binary segmentation, horizontal ternary segmentation, and vertical ternary segmentation.

[0078] In some embodiments, the video encoder 20 may further partition the coding block of the CU into one or more M×N prediction blocks (PBs). A prediction block is a rectangular (square or non-square) block of samples to which the same prediction (inter or intra) is applied. The prediction unit (PU) of a CU may include a prediction block of luma samples, two corresponding prediction blocks of chroma samples, and syntax elements for predicting the prediction blocks. In a monochrome picture or a picture with three separate color planes, a PU may include a single prediction block and a syntax structure for predicting the prediction block. The video encoder 20 may generate a predicted luma block, a predicted Cb block, and a predicted Cr block for the luma prediction block, the Cb prediction block, and the Cr prediction block of each PU of the CU.

[0079] Video encoder 20 may use intra prediction or inter prediction to generate a prediction block for a PU. If video encoder 20 uses intra prediction to generate a prediction block for a PU, video encoder 20 may generate the prediction block for the PU based on decoded samples of the frame associated with the PU. If video encoder 20 uses inter prediction to generate a prediction block for a PU, video encoder 20 may generate the prediction block for the PU based on decoded samples of one or more frames other than the frame associated with the PU.

[0080] After the video encoder 20 generates the predicted luma block, the predicted Cb block, and the predicted Cr block for one or more PUs of a CU, the video encoder 20 may generate a luma residual block for the CU by subtracting the predicted luma block of the CU from the original luma coding block of the CU, such that each sample in the luma residual block of the CU indicates the difference between a luma sample in one of the predicted luma blocks of the CU and a corresponding sample in the original luma coding block of the CU. Similarly, the video encoder 20 may generate a Cb residual block and a Cr residual block for the CU, respectively, such that each sample in the Cb residual block of the CU indicates the difference between a Cb sample in one of the predicted Cb blocks of the CU and a corresponding sample in the original Cb coding block of the CU, and each sample in the Cr residual block of the CU may indicate the difference between a Cr sample in one of the predicted Cr blocks of the CU and a corresponding sample in the original Cr coding block of the CU.

[0081] In addition, if Figure 4CAs shown in , video encoder 20 may use quadtree partitioning to decompose the luma residual block, Cb residual block, and Cr residual block of a CU into one or more luma transform blocks, Cb transform blocks, and Cr transform blocks. A transform block is a rectangular (square or non-square) block of samples to which the same transform is applied. A transform unit (TU) of a CU may include a transform block of luma samples, two corresponding transform blocks of chroma samples, and syntax elements for transforming the transform block samples. Thus, each TU of a CU may be associated with a luma transform block, a Cb transform block, and a Cr transform block. In some examples, the luma transform block associated with a TU may be a sub-block of the luma residual block of the CU. The Cb transform block may be a sub-block of the Cb residual block of the CU. The Cr transform block may be a sub-block of the Cr residual block of the CU. In a monochrome picture or a picture with three separate color planes, a TU may include a single transform block and a syntax structure for transforming the samples of the transform block.

[0082] Video encoder 20 may apply one or more transforms to the luma transform block of a TU to generate a luma coefficient block for the TU. A coefficient block may be a two-dimensional array of transform coefficients. A transform coefficient may be a scalar. Video encoder 20 may apply one or more transforms to the Cb transform block of a TU to generate a Cb coefficient block for the TU. Video encoder 20 may apply one or more transforms to the Cr transform block of a TU to generate a Cr coefficient block for the TU.

[0083] After generating a coefficient block (e.g., a luma coefficient block, a Cb coefficient block, or a Cr coefficient block), video encoder 20 may quantize the coefficient block. Quantization generally refers to the process by which transform coefficients are quantized to potentially reduce the amount of data used to represent the transform coefficients, thereby providing further compression. After video encoder 20 quantizes the coefficient block, video encoder 20 may entropy encode syntax elements indicating the quantized transform coefficients. For example, video encoder 20 may perform context-adaptive binary arithmetic coding (CABAC) on the syntax elements indicating the quantized transform coefficients. Finally, video encoder 20 may output a bitstream comprising a sequence of bits forming a representation of an encoded frame and associated data, which is stored in storage device 32 or sent to target device 14.

[0084] After receiving the bitstream generated by the video encoder 20, the video decoder 30 may parse the bitstream to obtain syntax elements from the bitstream. The video decoder 30 may reconstruct a frame of video data based at least in part on the syntax elements obtained from the bitstream. The process of reconstructing the video data is generally the inverse of the encoding process performed by the video encoder 20. For example, the video decoder 30 may perform an inverse transform on the coefficient blocks associated with the TUs of the current CU to reconstruct the residual blocks associated with the TUs of the current CU. The video decoder 30 also reconstructs the coding blocks of the current CU by adding samples of the prediction blocks for the PUs of the current CU to corresponding samples of the transform blocks of the TUs of the current CU. After reconstructing the coding blocks for each CU of the frame, the video decoder 30 may reconstruct the frame.

[0085] In some embodiments, there are different ways to signal prediction modes. In VVC, each CU can be coded in skip mode or non-skip mode. For CUs coded in non-skip mode, syntax elements are further signaled to indicate whether the current CU is coded in intra mode, inter mode, intra block copy (IBC) mode, or palette (PLT) mode. These modes are referred to as "MODE_INTRA", "MODE_INTER", "MODE_IBC", and "MODE_PLT" in the VVC specification, respectively. For CUs coded in inter mode, the prediction signal can be generated by pixels from a reference picture that is not the current picture. In addition, for CUs coded in inter mode, a flag is further signaled to indicate whether the current CU is in merge mode. For CUs coded in merge mode, several different types of merge modes are further signaled using additional syntax elements. These different types of merge modes include normal merge mode, sub-block merge mode, merge mode with MV difference (MMVD), combined inter and intra prediction (CIIP), and triangle merge mode. The merge modes are shown in the following sections.

[0086] In some embodiments, in VVC, the merge candidate list is constructed by including the following five types of candidates in order: spatial MVP (i.e., motion vector predictor) from spatially neighboring CUs, temporal MVP from co-located CUs, history-based MVP from FIFO tables, pairwise average MVP, and zero MV.

[0087] In VVC, the size of the merge list is signaled in the slice header, and the maximum allowed size of the merge list is 6. For each CU encoded in merge mode, the index of the best merge candidate is encoded using truncated unary binarization (TU). The first binary bit of the merge index is encoded using context, and bypass encoding is used for the other binary bits. In the following context of this disclosure, this extended merge mode is also referred to as normal merge mode because its concept is the same as the merge mode used in HEVC.

[0088] In some embodiments, in addition to the merge mode in which the implicitly derived motion information is directly used for the prediction sample generation of the current CU, a merge mode using motion vector difference (MMVD) is introduced in VVC. The MMVD flag is signaled immediately after the skip flag and the merge flag to specify whether the MMVD mode is used for the CU.

[0089] In MMVD, after a merge candidate is selected, it is further refined using signaled MVD information. Additional information includes a merge candidate flag, an index specifying the magnitude of motion, and an index indicating the direction of motion. In MMVD mode, one of the first two candidates in the merge list is selected as the motion vector (MV) basis. A merge candidate flag is signaled to specify which candidate to use.

[0090] The distance index specifies the motion magnitude information and indicates a predefined offset from the starting point. The offset is added to the horizontal component or vertical component of the starting MV. The relationship between the distance index and the predefined offset is specified in Table 1.

[0091]

[0092] Table 1: Relationship between distance index and predefined offset

[0093] The direction index indicates the direction of the MVD relative to the starting point. The direction index can indicate four directions as shown in Table 2. It should be noted that the meaning of the MVD symbol may vary depending on the information of the starting MV. When the starting MV is a unidirectionally predicted MV or a bidirectionally predicted MV in the case where two lists point to the same side of the current picture (i.e., the POCs of both references are greater than the POC of the current picture, or both are less than the POC of the current picture), the symbol in Table 2 specifies the sign of the MV offset added to the starting MV. When the starting MV is a bidirectionally predicted MV in the case where two MVs point to different sides of the current picture (i.e., the picture order count (POC) of one reference is greater than the POC of the current picture, and the POC of the other reference is less than the POC of the current picture), the symbol in Table 2 specifies that the sign of the MV offset added to the list 0 MV component of the starting MV has an opposite value to that for the list 1 MV.

[0094] Direction Index 00 01 10 11 X-axis + – N / A N / A Y-axis N / A N / A + –

[0095] Table 2: Sign of MV offset specified by direction index

[0096] In some embodiments, in VVC, a combined sub-block based merge list is used for signaling of sub-block based merge mode, wherein the combined sub-block based merge list contains both sub-block based temporal motion vector prediction (SbTMVP) candidates and affine merge candidates. In the following, it is referred to as sub-block merge mode. SbTMVP mode is enabled / disabled by a sequence parameter set (SPS) flag. If SbTMVP mode is enabled, the SbTMVP predictor is added as the first entry of the list of sub-block based merge candidates, followed by the affine merge candidates. The size of the sub-block based merge list is signaled in the SPS, and the maximum allowed size of the sub-block based merge list is 5 in VVC.

[0097] The sub-CU size used in SbTMVP is fixed to 8x8, and as done for affine merge mode, SbTMVP mode is only applicable to CUs whose width and height are both greater than or equal to 8.

[0098] The encoding logic for the additional SbTMVP merge candidates is the same as that for other merge candidates, i.e., for each CU in a forward predicted frame (P) or bidirectional predicted frame (B) slice, an additional rate-distortion (RD) check is performed to decide whether to use the SbTMVP candidate.

[0099] In some embodiments, VVC supports a sub-block based temporal motion vector prediction (SbTMVP) method. Similar to the temporal motion vector prediction (TMVP) in HEVC, SbTMVP uses the motion field in the co-located picture to improve the motion vector prediction and merge mode for the CU in the current picture. The same co-located picture used by TMVP is used for SbTMVP. SbTMVP differs from TMVP in two main aspects. First, TMVP predicts motion at the CU level, while SbTMVP predicts motion at the sub-CU level; second, TMVP extracts the temporal motion vector from the co-located block in the co-located picture (the co-located block is the lower right block or center block relative to the current CU), while SbTMVP applies a motion offset before extracting the temporal motion information from the co-located picture, where the motion offset is obtained from the motion vector of one of the spatially neighboring blocks from the current CU.

[0100] Figure 5A and Figure 5Bis a block diagram illustrating an exemplary SbTMVP process in VVC according to some embodiments of the present disclosure. In some embodiments, the sub-CU motion field is derived by applying the temporal vector (or motion offset) from the spatial neighbors and scaling the motion information from the corresponding co-located sub-CU.

[0101] Figure 5A and Figure 5B The SbTMVP process is shown in Figure 2. SbTMVP uses two steps to predict the motion vector of the sub-CU in the current CU. In the first step, check Figure 5A The spatial neighbor A1 in . If A1 has a motion vector that uses the co-located picture as its reference picture, this motion vector is selected as the motion offset to be applied. If no such motion is identified, the motion offset is set to (0, 0).

[0102] In the second step, the motion offset identified in step 1 is applied (ie, added to the coordinates of the current block) to obtain the sub-CU level motion information (motion vector and reference index) from the co-located picture, as shown in FIG5b . Figure 5B The example in assumes that the motion offset is set to the motion of block A1. Then, for each sub-CU, the motion information of its corresponding block in the co-located picture (the minimum motion grid covering the center sample) is used to derive the motion information of the sub-CU. After identifying the motion information of the co-located sub-CU, it is converted into the motion vector and reference index of the current sub-CU in a manner similar to the TMVP process of HEVC, where temporal motion scaling is applied to align the reference picture of the temporal motion vector with the reference picture of the current CU.

[0103] In VVC, a combined sub-block based merge list is used for signaling of sub-block based merge mode, wherein the combined sub-block based merge list contains both SbTMVP candidates and affine merge candidates. The SbTMVP mode is enabled / disabled by a sequence parameter set (SPS) flag. If the SbTMVP mode is enabled, the SbTMVP predictor is added as the first entry of the list of sub-block based merge candidates, followed by the affine merge candidates. The size of the sub-block based merge list is signaled in the SPS, and the maximum allowed size of the sub-block based merge list is 5 in VVC.

[0104] The sub-CU size used in SbTMVP is fixed to 8x8, and as done for affine merge mode, SbTMVP mode is only applicable to CUs whose width and height are both greater than or equal to 8.

[0105] The encoding logic for the additional SbTMVP merge candidate is the same as that for other merge candidates, ie, for each CU in a P or B slice, an additional RD check is performed to decide whether to use the SbTMVP candidate.

[0106] In some embodiments, in HEVC, only the translational motion model is applied to motion compensated prediction (MCP). In the real world, there are many types of motion, such as zooming in / out, rotation, projection motion, and other irregular motions. In VVC, block-based affine transformation motion compensated prediction is applied.

[0107] Figure 6A and Figure 6B is a block diagram illustrating an exemplary control point-based affine motion model according to some embodiments of the present disclosure. Figure 6A A 4-parameter affine model is shown. Figure 6B A 6-parameter affine model is shown.

[0108] like Figure 6A and Figure 6B As shown, the affine motion field of the block is described by motion information of two control point motion vectors (4 parameters) V0 and V1 or three control point motion vectors (6 parameters) V0, V1 and V2.

[0109] For the 4-parameter affine motion model, the motion vector at the sample position (x, y) in the block is derived as:

[0110]

[0111] For the 6-parameter affine motion model, the motion vector at the sample position (x, y) in the block is derived as:

[0112]

[0113] Among them (mv 0x ,mv 0y ) is the motion vector of the upper left control point, (mv 1x ,mv 1y ) is the motion vector of the upper right control point, and (mv 2x ,mv 2y ) is the motion vector of the lower left control point. W is the width of the block and H is the height of the block.

[0114] Figure 7 is a block diagram illustrating an exemplary affine motion vector prediction (AMVP) for sub-blocks according to some embodiments of the present disclosure. To simplify motion compensated prediction, block-based affine transform prediction is applied. To derive the motion vector for each 4×4 luma sub-block, the motion vector of the center sample of each sub-block is calculated according to the above equation (e.g. Figure 7) and rounded to 1 / 16 fractional precision. A motion compensated interpolation filter is then applied to generate a prediction for each subblock using the derived motion vectors. The subblock size of the chroma component is also set to 4×4. The MV of a 4×4 chroma subblock is calculated as the average of the MVs of the four corresponding 4×4 luma subblocks.

[0115] As done for translational motion inter prediction, there are also two affine motion inter prediction modes: affine merge mode and AMVP mode.

[0116] In some embodiments, the affine merge prediction mode (AF_MERGE mode) may be applied to CUs whose width and height are both greater than or equal to 8. In this mode, the control point motion vector (CPMV) of the current CU is generated based on the motion information of spatially neighboring CUs. There may be up to five CPMV predictor (CPMVP) candidates, and the index is signaled to indicate the index for the current CU. The following three types of CPMVP candidates are used to form the affine merge candidate list: (1) inherited affine merge candidates extrapolated from the CPMV of neighboring CUs; ​​(2) constructed affine merge CPMVP derived using the translated MV of neighboring CUs; ​​and (3) zero MV.

[0117] Figure 8 is a block diagram showing an exemplary location of inherited affine motion predictors according to some embodiments of the present disclosure. In VVC, there are at most two inherited affine candidates derived from the affine motion models of neighboring blocks, one from the left neighboring CU and one from the top neighboring CU. The candidate blocks are in Figure 8 -> A1 for the left predictor and B0->B1->B2 for the top predictor. Only the first inherited candidate from each side is selected. No clipping check is performed between two inherited candidates. When a neighboring affine CU is identified, its control point motion vectors are used to derive the CPMVP candidates in the affine merge list of the current CU.

[0118] Figure 9 is a block diagram illustrating exemplary control point motion vector inheritance according to some embodiments of the present disclosure. Figure 9 As shown in , if the adjacent lower left block A is encoded and decoded in affine mode, the motion vectors v2, v3, and v4 of the upper left, upper right, and lower left corners of the CU containing block A are obtained. When block A is encoded and decoded using a 4-parameter affine model, two CPMVs of the current CU are calculated based on v2 and v3. In the case where block A is encoded and decoded using a 6-parameter affine model, three CPMVs of the current CU are calculated based on v2, v3, and v4.

[0119] Figure 10is a block diagram illustrating exemplary positions of candidate positions for constructing an affine merge mode according to some embodiments of the present disclosure. The constructed affine candidate is constructed by combining adjacent translation motion information of each control point. The motion information about the control point is obtained from Figure 10 The specified spatial neighbors and temporal neighbors are derived as shown in CPMV. k (k=1, 2, 3, 4) represents the kth control point. For CPMV1, the B2->B3->A2 blocks are checked in order, and the MV of the first available block is used. Similarly, CPMV2 is derived by checking the B1->B0 blocks in order. CPMV3 is derived by checking the A1->A0 blocks in order. If TMVP is available, it is used as CPMV4.

[0120] After obtaining the MVs of the four control points, an affine merge candidate is constructed based on that motion information. The following combinations of control point MVs are used to construct in order:

[0121] {CPMV1, CPMV2, CPMV3}, {CPMV1, CPMV2, CPMV4}, {CPMV1, CPMV3, CPMV4}, {CPMV2, CPMV3, CPMV4}, {CPMV1, CPMV2}, {CPMV1, CPMV3}

[0122] A combination of 3 CPMVs can be used to construct a 6-parameter affine merge candidate, and a combination of 2 CPMVs can be used to construct a 4-parameter affine merge candidate. To avoid the motion scaling process, the relevant combination of control point MVs is discarded if the reference indices of the control points are different.

[0123] After checking inherited affine merge candidates and constructed affine merge candidates, if the list is still not full, a zero MV is inserted at the end of the list.

[0124] In some embodiments, in VVC, when a CU is encoded and decoded in merge mode, if the CU contains at least 64 luma samples (i.e., the CU width multiplied by the CU height is equal to or greater than 64), and if both the CU width and the CU height are less than 128 luma samples, an additional flag is signaled to indicate whether the combined inter / intra prediction (CIIP) mode is applied to the current CU. As its name suggests, CIIP prediction combines the inter prediction signal with the intra prediction signal. The inter prediction signal P in CIIP mode is derived using the same inter prediction process applied to the regular merge mode. inter ; and derive the intra prediction signal P after conventional intra prediction processing using the planar mode intra Then, the intra prediction signal and the inter prediction signal are combined using a weighted average, where the weights are calculated based on the top neighboring block and the left neighboring block (e.g. Figure 10 The weight values ​​are calculated for the encoding and decoding modes of blocks A1 and B1 as shown:

[0125] If the top neighbor is available and is intra-coded, set isIntraTop to 1, otherwise set isIntraTop to 0;

[0126] If the left neighbor is available and is intra-coded, set isIntraLeft to 1, otherwise set isIntraLeft to 0;

[0127] If (isIntraLeft+isIntraLeft) is equal to 2, set wt to 3;

[0128] Otherwise, if (isIntraLeft + isIntraLeft) is equal to 1, set wt to 2;

[0129] Otherwise, set wt to 1.

[0130] The CIIP forecast is formed as follows:

[0131] <![CDATA[P CIIP =((4-wt)*P inter +wt*P intra +2)>>2]]>

[0132] In some embodiments, in VVC, a new triangle partitioning mode is introduced for inter prediction. The triangle partitioning mode is only applied to CUs that are 8×8 or larger and are encoded and decoded in skip mode or merge mode. For CUs that meet these conditions and the merge flag is on, a CU-level flag is signaled to indicate whether the triangle partitioning mode is applied.

[0133] When this mode is used, the CU is evenly split into two triangular partitions using diagonal or anti-diagonal splitting. Each triangular partition in the CU uses its own motion for inter prediction; only unidirectional prediction is allowed for each partition, that is, each partition has one motion vector and one reference index. Unidirectional prediction motion constraints are applied to ensure that in triangular prediction mode, the CU only requires two motion compensated predictions, which is the same as regular bidirectional prediction.

[0134] If the CU level flag indicates that the current CU is encoded using triangle partitioning mode, a flag is signaled to indicate the triangle partitioning direction (i.e., diagonal or anti-diagonal). Then, an index is signaled for each of the two partitions to indicate the merged motion vector candidate for each triangle partition. After predicting each triangle partition, a blending process with adaptive weights is used to adjust the sample values ​​along the diagonal or anti-diagonal edges. After the prediction process, the transformation and quantization process will be applied to the entire CU. It is worth mentioning that the motion field of the CU predicted using triangle partitioning mode is stored in 4×4 units.

[0135] In some embodiments, intra block copying (IBC) is a tool implemented in the HEVC extended profile for screen content coding (SCC). It is well known that it significantly improves the codec efficiency of screen content video material. In particular, IBC can efficiently generate prediction factors from reconstructed regions of the current picture. The IBC mode is signaled at the block level. Block matching (BM) is performed at the encoder to find the optimal block vector (or motion vector) for each CU. Here, the block vector is used to indicate the displacement from the current block to a reference block that has been reconstructed within the current picture. The luma block vector of a CU encoded and decoded with IBC has integer precision. For some video formats (such as 420), the chroma block vector can be derived from the luma block vector through rounding, and the result is also integer precision. When combined with AMVR (Adaptive Motion Vector Resolution), the IBC mode can switch between 1-pixel motion vector precision and 4-pixel motion vector precision. In addition to intra prediction mode or inter prediction mode, a CU encoded and decoded with IBC is also considered a third prediction mode. IBC mode is applicable to CUs with a width and height less than or equal to 64 luma samples.

[0136] At CU level, IBC mode is signaled using a flag, and it can be signaled as IBC AMVP mode or IBC Skip / Merge mode as follows:

[0137] IBC skip / merge mode: The merge candidate index is used to indicate which of the block vectors from the neighboring candidate IBC codec blocks in the list is used to predict the current block. The merge list consists of at most two spatial candidates (A1 and B1) and at most two HMVP candidates.

[0138] For IBC-encoded CUs smaller than or equal to 4×4, only HMVP candidates are used to construct the IBC merge candidate list.

[0139] In IBC AMVP mode, block vector differences are encoded and decoded in the same way as motion vector differences. The block vector prediction method uses two candidates as predictors and constructs the IBC AMVP candidate list using the same construction process as the IBC merge list.

[0140] In some embodiments, in history-based motion vector prediction (HMVP), HMVP candidates are defined as motion information of previously coded blocks. A table with multiple HMVP candidates (referred to as the historical MV table in the rest of this document) is maintained during the encoding / decoding process. The table is cleared when a new slice or the first CTU of each CTU row is encountered. Whenever a non-sub-block inter-coded CU is present, the associated motion information is added to the last entry of the table as a new HMVP candidate. In VVC, the HMVP table size is set to 6, which indicates that up to 6 history-based MVP (HMVP) candidates can be added to the table. When a new motion candidate is inserted into the table, a constrained first-in-first-out (FIFO) rule is utilized, where a redundancy check (i.e., MV pruning) is first applied to find out whether the same HMVP exists in the table. If found, the same HMVP is removed from the table, and then all HMVP candidates are moved forward (towards the first entry). It is worth noting that the MVP stored in the last entry is the latest MVP, and the MVP stored in the first entry is the oldest MVP.

[0141] As explained in the previous section, the HMVP candidate can be used in the merge candidate list construction process after the TMVP candidate. The HMVP candidates in the table are checked in order from the last entry to the first entry in the HMVP table. A redundancy check is applied to the first two HMVP candidates, where each of these two HMVP candidates is compared with the left spatial merge candidate A1 and the upper spatial merge candidate B1. In other words, when inserting the HMVP candidate, the same MV candidate is checked at most 4 times. Once the total number of available merge candidates reaches the maximum allowed merge candidate minus 1, the merge candidate list construction process from the HMVP is terminated.

[0142] HMVP candidates can also be used in the AMVP candidate list construction process. The HMVP candidates in the table are checked in order from the first entry to the last entry in the HMVP table and are inserted into the candidate list after the TMVP candidate. Redundancy checking is not applied to HMVP candidates.

[0143] As explained in the previous section, HMVP candidates can be used in the IBC merge candidate list construction process as well as the IBC AMVP list construction. The HMVP candidates in the table are checked in order from the last entry to the first entry in the HMVP table and are inserted into the candidate list after the TMVP candidate. Redundancy check is applied to the first HMVP candidate, and the checked HMVP candidate only needs to be compared with the left spatial merge candidate and the upper spatial merge candidate. In other words, when inserting the HMVP candidate, the same MV candidate is checked at most 2 times.

[0144] In some embodiments, motion estimation regions (MERs) for parallel motion estimation are used. To speed up the encoding process, motion estimation can be performed in parallel, thereby deriving motion vectors for all prediction units within a given region simultaneously. Deriving merge candidates from spatial neighbors may interfere with parallel processing because one prediction unit cannot derive motion parameters from neighboring PUs until its associated motion estimation is complete. To alleviate the trade-off between codec efficiency and processing latency, HEVC defines a motion estimation region (MER), the size of which is signaled in the picture parameter set using the "log2_parallel_merge_level_minus2" syntax element. When a MER is defined, merge candidates falling into the same region are marked as unavailable and are therefore not considered in list construction.

[0145] In some embodiments, the way of constructing the merge candidate list introduces dependencies between adjacent blocks. In particular, in embedded encoder implementations, the motion estimation stage for adjacent blocks is usually performed in parallel or at least pipelined to increase throughput. For AMVP, this is not a big problem because MVP is only used to differentially encode and decode the MV found by the motion search. However, the motion estimation stage for the merge mode will usually only consist of candidate list construction and the decision of which candidate to select based on the cost function. Due to the dependencies between adjacent blocks mentioned above, the merge candidate list of adjacent blocks cannot be generated in parallel and presents a bottleneck for parallel encoder design. Therefore, several methods are introduced to make the merge-related mode more parallel-friendly. In the following section, the width and height of PMER are denoted as W and H, respectively.

[0146] SbTMVP for Parallel Motion Estimation Region (PMER)

[0147] In the first embodiment of the present disclosure, when the position of the neighboring block used to derive the TV for SbTMVP is located within the same PMER as the current block, the TV derived from the spatial block is considered unavailable and the default TV (e.g., zero vector) is used.

[0148] In a second embodiment of the present disclosure, the derivation of spatial neighboring blocks for deriving the temporal vector (TV) of the SbTMVP is modified to derive blocks outside a predefined parallel motion estimation region (PMER). Different schemes can be used to locate spatial neighboring blocks for TV derivation. Several examples are shown below. The embodiments are not limited to the examples shown. The position of the neighboring blocks is derived as a block containing samples located at one of the positions shown below. In the example shown, the position of the upper left sample of the current luma coding block relative to the upper left luma sample of the current picture is represented as (xCb, yCb); positions (xA0, yA0), (xA1, yA1), (xA2, yA2), (xB0, yB0), (xB1, yB1), (xB2, yB2) are sample positions covered by neighboring blocks; two variables cbWidth and cbHeight specify the width and height of the luma coding block.

[0149] 1)(xA0,yA0)=(xCb / W*W-1,yCb+cbHeight)

[0150] 2)(xA0,yA0)=(xCb / W*W-1,(yCb+cbHeight) / H*H)

[0151] 3)(xA1,yA1)=(xCb / W*W-1,yCb+cbHeight-1)

[0152] 4)(xA1,yA1)=(xCb / W*W-1,(yCb+cbHeight) / H*H-1)

[0153] 5)(xA2,yA2)=(xCb / W*W-1,yCb)

[0154] 6)(xA2,yA2)=(xCb / W*W-1,yCb / H*H)

[0155] 7)(xB0,yB0)=(xCb+cbWidth,yCb / H*H-1)

[0156] 8)(xB0,yB0)=((xCb+cbWidth) / W*W,yCb / H*H-1)

[0157] 9)(xB1,yB1)=(xCb+cbWidth-1,yCb / H*H-1)

[0158] 10)(xB1,yB1)=((xCb+cbWidth) / W*W-1,yCb / H*H-1)

[0159] 11)(xB2,yB2)=(xCb / W*W-1,yCb / H*H-1)

[0160] 12)(xB2,yB2)=(xCb / W*W-1,yCb-1)

[0161] 13)(xB2,yB2)=(xCb-1,yCb / H*H-1)

[0162] Figure 11 is a block diagram illustrating exemplary locations of spatially neighboring blocks for deriving TV according to some embodiments of the present disclosure. Figure 11 An example is shown where the neighboring block A1 used to derive the TV for the SbTMVP for CU 6 is located outside the MER according to the position derived using Equation 1 above. In current VVC, the spatial blocks used to derive the TV for the SbTMVP for CU 6 depend on the reconstruction of CU 5. Therefore, this dependency is problematic for parallel motion search.

[0163] In the third embodiment of the present disclosure, the spatial neighboring blocks used to derive the time vector (TV) for SbTMVP are modified to be blocks at the upper boundary of the current block. For some hardware encoder designs, the neighboring blocks located above the current block (e.g., Figure 10 The MVs of blocks B0, B1, and B2 in FIG1 have already been reconstructed and are available. Therefore, deriving the temporal vector from the upper neighboring blocks presents fewer problems for parallel motion estimation than using the neighboring blocks to the left of the current block (e.g., block A1). Different schemes can be used to locate a spatial neighboring block for TV derivation. Several examples are shown below. Embodiments are not limited to the examples shown. The location of the neighboring block is derived as a block containing a sample located at one of the following positions:

[0164] 1)(xB0,yB0)=(xCb+cbWidth,yCb-1)

[0165] 2)(xB1,yB1)=(xCb+cbWidth-1,yCb-1)

[0166] 3)(xB2,yB2)=(xCb-1,yCb-1)

[0167] In a fourth embodiment of the present disclosure, the HMVP candidate in the first entry of the HMVP table is used to derive TV for SbTMVP.

[0168] In a fifth embodiment of the present disclosure, temporal motion vectors derived from blocks in a co-located picture are used to derive TV for SbTMVP.

[0169] IBC consolidation for MER

[0170] In a sixth embodiment of the present disclosure, when the location of the neighboring block used to derive the IBC merge candidate is located within the same PMER as the current block, the derived IBC merge candidate is considered unavailable.

[0171] Figure 12 is a block diagram illustrating exemplary locations of IBC space candidates according to some embodiments of the present disclosure. Figure 12 It is an example of checking two neighboring blocks (A1, B1) according to a predefined order to derive IBC merge candidates for the current block (CU1). Since all neighboring two blocks are located in different PMERs, the two neighboring blocks are considered available.

[0172] Figure 13 is a block diagram illustrating exemplary locations of IBC space candidates according to some embodiments of the present disclosure. Figure 13 In another example shown, the neighboring block A1 of CU5 is located in a different MER, so it is considered available. The neighboring block B1 is located in the same MER and is considered unavailable.

[0173] In a seventh embodiment of the present disclosure, the derivation of spatial neighboring blocks for deriving IBC merge candidates is modified to derive blocks outside a predefined parallel motion estimation region (PMER). The positions of the neighboring blocks are derived as blocks containing samples located at one of the following positions:

[0174] 1)(xA0,yA0)=(xCb / W*W-1,yCb+cbHeight)

[0175] 2)(xA0,yA0)=(xCb / W*W-1,(yCb+cbHeight) / H*H)

[0176] 3)(xA1,yA1)=(xCb / W*W-1,yCb+cbHeight-1)

[0177] 4)(xA1,yA1)=(xCb / W*W-1,(yCb+cbHeight) / H*H–1)

[0178] 5)(xA2,yA2)=(xCb / W*W-1,yCb)

[0179] 6)(xA2,yA2)=(xCb / W*W-1,yCb / H*H)

[0180] 7)(xB0,yB0)=(xCb+cbWidth,yCb / H*H-1)

[0181] 8)(xB0,yB0)=((xCb+cbWidth) / W*W,yCb / H*H-1)

[0182] 9)(xB1,yB1)=(xCb+cbWidth-1,yCb / H*H-1)

[0183] 10)(xB1,yB1)=((xCb+cbWidth) / W*W-1,yCb / H*H-1)

[0184] 11)(xB2,yB2)=(xCb / W*W-1,yCb / H*H-1)

[0185] 12)(xB2,yB2)=(xCb / W*W-1,yCb-1)

[0186] 13)(xB2,yB2)=(xCb-1,yCb / H*H-1)

[0187] Figure 14 and Figure 15 is a block diagram illustrating exemplary locations of non-adjacent spatial candidates according to some implementations of the present disclosure. Figure 14 and Figure 15 In an example shown in , spatially neighboring blocks located immediately above the PMER region boundary are used to derive IBC merge candidates. In the example, the above position equation 9 is used.

[0188] exist Figure 15 In

[15] , spatially neighboring blocks located immediately adjacent to the left boundary of the merge region are used to derive IBC merge candidates. In this example, the position equation 3 above is used.

[0189] In an eighth embodiment of the present disclosure, when inserting an HMVP candidate into the IBC merge candidate list, the HMVP candidates in the table are checked in order from the first entry to the last entry in the HMVP table.

[0190] CIIP for Parallel Motion Estimation Regions (PMER)

[0191] In a ninth embodiment of the present disclosure, when a position of a neighboring block used to derive a weight (eg, wt) for a CIIP coding block is located within the same PMER as the current block, the neighboring block is considered unavailable.

[0192] In the tenth embodiment of the present disclosure, the derivation of spatially neighboring blocks for deriving weights for CIIP coding blocks is modified to derive blocks outside a predefined parallel motion estimation region (PMER). Different schemes can be used to locate spatially neighboring blocks for weight determination. Several examples are shown below. The embodiments are not limited to the examples shown. The positions of the neighboring blocks are derived as blocks containing samples located at one of the positions shown below. In the example shown, the position of the upper left sample of the current luma coding block relative to the upper left luma sample of the current picture is represented as (xCb, yCb); positions (xA0, yA0), (xA1, yA1), (xA2, yA2), (xB0, yB0), (xB1, yB1), (xB2, yB2) are sample positions overlapped by neighboring blocks; two variables, cbWidth and cbHeight, specify the width and height of the luma coding block.

[0193] 1)(xA0,yA0)=(xCb / W*W-1,yCb+cbHeight)

[0194] 2)(xA0,yA0)=(xCb / W*W-1,(yCb+cbHeight) / H*H)

[0195] 3)(xA1,yA1)=(xCb / W*W-1,yCb+cbHeight-1)

[0196] 4)(xA1,yA1)=(xCb / W*W-1,(yCb+cbHeight) / H*H-1)

[0197] 5)(xA2,yA2)=(xCb / W*W-1,yCb)

[0198] 6)(xA2,yA2)=(xCb / W*W-1,yCb / H*H)

[0199] 7)(xB0,yB0)=(xCb+cbWidth,yCb / H*H-1)

[0200] 8)(xB0,yB0)=((xCb+cbWidth) / W*W,yCb / H*H-1)

[0201] 9)(xB1,yB1)=(xCb+cbWidth-1,yCb / H*H-1)

[0202] 10)(xB1,yB1)=((xCb+cbWidth) / W*W-1,yCb / H*H-1)

[0203] 11)(xB2,yB2)=(xCb / W*W-1,yCb / H*H-1)

[0204] 12)(xB2,yB2)=(xCb / W*W-1,yCb-1)

[0205] 13)(xB2,yB2)=(xCb-1,yCb / H*H-1)

[0206] Figure 16 is a flow chart illustrating an exemplary process 1600 by which a video codec implements a technique for determining a current merge candidate replacement when a current coding unit / block and a spatially neighboring block used to determine a current merge candidate are in the same motion estimation region (MER) according to some embodiments of the present disclosure. For ease of description, process 1600 is described as being performed by a video decoder (e.g., Figure 3 The video decoder 30) is executed.

[0207] First embodiment:

[0208] In some embodiments, video decoder 30 receives video data corresponding to a coding unit from a bitstream ( 1610 ), and then receives a first syntax element defining a motion estimation region (MER) from the video data ( 1620 ).

[0209] Video decoder 30 receives a plurality of second syntax elements from the video data, the plurality of second syntax elements indicating whether the coding unit is in sub-block based temporal motion vector prediction (SbTMVP) mode ( 1630 ).

[0210] In some embodiments, the plurality of second syntax elements include a third syntax element indicating whether the coding unit is in subblock merge mode and a fourth syntax element indicating which subblock-based merge candidate to use to reconstruct the coding unit. The third syntax element is "merge_subblock_flag", and the fourth syntax element is "merge_subblock_idx".

[0211] Based on determining that the coding unit is in sub-block merge mode, video decoder 30 receives a plurality of sub-block-based merge candidates for the coding unit derived from neighboring blocks. Each sub-block merge candidate is constructed from a plurality of neighboring blocks. The neighboring blocks include spatially neighboring blocks and temporally neighboring blocks. In some embodiments, the plurality of sub-block-based merge candidates include both SbTMVP candidates and affine merge candidates.

[0212] Based on determining that the coding unit is in SbTMVP mode, video decoder 30 determines whether both the coding unit and the spatially neighboring blocks used to determine the temporal vector of the coding unit are within the same MER ( 1640 ).

[0213] Based on determining that the spatial neighboring blocks are SbTMVP candidates, the coding unit is in SbTMVP mode.

[0214] Based on determining that both the coding unit and the spatially neighboring blocks are within the same MER, video decoder 30 sets the temporal vector of the coding unit to zero ( 1650 ).

[0215] Second embodiment:

[0216] In some embodiments, video decoder 30 receives video data corresponding to a coding unit from a bitstream and then receives a first syntax element defining a motion estimation region (MER) from the video data. Video decoder 30 receives a plurality of second syntax elements from the video data, wherein the plurality of second syntax elements indicate whether the coding unit is in sub-block-based temporal motion vector prediction (SbTMVP) mode. Based on determining that the coding unit is in SbTMVP mode, video decoder 30 determines whether the coding unit and a first spatial neighboring block used to determine a temporal vector for the coding unit are both within the same MER. Based on determining that the coding unit and the first spatial neighboring block are both within the same MER, video decoder 30 selects a second spatial neighboring block outside the MER as an alternative to the first spatial neighboring block to determine the temporal vector for the coding unit.

[0217] In some embodiments, the second spatially neighboring block outside the MER is a spatially neighboring block at a left position of the first spatially neighboring block.

[0218] In some embodiments, the second spatially-neighboring block outside the MER is a spatially-neighboring block at an upper position of the first spatially-neighboring block.

[0219] In some embodiments, the second spatially neighboring block outside the MER is a spatially neighboring block positioned above and to the left of the first spatially neighboring block.

[0220] Third embodiment:

[0221] In some embodiments, video decoder 30 receives video data corresponding to a coding unit from a bitstream. Video decoder 30 then receives a first syntax element defining a motion estimation region (MER) from the video data. Video decoder 30 receives a plurality of second syntax elements from the video data, wherein the plurality of second syntax elements indicates whether the coding unit is in merge mode. Based on determining that the coding unit is in merge mode, video decoder 30 receives a plurality of merge candidates for the coding unit and determines whether the coding unit and a first spatially neighboring block used to determine a first merge candidate from the plurality of merge candidates are both within the same MER. Based on determining that the coding unit and the first spatially neighboring block are both within the same MER, video decoder 30 selects a second merge candidate from the plurality of merge candidates as a replacement for the first merge candidate.

[0222] In some embodiments, the merge mode is an intra block copy (IBC) merge.

[0223] In some embodiments, the merge mode is a combined inter and intra prediction (CIIP) merge.

[0224] Fourth embodiment:

[0225] In some embodiments, video decoder 30 receives video data corresponding to a coding unit from a bitstream. Video decoder 30 then receives a first syntax element defining a motion estimation region (MER) from the video data. Video decoder 30 also receives a plurality of second syntax elements from the video data, wherein the plurality of second syntax elements indicate whether the coding unit is in merge mode. Based on determining that the coding unit is in merge mode, video decoder 30 determines whether the coding unit and a first spatially neighboring block used to determine a merge candidate are both within the same MER. Based on determining that the coding unit and the first spatially neighboring block are both within the same MER, video decoder 30 selects a second spatially neighboring block outside the MER as an alternative to the first spatially neighboring block to determine the merge candidate.

[0226] In some embodiments, the second spatial neighboring block outside the MER is a spatial neighboring block at one of the merging candidate positions for the coding block of the adjacent MER.

[0227] In one or more examples, the functions described can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions can be stored on or transmitted through a computer-readable medium as one or more instructions or codes and executed by a hardware-based processing unit. A computer-readable medium may include a computer-readable storage medium, which corresponds to a tangible medium such as a data storage medium, or to a communication medium including any medium that facilitates the transfer of a computer program from one place to another (e.g., according to a communication protocol). In this manner, a computer-readable medium may generally correspond to (1) a non-volatile tangible computer-readable storage medium, or (2) a communication medium, such as a signal or carrier wave. A data storage medium can be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, codes, and / or data structures for implementing the implementations described in this application. A computer program product may include a computer-readable medium.

[0228] The terms used in the description of the embodiments herein are only used for the purpose of describing specific embodiments and are not intended to limit the scope of the claims. As used in the description of the embodiments and the appended claims, the singular forms "a", "an" and "the" are intended to also include the plural forms, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. It will be further understood that the terms "including" and / or "comprising..." when used in this specification specify the presence of stated features, elements and / or components, but do not exclude the presence or addition of one or more other features, elements, components and / or groups thereof.

[0229] It will also be understood that although the terms first, second, etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, without departing from the scope of the embodiments, a first electrode may be referred to as a second electrode, and similarly, a second electrode may be referred to as a first electrode. The first electrode and the second electrode are both electrodes, but they are not the same electrode.

[0230] References throughout this specification to "one example," "an example," "an illustrative example," and the like in the singular or plural form indicate that one or more specific features, structures, or characteristics described in connection with the example are included in at least one example of the present disclosure. Thus, the appearance of the phrases "in one example," "in an example," "in an illustrative example," and the like in the singular or plural form throughout this specification are not necessarily all referring to the same example. Furthermore, the specific features, structures, or characteristics of one or more examples may be combined in any suitable manner.

[0231] The description of the present application has been presented for the purpose of illustration and description and is not intended to be exhaustive or limited to the invention in the form disclosed. Many modifications, variations and alternative embodiments will be apparent to those of ordinary skill in the art having the benefit of the teachings presented in the foregoing description and the associated drawings. The embodiments are chosen and described in order to best explain the principles of the invention, the practical application, and to enable others skilled in the art to understand the various embodiments of the invention and to best utilize the basic principles and various embodiments with various modifications suitable for the specific use intended. Therefore, it will be understood that the scope of the claims is not limited to the specific examples of the disclosed embodiments, and that modifications and other embodiments are intended to be included within the scope of the appended claims.

Claims

1. A method for encoding video data, comprising: Determining whether the coding unit is in sub-block based temporal motion vector prediction (SbTMVP) mode; determining, when the coding unit is in the SbTMVP mode, whether the coding unit and a spatially neighboring block used to determine a temporal vector of the coding unit are both within a same motion estimation region MER, wherein if the coding unit and the spatially neighboring block are both within the same MER, the temporal vector of the coding unit is equal to zero; Sending a first syntax element and a second syntax element; The first syntax element is used to define MER, and the second syntax element is used to indicate whether the coding unit is in the SbTMVP mode. The temporal vector is a motion offset used to obtain sub-CU level motion information of the coding unit from the co-located block of the co-located picture.

2. The method of encoding video data according to claim 1, wherein the second syntax element includes a third syntax element indicating whether the coding unit is in subblock merge mode and a fourth syntax element indicating which subblock-based merge candidate to use to reconstruct the coding unit.

3. The method for encoding video data according to claim 2, wherein: The third syntax element is "merge_subblock_flag", and the fourth syntax element is "merge_subblock_idx".

4. The method for encoding video data according to claim 2, further comprising: When the coding unit is in the subblock merge mode, a plurality of subblock-based merge candidates derived from neighboring blocks are determined for the coding unit. 5 . The method of encoding video data according to claim 4 , wherein the plurality of sub-block based merge candidates include both SbTMVP candidates and affine merge candidates.

6. The method for encoding video data according to claim 5, further comprising: In case the sub-block based merge candidate is an SbTMVP candidate, the coding unit is in the SbTMVP mode.

7. An electronic device comprising: one or more processing units; a memory coupled to the one or more processing units; as well as A plurality of programs are stored in the memory, and when the plurality of programs are executed by the one or more processing units, the plurality of programs enable the electronic device to perform the method according to any one of claims 1 to 6.

8. A non-volatile computer-readable storage medium storing a plurality of programs for execution by an electronic device having one or more processing units, wherein the plurality of programs, when executed by the one or more processing units, causes the electronic device to perform the method according to any one of claims 1 to 6.

9. A computer program product comprising a plurality of programs for execution by a computing device having one or more processing units, wherein the plurality of programs, when executed by the one or more processors, causes the computing device to perform the method of any one of claims 1 to 6.

10. A method for storing a bitstream, comprising: Performing the method according to any one of claims 1 to 6 to generate a bit stream; as well as The bitstream is stored.

Images