Motion compensation considering out-of-boundary conditions in video coding.

Abstract

A method, an apparatus, and a non-transitory computer-readable storage medium for video coding are provided. In one method, a decoder derives a first reference picture and a second reference picture for a current coding block, the decoder uses a motion compensation process to derive a first predictor sample and a second predictor sample from the first reference picture based on a motion vector associated with the first reference picture, the decoder determines whether the first predictor sample and the second predictor sample are out of bounds (OOB), the decoder replaces the first predictor sample or the second predictor sample that is OOB with a first substitute predictor sample or a second substitute predictor sample in response to determining that the first predictor sample or the second predictor sample is OOB, and the decoder generates a final predicted sample by averaging sample values ​​of the first predictor sample and the second predictor sample.

Description

[Technical field]

[0001]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is based on and claims priority to Provisional Application No. 63 / 300,631, filed January 18, 2022, the entire contents of which are incorporated herein by reference for all purposes.

[0002]

[0002] This disclosure relates to video coding and compression. More particularly, this disclosure relates to methods and apparatus for inter prediction in video coding. [Background technology]

[0003]

[0003] Digital video is supported by various electronic devices, such as digital televisions, laptop or desktop computers, tablet computers, digital cameras, digital recording devices, digital media players, video gaming consoles, smartphones, video teleconferencing devices, video streaming devices, etc. The electronic devices transmit and receive digital video data or otherwise communicate over a communication network and / or store the digital video data in a storage device. Due to limited bandwidth capacity of the communication network and limited memory resources of the storage device, video coding may be used to compress the video data according to one or more video coding standards before the video data is communicated or stored. For example, video coding standards include Versatile Video Coding (VVC), Joint Exploration test Model (JEM), High-Efficiency Video Coding (HEVC / H.265), Advanced Video Coding (AVC / H.264), Moving Picture Expert Group (MPEG) encoding, etc. Video coding generally employs prediction methods (e.g., inter-prediction, intra-prediction, etc.) that exploit the redundancy inherent in video data. Video coding aims to compress video data into a format that uses a lower bitrate while avoiding or minimizing degradation of video quality. Summary of the Invention [Problem to be solved by the invention]

[0004]

[0004] Examples of this disclosure provide methods and apparatus for inter prediction in video coding. [Means for solving the problem]

[0005] According to a first aspect of the present disclosure, a method for decoding a bidirectionally predictively coded block is provided, the method includes: a decoder deriving a first reference picture for a current coding block; a decoder using a motion compensation process to derive a first predictor sample from the first reference picture based on a motion vector associated with the first reference picture; a decoder determining whether the first predictor sample is out of bounds (OOB); in response to determining that the first predictor sample is OOB, the decoder replacing the first predictor sample that is OOB with a first substitute predictor sample; and a decoder deriving a second reference picture for the current coding block. the decoder may include determining whether the second predictor sample is out of bounds (OOB), and in response to determining that the second predictor sample is OOB, the decoder may include replacing the second predictor sample that is OOB with a second substitute predictor sample, and generating a final predicted sample by averaging sample values ​​of the first predictor sample and the second predictor sample.

[0006]

[0006] According to a second aspect of the present disclosure, a method for coding a bidirectionally predictively encoded block is provided, the method includes: an encoder deriving a first reference picture for a current encoding block; an encoder using a motion compensation process to derive a first predictor sample from the first reference picture based on a motion vector associated with the first reference picture; an encoder determining whether the first predictor sample is out of bounds (OOB); in response to determining that the first predictor sample is OOB, the encoder replacing the predictor sample that is OOB with a first substitute predictor sample; and an encoder deriving a second reference picture for the current encoding block. the encoder using a motion compensation process to derive a second predictor sample from the second reference picture based on a motion vector associated with the second reference picture; the encoder determining whether the second predictor sample is out of bounds (OOB); and the decoder, in response to determining that the second predictor sample is OOB, replacing the second predictor sample that is OOB with a second substitute predictor sample; and generating a final predicted sample by averaging sample values ​​of the first predictor sample and the second predictor sample.

[0007]

[0007] According to a third aspect of the present disclosure, a method for decoding a unidirectional predictive coding block is provided. The method may include: a decoder deriving a first reference picture for a current coding block; a decoder deriving a predictor sample from the first reference picture using a motion compensation process based on a motion vector associated with the first reference picture; a decoder determining whether the predictor sample is out of bounds (OOB); and a decoder replacing the predictor sample that is OOB with a substitute predictor sample in response to determining that the predictor sample is OOB.

[0008]

[0008] According to a fourth aspect of the present disclosure, a method for coding a unidirectional predictive coding block is provided. The method may include: an encoder deriving a first reference picture for a current coding block; an encoder using a motion compensation process to derive a predictor sample from the first reference picture based on a motion vector associated with the first reference picture; an encoder determining whether the predictor sample is out of bounds (OOB); and an encoder replacing the predictor sample that is OOB with a substitute predictor sample in response to determining that the predictor sample is OOB.

[0009]

[0009] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not intended to be restrictive of the present disclosure.

[0010]

[0010] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate examples consistent with the present disclosure and, together with the description, serve to explain the principles of the disclosure. [Brief description of the drawings]

[0011] [Figure 1]

[0011] A block diagram showing an example system for encoding and decoding video blocks in accordance with some implementations of the present disclosure. [Diagram 2]

[0012] 1 is a block diagram illustrating an example video encoder according to some implementations of the present disclosure. [Diagram 3]

[0013] 1 is a block diagram illustrating an example video decoder in accordance with certain implementations of the present disclosure. [Figure 4A]

[0014] A block diagram showing how a frame is recursively divided into multiple video blocks of different sizes and shapes in accordance with some implementations of this disclosure. [Figure 4B]A block diagram showing how a frame is recursively divided into multiple video blocks of different sizes and shapes in accordance with some implementations of this disclosure. [Figure 4C] A block diagram showing how a frame is recursively divided into multiple video blocks of different sizes and shapes in accordance with some implementations of this disclosure. [Figure 4D] A block diagram showing how a frame is recursively divided into multiple video blocks of different sizes and shapes in accordance with some implementations of this disclosure. [Figure 4E] A block diagram showing how a frame is recursively divided into multiple video blocks of different sizes and shapes in accordance with some implementations of this disclosure. [Diagram 5]

[0015] A diagram showing example locations of spatial merge candidates, according to some implementations of the present disclosure. [Figure 6]

[0016] A diagram showing examples of candidate pairs considered for redundancy check of spatial merge candidates, according to some implementations of the present disclosure. [Figure 7]

[0017] A diagram illustrating an example motion vector scaling of temporal merge candidates in accordance with some implementations of the present disclosure. [Figure 8]

[0018] A diagram illustrating candidate positions of temporal merge candidates according to some implementations of the present disclosure. [Figure 9A]

[0019] FIG. 13 illustrates an example of an MMVD search point, according to some implementations of the present disclosure. [Figure 9B]

[0020] FIG. 13 illustrates an example of an MMVD search point, according to some implementations of the present disclosure. [Figure 10A]

[0021] A diagram showing an example of a control point-based affine motion model in accordance with some implementations of the present disclosure. [Figure 10B]

[0022] A diagram showing an example of a control point-based affine motion model in accordance with some implementations of the present disclosure. [Figure 11]

[0023] A diagram showing an example of affine MVF per sub-block in accordance with some implementations of the present disclosure. [Figure 12]

[0024] A diagram illustrating the location of inherited affine motion predictors in accordance with some implementations of the present disclosure. [Figure 13]

[0025] A diagram showing locations of candidate positions for configured affine merge modes in accordance with some implementations of the present disclosure. [Figure 14]

[0026] A diagram showing an example of sub-blocks MV and pixels according to some implementations of the present disclosure. [Figure 15A]

[0027] A diagram showing an example of spatially adjacent blocks used by the ATVMP in accordance with some implementations of the present disclosure. [Figure 15B]

[0028] FIG. 1 illustrates an example of an SbTMVP process, according to some implementations of the present disclosure. [Figure 16]

[0029] A diagram showing an example of an extended CU region used in BDOF in accordance with some implementations of the present disclosure. [Figure 17]

[0030] A diagram showing an example of decoding side motion vector refinement according to some implementations of the present disclosure. [Figure 18]

[0031] FIG. 13 illustrates an example of GPM divisions grouped by identical angles, according to some implementations of the present disclosure. [Figure 19]

[0032] A diagram showing an example of one-way prediction MV selection for geometric partitioning mode according to some implementations of the present disclosure. [Figure 20]

[0033] A diagram showing examples of upper and left neighboring blocks used in CIIP weight derivation, according to some implementations of the present disclosure. [Figure 21]

[0034] A diagram showing an example of spatially adjacent blocks used to derive spatial merging candidates in accordance with some implementations of the present disclosure. [Figure 22]

[0035] A diagram showing an example of template matching performed on a search area around an initial MV in accordance with some implementations of the present disclosure. [Diagram 23]

[0036] A diagram showing an example of a diamond region within a search area, according to some implementations of the present disclosure. [Figure 24]

[0037] A diagram showing the frequency responses of an interpolation filter and a VVC interpolation filter in the phase of half (1 / 2) pels in accordance with some implementations of the present disclosure. [Diagram 25]

[0038] A diagram showing an example of a template and a reference sample for the template in a reference picture in accordance with some implementations of the present disclosure. [Figure 26]

[0039] A diagram showing an example of a template for a block with sub-block motion and a reference sample for the template using motion information of the sub-blocks of a current block, in accordance with some implementations of the present disclosure. [Figure 27]

[0040] A diagram showing an example of a padded reference picture for motion compensation, according to some implementations of the present disclosure. [Figure 28]

[0041] FIG. 1 illustrates an example of fractional interpolation in accordance with some implementations of the present disclosure. [Figure 29]

[0042] FIG. 1 is a block diagram illustrating a computing environment coupled with a user interface in accordance with some implementations of the present disclosure. [Diagram 30]

[0043] A block diagram illustrating a video coding process according to some implementations of the present disclosure. [Diagram 31]

[0044] A diagram showing an example of a picture divided into CTUs, according to some implementations of the present disclosure. [Diagram 32]

[0045] A diagram showing an example of a picture divided into tiles and raster scan slices in accordance with some implementations of the present disclosure. [Diagram 33]

[0046] A diagram showing an example of a picture divided into tiles and rectangular slices, according to some implementations of the present disclosure. [Diagram 34]

[0047] A diagram showing an example of a picture divided into four tiles and four rectangular slices, according to some implementations of the present disclosure. [Diagram 35]

[0048] A diagram showing an example of a picture divided into 28 sub-pictures in accordance with some implementations of the present disclosure. [Diagram 36]

[0049] A diagram showing an example of horizontal wraparound motion compensation in VVC according to some implementations of the present disclosure. [Figure 37]

[0050] 1 is a flowchart illustrating a video coding method according to some implementations of the present disclosure. [Figure 38]

[0051] 1 is a flowchart illustrating a method for decoding a bidirectionally predictively coded block according to some implementations of the present disclosure. [Figure 39]

[0052] 1 is a flowchart illustrating a method for coding a bidirectionally predictively coded block according to some implementations of the present disclosure. [Diagram 40]

[0053] 1 is a flowchart illustrating a method for decoding a unidirectional predictively coded block according to some implementations of the present disclosure. [Diagram 41]

[0054] 1 is a flowchart illustrating a method for coding a unidirectional predictively coded block according to some implementations of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012]

[0055] Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. The following description refers to the accompanying drawings, in which the same numbers in different drawings represent the same or similar elements unless otherwise stated. The implementations described in the following description of the exemplary embodiments do not represent all implementations consistent with the present disclosure. Rather, the implementations are merely examples of apparatuses and methods consistent with aspects related to the present disclosure as described in the appended claims.

[0013]

[0056] The terms used in this disclosure are for the purpose of describing particular embodiments only and are not intended to be limiting of the disclosure. As used in this disclosure and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. It is also to be understood that the term "and / or" as used herein is intended to mean and include any and all possible combinations of one or more of the associated listed items.

[0014]

[0057] It should be understood that terms such as "first", "second", and "third" may be used herein to describe various information, but the information should not be limited by these terms. These terms are used only to distinguish one category of information from another. For example, the first information may be referred to as the second information, and similarly, the second information may be referred to as the first information, without departing from the scope of this disclosure. The term "if" as used herein may be understood to mean "when", "in the event of", or "at the discretion of", depending on the context.

[0015]

[0058] To compress video data, various video coding techniques may be used. Video coding is performed according to one or more video coding standards. For example, today, well-known video coding standards include Versatile Video Coding (VVC), High Efficiency Video Coding (HEVC, also called H.265 or MPEG-H Part2), and Advanced Video Coding (AVC, also called H.264 or MPEG-4 Part10), which are jointly developed by ISO / IEC MPEG and ITU-T VECG. AOMedia Video 1 (AV1) was developed by the Alliance for Open Media (AOM) as a successor to its previous standard, VP9. Audio Video Coding (AVS), which refers to digital audio and digital video compression standards, is another series of video compression standards developed by the Chinese Audio and Video Coding Standards Workgroup. Most of the existing video coding standards are built on the well-known hybrid video coding framework, i.e., using block-based prediction methods (e.g., inter-prediction, intra-prediction) to reduce the redundancy present in a video image or sequence, and transform coding to compact the energy of the prediction error. An important goal of video coding techniques is to compress video data into a format that uses a lower bit rate while avoiding or minimizing the loss of video quality.

[0016]

[0059] The first generation AVS standards include the Chinese national standard "Information Technology, Advanced Audio-Video Coding, Part 2: Video" (known as AVS1) and "Information Technology, Advanced Audio-Video Coding, Part 16: Radio-Television Video" (known as AVS+). It can achieve about 50% bitrate savings with the same perceptual quality compared to the MPEG-2 standard. The video part of the AVS1 standard was promulgated as the Chinese national standard in February 2006. The second generation AVS standards include a series of Chinese national standards "Information Technology, Efficient Multimedia Coding" (known as AVS2), which are mainly aimed at the transmission of additional HD TV programs. The coding efficiency of AVS2 is twice that of AVS+. In May 2016, AVS2 was issued as the Chinese national standard. Meanwhile, the video part of the AVS2 standard was put forward by the Institute of Electrical and Electronics Engineers (IEEE) as one of the international standards for applications. The AVS3 standard is one of the new generation video coding standards for UHD video applications, aiming to exceed the coding efficiency of the latest international standard, HEVC. In March 2019, the 68th AVS Conference finalized the AVS3-P2 baseline, which achieves approximately 30% bitrate savings compared to the HEVC standard. Currently, there is one reference software, called the High Performance Model (HPM), maintained by the AVS group to demonstrate a reference implementation of the AVS3 standard.

[0017]

[0060] 1 is a block diagram illustrating an example system 10 for encoding and decoding video blocks in parallel, according to some implementations of the present disclosure. As shown in FIG. 1, system 10 includes a source device 12 that generates and encodes video data that is subsequently decoded by a destination device 14. Source device 12 and destination device 14 may include any of a wide variety of electronic devices, including desktop or laptop computers, tablet computers, smartphones, set-top boxes, digital televisions, cameras, display devices, digital media players, video gaming consoles, video streaming devices, and the like. In some implementations, source device 12 and destination device 14 are equipped with wireless communication capabilities.

[0018]

[0061] In some implementations, the destination device 14 may receive the encoded video data to be decoded via a link 16. The link 16 may comprise any type of communication medium or device capable of moving the encoded video data from the source device 12 to the destination device 14. In one example, the link 16 may comprise a communication medium that allows the source device 12 to transmit the encoded video data directly to the destination device 14 in real time. The encoded video data may be modulated according to a communication standard, such as a wireless communication protocol, and transmitted to the destination device 14. The communication medium may include any wireless or wired communication medium, such as the radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form part of a packet-based network, such as a local area network, a wide area network, or a global network such as the Internet. The communication medium may include routers, switches, base stations, or any other equipment that may be useful in facilitating communication from the source device 12 to the destination device 14.

[0019]

[0062] In some other implementations, the encoded video data may be transmitted from output interface 22 to storage device 32. The encoded video data in storage device 32 may then be accessed by destination device 14 via input interface 28. Storage device 32 may include any of a variety of distributed or locally accessed data storage media, such as a hard drive, Blu-ray® disc, digital versatile disc (DVD), compact disc read only memory (CD-ROM), flash memory, volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded video data. In a further example, storage device 32 may correspond to a file server or another intermediate storage device capable of holding the encoded video data generated by source device 12. Destination device 14 may access the stored video data from storage device 32 via streaming or download. The file server may be any type of computer capable of storing encoded video data and transmitting the encoded video data to destination device 14. Exemplary file servers include web servers (e.g., for websites), File Transfer Protocol (FTP) servers, Network Attached Storage (NAS) devices, or local disk drives. Destination device 14 may access the encoded video data through any standard data connection, including wireless channels (e.g., Wireless Fidelity (Wi-Fi) connections), wired connections (e.g., Digital Subscriber Line (DSL), cable modems, etc.), or a combination of both suitable for accessing encoded video data stored in a file server. Transmission of the encoded video data from storage device 32 may be a streaming transmission, a download transmission, or a combination of both.

[0020]

[0063] As shown in FIG. 1, source device 12 includes a video source 18, a video encoder 20, and an output interface 22. Video source 18 may include sources such as a video imaging device, e.g., a video camera, a video archive containing previously captured video, a video feed interface for receiving video from a video content provider, and / or a computer graphics system for generating computer graphics data as the source video, or a combination of such sources. As an example, if video source 18 is a video camera of a security surveillance system, source device 12 and destination device 14 may form a camera phone or a video phone. However, implementations described in this application may be applicable to video coding generally and may be applicable to wireless and / or wired applications.

[0021]

[0064] The captured, pre-captured, or computer-generated video may be encoded by video encoder 20. The encoded video data may be transmitted directly to destination device 14 via output interface 22 of source device 12. The encoded video data may also (or alternatively) be stored in storage device 32 for later access by destination device 14 or other devices for decoding and / or playback. Output interface 22 may further include a modem and / or a transmitter.

[0022]

[0065] Destination device 14 includes an input interface 28, a video decoder 30, and a display device 34. Input interface 28 may include a receiver and / or modem and receive encoded video data via link 16. The encoded video data communicated via link 16 or provided on storage device 32 may include various syntax elements generated by video encoder 20 for use by video decoder 30 in decoding the video data. Such syntax elements may be included within the encoded video data transmitted over a communication medium, stored on a storage medium, or stored on a file server.

[0023]

[0066] In some implementations, destination device 14 may include a display device 34, which may be an integrated display device as well as an external display device configured to communicate with destination device 14. Display device 34 displays the decoded video data to a user and may comprise any of a variety of display devices, such as a Liquid Crystal Display (LCD), a plasma display, an Organic Light Emitting Diode (OLED) display, or another type of display device.

[0024]

[0067] Video encoder 20 and video decoder 30 may operate according to a proprietary or industry standard, such as VVC, HEVC, MPEG-4 Part 10, AVC, or extensions of such standards. It should be understood that the present application is not limited to a particular video encoding / decoding standard, but is applicable to other video encoding / decoding standards. It is generally contemplated that video encoder 20 of source device 12 may be configured to encode video data according to any of these current or future standards. Similarly, it is generally contemplated that video decoder 30 of destination device 14 may be configured to decode video data according to any of these current or future standards.

[0025]

[0068] Each of the video encoder 20 and the video decoder 30 may be implemented as any of a variety of suitable encoder and / or decoder circuits, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combination thereof. If implemented partially in software, the electronic device may store instructions for the software on a suitable non-transitory computer-readable medium and execute the instructions in hardware using one or more processors to perform the video encoding / decoding operations disclosed in this disclosure. Each of the video encoder 20 and the video decoder 30 may be included in one or more encoders or decoders, any of which may be integrated as part of a combined encoder / decoder (CODEC) within the respective device.

[0026]

[0069] 2 is a block diagram illustrating an example video encoder 20 according to some implementations described in this application. The video encoder 20 may perform intra-predictive and inter-predictive coding of video blocks within a video frame. Intra-predictive coding relies on spatial prediction to reduce or remove spatial redundancy in video data within a given video frame or picture. Inter-predictive coding relies on temporal prediction to reduce or remove temporal redundancy in video data within adjacent video frames or pictures of a video sequence. It should be noted that the term "frame" is sometimes used synonymously with the term "image" or "picture" in the field of video coding.

[0027]

[0070] As shown in FIG. 2, the video encoder 20 includes a video data memory 40, a prediction processor 41, a decoded picture buffer (DPB) 64, an adder 50, a transform processor 52, a quantizer 54, and an entropy encoder 56. The prediction processor 41 further includes a motion estimator 42, a motion compensation unit 44, a partitioner 45, an intra prediction processor 46, and an intra block copy (BC) unit 48. In some implementations, the video encoder 20 also includes an inverse quantizer 58 for reconstruction of video blocks, an inverse transform processor 60, and an adder 62. An in-loop filter 63, such as a deblocking filter, may be disposed between the adder 62 and the DPB 64 to filter block boundaries to remove blocky artifacts from the reconstructed video. In addition to the deblocking filter, another in-loop filter, such as a Sample Adaptive Offset (SAO) filter and / or an adaptive in-loop filter (ALF), may also be used to filter the output of summer 62. In some examples, the in-loop filter may be omitted and the decoded video block may be provided directly to DPB 64 by summer 62. Video encoder 20 may take the form of a fixed or programmable hardware unit, or may be divided into one or more of the illustrated fixed or programmable hardware units.

[0028]

[0071] Video data memory 40 may store video data to be encoded by components of video encoder 20. The video data in video data memory 40 may be obtained, for example, from video source 18 shown in FIG. 1. DPB 64 is a buffer that stores reference video data (e.g., reference frames or pictures) for use in encoding video data by video encoder 20 (e.g., in an intra-predictive or inter-predictive coding mode). Video data memory 40 and DPB 64 may be formed by any of a variety of memory devices. In various examples, video data memory 40 may be on-chip with other components of video encoder 20 or may be off-chip relative to those components.

[0029]

[0072] As shown in FIG. 2, after receiving the video data, the partitioning unit 45 in the prediction processing unit 41 partitions the video data into video blocks. This partitioning may also include partitioning the video frame into slices, tiles (e.g., a set of video blocks), or other larger coding units (CUs) according to a predefined partitioning structure, such as a Quad-Tree (QT) structure associated with the video data. A video frame may be or be considered as a two-dimensional array or matrix of samples having sample values. The samples in the array may also be referred to as pixels or pels. The number of samples in the horizontal and vertical directions (or axes) of the array or picture defines the size and / or resolution of the video frame. The video frame may be partitioned into multiple video blocks, for example, by using QT partitioning. The video blocks may also be or be considered as a two-dimensional array or matrix of samples having sample values, although with smaller dimensions than the video frame. The number of samples in the horizontal and vertical directions (or axes) of the video block defines the size of the video block. A video block may be further divided into one or more block divisions or sub-blocks (which may again form blocks), e.g., by repeatedly using QT, Binary-Tree (BT) or Triple-Tree (TT) divisions, or any combination thereof. Note that the term "block" or "video block" as used herein may be a portion of a frame or picture, in particular a rectangular (square or non-square) portion. For example, with reference to HEVC and VVC, a block or video block may be or correspond to a coding tree unit (CTU), a CU, a prediction unit (PU), or a transform unit (TU), and / or a corresponding block, e.g., a coding tree block (CTB), a coding block (CB), a prediction block (PB), or a transform block (TB), and / or a sub-block.

[0030]

[0073] Prediction processor 41 may select one of a number of possible predictive coding modes, such as one of a number of intra-predictive coding modes or one of a number of inter-predictive coding modes, for the current video block based on the error result (e.g., coding rate and level of distortion). Prediction processor 41 may provide the resulting intra-predictive or inter-predictive coded block to summer 50 to generate a residual block and to summer 62 to reconstruct the coded block for later use as part of a reference frame. Prediction processor 41 also provides syntax elements, such as motion vectors, intra-mode indicators, partitioning information, and other such syntax information to entropy coder 56.

[0031]

[0074] To select an appropriate intra-prediction coding mode for a current video block, intra prediction unit 46 within prediction unit 41 may perform intra-prediction coding of the current video block relative to one or more neighboring blocks in the same frame as the current block to be coded to provide spatial prediction. Motion estimation unit 42 and motion compensation unit 44 within prediction unit 41 may perform inter-prediction coding of the current video block relative to one or more predictive blocks in one or more reference frames to provide temporal prediction. Video encoder 20 may perform multiple coding passes to select an appropriate coding mode for each block of video data, for example.

[0032]

[0075] In some implementations, motion estimator 42 determines the inter prediction mode for the current video frame by generating motion vectors that indicate the displacement of video blocks in the current video frame relative to a predictive block in a reference video frame according to a predetermined pattern in the sequence of video frames. Motion estimation performed by motion estimator 42 is the process of generating motion vectors that estimate the motion of video blocks. The motion vectors may indicate, for example, the displacement of video blocks in a current video frame or picture relative to a predictive block in a reference frame relative to a current block being coded in the current frame. The predetermined pattern may designate a video frame in the sequence as a P frame or a B frame. Intra BC unit 48 may determine vectors, e.g., block vectors, for intra BC coding in a manner similar to the determination of motion vectors by motion estimator 42 for inter prediction, or may utilize motion estimator 42 to determine block vectors.

[0033]

[0076] A prediction block of a video block may be or correspond to a block of a reference frame or reference block that is deemed to closely match the video block to be encoded in terms of pixel differences, which may be determined by sum of absolute difference (SAD), sum of square difference (SSD), or other difference metric. In some implementations, video encoder 20 may calculate values ​​for sub-integer pixel locations of the reference frame stored in DPB 64. For example, video encoder 20 may interpolate values ​​for quarter-pixel, eighth-pixel, or other fractional pixel locations of the reference frame. Thus, motion estimation unit 42 may perform motion search for whole pixel locations and fractional pixel locations and output motion vectors with fractional pixel precision.

[0034]

[0077] Motion estimation unit 42 calculates a motion vector for a video block in an inter-predictively coded frame by comparing the position of the video block to the position of a predictive block of a reference frame selected from a first reference frame list (List 0) or a second reference frame list (List 1), each of which identifies one or more reference frames stored in DPB 64. Motion estimation unit 42 sends the calculated motion vector to motion compensation unit 44 and then to entropy coding unit 56.

[0035]

[0078] The motion compensation performed by motion compensation unit 44 may include obtaining or generating a predictive block based on the motion vector determined by motion estimation unit 42. Upon receiving the motion vector for the current video block, motion compensation unit 44 may locate the predictive block to which the motion vector points in one of the reference frame lists, obtain the predictive block from DPB 64, and forward the predictive block to summer 50. Summer 50 then forms a residual video block of pixel difference values ​​by subtracting pixel values ​​of the predictive block provided by motion compensation unit 44 from pixel values ​​of the current video block being coded. The pixel difference values ​​forming the residual video block may include luma or chroma difference components, or both. Motion compensation unit 44 may also generate syntax elements associated with the video blocks of the video frames for use by video decoder 30 in decoding the video blocks of the video frames. The syntax elements may include, for example, syntax elements defining the motion vector used to identify the predictive block, any flags indicating a prediction mode, or any other syntax information described herein. It should be noted that motion estimator 42 and motion compensator 44 may be highly integrated, but are illustrated separately for conceptual purposes.

[0036]

[0079] In some implementations, intra BC unit 48 may generate a vector to obtain a predictive block in a manner similar to that described above in connection with motion estimation unit 42 and motion compensation unit 44, except that the predictive block is in the same frame as the current block being coded, and the vector is referred to as a block vector rather than a motion vector. Specifically, intra BC unit 48 may determine an intra prediction mode to use to code the current block. In some examples, intra BC unit 48 may code the current block using various intra prediction modes, e.g., during separate coding passes, and test their performance through rate-distortion analysis. Intra BC unit 48 may then select an appropriate intra prediction mode to use from among the various tested intra prediction modes and generate an intra mode indicator accordingly. For example, intra BC unit 48 may calculate rate-distortion values ​​using the rate-distortion analysis for the various tested intra prediction modes, and select the intra prediction mode with the best rate-distortion characteristics from among the tested modes as the appropriate intra prediction mode to use. The rate-distortion analysis generally determines the amount of distortion (or error) between a coded block and the original uncoded block that was coded to create the coded block, and the bitrate (i.e., number of bits) used to create the coded block. Intra BC unit 48 may calculate ratios from the distortions and rates of the various coded blocks to determine which intra prediction mode exhibits the best rate-distortion value for the block.

[0037]

[0080] In other examples, intra BC unit 48 may use, in whole or in part, motion estimation unit 42 and motion compensation unit 44 to perform such functions for intra BC prediction according to implementations described herein. In either case, for intra block copying, the predictive block may be a block that is deemed to closely match the block to be coded in terms of pixel differences, which may be determined by SAD, SSD, or other difference metrics, and identification of the predictive block may include calculation of values ​​of sub-integer pixel positions.

[0038]

[0081] Regardless of whether the predictive block is a block from the same frame via intra prediction or a block from a different frame via inter prediction, video encoder 20 may form a residual video block by subtracting pixel values ​​of the predictive block from pixel values ​​of the current video block being coded to form pixel difference values. The pixel difference values ​​that form the residual video block may include both luma and chroma component differences.

[0039]

[0082] The intra prediction unit 46 may intra predict the current video block as an alternative to the inter prediction performed by the motion estimation unit 42 and the motion compensation unit 44, or the intra block copy prediction performed by the intra BC unit 48, as described above. Specifically, the intra prediction unit 46 may determine an intra prediction mode to use to encode the current block. To do so, the intra prediction unit 46 may encode the current block using various intra prediction modes, e.g., during separate encoding passes, and the intra prediction unit 46 (or, in some examples, a mode selection unit) may select an appropriate intra prediction mode to use from the tested intra prediction modes. The intra prediction unit 46 may provide information to the entropy coder 56 indicating the selected intra prediction mode for the block. The entropy coder 56 may encode the information indicating the selected intra prediction mode into the bitstream.

[0040]

[0083] After prediction processor 41 determines a predictive block for the current video block by inter- or intra-prediction, adder 50 forms a residual video block by subtracting the predictive block from the current video block. The residual video data in the residual block, which may be included in one or more TUs, is provided to transform processor 52. Transform processor 52 converts the residual video data into residual transform coefficients using a transform, such as a Discrete Cosine Transform (DCT) or a conceptually similar transform.

[0041]

[0084] The transform processor 52 may send the resulting transform coefficients to a quantizer 54, which quantizes the transform coefficients to further reduce the bit rate. The quantization process may also reduce the bit depth associated with some or all of the coefficients. The degree of quantization may be modified by adjusting a quantization parameter. In some examples, the quantizer 54 may then perform a scan of a matrix containing the quantized transform coefficients. Alternatively, the entropy coder 56 may perform the scan.

[0042]

[0085] Following quantization, entropy encoder 56 entropy encodes the quantized transform coefficients into a video bitstream using, for example, Context Adaptive Variable Length Coding (CAVLC), Context Adaptive Binary Arithmetic Coding (CABAC), Syntax-based context-adaptive Binary Arithmetic Coding (SBAC), Probability Interval Partitioning Entropy (PIPE) coding, or another entropy encoding methodology or technique. The encoded bitstream may then be transmitted to video decoder 30 as shown in FIG. 1 or stored in storage device 32 as shown in FIG. 1 for later transmission to or retrieval by video decoder 30. Entropy encoder 56 may also entropy encode motion vectors and other syntax elements of the current video frame being encoded.

[0043]

[0086] Inverse quantization unit 58 and inverse transform unit 60 apply inverse quantization and inverse transform, respectively, to reconstruct the residual video block in the pixel domain to generate reference blocks for prediction of other video blocks. As described above, motion compensation unit 44 may generate a motion compensated prediction block from one or more reference blocks of a frame stored in DPB 64. Motion compensation unit 44 may also apply one or more interpolation filters to the prediction block to calculate sub-integer pixel values ​​for use in motion estimation.

[0044]

[0087] Summer 62 adds the reconstructed residual block to the motion compensated prediction block produced by motion compensation unit 44 to create a reference block for storage in DPB 64. The reference block may then be used by intra BC unit 48, motion estimation unit 42, and motion compensation unit 44 as a prediction block to inter predict another video block in a subsequent video frame.

[0045]

[0088] 3 is a block diagram illustrating an exemplary video decoder 30 according to some implementations of the present application. The video decoder 30 includes a video data memory 79, an entropy decoding unit 80, a prediction processing unit 81, an inverse quantization unit 86, an inverse transform processing unit 88, an adder 90, and a DPB 92. The prediction processing unit 81 further includes a motion compensation unit 82, an intra prediction unit 84, and an intra BC unit 85. The video decoder 30 may perform a decoding process that is generally the reverse of the encoding process described above for the video encoder 20 in connection with FIG. 2. For example, the motion compensation unit 82 may generate prediction data based on a motion vector received from the entropy decoding unit 80, while the intra prediction unit 84 may generate prediction data based on an intra prediction mode indicator received from the entropy decoding unit 80.

[0046]

[0089] In some examples, the processing units of the video decoder 30 may be tasked with performing implementations of the present application. Also, in some examples, implementations of the present disclosure may be split among one or more of the processing units of the video decoder 30. For example, the intra BC unit 85 may perform implementations of the present application alone or in combination with other processing units of the video decoder 30, such as the motion compensation unit 82, the intra prediction unit 84, and the entropy decoding unit 80. In some examples, the video decoder 30 may not include the intra BC unit 85, and the functions of the intra BC unit 85 may be performed by other components of the prediction processing unit 81, such as the motion compensation unit 82.

[0047]

[0090] Video data memory 79 may store video data, such as an encoded video bitstream, that is decoded by other components of video decoder 30. The video data stored in video data memory 79 may be obtained, for example, from storage device 32, from a local video source such as a camera, via wired or wireless network communication of video data, or by accessing a physical data storage medium (e.g., a flash drive or hard disk). Video data memory 79 may include a coded picture buffer (CPB) that stores coded video data from the coded video bitstream. DPB 92 of video decoder 30 stores reference video data for use in decoding video data by video decoder 30 (e.g., in an intra-predictive coding mode or an inter-predictive coding mode). Video data memory 79 and DPB 92 may be formed by any of a variety of memory devices, such as dynamic random access memory (DRAM), including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. For purposes of illustration, video data memory 79 and DPB 92 are shown in FIG. 3 as two separate components of video decoder 30. However, it will be apparent to one skilled in the art that video data memory 79 and DPB 92 may be provided by the same memory device or separate memory devices. In some examples, video data memory 79 may be on-chip with other components of video decoder 30 or may be off-chip with respect to those components.

[0048]

[0091] During the decoding process, video decoder 30 receives an encoded video bitstream representing video blocks of encoded video frames and associated syntax elements. Video decoder 30 may receive the syntax elements at the video frame level and / or the video block level. An entropy decoding unit 80 of video decoder 30 entropy decodes the bitstream to generate quantized coefficients, motion vectors or intra-prediction mode indicators, and other syntax elements. Entropy decoding unit 80 then forwards the motion vectors or intra-prediction mode indicators and other syntax elements to a prediction processor 81.

[0049]

[0092] When a video frame is coded as an intra-predictive coded (I) frame or for intra-coded predictive blocks in other types of frames, the intra prediction unit 84 of the prediction processing unit 81 may generate predictive data for video blocks of the current video frame based on a signaled intra-prediction mode and reference data from previously decoded blocks of the current frame.

[0050]

[0093] When a video frame is coded as an inter-predictive (i.e., B or P) frame, motion compensation unit 82 of prediction unit 81 creates one or more predictive blocks for video blocks of the current video frame based on the motion vectors and other syntax elements received from entropy decoding unit 80. Each of the predictive blocks may be created from a reference frame in one of the reference frame lists. Video decoder 30 may configure the reference frame lists, List 0 and List 1, using a default configuration technique based on the reference frames stored in DPB 92.

[0051]

[0094] In some examples, when a video block is encoded according to the intra BC mode described herein, intra BC unit 85 of prediction processor 81 creates a predictive block of the current video block based on the block vectors and other syntax elements received from entropy decoding unit 80. The predictive block may be within the same reconstructed region of the picture as the current video block defined by video encoder 20.

[0052]

[0095] Motion compensation unit 82 and / or intra BC unit 85 determine prediction information for video blocks of the current video frame by analyzing the motion vectors and other syntax elements, and then use the prediction information to create a predictive block for the current video block being decoded. For example, motion compensation unit 82 uses some of the received syntax elements to determine the prediction mode (e.g., intra prediction or inter prediction) used to encode the video blocks of the video frame, the inter prediction frame type (e.g., B or P), configuration information regarding one or more of the reference frame lists of the frame, the motion vectors of each inter predictive coded video block of the frame, the inter prediction status of each inter predictive coded video block of the frame, and other information for decoding video blocks in the current video frame.

[0053]

[0096] Similarly, intra BC unit 85 may use some of the received syntax elements, such as flags, to determine that the current video block was predicted using intra BC mode, configuration information regarding which video blocks of the frame are within the reconstruction domain and should be stored in DPB 92, block vectors for each intra BC predicted video block of the frame, the intra BC prediction status for each intra BC predicted video block of the frame, and other information for decoding video blocks in the current video frame.

[0054]

[0097] Motion compensation unit 82 may also perform interpolation using an interpolation filter used by video encoder 20 during encoding of the video block to calculate sub-integer pixel interpolated values ​​of the reference block. In this case, motion compensation unit 82 may determine the interpolation filter used by video encoder 20 from the received syntax elements and use that interpolation filter to create the prediction block.

[0055]

[0098] An inverse quantizer 86 uses the same quantization parameters calculated by video encoder 20 for each video block in a video frame to inverse quantize the quantized transform coefficients provided in the bitstream and entropy decoded by entropy decoding unit 80 to determine the degree of quantization. An inverse transform processor 88 applies an inverse transform, e.g., an inverse DCT, an inverse integer transform, or a conceptually similar inverse transform process, to the transform coefficients in order to reconstruct the residual block in the pixel domain.

[0056]

[0099] After the motion compensation unit 82 or the intra BC unit 85 generates a predictive block for the current video block based on the vectors and other syntax elements, the adder 90 reconstructs a decoded video block for the current video block by adding the residual block from the inverse transform processor 88 and the corresponding predictive block generated by the motion compensation unit 82 and the intra BC unit 85. To further process the decoded video block, an in-loop filter 91, such as a deblocking filter, an SAO filter, and / or an ALF, may be disposed between the adder 90 and the DPB. In some examples, the in-loop filter 91 may be omitted and the decoded video block may be provided directly to the DPB 92 by the adder 90. The decoded video block in a given frame is then stored in the DPB 92, which stores reference frames used for subsequent motion compensation of the next video block. The DPB 92 or a memory device separate from the DPB 92 may store the decoded video for later presentation on a display device, such as the display device 34 of FIG. 1.

[0057]

[0100] In a typical video coding process, a video sequence typically includes an ordered set of frames or pictures. Each frame may include three sample arrays, denoted SL, SCb, and SCr. SL is a two-dimensional array of luma samples. SCb is a two-dimensional array of Cb chroma samples. SCr is a two-dimensional array of Cr chroma samples. In another example, a frame may be monochromatic and thus include only one two-dimensional array of luma samples.

[0058]

[0101] As shown in FIG. 4A, video encoder 20 (or, more specifically, divider 45) generates an encoded representation of a frame by first dividing the frame into a set of CTUs. A video frame may include an integer number of CTUs ordered consecutively from left to right and top to bottom in raster scan order. Each CTU is the largest logical coding unit, and the width and height of the CTUs are signaled by video encoder 20 in a sequence parameter set such that all CTUs in a video sequence have the same size, which is one of 128×128, 64×64, 32×32, and 16×16. However, it should be noted that the present application is not necessarily limited to a particular size. As shown in FIG. 4B, each CTU may include one CTB of luma samples, two corresponding coding tree blocks of chroma samples, and syntax elements used to code the samples of the coding tree blocks. The syntax elements describe how a video sequence may be reconstructed in video decoder 30, including characteristics of different types of units of coding blocks of pixels, as well as inter or intra prediction, intra prediction modes, motion vectors, and other parameters. For a monochromatic picture or a picture with three separate color planes, a CTU may include a single coding tree block and syntax elements used to code samples of the coding tree block. A coding tree block may be an N×N block of samples.

[0059]

[0102] To achieve better performance, video encoder 20 may recursively perform tree partitioning, such as binary tree partitioning, ternary tree partitioning, quad tree partitioning, or a combination thereof, on the coding tree blocks of the CTU to partition the CTU into smaller CUs. As shown in FIG. 4C, 64×64 CTU 400 is first partitioned into four smaller CUs, each with a block size of 32×32. Of the four smaller CUs, CU 410 and CU 420 are partitioned by block size into four 16×16 CUs, respectively. Two 16×16 CUs 430 and CU 440 are further partitioned by block size into four 8×8 CUs, respectively. FIG. 4D shows a quad tree data structure illustrating the final result of the partitioning process of CTU 400 shown in FIG. 4C, where each terminal node of the quad tree corresponds to one CU of a respective size ranging from 32×32 to 8×8. Similar to the CTU shown in FIG. 4B, each CU may include a CB of luma samples, two corresponding coding blocks of chroma samples of the same size frame, and syntax elements used to code the samples of the coding blocks. In a monochromatic picture or a picture with three separate color planes, a CU may include a single coding block and syntax structures used to code the samples of the coding blocks. It should be noted that the quadtree division shown in FIG. 4C and FIG. 4D is for illustration purposes only, and one CTU may be divided into CUs based on quadtree / ternary tree / binary tree division to accommodate various local characteristics. In a multi-type tree structure, one CTU is divided by a quadtree structure, and the CUs at each quadtree end may be further divided by binary tree structure and ternary tree structure. As shown in FIG. 4E, there are five possible division types of a coding block with width W and height H, namely, 4-way division, horizontal 2-way division, vertical 2-way division, horizontal 3-way division, and vertical 3-way division.

[0060]

[0103] In some implementations, video encoder 20 may further divide the coding blocks of a CU into one or more M×N PBs. A PB is a rectangular (square or non-square) block of samples to which the same prediction, inter or intra prediction, is applied. A PU of a CU may include a PB of luma samples, two corresponding PBs of chroma samples, and syntax elements used to predict the PBs. In a monochromatic picture or a picture with three separate color planes, a PU may include a single PB and syntax structures used to predict the PBs. Video encoder 20 may generate a predictive luma block, a predictive Cb block, and a predictive Cr block for the luma, Cb, and Cr PBs of each PU of a CU.

[0061]

[0104] Video encoder 20 may generate the predictive blocks of a PU using intra prediction or inter prediction. If video encoder 20 generates the predictive blocks of a PU using intra prediction, video encoder 20 may generate the predictive blocks of the PU based on decoded samples of a frame associated with the PU. If video encoder 20 generates the predictive blocks of a PU using inter prediction, video encoder 20 may generate the predictive blocks of the PU based on decoded samples of one or more frames other than the frame associated with the PU.

[0062]

[0105] After video encoder 20 generates the predicted luma block, the predicted Cb block, and the predicted Cr block for one or more PUs of a CU, video encoder 20 may generate a luma residual block of the CU by subtracting the predicted luma block of the CU from its original luma coding block, such that each sample in the luma residual block of the CU indicates a difference between a luma sample in one of the predicted luma blocks of the CU and a corresponding sample in the original luma coding block of the CU. Similarly, video encoder 20 may generate a Cb residual block and a Cr residual block of the CU, such that each sample in the Cb residual block of the CU indicates a difference between a Cb sample in one of the predicted Cb blocks of the CU and a corresponding sample in the original Cb coding block of the CU, and such that each sample in the Cr residual block of the CU indicates a difference between a Cr sample in one of the predicted Cr blocks of the CU and a corresponding sample in the original Cr coding block of the CU.

[0063]

[0106] Further, as shown in FIG. 4C, video encoder 20 may use quadtree partitioning to decompose the luma residual block, Cb residual block, and Cr residual block of a CU into one or more luma transform blocks, Cb transform blocks, and Cr transform blocks, respectively. A transform block is a rectangular (square or non-square) block of samples to which the same transform is applied. A TU of a CU may include a transform block of luma samples, two corresponding transform blocks of chroma samples, and syntax elements used to transform the transform block samples. Thus, each TU of a CU may be associated with a luma transform block, a Cb transform block, and a Cr transform block. In some examples, the luma transform block associated with a TU may be a sub-block of the luma residual block of the CU. The Cb transform block may be a sub-block of the Cb residual block of the CU. The Cr transform block may be a sub-block of the Cr residual block of the CU. In a monochrome picture or a picture with three separate color planes, a TU may include a single transform block and syntax structures used to transform samples of the transform block.

[0064]

[0107] Video encoder 20 may apply one or more transforms to a luma transform block of the TU to generate a luma coefficient block of the TU. The coefficient block may be a two-dimensional array of transform coefficients. The transform coefficients may be scalar quantities. Video encoder 20 may apply one or more transforms to a Cb transform block of the TU to generate a Cb coefficient block of the TU. Video encoder 20 may apply one or more transforms to a Cr transform block of the TU to generate a Cr coefficient block of the TU.

[0065]

[0108] After the video encoder 20 generates a coefficient block (e.g., a luma coefficient block, a Cb coefficient block, or a Cr coefficient block), it may quantize the coefficient block. Quantization generally refers to a process in which transform coefficients are quantized to potentially reduce the amount of data used to represent the transform coefficients to achieve further compression. After the video encoder 20 quantizes the coefficient block, the video encoder 20 may entropy code syntax elements indicating the quantized transform coefficients. For example, the video encoder 20 may perform CABAC on the syntax elements indicating the quantized transform coefficients. Finally, the video encoder 20 may output a bitstream including a sequence of bits forming a representation of the encoded frame and associated data, which may be stored in the storage device 32 or transmitted to the destination device 14.

[0066]

[0109] After receiving the bitstream generated by video encoder 20, video decoder 30 may parse the bitstream to obtain syntax elements from the bitstream. Video decoder 30 may reconstruct a frame of video data based at least in part on the syntax elements obtained from the bitstream. The process of reconstructing video data is approximately the reverse of the encoding process performed by video encoder 20. For example, video decoder 30 may perform an inverse transform on coefficient blocks associated with TUs of the current CU to reconstruct residual blocks associated with TUs of the current CU. Video decoder 30 also reconstructs coding blocks of the current CU by adding samples of the predictive blocks of the PUs of the current CU to corresponding samples of the transform blocks of the TUs of the current CU. Video decoder 30 may reconstruct the frame after reconstructing the coding blocks for each CU of the frame.

[0067]

[0110] As mentioned above, video coding mainly uses two modes, intraframe prediction (or intra prediction) and interframe prediction (or inter prediction), to achieve video compression. It should be noted that IBC can be considered as intraframe prediction or a third mode. Of the two modes, interframe prediction contributes more to coding efficiency than intraframe prediction, since it uses motion vectors to predict a current video block from a reference video block.

[0068]

[0111] However, as the imaging technology of video data is constantly improving and the video block size is becoming finer to preserve the details of the video data, the amount of data required to represent the motion vector of the current frame is also increasing significantly. One way to overcome this challenge is to benefit from the fact that a group of adjacent CUs in both spatial and temporal domains not only have similar video data for prediction purposes, but also the motion vectors between these adjacent CUs are similar. Therefore, the motion information of spatially adjacent CUs and / or temporally co-located CUs can be used as an approximation of the motion information (e.g., motion vector) of the current CU by investigating their spatial and temporal correlations, which is also called the "Motion Vector Predictor (MVP)" of the current CU.

[0069]

[0112] Instead of encoding the actual motion vector of the current CU determined by motion estimator 42 into the video bitstream as described above in connection with Figure 2, the motion vector predictor of the current CU is subtracted from the actual motion vector of the current CU to create a Motion Vector Difference (MVD) for the current CU. By doing so, the motion vectors determined for each CU of a frame by motion estimator 42 do not need to be encoded into the video bitstream, and the amount of data used to represent motion information in the video bitstream may be significantly reduced.

[0070]

[0113] Similar to the process of choosing a prediction block in a reference frame during inter-frame prediction of a codeblock, a set of rules needs to be adopted by both the video encoder 20 and the video decoder 30 to construct a motion vector candidate list (also called a "merge list") for the current CU using potential candidate motion vectors associated with spatially neighboring CUs and / or temporally co-located CUs of the current CU, and then select one element from the motion vector candidate list as the motion vector predictor for the current CU. By doing so, the motion vector candidate list itself does not need to be transmitted from the video encoder 20 to the video decoder 30, and the index of the selected motion vector predictor in the motion vector candidate list is sufficient for the video encoder 20 and the video decoder 30 to use the same motion vector predictor in the motion vector candidate list to encode and decode the current CU.

[0071]

[0114] For each inter-predicted CU, the motion parameters consist of a motion vector, a reference picture index and a reference picture list usage index, as well as additional information required for the new coding features of VVC used for inter-prediction sample generation. The motion parameters may be signaled in an explicit or implicit manner. If a CU is coded in skip mode, it is associated with one PU and has no significant residual coefficients, no coded motion vector deltas, and no reference picture indexes. A merge mode is specified, whereby the motion parameters of the current CU, including spatial and temporal candidates and additional schedules introduced in VVC, are obtained from neighboring CUs. The merge mode may be applied to any inter-predicted CU, not just skip mode. An alternative to the merge mode is explicit transmission of motion parameters, where the motion vector, the corresponding reference picture index of each reference picture list, the reference picture list usage flag, and other required information are explicitly signaled for each CU.

[0072]

[0115] Additionally, ITU-T VCEG (Q6 / 16) and ISO / IEC MPEG (JTC1 / SC29 / WG11) are studying the potential need for standardization of future video coding technologies with compression capabilities significantly beyond those of the current VVC standard. Such future standardization actions could take the form of additional extensions to VVC or entirely new standards. These groups are working together on this exploration in a collaborative effort known as the Joint Video Exploration Team (JVET) to evaluate compression technology designs proposed by experts in the field. The first exploration experiment (EE) has been established at the JVET meeting on January 6-15, 2021, and this exploration software model has been named the Enhanced Compression Model (ECM), with ECM version 2 (ECM2) being published in August 2021. The newly developed inter-prediction scheme in ECM2 is described in detail in the next section.

[0073]

[0116] The following sections provide details of the VVC and its inter-prediction methodology as specified in the ECM model under development.

[0074]

[0117] Enhanced Merge Prediction in VVC

[0075]

[0118] In VVC, a merge candidate list is constructed by including, in order, the following five types of candidates:

[0119] 1) Spatial MVP from spatially adjacent CUs

[0120] 2) Temporal MVP from collocated CUs

[0121] 3) History-based MVP from FIFO tables

[0122] 4) Average MVP per pair

[0123] 5) Zero's music video

[0076]

[0124] The size of the merge list is signaled in the Sequence Parameter Set header, and the maximum allowed size of the merge list is 6. For each CU code in the merge mode, the index of the best merge candidate is encoded using truncated unary binarization (TU). The first bin of the merge index is encoded with the context, and bypass encoding is used for the other bins.

[0077]

[0125] In this section, the derivation process for each category of merge candidates is provided. As is done in HEVC, VVC also supports parallel derivation of merge candidate lists for all CUs within an area of ​​a certain size.

[0078]

[0126] Spatial Candidate Derivation in VVC

[0079]

[0127] The derivation of spatial merge candidates in VVC is the same as that in HEVC, except that the positions of the first two merge candidates are swapped. Up to four merge candidates are selected from the candidates at the positions shown in FIG. 5. The order of derivation is B0, A0, B1, A1, and B2. Position B2 is considered only if one or more CUs at positions B0, A0, B1, A1 are unavailable (e.g., because they belong to another slice or tile) or are intra-coded. After the candidate at position A1 is added, the addition of the remaining candidates is subject to a redundancy check that ensures that candidates with the same motion information are removed from the list so that the coding efficiency is improved. To reduce computational complexity, the aforementioned redundancy check does not consider all possible candidate pairs. Instead, only the pairs linked by arrows in FIG. 6 are considered, and a candidate is added to the list only if the corresponding candidates used for the redundancy check do not have the same motion information.

[0080]

[0128] Derivation of time candidates in VVC

[0081]

[0129] In this step, only one candidate is added to the list. In particular, in this derivation of the temporal merge candidate, a scaled motion vector is derived based on the co-located CU belonging to the co-located reference picture. The reference picture list and reference index used for derivation of the co-located CU are explicitly signaled in the slice header. The scaled motion vector of the temporal merge candidate is obtained as shown by the dotted line in FIG. 7, and this motion vector is scaled from the motion vector of the co-located CU using POC distances tb and td, where tb is defined to be the POC difference between the reference picture of the current picture and the current picture, and td is defined to be the POC difference between the reference picture of the co-located picture and the co-located picture. The reference picture index of the temporal merge candidate is set equal to zero.

[0082]

[0130] The location of the temporal candidate is selected between candidate C0 and candidate C1 as shown in Figure 8. If the CU in location C0 is unavailable, is intra-coded, or is outside the current row of the CTU, location C1 is used. Otherwise, location C0 is used to derive the temporal merge candidate.

[0083]

[0131] History-Based Merge Candidate Deriving in VVC

[0084]

[0132] History-based MVP (HMVP) merge candidates are added to the merge list after spatial MVP and TMVP. In this method, the motion information of a previously coded block is stored in a table and used as the MVP of the current CU. During the encoding / decoding process, a table with multiple HMVP candidates is maintained. When a new CTU row occurs, the table is reset (emptied). Whenever there is a non-subblock inter-coded CU, the associated motion information is added to the last entry of the table as a new HMVP candidate.

[0085]

[0133] The HMVP table size S is set to 6, which indicates that up to five history-based MVP (HMVP) candidates can be added to the table. When inserting a new motion candidate into the table, a constrained first-in-first-out (FIFO) rule is utilized, and a redundancy check is first applied to find whether there is an identical HMVP in the table. If found, the identical HMVP is removed from the table, all subsequent HMVP candidates are moved forward, and the identical HMVP is inserted into the last entry of the table.

[0086]

[0134] HMVP candidates may be used in the merge candidate list construction process. The most recent few HMVP candidates in the table are checked in order and inserted after the TMVP candidate in the candidate list. Redundancy checks are applied to spatial or temporal merge candidates in the HMVP candidates.

[0087]

[0135] To reduce the number of redundancy checking operations, the following simplifications are introduced.

[0088]

[0136] 1. The last two entries in the table are redundancy checked against the A1 and B1 space candidates, respectively.

[0089]

[0137] 2. When the total number of available merge candidates reaches the maximum allowed merge candidates minus one, the merge candidate list construction process from HMVP ends.

[0090]

[0138] Pairwise Average Merging Candidate Derivation in VVC

[0091]

[0139] The pairwise average candidate is generated by averaging a predefined candidate pair in an existing merge candidate list using the first two merge candidates. The first merge candidate may be defined as p0Cand and the second merge candidate may be defined as p1Cand, respectively. The averaged motion vector is calculated separately for each reference list depending on the availability of motion vectors in p0Cand and p1Cand. If both motion vectors are available in one list, these two motion vectors are averaged even if they point to different reference pictures, and the reference picture is set to one of p0Cand; if only one motion vector is available, it is used directly, and if no motion vector is available, this list is left invalid. Also, if the half (1 / 2) pel interpolation filter index of p0Cand and p1Cand is different, it is set to 0.

[0092]

[0140] If the merge list is not full after the pairwise average merge candidates are added, MVPs of zeros are inserted at the end until the maximum number of merge candidates is reached.

[0093]

[0141] High-precision (1 / 16-pel) motion compensation and storage of motion vectors in VVC.

[0094]

[0142] VVC increases the MV precision to 1 / 16 luma samples to improve prediction efficiency for slow-motion videos. This higher motion precision is especially useful for video content with locally varying non-translational motion, such as in the affine mode. For higher MV precision fractional position sample generation, the 8-tap luma and 4-tap chroma interpolation filters of HEVC are extended to 16 phases for luma and 32 phases for chroma. This extended filter set is applied to the MC process of inter-coded CUs except for affine mode CUs. In the affine mode, a set of 6-tap luma interpolation filters with 16 phases is used for reduced computational complexity and memory bandwidth savings.

[0095]

[0143] In VVC, the highest precision of explicitly signaled motion vectors for non-affine CUs is 1 / 4 luma sample. In some inter prediction modes, such as affine mode, motion vectors may be signaled at 1 / 16 luma sample precision. In all inter coded CUs with implicitly inferred MVs, the MVs are derived at 1 / 16 luma sample precision and motion compensation prediction is performed at 1 / 16 luma sample precision. For storage of the intra motion field, all motion vectors are stored at 1 / 16 luma sample precision.

[0096]

[0144] For the temporal motion field storage used by TMVP and SbTVMP, the motion field compression is performed at a granularity of 8x8 size, as opposed to the granularity of 16x16 size in HEVC.

[0097]

[0145] Merge Mode with MVD in VVC (MMVD)

[0098]

[0146] In addition to the merge mode where the implicitly derived motion information is directly used to generate the prediction samples of the current CU, a merge mode with motion vector difference (MMVD) is introduced in VVC. Immediately after sending the normal merge flag, an MMVD flag is signaled to specify whether the MMVD mode is used for the CU.

[0099]

[0147] In MMVD, after a merge candidate is selected, the merge candidate is further refined by the signaled MVD information. The further information includes a merge candidate flag, an index for specifying the magnitude of motion, and an index for indicating the direction of motion. In MMVD mode, one of the first two candidates in the merge list is selected to be used as the MV basis. The mmvd candidate flag is signaled to specify which one is used between the first and second merge candidates.

[0100]

[0148] The distance index specifies the information of the magnitude of the motion and indicates a predefined offset from the starting point. As shown in Figures 9A and 9B, the offset is added to either the horizontal or vertical component of the starting MV. Table 1 specifies the relationship between the distance index and the predefined offset.

[0101] [Table 1]

[0102]

[0150] The direction index represents the direction of the MVD relative to the starting point. The direction index can represent four directions shown in Table 2. Note that the meaning of the MVD code can change according to the information of the starting MV. If the starting MV is a unidirectionally predicted MV or a bidirectionally predicted MV and both lists point to the same side of the current picture (i.e., the POCs of the two references are both greater than the POC of the current picture or both less than the POC of the current picture), the code in Table 2 specifies the sign of the MV offset added to the starting MV. If the starting MV is a bidirectionally predicted MV and the two MVs point to different sides of the current picture (i.e., the POC of one reference is greater than the POC of the current picture and the POC of the other reference is less than the POC of the current picture), if the difference of the POC in list 0 is greater than the difference of the POC in list 1, the code in Table 2 specifies the sign of the MV offset added to the MV component of the starting MV of list 0, and the sign of the MV in list 1 has the opposite value. Otherwise, if the difference in POCs in List 1 is greater than the difference in POCs in List 0, the sign in Table 2 specifies the sign of the MV offset added to the MV component of the starting MV of List 1, and the sign of the MV in List 0 has the opposite value.

[0103]

[0151] The MVD is scaled according to the POC difference in each direction. If the POC difference in both lists is the same, no scaling is required. Otherwise, if the POC difference in list 0 is larger than the POC difference in list 1, the MVD of list 1 is scaled by defining the POC difference of L0 as td and the POC difference of L1 as tb as described in Figure 7. If the POC difference of L1 is larger than L0, the MVD of list 0 is also scaled in the same way. If the starting MV is unidirectionally predicted, the MVD is added to the available MV.

[0104] [Table 2]

[0105]

[0153] Symmetric MVD coding in VVC

[0106]

[0154] In VVC, in addition to the normal unidirectional and bidirectional prediction mode MVD signaling, a symmetric MVD mode for bidirectional prediction MVD signaling is applied. In the symmetric MVD mode, motion information including reference picture indexes of both list 0 and list 1 and the MVD of list 1 is derived without being signaled.

[0107]

[0155] The decoding process for the symmetric MVD mode is as follows.

[0108]

[0156] 1. At the slice level, the variables BiDirPredFlag, RefIdxSymL0, and RefIdxSymL1 are derived as follows:

[0157] - If mvd_l1_zero_flag is 1, then BiDirPredFlag is set equal to 0.

[0158] - Otherwise, if the closest reference picture in list 0 and the closest reference picture in list 1 form a forward-backward or backward-forward reference picture pair, BiDirPredFlag is set to 1 and the reference pictures in both list 0 and list 1 are short-term reference pictures. Otherwise, BiDirPredFlag is set to 0.

[0159] 2. At the CU level, if the CU is bidirectionally predictively coded and BiDirPredFlag is equal to 1, then a symmetric mode flag is explicitly signaled indicating whether symmetric mode is used or not.

[0109]

[0160] If the symmetric mode flag is true, only mvp_l0_flag, mvp_l1_flag, and MVD0 are explicitly signaled. The reference indexes of list 0 and list 1 are set equal to their respective reference picture pairs. MVD1 is set equal to (-MVD0). The final motion vector is shown in the following equation:

[0110]

number

[0111]

[0162] In the encoder, symmetric MVD motion estimation starts with initial MV estimation. The set of initial MV candidates consists of the MVs obtained from the unidirectional predictive search, the MVs obtained from the bidirectional predictive search, and the MVs from the AMVP list. The one with the lowest rate-distortion cost is selected as the initial MV for the symmetric MVD motion search.

[0112]

[0163] Affine motion compensation prediction in VVC

[0113]

[0164] In HEVC, only translational motion model is applied for motion compensation prediction (MCP). Meanwhile, in the real world, there are many kinds of motion, such as zoom, rotation, perspective motion, and other irregular motion. In VVC, block-based affine transformation motion compensation prediction is applied. As shown in Figure 10A and Figure 10B, the affine motion field of a block is described by the motion information of two control points (4 parameters) as shown in Figure 10A or three control points motion vectors (6 parameters) as shown in Figure 10B.

[0114]

[0165] In the case of a four-parameter affine motion model, the motion vector at a sample position (x, y) within a block is derived as follows:

[0115]

number

[0116]

[0167] In the case of a six-parameter affine motion model, the motion vector at a sample position (x, y) within a block is derived as follows:

[0117]

number

[0118]

[0169] Here, (mv 0x ,mv 0y ) is the motion vector of the upper left corner control point, and (mv 1x ,mv 1y ) is the motion vector of the control point in the upper right corner, and (mv 2x ,mv 2y ) is the motion vector of the bottom-left corner control point.

[0119]

[0170] To simplify the motion compensation prediction, block-based affine transformation prediction is applied. To derive the motion vector of each 4x4 luma sub-block, the motion vector of the center sample of each sub-block as shown in Figure 11 is calculated according to the above equation and rounded to 1 / 16 decimal precision. Then, a motion compensation interpolation filter is applied to generate a prediction of each sub-block using the derived motion vector. The sub-block size of the chroma component is also set to 4x4. The MV of a 4x4 chroma sub-block is calculated as the average of the MVs of the top-left and bottom-right luma sub-blocks in the collocated 8x8 luma region.

[0120]

[0171] As with translational motion inter prediction, there are also two affine motion inter prediction modes: affine merge mode and affine AMVP mode.

[0121]

[0172] Affine Merge Prediction

[0122]

[0173] The AF_MERGE mode may be applied to CUs whose width and height are both equal to or greater than 8. In this mode, the CPVM of the current CU is generated based on the motion information of spatially neighboring CUs. There may be up to five CPMVP candidates, and an index is signaled to indicate the CPMVP candidate to be used for the current CU. The following three types of CPVM candidates are used to form the affine merge candidate list:

[0174] - Inherited affine merge candidates extrapolated from the CPMVs of neighboring CUs

[0175] - Constructed affine merge candidate CPMVP derived using translation MVs of neighboring CUs

[0176] - Zero's MV

[0123]

[0177] In VVC, there are up to two inherited affine candidates, which are derived from the affine motion models of the neighboring blocks, one from the left neighboring CU and the other from the top neighboring CU. The candidate blocks are shown in Fig. 12. For the left predictor, the scanning order is A0->A1, and for the top predictor, the scanning order is B0->B1->B2. Only the first inherited candidate from each side is selected. No pruning check is performed between the two inherited candidates. Once a neighboring affine CU is identified, its control point motion vector is used to derive the CPMVP candidate in the affine merge list of the current CU. If the neighboring bottom-left block A is coded in affine mode, the motion vectors v_2, v_3, and v_4 for the top-left, top-right, and bottom-left corners of the CU containing block A are obtained. If block A is coded with a four-parameter affine model, the two CPMVs of the current CU are calculated according to v_2 and v_3. If block A is coded with a 6-parameter affine model, the three CPMVs of the current CU are calculated according to v_2, v_3, and v_4.

[0124]

[0178] Constructed affine candidates means that the candidates are constructed by combining the neighboring translational motion information of each control point. The motion information of the control points is derived from the designated spatial and temporal neighbors shown in FIG. 13. CPMVk (k=1, 2, 3, 4) represents the kth control point. For CPMV1, the block B2->B3->A2 is checked and the MV of the first available block is used. For CPMV2, the block B1->B0 is checked and for CPMV3, the block A1->A0 is checked. TMVP is used as CPMV4 if available.

[0125]

[0179] After the MVs of the four control points are obtained, an affine merge candidate is constructed based on their motion information. The following combinations of control point MVs are used for construction in order:

[0126]

[0180] {CPMV1, CPMV2, CPMV3}, {CPMV1, CPMV2, CPMV4}, {CPMV1, CPMV3, CPMV4}, {CPMV2, CPMV3, CPMV4}, {CPMV1, CPMV2}, {CPMV1, CPMV3}

[0127]

[0181] A combination of three CPMVs constitutes a 6-parameter affine merging candidate, and a combination of two CPMVs constitutes a 4-parameter affine merging candidate. To avoid the motion scaling process, if the reference indices of the control points are different, the associated control point MV combination is discarded.

[0128]

[0182] After inherited and constructed affine merge candidates are checked, if the list is not yet filled, a MV of zero is inserted at the end of the list.

[0129]

[0183] Affine AMVP Prediction in VVC

[0130]

[0184] Affine AMVP mode may be applied to CUs whose width and height are both 16 or greater. A CU-level affine flag is signaled in the bitstream to indicate whether affine AMVP mode is used, and then another flag is signaled to indicate whether it is 4-parameter affine or 6-parameter affine. In this mode, the difference between the CPMV of the current CU and its predictor CPMVP is signaled in the bitstream. The size of the affine AVMP candidate list is 2, and this candidate list is generated by using the following four types of CPVM candidates in order:

[0131]

[0185] - Inherited affine AMVP candidates extrapolated from the CPMVs of neighboring CUs

[0186] - A constructed affine AMVP candidate CPMVP derived using the translational MVs of neighboring CUs

[0187] - Translational MV from adjacent CU

[0188] - Zero's MV

[0132]

[0189] The checking order of inherited affine AMVP candidates is the same as the checking order of inherited affine merge candidates. The only difference is that for AVMP candidates, only affine CUs with the same reference picture as in the current block are considered. No pruning process is applied when inserting inherited affine motion predictors into the candidate list.

[0133]

[0190] The constructed AMVP candidates are derived from the specified spatial neighborhood shown in Figure 13. The same check order is used as done in the construction of affine merge candidates. In addition, the reference picture indexes of the neighboring blocks are also checked. The first block in the check order that is inter-coded and has the same reference picture as in the current CU is used. There is only one. If the current CU is coded in 4-parameter affine mode and both mv0 and mv1 are available, they are added as one candidate in the affine AMVP list. If the current CU is coded in 6-parameter affine mode and all three CPMVs are available, they are added as one candidate in the affine AMVP list. Otherwise, the constructed AMVP candidate is set as unavailable.

[0134]

[0191] After inserting valid inherited and constructed affine AMVP candidates, if the number of affine AMVP list candidates is still less than 2, then mv0, mv1, and mv2 are added in order as translation MVs to predict all control point MVs of the current CU, when available. Finally, if the affine AMVP list is not yet filled, MVs of zero are used to fill the affine AMVP list.

[0135]

[0192] Prediction refinement using optical flow of affine modes in VVC

[0136]

[0193] Subblock-based affine motion compensation can save memory access bandwidth and reduce computational complexity compared to pixel-based motion compensation, but it comes at the expense of prediction accuracy. To achieve finer granularity of motion compensation, prediction refinement with optical flow (PROF) is used to refine subblock-based affine motion compensation prediction without increasing memory access bandwidth for motion compensation. In VVC, after subblock-based affine motion compensation is performed, luma prediction samples are refined by adding the difference derived by the optical flow equation. PROF is explained in the following four steps:

[0137]

[0194] Step 1) Sub-block based affine motion compensation is performed to generate the sub-block prediction I(i,j).

[0138]

[0195] Step 2) Spatial gradients of subblock predictions g x (i,j) and g y (i,j) is computed at each sample position using a 3-tap filter [-1,0,1]. The gradient computation is exactly the same as the BDOF gradient computation.

[0139]

number

[0140]

[0198] shift1 is used to control the precision of the gradient. Sub-block (i.e., 4x4) prediction is extended by one sample on each side for gradient calculation. To avoid additional memory bandwidth and additional interpolation calculations, the extended samples on the extension boundary are copied from the nearest integer pixel location in the reference picture.

[0141]

[0199] Step 3) The brightness prediction refinement is derived using the following optical flow equation:

[0142]

number

[0143]

[0201] where Δv(i,j) is the difference between the sample MV calculated for sample position (i,j) denoted by v(i,j) and the sub-block MV of the sub-block to which sample (i,j) belongs, as shown in Figure 14. Δv(i,j) is quantized in units of 1 / 32 luma sample precision.

[0144]

[0202] Since the affine model parameters and the sample position relative to the center of the subblock do not change for each subblock, Δv(i,j) can be calculated for the first subblock and reused for other subblocks in the same CU. dx(i,j) and dy(i,j) are calculated from the sample position (i,j) to the subblock (x SB ,y SB ), Δv(x,y) can be derived as follows:

[0145]

number

[0146]

[0205] To maintain accuracy, we use subblocks (x SB ,y SB The input for ((W SB -1) / 2,(H SB -1) / 2), where W SB and H SB are the width and height of the sub-block, respectively.

[0147]

[0206] For the four-parameter affine model,

[0148]

number

[0149]

[0208] For the six-parameter affine model,

[0150]

number

[0151]

[0210] Here, (v 0x ,v 0y ), (v 1x ,v 1y ), (v 2x ,v 2y ) are the motion vectors of the top-left, top-right, and bottom-left control points, and w and h are the width and height of the CU.

[0152]

[0211] Step 4) Finally, the sub-block prediction I(i,j) is added with the luma prediction refinement ΔI(i,j). The final prediction I′ is generated by the following equation:

[0153]

number

[0154]

[0213] PROF is not applied in two cases for affine coded CUs: 1) all control points MV are the same, which indicates that the CU has only translational motion, and 2) the affine motion parameters are larger than the specified limit, since sub-block based affine MC is downgraded to CU based MC to avoid large memory access bandwidth requirements.

[0155]

[0214] In order to reduce the coding complexity of affine motion estimation using PROF, a fast coding method is applied. PROF is not applied in the affine motion estimation stage in the following two situations: a) if this CU is not a root block and its parent block does not select affine mode as its best mode, PROF is not applied because the current CU is unlikely to select affine mode as its best mode; b) if the magnitudes of the four affine parameters (C, D, E, F) are all smaller than a predefined threshold and the current picture is not a low-latency picture, PROF is not applied because the improvement brought by PROF in this case is small. In this way, affine motion estimation using PROF can be accelerated.

[0156]

[0215] Sub-block-based Temporal Motion Vector Prediction (SbTMVP) in VVC

[0157]

[0216] VVC supports the sub-block-based temporal motion vector prediction (SbTMVP) method. Similar to temporal motion vector prediction (TMVP) in HEVC, SbTMVP uses motion fields in co-located pictures to improve the motion vector prediction and merge mode of CUs in the current picture. The same co-located pictures used by TMVP are also used for SbTVMP. SbTMVP differs from TMVP in two main ways:

[0158]

[0217] - TMVP predicts motion at the CU level, while SbTMVP predicts motion at the sub-CU level.

[0218] - Where TMVP obtains temporal motion vectors from a co-located block in a co-located picture (the co-located block is the bottom-right or center block with respect to the current CU), SbTMVP applies a motion shift amount before obtaining temporal motion information from the co-located picture, where the motion shift amount is obtained from a motion vector from one of the spatially neighboring blocks of the current CU.

[0159]

[0219] The SbTVMP process is shown in Figure 15A and Figure 15B. SbTMVP predicts the motion vectors of sub-CUs in the current CU in two steps. In the first step, the spatial neighbor A1 in Figure 15A is examined. If A1 has a motion vector that uses a co-located picture as a reference picture, this motion vector is selected as the motion shift amount to be applied. If no such motion is identified, the motion shift amount is set to (0,0).

[0160]

[0220] In the second step, the motion shift amount identified in step 1 is applied (i.e., added to the coordinates of the current block) to obtain sub-CU level motion information (motion vector and reference index) from the co-located picture, as shown in FIG. 15B. The example of FIG. 15B assumes that the motion shift amount is set to the motion of block A1. Then, for each sub-CU, the motion information of its corresponding block in the co-located picture (the smallest motion grid covering the center sample) is used to derive the motion information of the sub-CU. After the motion information of the co-located sub-CU is identified, this motion information is converted to the motion vector and reference index of the current sub-CU in a manner similar to the TMVP process of HEVC, and temporal motion scaling is applied to align the reference picture of the temporal motion vector to the picture of the current CU.

[0161]

[0221] In VVC, a combined subblock-based merge list containing both SbTVMP and affine merge candidates is used to signal the subblock-based merge mode. The SbTVMP mode is enabled / disabled by a Sequence Parameter Set (SPS) flag. If the SbTMVP mode is enabled, the SbTMVP predictor is added as the first entry in the list of subblock-based merge candidates, followed by the affine merge candidates. The size of the subblock-based merge list is signaled in the SPS, and the maximum allowed size of the subblock-based merge list is 5 in VVC.

[0162]

[0222] The sub-CU size used in SbTMVP is fixed to 8x8, and similar to the affine merge mode, the SbTMVP mode only applies to CUs whose width and height are both 8 or greater.

[0163]

[0223] The encoding logic of the additional SbTMVP merge candidate is the same as other merge candidates, i.e., for each CU of a P slice or a B slice, an additional RD check is performed to determine whether to use the SbTMVP candidate.

[0164]

[0224] Adaptive Motion Vector Resolution (AMVR) for VVC

[0165]

[0225] In HEVC, when use_integer_mv_flag in the slice header is equal to 0, the motion vector differential (MVD) (between the motion vector of the CU and the predicted motion vector) is signaled in units of quarter-luminance samples. In VVC, a CU-level adaptive motion vector resolution (AMVR) scheme is introduced. AMVR allows the MVD of a CU to be coded with different precisions. Depending on the mode of the current CU (normal AMVP mode or affine AVMP mode), the MVD of the current CU may be adaptively selected as follows:

[0166]

[0226] - Normal AMVP mode: quarter luma samples, half luma samples, integer luma samples, or 4 luma samples

[0227] - Affine AMVP modes: quarter luma samples, integer luma samples, or 1 / 16 luma samples

[0167]

[0228] If the current CU has at least one non-zero MVD component, a CU-level MVD resolution indication is conditionally signaled. If all MVD components (i.e., both horizontal and vertical MVDs in reference list L0 and reference list L1) are zero, a quarter-luminance sample MVD resolution is inferred.

[0168]

[0229] For a CU with at least one non-zero MVD component, a first flag is signaled to indicate whether quarter luma sample MVD precision is used for that CU. If the first flag is 0, no further signaling is required and quarter luma sample MVD precision is used for the current CU. Otherwise, a second flag is signaled to indicate that half luma sample or other MVD precision (integer or 4 luma sample) is used for the regular AMVP CU. For half luma samples, a 6-tap interpolation filter is used instead of the default 8-tap interpolation filter for the half luma sample positions. Otherwise, a third flag is signaled to indicate whether integer luma sample or 4 luma sample MVD precision is used for the regular AMVP CU. For affine AMVP CUs, the second flag is used to indicate whether integer luma sample or 1 / 16 luma sample MVD precision is used. To ensure that the reconstructed MV has the intended precision (quarter luma samples, half luma samples, integer luma samples, or 4 luma samples), the motion vector predictor of a CU may be rounded to the same precision as that of the MVD before being added with the MVD. The motion vector predictor is rounded towards zero (i.e., negative motion vector predictors are rounded towards positive infinity and positive motion vector predictors are rounded towards negative infinity).

[0169]

[0230] The encoder uses the RD check to determine the motion vector resolution of the current CU. To avoid always performing CU-level RD checks four times for each MVD resolution, in VTM14, the RD checks of MVD precisions other than quarter luma samples are only conditionally invoked. For normal AVMP mode, the RD costs of quarter luma samples MVD precision and integer luma samples MV precision are calculated first. Then, the RD cost of integer luma samples MVD precision is compared with the RD cost of quarter luma samples MVD precision to determine whether the RD cost of the four luma samples MVD precision needs to be further checked. If the RD cost of quarter luma samples MVD precision is much smaller than the RD cost of integer luma samples MVD precision, the RD check of the four luma samples MVD precision is skipped. Then, if the RD cost of integer luma samples MVD precision is significantly larger than the best RD cost of the previously tested MVD precision, the check of half luma samples MVD precision is skipped. For affine AMVP mode, if affine inter mode is not selected after checking the rate-distortion costs of affine merge / skip mode, merge / skip mode, normal AMVP mode with quarter luma sample MVD precision, and affine AMVP mode with quarter luma sample MVD precision, affine inter mode with 1 / 16 luma sample MV precision and 1 pel MV precision are not checked. Additionally, the affine parameters obtained in affine inter mode with quarter luma sample MV precision are used as the starting search point for affine inter mode with 1 / 16 luma sample and quarter luma sample MV precision.

[0170]

[0231] Bi-prediction with CU-level weight (BCW) in VVC

[0171]

[0232] In HEVC, a bi-predictive signal is generated by averaging two prediction signals obtained from two different reference pictures and / or by using two different motion vectors. In VVC, the bi-predictive mode is extended beyond the scope of a simple average to allow a weighted average of the two prediction signals.

[0172]

number

[0173]

[0234] Weighted average bi-prediction allows five weights w∈{-2,3,4,5,10}. For each bi-predicted CU, the weight w is determined in one of two ways: 1) for non-merged CUs, the weight index is signaled after the motion vector differential; 2) for merged CUs, the weight index is inferred from the neighboring blocks based on the merge candidate index. BCW is applied only to CUs with 256 or more luma samples (i.e., CU width × CU height is 256 or more). For low latency pictures, all five weights are used. For non-low latency pictures, only three weights (w∈{3,4,5}) are used.

[0174]

[0235] - In the encoder, fast search algorithms are applied to find the weight indexes without significantly increasing the encoder complexity. These algorithms are summarized as follows. For more details, see VTM software and document JVET-L0646. When combined with AMVR, for 1-pel and 4-pel motion vector precision, unequal weights are only conditionally checked if the current picture is a low-latency picture.

[0236] - When combined with affine, affine ME may be performed for unequal weights if and only if the affine mode is selected as the current best mode.

[0237] - When the two reference pictures of bi-prediction are the same, unequal weights are only conditionally checked.

[0238] - Unequal weights are not explored if certain conditions are met, depending on the POC distance between the current picture and its reference pictures, the coding QP, and the temporal level.

[0175]

[0239] The BCW weight index is coded using one context coding bin followed by a bypass coding bin: the first context coding bin indicates whether equal weights are used, and if unequal weights are used, an additional bin is signaled using bypass coding to indicate which unequal weights are used.

[0176]

[0240] Weighted prediction (WP) is a coding tool for efficiently encoding video content with fading, supported by H.264 / AVC and HEVC standards. Support for WP was also added to the VVC standard. WP allows for signaling weighting parameters (weight and offset) for each reference picture in each of the reference picture lists L0 and L1. Then, during motion compensation, the weight and offset of the corresponding reference picture are applied. WP and BCW are designed for various types of video content. To avoid interactions between WP and BCW that may complicate VVC decoder design, when a CU uses WP, the BCW weight index is not signaled and w is inferred to be 4 (i.e., equal weights are applied). For merge CUs, the weight index is inferred from neighboring blocks based on the merge candidate index. This can be applied to both normal merge mode and inherited affine merge mode. For configured affine merge mode, the affine motion information is configured based on the motion information of up to three blocks. The BCW index of a CU using the configured affine merge mode is simply set equal to the BCW index of the first control point MV.

[0177]

[0241] In VVC, CIIP and BCW cannot be applied jointly to a CU. If a CU is coded in CIIP mode, the BCW index of the current CU is set to 2, i.e., equal weight.

[0178]

[0242] Bi-directional optical flow (BDOF) in VVC

[0179]

[0243] VVC includes a bidirectional optical flow (BDOF) tool. BDOF was previously called BIO and was included in JEM. Compared to the JEM version, BDOF in VVC is a simpler version and requires significantly less computation, especially in terms of the number of multiplications and the size of the multipliers.

[0180]

[0244] BDOF is used to refine the bidirectional prediction signal of a CU at a 4×4 sub-block level. BDOF is applied to a CU if the CU satisfies all of the following conditions:

[0181]

[0245] - The CU is coded using "true" bidirectional prediction mode, i.e., one of the two reference pictures is before the current picture in display order and the other is after the current picture in display order.

[0246] - The distance from the two reference pictures to the current picture (i.e., the POC difference) is the same.

[0247] - Both reference pictures are short-term reference pictures.

[0248] - The CU is not coded using affine mode or SbTMVP merge mode.

[0249] - The CU has more than 64 luma samples.

[0250] - Both the CU height and CU width are 8 or more luma samples.

[0251] - The BCW weight index indicates equal weighting.

[0252] - The WP is not valid for the current CU.

[0253] - CIIP mode is not used for the current CU.

[0182]

[0254] BDOF is applied only to the luma component. As the name suggests, the BDOF mode is based on the concept of optical flow, which assumes that the object motion is smooth. For each 4 × 4 subblock, the motion refinement (v x ,v y ) is calculated. The motion refinement is then used to adjust the bidirectionally predicted sample values ​​in the 4x4 sub-block. The following steps are applied in the BDOF process:

[0183]

[0255] First, the horizontal and vertical gradients of the two prediction signals

number

[0184]

number

[0185]

[0258] Here, I (k) (i,j) is the sample value at coordinate (i,j) of the predicted signal in list k (k=0,1), and shift1 is calculated based on the luma bit depth bitDepth as shift1=max(6,bitDepth-6).

[0186]

[0259] The autocorrelation and cross-correlation of the gradients S1, S2, S3, S5, and S6 are then calculated as follows:

[0187]

number

[0188]

[0260] Here,

number

[0189]

[0261] where Ω is a 6×6 window around the 4×4 subblock, with values ​​n a and value n b are set equal to min(1, bitDepth-11) and min(4, bitDepth-8), respectively.

[0190]

[0262] The motion refinement measure (v x ,v y ) is derived.

[0191]

number

[0192]

[0263] Where:

number

number

number

[0193]

[0264] Based on the motion refinement and gradient, the following adjustments are calculated for each sample in the 4x4 sub-block:

[0194]

number

[0195]

[0265] Finally, the BDOF samples of the CU are calculated by adjusting the bi-directional prediction samples as follows:

[0196]

number

[0197]

[0266] These values ​​are chosen such that the multipliers of the BDOF process do not exceed 15 bits and the maximum bit width of the intermediate parameters of the BDOF process is kept within 32 bits.

[0198]

[0267] To derive the gradient value, we first select some prediction samples I in list k (k=0, 1) outside the current CU boundary. (k) (i,j) need to be generated. As shown in Figure 16, BDOF in VVC uses one extended row / column around the boundary of the CU. To control the computational complexity of generating predicted samples outside the boundary, predicted samples in the extended area (white positions) are generated by directly taking reference samples at nearby integer positions (using floor() operation on the coordinates) without interpolation, and a normal 8-tap motion compensation interpolation filter is used to generate predicted samples within the CU (gray positions). These extended sample values ​​are only used in gradient calculation. In the remaining steps of the BDOF process, if sample and gradient values ​​outside the CU boundary are needed, they are padded (i.e., repeated) from their nearest neighbors.

[0199]

[0268] If the width and / or height of a CU is greater than 16 luma samples, the CU may be divided into sub-blocks with width and / or height equal to 16 luma samples, and the sub-block boundaries are treated as CU boundaries in the BDOF process. The maximum unit size of the BDOF process is limited to 16x16. For each sub-block, the BDOF process may be skipped. If the SAD between the initial L0 predicted sample and the L1 predicted sample is less than a threshold, the BDOF process is not applied to the sub-block. The threshold is set to be equal to (8*W*(H>>1), where W denotes the width of the sub-block and H denotes the height of the sub-block. To avoid further complexity of the SAD calculation, the SAD between the initial L0 predicted sample and the L1 predicted sample calculated in the DVMR process is reused here.

[0200]

[0269] If BCW is enabled for the current block, i.e., the BCW weight index indicates unequal weights, bidirectional optical flow is disabled. Similarly, if WP is enabled for the current block, i.e., luma_weight_lx_flag is 1 for either of the two reference pictures, BDOF is also disabled. If the CU is coded in symmetric MVD or CIIP mode, BDOF is also disabled.

[0201]

[0270] Decoder side motion vector refinement (DMVR) in VVC

[0202]

[0271] In order to improve the accuracy of MV in merge mode, bilateral matching (BM) based decoder-side motion vector refinement is applied in VVC. In bidirectional prediction operation, refined MVs around the initial MV in reference picture list L0 and reference picture list L1 are searched. The BM method calculates the distortion between two candidate blocks in reference picture list L0 and list L1. As shown in Figure 17, the SAD between red blocks based on each MV candidate around the initial MV is calculated. The MV candidate with the lowest SAD becomes the refined MV and is used to generate the bidirectional prediction signal.

[0203]

[0272] In VVC, the application of DMVR is restricted and applies only to CUs coded with the following modes and features:

[0204]

[0273] - CU level merge mode with bi-predictive MV

[0274] - Relative to the current picture, one reference picture is in the past and the other reference picture is in the future.

[0275] - The distance from the two reference pictures to the current picture (i.e., the POC difference) is the same.

[0276] - Both reference pictures are short-term reference pictures.

[0277] - The CU has more than 64 luma samples.

[0278] - Both the CU height and CU width are 8 or more luma samples.

[0279] - The BCW weight index indicates equal weighting.

[0280] - WP is not enabled for the current block.

[0281] - CIIP mode is not used for the current block.

[0205]

[0282] The refined MVs derived by the DMVR process are used to generate inter prediction samples and are also used for temporal motion vector prediction for future picture encoding, while the original MVs are used in the deblocking process and are also used for spatial motion vector prediction for future CU encoding.

[0206]

[0283] The next section describes additional features of DMVR.

[0207]

[0284] Search method

[0208]

[0285] In DVMR, the search points are around the initial MV, and the MV offset follows the mirroring rule of MV difference. In other words, any point checked by DMVR, indicated by a candidate MV pair (MV0,MV1), follows the following two equations:

[0209]

number

[0210]

[0288] Here, MV_offset represents the refinement offset between the initial MV and the refined MV in one of the reference pictures. The refined search range is two integer luma samples from the initial MV. The search includes an integer sample offset search stage and a fractional sample refinement stage.

[0211]

[0289] A 25-point full search is applied for the integer sample offset search. First, the SAD of the initial MV pair is calculated. If the SAD of the initial MV pair is smaller than a threshold, the integer sample stage of DMVR is terminated. Otherwise, the SAD of the remaining 24 points is calculated and checked in raster scan order. The point with the smallest SAD is selected as the output of the integer sample offset search stage. To mitigate the penalty due to uncertainty in DMVR refinement, it is proposed to prioritize the original MV during the DMVR process. The SAD between the reference blocks referenced by the initial MV candidates is reduced by ¼ of the SAD value.

[0212]

[0290] The integer sample search is followed by fractional sample refinement. To reduce computational complexity, the fractional sample refinement is derived by using a parametric error surface equation instead of an additional search with SAD comparison. The fractional sample refinement is conditionally invoked based on the output of the integer sample search stage. When the integer sample search stage ends with the center with the smallest SAD in either the first iteration search or the second iteration search, the fractional sample refinement is further applied.

[0213]

[0291] In the parametric error surface based sub-pixel offset estimation, the cost of the central location and the costs of the four centrally adjacent locations are used to fit a 2D parabolic error surface equation of the form:

[0214]

number

[0215]

[0293] Here, (x min ,y min ) corresponds to the least costly fractional position, and C corresponds to the minimum cost value. By solving the above equation using the cost values ​​of the five search points, (x min ,y min ) is calculated as follows:

[0216]

number

[0217]

[0296] Since all cost values ​​are positive and the smallest value is E(0,0), x min and y min The value of is automatically constrained to be between -8 and 8. This corresponds to a half (1 / 2) pel offset with 1 / 16 pel MV precision in VVC. The calculated fraction (xmin ,y min ) is added to the integer distance refinement MV to obtain the sub-pixel accurate refined delta MV.

[0218]

[0297] Bilinear Interpolation and Sample Padding

[0219]

[0298] In VVC, the resolution of the MV is 1 / 16 luma sample. The samples at fractional positions are interpolated using an 8-tap interpolation filter. In DMVR, the search points surround the initial fractional pel MV with integer sample offsets, so the samples at these fractional positions need to be interpolated in the DMVR search process. To reduce the computational complexity, a bilinear interpolation filter is used to generate fractional samples for the search process in DMVR. Another important effect is that by using a bilinear filter, the search range of 2 samples means that DVMR does not access more reference samples compared to the normal motion compensation process. After the refined MV is obtained in the DMVR search process, a normal 8-tap interpolation filter is applied to generate the final prediction. To avoid accessing more reference samples for the normal MC process, samples that are not needed for the interpolation process based on the original MV but are needed for the interpolation process based on the refined MV are padded from the available samples.

[0220]

[0299] Geometric partitioning mode (GPM) in VVC

[0221]

[0300] VVC supports geometric partitioning mode for inter prediction. The geometric partitioning mode is signaled using a CU-level flag as a type of merge mode, and other merge modes include normal merge mode, MMVD mode, CIIP mode, and sub-block merge mode. The geometric partitioning mode allows for all possible CU sizes w×h=2, except for 8×64 and 64×8. m ×2 n, for each m, n∈{3···6}, a total of 64 partitions are supported.

[0222]

[0301] When this mode is used, a CU is divided into two parts by a geometrically positioned straight line (Figure 18). The position of the division line is mathematically derived from the angle and offset parameters of the particular division. Each part of the geometric division within a CU is inter-predicted using its own motion, and only uni-prediction is allowed for each division, i.e., each part has one motion vector and one reference index. As with conventional bi-prediction, a uni-prediction motion constraint is applied to ensure that only two motion-compensated predictions are required for each CU. The uni-prediction motion for each division is derived using the process described in 3.4.11.1.

[0223]

[0302] If a geometric partition mode is used for the current CU, a geometric partition index indicating the partition mode (angle and offset) of the geometric partition and two merge indices (one for each partition part) are further signaled. The number of maximum GPM candidate sizes is explicitly signaled in the SPS, which specifies the syntax binarization of the GPM merge index. After predicting each part of the geometric partition, the sample values ​​along the edges of the geometric partition are adjusted using a blending process with adaptive weights, as in 3.4.11.2. This is the prediction signal for the entire CU, and similar to other prediction modes, the transformation and quantization process may be applied to the entire CU. Finally, the motion field of the CU predicted using the geometric partition mode is stored, as in 3.4.11.3.

[0224]

[0303] One-way prediction candidate list configuration

[0225]

[0304] The unidirectional prediction candidate list is directly derived from the merge candidate list constructed according to the extended merge prediction process in 3.4.1. Let n be the index of the unidirectional prediction motion in the geometric unidirectional prediction candidate list. The LX motion vector (X is equal to the parity of n) of the nth extended merge candidate is used as the nth unidirectional prediction motion vector for the geometric partition mode. In FIG. 19, these motion vectors are marked with "x". If the corresponding LX motion vector of the nth extended merge candidate does not exist, the L(1-X) motion vector of the same candidate is used instead as the unidirectional prediction motion vector for the geometric partition mode.

[0226]

[0305] Combined inter and intra prediction (CIIP) in VVC

[0227]

[0306] In VVC, when a CU is coded in merge mode, if the CU contains at least 64 luma samples (i.e., CU width × CU height is 64 or more), and if both the CU width and CU height are less than 128 luma samples, an additional flag is signaled to indicate whether a combined inter / intra prediction (CIIP) mode is applied to the current CU. As the name suggests, CIIP prediction combines an inter prediction signal and an intra prediction signal. The inter prediction signal P of the CIIP mode inter is derived using the same inter prediction process as applied in the normal merge mode, resulting in an intra prediction signal P intra is derived according to the normal intra prediction process with planar mode. The intra prediction signal and the inter prediction signal are then combined using a weighted average, with the weight value calculated according to the coding modes of the upper and left neighboring blocks (shown in FIG. 20) as follows:

[0228]

[0307] - If the top neighbor is available and is intra-coded, set isIntraTop to 1, otherwise set isIntraTop to 0.

[0308] - If the left neighbor is available and is intra-coded, set isIntraLeft to 1, otherwise set isIntraLeft to 0.

[0309] - If (isIntraLeft+isIntraTop) is equal to 2, then wt is set to 3.

[0310] - Otherwise, if (isIntraLeft+isIntraTop) is equal to 1, then wt is set to 2.

[0311] - Else, set wt to 1.

[0229]

[0312] The CIIP forecast is formed as follows:

[0230]

number

[0231]

[0313] Intra Block Copy (IBC) in VVC

[0232]

[0314] Intra Block Copy (IBC) is a tool adopted in the HEVC extension on SCC. It is well known that IBC significantly improves the coding efficiency of screen content material. Since IBC mode is implemented as a block-level coding mode, block matching (BM) is performed in the encoder to find the optimal block vector (or motion vector) for each CU. Here, the block vector is used to indicate the displacement from the current block to a reference block already reconstructed in the current picture. The luma block vectors of IBC coded CUs are integer precision. The chroma block vectors are also rounded to integer precision. When combined with AMVR, IBC mode can switch between 1-pel and 4-pel motion vector precision. IBC coded CUs are treated as a third prediction mode other than intra or inter prediction modes. IBC mode is applicable to CUs whose width and height are both 64 luma samples or less.

[0233]

[0315] On the encoder side, hash-based motion estimation is performed for IBC. The encoder performs an RD check for blocks whose width or height is less than or equal to 16 luma samples. For non-merge modes, a block vector search is first performed using a hash-based search. If the hash search does not return a valid candidate, a block matching-based local search is performed.

[0234]

[0316] In hash-based search, hash key matching (32-bit CRC) between the current block and the reference block is extended to all allowed block sizes. The hash key calculation for all positions in the current picture is based on 4x4 sub-blocks. For larger sized current blocks, the hash key is determined to match the hash key of the reference block when all the hash keys of all 4x4 sub-blocks match the hash keys of the corresponding reference positions. If the hash keys of multiple reference blocks are found to match the hash key of the current block, the block vector cost of each matched reference is calculated and the one with the minimum cost is selected.

[0235]

[0317] In the block matching search, the search range is set to cover both the previous CTU and the current CTU. At the CU level, the IBC mode is signaled by a flag, which is signaled as IBC·AMVP mode or IBC skip / merge mode as follows:

[0236]

[0318] IBC skip / merge mode: The merge candidate index is used to indicate which block vector in the list from neighboring candidate IBC coded blocks is used to predict the current block. The merge list consists of spatial candidates, HMVP candidates, and pairwise candidates.

[0237]

[0319] IBC AMVP mode: Block vector differentials are coded in the same way as motion vector differentials. The block vector prediction method uses two candidates as predictors: one from the left neighbor and one from the top neighbor (for IBC coding). If neither neighbor is available, a default block vector is used as predictor. A flag is signaled to indicate the block vector predictor index.

[0238]

[0320] IBC reference area

[0239]

[0321] To reduce memory consumption and decoder complexity, IBC in VVC only allows reconstructing parts of a predefined area, including the area of ​​the current CTU and some area of ​​the left CTU. Depending on the position of the current coding CU within the current CTU, the following applies:

[0240]

[0322] If the current block falls in the top-left 64x64 block of the current CTU, in addition to the already reconstructed samples in the current CTU, the current block can also refer to reference samples in the bottom-right 64x64 block of the left CTU using CPR mode.The current block can also refer to reference samples in the bottom-left 64x64 block of the left CTU and reference samples in the top-right 64x64 block of the left CTU using CPR mode.

[0241]

[0323] If the current block falls in the top-right 64x64 block of the current CTU, when the luma position (0,64) for the current CTU has not yet been reconstructed, the current block can also refer to reference samples in the bottom-left 64x64 block and bottom-right 64x64 block of the left CTU using the CPR mode, in addition to the already reconstructed samples in the current CTU. Otherwise, the current block can also refer to reference samples in the bottom-right 64x64 block of the left CTU.

[0242]

[0324] If the current block falls in the bottom-left 64x64 block of the current CTU, and the luma position (64,0) for the current CTU has not yet been reconstructed, the current block can also refer to reference samples in the top-right 64x64 block and bottom-right 64x64 block of the left CTU using the CPR mode, in addition to the already reconstructed samples in the current CTU. Otherwise, the current block can also refer to reference samples in the bottom-right 64x64 block of the left CTU using the CPR mode.

[0243]

[0325] If the current block falls in the bottom-right 64x64 block of the current CTU, the CPR mode can be used to refer only to samples already reconstructed within the current CTU.

[0244]

[0326] This restriction allows the hardware implementation to implement the IBC mode using local on-chip memory.

[0245]

[0327] Local illumination compensation (LIC) in ECM

[0246]

[0328] LIC is an inter-prediction technique for modeling the local illumination variation between a current block and its predicted block as a function of the local illumination variation between the current block template and the reference block template. The parameters of the function can be represented by a scale α and an offset β, forming a linear equation, i.e., α*p[x]+β, for compensating for illumination changes. Here, p[x] is the reference sample pointed to by the MV at position x on the reference picture. Since α and β can be derived based on the current block template and the reference block template, no signaling overhead is required for α and β, except that a LIC flag is signaled for the AMVP mode to indicate the use of LIC.

[0247]

[0329] Local illumination compensation proposed in JVET-O0066 is used for unidirectionally predicted inter CUs with the following modifications:

[0248]

[0330] - Intra adjacent samples may be used in the LIC parameter derivation.

[0331] - LIC is disabled for blocks with less than 32 luma samples.

[0332] - For both non-subblock and affine modes, the LIC parameter derivation is performed based on the template block samples corresponding to the current CU instead of the partial template block samples corresponding to the first top-left 16x16 unit.

[0333] - The samples of the reference block template are generated by using MC together with block MV without rounding to integer pel precision.

[0249]

[0334] Non-adjacent spatial candidates in ECM

[0250]

[0335] Non-adjacent spatial merge candidates as in JVET-L0399 are inserted after TMVP in the regular merge candidate list. The spatial merge candidate pattern is shown in Figure 21. The distance between the non-adjacent spatial candidate and the current coding block is based on the width and height of the current coding block. No line buffer restrictions are applied.

[0251]

[0336] Template Matching (TM) in ECM

[0252]

[0337] Template matching (TM) is a decoder-side MV derivation method to refine the motion information of the current CU by finding the closest match between a template in the current picture (i.e., a block adjacent to the top and / or left of the current CU) and a block in a reference picture (i.e., the same size as the template). As shown in Figure 22, a better MV is searched for within a search range of [-8, +8] pels centered on the initial motion of the current CU. The template matching method in JVET-J0021 is used with the following modifications: the size of the search step is determined based on the AMVR mode, and TM can be cascaded with the bilateral matching process in merge mode.

[0253]

[0338] In AMVP mode, the MVP candidate is determined based on the template matching error to select the MVP candidate that achieves the minimum error between the current block template and the reference block template, and then TM is performed only on this particular MVP candidate for MV refinement. TM refines this MVP candidate starting with full-pel MVD accuracy (or 4-pel for 4-pel AMVR mode) in the search range of [-8,+8]-pel using an iterative diamond search. The AMVP candidate may be further refined by using a cross search of full-pel MVD accuracy (or 4-pel for 4-pel AMVR mode) followed by half-pel and quarter-pel MVD accuracy depending on the AMVR mode, as specified in Table 3. This search process ensures that the MVP candidate maintains the same MV accuracy as exhibited in AMVR mode after the TM process.

[0254] [Table 3]

[0255]

[0340] In the merge mode, a similar search method is applied to the merge candidates indicated by the merge index. As Table 3 shows, TM may run up to 1 / 8-pel MVD accuracy or skip accuracy beyond half-pel MVD accuracy depending on whether an alternative interpolation filter (used when AMVR is in half-pel mode) is used according to the merged motion information. Furthermore, when the TM mode is enabled, template matching may function as an independent process or an additional MV refinement process between the block-based bilateral matching (BM) method and the sub-block-based BM method depending on whether BM can be enabled according to the enablement condition check.

[0256]

[0341] Multi-pass decoder-side motion vector refinement in ECM

[0257]

[0342] Multi-pass decoder-side motion vector refinement is applied. In the first pass, bilateral matching (BM) is applied to the coding block. In the second pass, BM is applied to each 16x16 sub-block in the coding block. In the third pass, the MV in each 8x8 sub-block is refined by applying bidirectional optical flow (BDOF). The refined MV is stored for both spatial and temporal motion vector prediction.

[0258]

[0343] First Pass - Refining Block-Based Bilateral Matching MV

[0259]

[0344] In the first pass, refined MVs are derived by applying BM to the coding block. Similar to decoder-side motion vector refinement (DMVR), in bidirectional prediction operation, refined MVs are searched around two initial MVs (MV0 and MV1) in reference picture lists L0 and L1. Based on the minimum bilateral matching cost between two reference blocks in L0 and L1, refined MVs (MV0_pass1 and MV1_pass1) are derived around the initial MV.

[0260]

[0345] BM performs a local search to derive the integer sample precision intDeltaMV. The local search applies a 3x3 square search pattern to loop through the horizontal search range [-sHor,sHor] and the vertical search range [-sVer,sVer], where the values ​​of sHor and sVer are determined by the block dimension and the maximum value of sHor and sVer is 8.

[0261]

[0346] The bilateral matching cost is calculated as bilCost=mvDistanceCost+sadCost. When the block size cbW*cbH is larger than 64, the MRSAD cost function is applied to remove the DC effect of distortion between reference blocks. The intDeltaMV local search ends when bilCost at the center point of the 3x3 search pattern has the minimum cost. Otherwise, the current minimum cost search point becomes the new center point of the 3x3 search pattern and continues searching for the minimum cost until the end of the search range is reached.

[0262]

[0347] Further refinement of the existing fractional samples is applied to derive the final deltaMV. The refined MV after the first pass is then derived as follows:

[0263]

number

[0264]

[0350] Second Pass - Refining Subblock-Based Bilateral Matching MV

[0265]

[0351] In the second pass, refined MVs are derived by applying BM to the 16x16 grid of sub-blocks. For each sub-block, refined MVs are searched around the two MVs obtained in the first pass (MV0_pass1 and MV1_pass1) in the reference picture lists L0 and L1. Refined MVs (MV0_pass2(sbIdx2) and MV1_pass2(sbIdx2)) are derived based on the minimum bilateral matching cost between the two reference sub-blocks in L0 and L1.

[0266]

[0352] For each subblock, the BM performs a full search to derive the integer sample precision intDeltaMV. The full search has a horizontal search range of [-sHor,sHor] and a vertical search range of [-sVer,sVer], where the values ​​of sHor and sVer are determined by the block dimension, and the maximum value of sHor and sVer is 8.

[0267]

[0353] The bilateral matching cost is calculated by applying a cost factor to the SATD cost between two reference subblocks as bilCost=satdCost*costFactor. The search area (2*sHor+1)*(2*sVer+1) is divided into up to five diamond-shaped search regions as shown in Figure 23. Each search region is assigned a costFactor determined by the distance (intDeltaMV) between each search point and the starting MV, and each diamond region is processed sequentially starting from the center of the search area. In each region, the search points are processed in raster scan order starting from the top-left corner of the region toward the bottom-right corner. If the minimum bilCost in the current search region is less than a threshold equal to sbW*sbH, the integer pel exhaustive search ends, otherwise the integer pel exhaustive search proceeds to the next search region until all search points are examined.

[0268]

[0354] To derive the final deltaMV(sbIdx2), further refinement of the existing VVC·DMVR fractional samples is applied. The refined MV in the second pass is derived as follows:

[0269]

number

[0270]

[0357] Third Pass-Subblock Based Bidirectional Optical Flow MV Refinement

[0271]

[0358] In the third pass, refined MVs are derived by applying the BDOF to the 8x8 grid sub-blocks. For each 8x8 sub-block, starting from the refined MV of the parent sub-block in the second pass, BDOF refinement is applied to derive scaled Vx and Vy without clipping. The derived bioMv(Vx,Vy) is rounded to 1 / 16 sample precision and clipped between -32 and 32.

[0272]

[0359] The refined MVs in the third pass (MV0_pass3(sbIdx3) and MV1_pass3(sbIdx3)) are derived as follows:

[0273]

number

[0274]

[0362] OBMCs in ECM

[0275]

[0363] When OBMC is applied, the top and left boundary pixels of a CU are refined using motion information of neighboring blocks together with weighted prediction, as described in JVET-L0101.

[0276]

[0364] The conditions for which OBMC does not apply are as follows:

[0365] - If OBMC is disabled at the SPS level

[0366] - If the current block has intra or IBC mode

[0367] - If the current block applies LIC

[0368] - If the current luminance block area is 32 or less

[0277]

[0369] Sub-block boundary OBMC is performed by applying the same blending to the top, left, bottom, and right sub-block boundary pixels using the motion information of the neighboring sub-blocks, which makes it valid for the following sub-block-based coding tools:

[0278]

[0370] - Affine AMVP mode

[0371] - Affine merge mode and sub-block based temporal motion vector prediction (SbTMVP)

[0372] - Subblock-based bilateral matching

[0279]

[0373] Sample-based BDOF in ECM

[0280]

[0374] In sample-based BDOF, instead of deriving the motion refinement (Vx, Vy) on a block-by-block basis, the derivation is performed on a sample-by-sample basis.

[0281]

[0375] The coding block is divided into 8x8 subblocks. For each subblock, it is determined whether to apply BDOF by checking the SAD between two reference subblocks against a threshold. If it is determined that BDOF is applied to a subblock, a sliding 5x5 window is used for all samples in the subblock, and the existing BDOF process is applied for all sliding windows to derive Vx and Vy. The derived motion refinement (Vx, Vy) is applied to adjust the bidirectional predicted sample value of the center sample of the window.

[0282]

[0376] Interpolation in ECM

[0283]

[0377] The 8-tap interpolation filter used in VVC is replaced with a 12-tap filter. The interpolation filter is derived from a sinc function whose frequency response is cut off at the Nyquist frequency and cropped by a cosine window function. Table 4 shows the filter coefficients for all 16 phases. Figure 24 compares the frequency response of the interpolation filter with the VVC interpolation filter, all at half-pel phase.

[0284] [Table 4]

[0285]

[0379] Multi-hypothesis prediction (MHP) in ECM

[0286]

[0380] In the multi-hypothesis inter prediction mode (JVET-M0425), in addition to the conventional bi-predictive signal, one or more additional motion compensated prediction signals are signaled. The resulting overall prediction signal is obtained by weighted overlapping sample by sample. The bi-predictive signal p bi and by using the first additional inter prediction signal / hypothesis h3, the resulting prediction signal p3 is obtained as follows:

[0287]

number

[0288]

[0382] The weighting factor α is specified by the new syntax element add_hyp_weight_idx according to the following mapping:

[0289] [Table 5]

[0290]

[0383] Similar to above, two or more additional prediction signals may be used, with the resulting overall prediction signal being iteratively accumulated with each additional prediction signal.

[0291]

number

[0292]

[0385] The resulting overall prediction signal is the final p n (i.e., the p with the largest index n n ) Within this EE, up to two additional prediction signals may be used (i.e., n is limited to 2).

[0293]

[0386] The motion parameters of each additional prediction hypothesis are signaled either explicitly by specifying a reference index, a motion vector predictor index, and a motion vector differential, or implicitly by specifying a merge index. A separate multi-hypothesis merge flag distinguishes between these two signaling modes.

[0294]

[0387] For inter AMVP mode, MHP is applied only if unequal weights in BCW are selected in bi-predictive mode.

[0295]

[0388] A combination of MHP and BDOF is possible, but BDOF is only applied to the bi-predictive signal part of the prediction signal (ie, usually the first two hypotheses).

[0296]

[0389] Adaptive reordering of merge candidates with template matching in ECM (ARMC-TM)

[0297]

[0390] Merge candidates are adaptively reordered by template matching (TM). This reordering method applies to normal merge mode, template matching (TM) merge mode, and affine merge mode (except for SbTMVP candidates). For TM merge mode, merge candidates are reordered before the refinement process.

[0298]

[0391] After the merge candidate list is constructed, the merge candidates are divided into several subgroups. For normal merge mode and TM merge mode, the subgroup size is set to 5. For affine merge mode, the subgroup size is set to 3. The merge candidates in each subgroup are reordered in ascending order according to their cost values ​​based on template matching. For simplicity, the merge candidates in the last subgroup but not the first are not reordered.

[0299]

[0392] The template matching cost of a merge candidate is measured by the sum of absolute differences (SAD) between the template's samples of the current block and their corresponding reference samples. The template contains a set of reconstructed samples that are neighbors of the current block. The template's reference samples are located by the motion information of the merge candidate.

[0300]

[0393] If the merge candidate utilizes bi-prediction, the reference samples of the merge candidate's template are also generated by bi-prediction, as shown in FIG.

[0301]

[0394] For a subblock-based merging candidate with subblock size equal to W×H, the top template contains several subtemplates of size W×1, and the left template contains several subtemplates of size 1×H. As shown in Figure 26, the motion information of the subblocks in the first row and first column of the current block is used to derive the reference samples of each subtemplate.

[0302]

[0395] Geometric Partitioning Mode (GPM) with Merged Motion Vector Differential (MMVD) in ECM

[0303]

[0396] GPM in VVC is extended by applying motion vector refinement on top of the existing GPM's unidirectional MV. First, a flag is signaled to specify whether this mode is used for the CU of the GPM. If this mode is used, each geometric partition of the CU of the GPM can further determine whether to signal MVD. If MVD is signaled for the geometric partition, after the GPM merge candidate is selected, the motion of the partition is further refined by the signaled MVD information. All other procedures are the same as GPM.

[0304]

[0397] MVD is signaled as a distance and direction pair, similar to MMVD. There are 9 candidate distances (1 / 4-pel, 1 / 2-pel, 1-pel, 2-pel, 3-pel, 4-pel, 6-pel, 8-pel, 16-pel) and 8 candidate directions (4 horizontal / vertical and 4 diagonal) involved in GPM with MMVD (GPM-MMVD). Additionally, if pic_fpel_mmvd_enabled_flag is equal to 1, MVD is left-shifted by 2, similar to MMVD.

[0305]

[0398] Geometric Partitioning Mode (GPM) using Template Matching (TM) in ECM

[0306]

[0399] Template matching is applied to the GPM. If GPM mode is enabled for a CU, a CU-level flag is signaled to indicate whether TM is applied to both geometric partitions. The motion information of each geometric partition is refined using the TM. Once a TM is chosen, a template is constructed using the left, top, or left and top neighboring samples according to the partition angle, as shown in Table 5. The motion amount is then refined by minimizing the difference between the current template and the template in the reference picture, using the same search pattern in merge mode with the half-pel interpolation filter disabled.

[0307] [Table 6]

[0308]

[0401] The GPM candidate list is constructed as follows:

[0309]

[0402] 1. Interleaved list-0 MV candidates and list-1 MV candidates are derived directly from the normal merge candidate list. List-0 MV candidates have higher priority than list-1 MV candidates. A pruning method with an adaptive threshold based on the current CU size is applied to remove redundant MV candidates.

[0403] 2. From the normal merge candidate list, the interleaved list 1 MV candidates and list 0 MV candidates are further derived directly. The list 1 MV candidates have higher priority than the list 0 MV candidates. Similarly, a pruning method with an adaptive threshold is applied to remove redundant MV candidates.

[0404] 3. MV candidates are zero-padded until the GPM candidate list is filled.

[0310]

[0405] GPM-MMVD and GPM-TM are enabled exclusively for the CU of one GPM. This is done by signaling the GPM-MMVD syntax first. If the two GPM-MMVD control flags are both equal to false (i.e., GPM-MMVD is disabled for the two GPM splits), then the GPM-TM flag is signaled to indicate whether template matching is applied to the two GPM splits. Otherwise (if at least one GPM-MMVD flag is equal to true), the value of the GPM-TM flag is inferred to be false.

[0311]

[0406] Dividing a picture into CTUs

[0312]

[0407] A picture is partitioned into a sequence of coding tree units (CTUs). The concept of a CTU is the same as that of HEVC [6][7]. For a picture with three sample arrays, a CTU consists of an NxN block of luma samples and the corresponding two blocks of chroma samples. Figure 31 shows an example of a picture partitioned into CTUs.

[0313]

[0408] The maximum allowed size of a luma block in a CTU is specified to be 128x128 (except for the maximum size of a luma transform block, which is 64x64).

[0314]

[0409] Dividing a picture into subpictures, slices, and tiles

[0315]

[0410] A picture is divided into one or more tile rows and one or more tile columns. A tile is a sequence of CTUs that covers a rectangular area of ​​the picture.

[0316]

[0411] A slice consists of an integer number of complete tiles, or an integer number of contiguous complete CTU rows within a tile of a picture.

[0317]

[0412] Two modes of slices are supported: raster scan slice mode and rectangular slice mode. In raster scan slice mode, a slice contains a sequence of complete tiles within the tile raster scan of the picture. In rectangular slice mode, a slice contains either several complete tiles that collectively form a rectangular area of ​​the picture, or several consecutive complete CTU rows of one tile that collectively form a rectangular area of ​​the picture. The tiles in a rectangular slice are scanned in tile raster scan order within the rectangular area corresponding to the slice.

[0318]

[0413] A subpicture contains one or more slices that collectively cover a rectangular area of ​​the picture.

[0319]

[0414] FIG. 32 shows an example of raster scan slice division of a picture, where the picture is divided into 12 tiles and 3 raster scan slices.

[0320]

[0415] FIG. 33 shows an example of rectangular slice division of a picture, where the picture is divided into 24 tiles (6 tile columns and 4 tile rows) and 9 rectangular slices.

[0321]

[0416] FIG. 34 shows an example of a picture divided into tiles and rectangular slices, where the picture is divided into four tiles (two columns of tiles and two rows of tiles) and four rectangular slices.

[0322]

[0417] Figure 35 shows an example of sub-picture division of a picture, where the picture is divided into 18 tiles, with 12 tiles on the left hand side each covering one slice of a 4x4 CTU and 6 tiles on the right hand side each covering two vertically stacked slices of a 2x2 CTU, generating in total 24 slices of various dimensions and 24 sub-pictures (each slice is a sub-picture).

[0323]

[0418] 360° video coding tools

[0324]

[0419] Horizontal wraparound motion compensation

[0325]

[0420] Horizontal wraparound motion compensation in VTM10 is a 360-specific coding tool designed to improve the visual quality of 360-degree videos reconstructed in equi-rectangular projection (ERP) format [5]. In conventional motion compensation, when a motion vector references a sample that crosses the picture boundary of a reference picture, iterative padding is applied to derive the value of the out-of-bounds sample by copying from the nearest sample on the corresponding picture boundary. For 360-degree videos, this iterative padding method is not suitable and can result in visual artifacts called "seam artifacts" in the reconstructed viewport video. Because 360-degree videos are captured on a sphere and are essentially "borderless", reference samples that are outside the boundary of the reference picture in the projected region can always be obtained from neighboring samples in the spherical region. For common projection formats, it can be difficult to derive the corresponding neighboring samples in the spherical region, since it requires 2D-to-3D and 3D-to-2D coordinate transformations [5] as well as sample interpolation of fractional sample positions. For the left and right boundaries of the ERP projection format, this problem is much simpler, since the spherical neighborhood outside the left picture boundary can be obtained from samples in the right picture boundary, and vice versa. Considering that the ERP projection format is widely used and relatively easy to implement, horizontal wraparound motion compensation was adopted in VVC to improve the visual quality of 360-degree videos encoded in the ERP projection format.

[0326]

[0421] The horizontal wrap-around motion compensation process is as shown in Figure 36. If a portion of a reference block is outside the left (or right) boundary of a reference picture in the projected region, instead of repetitive padding, its "out-of-bounds" portion is obtained from the corresponding spherical neighborhood located in the reference picture toward the right (or left) boundary in the projected region. Repetitive padding is used only for the top and bottom picture boundaries. As shown in Figure 36, horizontal wrap-around motion compensation can be combined with a non-standard padding method often used in 360-degree video coding (see padded ERP in [5]). In VVC, this is achieved by signaling a high-level syntax element indicating the wrap-around offset, which should be set to the ERP picture width before padding, and this syntax is used to adjust the position of the horizontal wrap-around accordingly. This syntax is not affected by specific padding amounts on the left and right picture boundaries, and therefore naturally supports asymmetric padding of ERP pictures, i.e., when the left and right padding are different. When the reference samples are outside the left and right boundaries of the reference picture, horizontal wraparound motion compensation provides more meaningful information for motion compensation. Under the 360-degree video CTC [4], this tool improves compression performance not only in terms of rate-distortion performance, but also in terms of reducing seam artifacts and improving the subjective quality of the reconstructed 360-degree video. Horizontal wraparound motion compensation can also be used for other single-plane projection formats with constant sampling density in the horizontal direction, such as the adjusted equal-area projection in 360Lib [5].

[0327] [Table 7]

[0328]

[0423] In existing video codecs such as HEVC and VVC, a reference block of an inter-mode coded CU (hereinafter referred to as inter CU) may be located partially or entirely outside a reference picture because an iterative padding process is applied to the reference pictures to generate reference pixels around each reference picture.

[0329]

[0424] In the VVC specification, the padding process is implemented by modifying the integer reference sample acquisition process (as shown in Table 6): Whenever an integer reference sample to be acquired is located outside the reference picture, the nearest integer reference sample in the reference picture is used instead.

[0330]

[0425] With padded reference pictures, it is valid for inter CUs to have reference blocks that are partially or entirely outside the reference picture as shown in Figure 27, where bidirectional motion compensation is performed to generate inter predicted blocks for the current block. In this example, the reference blocks in list 0 are partially out of bounds (OOB), while the reference blocks in list 1 are entirely inside the reference picture.

[0331]

[0426] When an interblock is bidirectionally or multidirectionally predicted, the final predictor is simply the average of two or more motion compensated prediction blocks, and when BWC is further applied, different weights are applied to the MC predictors of list 0 and list 1 to generate the final predictor, respectively. However, the out-of-bounds (OOB) part of a motion compensation block has a low prediction effect because the OOB part is simply a repeating pattern generated by the boundary pixels in the reference picture. However, in existing video codecs, the low-effect OOB part of an MC block is not specially considered in inter prediction. Similarly, in VVC, wrap-around motion compensation may also be used. The same problem may be observed in the wrap-around MC scheme.

[0332]

[0427] In this disclosure, some methods are proposed to improve inter prediction by considering the less effective OOB part of MC block. Furthermore, "block" is used to explain the concept of this disclosure, and "block" may be easily replaced by any concrete definition used in existing codecs. For example, "block" may be coding unit (CU), prediction unit (PU), transform unit (TU), coding tree unit (CTU), coding block (CB), prediction block (PB), transform block (TB), coding tree block (CTB), sub-CU, sub-PU, or any other well-defined term in existing codecs.

[0333]

[0428] The following methods may be applied alone or in combination.

[0334]

[0429] In one embodiment according to the present disclosure, when combining two or more prediction blocks generated by a motion compensation process, an additional weighting is applied to the predictor, and the additional weighting is derived based on whether the prediction sample is OOB or not. The basic concept is that when performing a weighted average of two predictor blocks, the OOB predictor sample is given less weight because it has a lower effect.

[0335]

[0430] In one scheme, the final predictor sample for a bidirectional inter-coded block is generated by a weighted average of the predictors using the following formula:

[0336]

number

[0337]

[0432] Where:

number

number

[0338]

[0433] As shown in Figure 28, the predictor sample of a block is derived by performing motion compensation from a reference picture. The motion vector of the current block is used to locate the reference block. It should be noted that if the reference block is located at a fractional position between integer reference samples, a fractional sample interpolation process is further applied to derive the predicted sample. As described in VVC section 8.5.6.3.2 "Luminance Sample Interpolation Filtering Process", fractional sample interpolation first performs vertical one-dimensional interpolation accessing eight vertically adjacent integer samples, followed by horizontal one-dimensional interpolation accessing eight vertically interpolated horizontally adjacent samples. In ECM, the 8-tap interpolation filter used in VVC is further replaced by a 12-tap filter.

[0339]

[0434] Based on the motion compensation scheme described above, if at least one of the integer reference samples used to generate the predictor sample (through the interpolation process) is located outside the reference picture, the predictor sample

number

[0340]

[0435] The weights may be derived using the following method.

[0341]

number

number

number

number

[0342]

[0439] (this is

number

[0343]

[0440] In other cases

number

number

number

[0344]

number

[0345]

[0443] (this is

number

[0346]

[0444] In other cases

number

number

number

number

[0347]

[0447] (this is

number

[0348]

[0448] In other cases

number

number

number

number

[0349]

[0451] (this is

number

[0350]

[0452] FIG. 30 shows an example of a coding method according to the present disclosure. In step 3001, a decoder derives a first reference picture and a second reference picture for a current coding block. In step 3002, the decoder derives a first predictor sample from the first reference picture based on a first motion vector associated with the first reference picture using a motion compensation process. In step 3003, the decoder derives a second predictor sample from the second reference picture based on a second motion vector associated with the second reference picture using a motion compensation process. In step 3004, the decoder obtains a final predictor sample in the current coding block based on at least one of the first predictor sample or the second predictor sample and an out-of-bounds (OOB) condition. This example method may be performed by an encoder as well.

[0351]

[0453] In another embodiment of the present disclosure, when a block is coded as a BCW mode that combines two prediction blocks generated by a motion compensation process with a BCW weighted average, an additional weighting is applied to the predictor, and the additional weighting is derived based on whether the predictor is OOB or not. In one scheme, the final predictor sample of a BCW coded block is generated by a weighted average of the predictors using the following formula:

[0352]

number

[0353]

[0455] where w is the BCW weighting,

number

number

[0354]

number

number

number

number

[0355]

[0459] (this is

number

[0356]

[0461] In other cases

number

number

number

number

[0357]

[0464] (this is

number

[0358]

[0465] In other cases

number

number

number

number

[0359]

[0468] (this is

number

[0360]

[0469] In other cases

number

number

number

number

[0361]

[0472] (this is

number

[0362]

[0473] In another embodiment of the present disclosure, when BDOF is enabled for a block, an additional weighting is applied to the predictor, and the additional weighting is derived based on whether the predictor is OOB or not. In one scheme, the final predictor sample for a BDOF enabled block is generated by a weighted average of the predictors using the following formula:

[0363]

number

[0364]

[0475] Here, B i,j is the BDOF offset of each predictor sample,

number

number

[0365]

number

number

Number

Number

[0366]

[0479] (This is

Number

[0367]

[0481] In other cases

Number

Number

Number

Number

[0368]

[0484] (This is

Number

[0369]

[0485] In other cases

Number

Number

number

number

[0370]

[0488] (this is

number

[0371]

[0489] In other cases

number

number

number

number

[0372]

[0492] (this is

number

[0373]

[0493] In another embodiment of the present disclosure, when BDOF is enabled for a block, additional weighting is applied to the predictor, and the additional weighting is derived based on whether the predictor is OOB. In one approach, the final predictor sample of the BDOF-enabled block is generated by a weighted average of the predictors using the following equation.

[0374]

Number

[0375]

[0495] Here, B i,j is the BDOF offset of each predictor sample,

Number

Number

[0376]

Number

Number

Number

Number

[0377]

[0500] (This is

Number

[0378]

[0502] In other cases

Number

Number

Number

Number

[0379]

[0506] (This is

Number

[0380]

[0507] In other cases

Number

Number

Number

Number

[0381]

[0510] (this is

number

[0382]

[0511] In other cases

number

number

number

number

[0383]

[0514] (this is

number

[0384]

[0515] Based on the motion compensation scheme described above, if at least one of the integer reference samples used to generate a predictor sample (through the interpolation process) is located outside the reference picture, the predictor sample is defined as OOB.

[0385]

[0516] However, in some cases, only a few OOB reference integer samples are used to generate a predictor sample through an interpolation process, in which case the predictor sample may still provide an efficient prediction (e.g., only two of the eight reference integer samples are OOB). Therefore, in order to provide a tolerance for the OOB decision, various schemes have been proposed for determining whether a predicted sample is OOB or not.

[0386]

[0517] In another embodiment of the present disclosure, if at least one of the vertically closest integer reference samples is OOB or at least one of the horizontally closest integer reference samples is OOB, the predictor sample is determined to be OOB. For example, as shown in Figure 28, the horizontally closest integer reference samples of the predictor P_0,0^Lx are samples d and e.

[0387]

[0518] In another embodiment of the present disclosure, a predictor sample is determined to be OOB if at least N of the integer samples used to generate the predictor sample are OOB, where N is any integer.

[0388]

[0519] In another embodiment of the present disclosure, when performing a weighted average of two or more predictor blocks, predictor samples that use OOB integer reference samples to perform interpolation are given less weight. Furthermore, the weight is inversely proportional to the number of OOB integer reference samples.

[0389]

[0520] In another embodiment of the present disclosure, when performing weighted averaging of two or more predictor blocks, weights are given to predictor samples that use OOB integer reference samples to perform the interpolation, and the weights are signaled in the bitstream at different levels, such as sequence level (sequence parameter set), picture level (picture parameter set), slice level (slice header), or block level.

[0390]

[0521] However, in this disclosure, in order to realize flexible OOB judgment, the following scheme is proposed to judge whether a predictor sample is OOB or not. In one or more embodiments, a predictor sample is judged to be OOB according to whether the predictor sample is located outside the reference picture and the fractional position of the predictor sample. Specifically, the position of the predicted pixel in the current block is Pos i,j and the horizontal and vertical position indexes are denoted as Pos_x i,j and Pos_y i,j The MV of list 0 and the MV of list 1 of the current block are denoted as

number

number

number

[0391]

[0522] Note that in this disclosure, it is assumed that the position index starts from the top-left of the picture, e.g., the position index of the top-left sample of the picture is denoted as (0,0), and the position index increases from top to bottom and left to right. In one exemplary scheme, one predictor sample is determined to be OOB using the following condition check:

[0392]

[0523] In one exemplary scheme, one predictor sample is determined to be OOB using the following condition checks (M, N, O, P are any integers):

[0393]

[0524] Predictor Sample

number

[0394]

number

[0395]

[0529] Otherwise, if none of the above conditions hold, the predictor samples

number

[0396]

[0530] FIG. 37 shows a flowchart of a video coding process according to an example of the present disclosure. In step 3701, the encoder or decoder may derive a reference picture for a current coding block. In step 3702, the encoder or decoder may use a motion compensation process to derive a predictor sample from the reference picture based on a motion vector associated with the reference picture. In step 3703, the encoder or decoder may determine whether the predictor sample is located outside the reference picture by a certain margin. In step 3704, the encoder or decoder may determine that the predictor sample is OOB in response to determining that the predictor sample is located outside the reference picture by a certain margin.

[0397]

[0531] In yet another embodiment, the OOB determination of predictor samples may be made based on a group of predictor samples. In one example, a block of M×N predictor samples, where M is the block width and N is the block height, share the same OOB status (determined as OOB or not OOB) according to the OOB determination process described above (but not limited to the proposed scheme).

[0398]

[0532] DMVR considering OOB conditions

[0399]

[0533] In order to improve the accuracy of MV in merge mode, in VVC, bilateral matching (BM) based decoder-side motion vector refinement is applied. In bidirectional prediction operation, refined MVs around the initial MV in reference picture list L0 and reference picture list L1 are searched. The BM method calculates the distortion between two candidate blocks in reference picture list L0 and list L1. As shown in FIG. 17, the SAD between red blocks based on each MV candidate around the initial MV is calculated. The MV candidate with the lowest SAD becomes the refined MV and is used to generate the bidirectional prediction signal. In another embodiment, if a sample of a candidate block in reference picture list L0 or list L1 is determined to be OOB, the sample and its related samples of other corresponding candidate blocks are not used to calculate the matching cost (e.g., sample absolute difference sum or sample squared difference sum).

[0400]

[0534] Template matching considering OOB conditions

[0401]

[0535] Template matching (TM) is a decoder-side MV derivation method for refining the motion information of a current CU by finding the closest match between a template in a current picture (i.e., a neighboring block above and / or to the left of the current CU) and a reference block in a reference picture (i.e., the same size as the template). In another embodiment, if a sample of a reference block in a reference picture is determined to be OOB, that sample and its associated sample of the template in the current picture are not used to calculate a matching cost (e.g., sample absolute difference sum or sample squared difference sum).

[0402]

[0536] MHP considering OOB conditions

[0403]

[0537] In the multi-hypothesis inter prediction mode (JVET-M0425), in addition to the conventional bi-predictive signal, one or more additional motion compensated prediction signals are signaled. The resulting overall prediction signal is obtained by weighted overlapping sample by sample. The bi-predictive signal p bi and by using the first additional inter prediction signal / hypothesis h3, the resulting prediction signal p3 is obtained as follows:

[0404]

number

[0405]

[0539] In another embodiment, if a sample of the additional motion compensated prediction is determined to be OOB, that sample is not used to generate the resulting overall prediction signal for the MHP mode.

[0406]

[0540] LIC considering OOB conditions

[0407]

[0541] LIC is an inter-prediction technique for modeling the local illumination variation between a current block and its predicted block as a function derived from pixels between the current block template and the reference block of the current block template. The parameters of the function can be represented by a scale α and an offset β, forming a linear formula, i.e., α*p[x]+β to compensate for illumination changes, where p[x] is a reference sample pointed to by the MV at position x on the reference picture. Since α and β can be derived based on the current block template and the reference block of the current block template, no signaling overhead is required for α and β, except that a LIC flag is signaled to indicate the use of LIC. In another embodiment, if a sample of the reference block of the current block template is determined to be OOB, the sample and its associated sample of the current block template are not used to derive the LIC parameters.

[0408]

[0542] Affine PROF

[0409]

[0543] In another embodiment, when performing PROF, if the samples used to derive the horizontal / vertical gradient values ​​include the OOB reference samples, the PROF offset is set as zero.

[0410]

[0544] In another embodiment, when running PROF, if the samples used to derive the horizontal / vertical gradient values ​​include OOB reference samples, the horizontal / vertical gradient values ​​are set as zero.

[0411]

[0545] One-way prediction considering OOB conditions

[0412]

[0546] The unidirectional prediction method is similar to the inter-predicted block predicted using one single motion vector. In one or more examples of the present disclosure, for the unidirectional predicted block, a substitute predictor is used to replace the OOB predictor sample. In one scheme, the substitute predictor is derived by a predictor in another reference picture, which is pointed to by scaling the mv of the current block according to the temporal distance between the current picture, the reference picture, and the target reference picture, using a scaling method similar to that used to derive Temporal MV Prediction (TMVP) in VVC or HEVC.

[0413]

[0547] As shown in FIG. 38, according to one embodiment of the present invention, a method for decoding a bidirectional predictive coding block is provided. In step 3801, the decoder derives a first reference picture for a current coding block. In step 3802, the decoder derives a first predictor sample from the first reference picture based on a motion vector associated with the first reference picture using a motion compensation process. In step 3803, the decoder determines whether the first predictor sample is out of bounds (OOB), and in response to determining that the first predictor sample is OOB, the decoder replaces the first predictor sample that is OOB with a first substitute predictor sample. In step 3804, the decoder derives a second reference picture for the current coding block. In step 3805, the decoder derives a second predictor sample from the second reference picture based on a motion vector associated with the second reference picture using a motion compensation process. In step 3806, the decoder determines whether the second predictor sample is out of bounds (OOB), and in response to determining that the second predictor sample is OOB, the decoder replaces the second predictor sample that is OOB with a second substitute predictor sample. In step 3807, the decoder generates a final predicted sample by averaging sample values ​​of the first predictor sample and the second predictor sample.

[0414]

[0548] FIG. 39 is a block diagram illustrating a method for coding a bidirectional predictive coding block according to an example of the present disclosure. In step 3901, the encoder derives a first reference picture for a current coding block. In step 3902, the encoder derives a first predictor sample from the first reference picture based on a motion vector associated with the first reference picture using a motion compensation process. In step 3903, the encoder determines whether the first predictor sample is out of bounds (OOB), and in response to determining that the first predictor sample is OOB, the encoder replaces the first predictor sample that is OOB with a first substitute predictor sample. In step 3904, the encoder derives a second reference picture for the current coding block. In step 3905, the encoder derives a second predictor sample from the second reference picture based on a motion vector associated with the second reference picture using a motion compensation process. In step 3906, the encoder determines whether the second predictor sample is out of bounds (OOB), and in response to determining that the second predictor sample is OOB, the encoder replaces the second predictor sample that is OOB with a second substitute predictor sample. In step 3907, the encoder generates a final predicted sample by averaging sample values ​​of the first predictor sample and the second predictor sample.

[0415]

[0549] As shown in FIG. 40, according to one example of the present disclosure, a method for decoding a unidirectional predictive coding block is provided. In step 4001, a decoder derives a first reference picture for a current coding block. In step 4002, the decoder derives a first predictor sample from the first reference picture based on a motion vector associated with the first reference picture using a motion compensation process. In step 4003, the decoder determines whether the first predictor sample is out of bounds (OOB), and in response to determining that the first predictor sample is OOB, the decoder replaces the first predictor sample that is OOB with a substitute predictor sample. The decoder may derive the substitute predictor sample based on a second predictor sample in a second reference picture. For example, the decoder may derive the second predictor sample in the second reference picture by scaling a motion vector of the current coding block according to a distance between the current picture and the first and second reference pictures.

[0416]

[0550] FIG. 41 is a block diagram showing a method for coding a unidirectional predictive coding block according to an example of the present disclosure. In step 4101, the encoder derives a first reference picture for a current coding block. In step 4102, the encoder derives a first predictor sample from the first reference picture based on a motion vector associated with the first reference picture using a motion compensation process. In step 4103, the encoder determines whether the first predictor sample is out of bounds (OOB), and in response to determining that the first predictor sample is OOB, the encoder replaces the first predictor sample that is OOB with a substitute predictor sample.

[0417]

[0551] Implementation of the proposed method

[0418]

[0552] In an alternative method that meets the same motivation of the proposed method, in the fourteenth embodiment, the OOB predictor in one reference list LX (X can be 0 or 1) may be replaced by the corresponding non-OOB predictor in the other reference list L(1-X). After the replacement, the original bidirectional MC averaging process may be kept unchanged. An example with the same results as those shown in the first embodiment is shown below.

[0419]

number

number

[0420]

[0554] it is

number

[0421]

[0555] (Also, this is

number

[0422]

[0556] In other cases

number

number

[0423]

[0557] it is

number

[0424]

[0558] (this is

number

[0425]

[0559] In other cases

number

number

[0426]

[0560] it is

number

[0427]

[0561] In other cases

number

number

[0428]

[0562] it is

number

[0429]

[0563] High-level syntax signaling

[0430]

[0564] In another embodiment, one or more syntax elements may be signaled in the bitstream to indicate the enabling / disabling of the proposed MC considering the OOB conditions. Furthermore, one or more syntax elements may be signaled in the bitstream to indicate the parameters or thresholds to be used in the MC considering the OOB. The proposed syntax elements may be signaled at the sequence level (e.g., sequence parameter set), picture level (e.g., picture parameter set), slice level (e.g., slice header), CTU / CTB level, CU / CB level, PU / PB level. Furthermore, the control of the enabling / disabling may be hierarchical. For example, one syntax element is signaled at the sequence level to indicate the enabling / disabling of the proposed OOB scheme. If the syntax element indicates that the OOB scheme is disabled, no further syntax elements are signaled in the bitstream, and if the syntax element indicates that the OOB scheme is enabled, more syntax elements may be signaled at the lower levels to indicate the enabling / disabling of OOB at the lower levels. For example, if a sequence-level syntax element indicates that OOB is enabled, then one slice-level syntax element is signaled to indicate enabling / disabling of the OOB scheme at the slice level.

[0431]

[0565] In the motion compensation (MC) schemes of existing video coding standards, when a reference sample indicated by a motion vector is located outside a predetermined boundary, either padding of boundary samples or horizontal / vertical wraparound is utilized to enable the MC process to use the reference sample located outside the boundary. The predetermined boundary includes, but is not limited to, a coding picture boundary, a sub-picture boundary, a slice boundary, and / or a tile boundary. In another embodiment, the proposed MC scheme may be applied to any scheme when motion compensation may be performed based on a reference sample located outside the boundary as described above. It is also noted that the proposed scheme may be applied when reference picture resampling (RPR) is used.

[0432]

[0566] It should be noted that the proposed motion compensation (MC) scheme that takes into account the OOB condition is not limited to be applied to the encoding method in the proposed embodiment, but can also be applied to all inter-tools that perform MC to generate predictor samples, such as sub-block MC (sub-block TMVP, affine mode, etc.), OBMC, IBC, SMVD, DMVR, etc., as described in the previous section.

[0433]

[0567] The above methods may be implemented using an apparatus including one or more circuits including an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a controller, a microcontroller, a microprocessor, or other electronic components. The apparatus may use the circuits in combination with other hardware or software components to perform the above described methods. Each module, sub-module, processing unit, or sub-unit disclosed above may be at least partially implemented using one or more circuits.

[0434]

[0568] 29 illustrates a computing environment 1610 coupled with a user interface 1650. The computing environment 1610 may be part of a data processing server. The computing environment 1610 includes a processor 1620, a memory 1630, and an input / output (I / O) interface 1640.

[0435]

[0569] The processor 1620 typically controls the overall operation of the computing environment 1610, such as operations related to display, data acquisition, data communication, and image processing. The processor 1620 may include one or more processors for executing instructions called for performing all or a portion of the steps in the methods described above. Additionally, the processor 1620 may include one or more modules that facilitate interaction between the processor 1620 and other components. The processor may be a central processing unit (CPU), a microprocessor, a single chip machine, a graphics processing unit (GPU), or the like.

[0436]

[0570] The memory 1630 is configured to store various types of data to support the operation of the computing environment 1610. The memory 1630 may include predefined software 1632. Examples of such data include instructions for any application or method operated on the computing environment 1610, video data sets, image data, etc. The memory 1630 may be implemented by using any type of volatile or non-volatile memory device or combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic memory, a flash memory, a magnetic disk, or an optical disk.

[0437]

[0571] The I / O interface 1640 provides an interface between the processor 1620 and a peripheral interface module such as a keyboard, a click wheel, buttons, etc. The buttons may include, but are not limited to, a home button, a start scan button, and a stop scan button. The I / O interface 1640 may be coupled to an encoder and a decoder.

[0438]

[0572] In one embodiment, a non-transitory computer-readable storage medium is also provided that includes a plurality of programs executable by the processor 1620 in the computing environment 1610, for example in the memory 1630, for performing the above-described methods. In one example, the plurality of programs may be executed by the processor 1620 in the computing environment 1610 to receive (e.g., from the video encoder 20 of FIG. 2) a bitstream or datastream including encoded video information (e.g., video blocks representing encoded video frames, and / or one or more associated syntax elements, etc.) and to perform the above-described decoding method according to the received bitstream or datastream. In another example, the programs may be executed by the processor 1620 in the computing environment 1610 to perform the encoding method described above to encode video information (e.g., video blocks representing encoded video frames, and / or one or more associated syntax elements, etc.) into a bitstream or data stream, and may also be executed by the processor 1620 in the computing environment 1610 to transmit the bitstream or data stream (e.g., to the video decoder 30 of FIG. 3). Alternatively, a non-transitory computer-readable storage medium may store a bitstream or data stream including encoded video information (e.g., video blocks representing encoded video frames, and / or one or more associated syntax elements, etc.) generated by an encoder (e.g., the video encoder 20 of FIG. 2) using the encoding method described above, for use by the decoder (e.g., the video decoder 30 of FIG. 3) in decoding video data. The non-transitory computer-readable storage medium may be, for example, a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0439]

[0573] In one embodiment, a computing device is also provided that includes one or more processors (e.g., processor 1620) and a non-transitory computer-readable storage medium or memory 1630 storing a plurality of programs executable by the one or more processors, the one or more processors being configured to perform the methods described above upon execution of the plurality of programs.

[0440]

[0574] In one embodiment, a computer program product is also provided that includes a number of programs, e.g., in memory 1630, executable by the processor 1620 in the computing environment 1610 to perform the methods described above. For example, the computer program product may include a non-transitory computer-readable storage medium.

[0441]

[0575] In one embodiment, the computing environment 1610 may be implemented using one or more ASICs, DSPs, digital signal processing devices (DSPDs), programmable logic devices (PLDs), FPGAs, GPUs, controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0442]

[0576] The description of the present disclosure has been presented for purposes of illustration and is not intended to be exhaustive or limiting of the disclosure. Many modifications, variations and alternative implementations will be apparent to one skilled in the art having the benefit of the teachings presented in the foregoing descriptions and the associated drawings.

[0443]

[0577] Unless otherwise specified, the order of steps of the method according to the present disclosure is intended to be illustrative only, and the steps of the method according to the present disclosure are not limited to the order specifically described above, and may be changed according to actual conditions. Moreover, at least one of the steps of the method according to the present disclosure may be adjusted, combined, or deleted according to actual requirements.

[0444]

[0578] The examples have been chosen and described in order to explain the principles of the disclosure, to enable those skilled in the art to understand the disclosure in its various implementations, and to make best use of the underlying principles and various implementations with various modifications as appropriate for the particular use contemplated. Thus, it should be understood that the scope of the disclosure is not limited to the particular examples of implementations disclosed, and that modifications and other implementations are intended to be included within the scope of the disclosure.

Claims

1. 1. A method for decoding a bidirectionally predictively coded block, comprising: a decoder deriving a first reference picture for a current coding block; the decoder deriving, using a motion compensation process, from the first reference picture a first predictor sample based on a motion vector associated with the first reference picture; the decoder determining whether the first predictor sample is out of bounds (OOB), and in response to the decoder determining that the first predictor sample is OOB, replacing the first predictor sample that is OOB with a first substitute predictor sample; the decoder deriving a second reference picture for the current coding block; deriving, by the decoder, a second predictor sample from the second reference picture using a motion compensation process based on a motion vector associated with the second reference picture; the decoder determining whether the second predictor sample is out of bounds (OOB), and in response to the decoder determining that the second predictor sample is OOB, replacing the second predictor sample that is OOB with a second substitute predictor sample; generating a final predicted sample by averaging sample values ​​of the first predictor sample and the second predictor sample; A method comprising:

2. Replacing the first predictor sample that is OOB with the first substitute predictor sample comprises:

2. The method for video decoding of claim 1, further comprising setting the first substitute predictor sample as the second predictor sample.

3. Replacing the second predictor samples that are OOB with the second substitute predictor samples comprises:

2. The method for video decoding of claim 1, further comprising setting the second alternative predictor sample as the first predictor sample.

4. 1. A method for coding a bidirectionally predictively coded block, comprising: an encoder deriving a first reference picture for a current coding block; the encoder deriving a first predictor sample from the first reference picture using a motion compensation process based on a motion vector associated with the first reference picture; the encoder determining whether the first predictor sample is out of bounds (OOB), and in response to the encoder determining that the first predictor sample is OOB, replacing the predictor sample that is OOB with a first substitute predictor sample; the encoder deriving a second reference picture for the current coding block; the encoder deriving, using a motion compensation process, a second predictor sample from the second reference picture based on a motion vector associated with the second reference picture; the encoder determining whether the second predictor sample is out of bounds (OOB), and the decoder, in response to determining that the second predictor sample is OOB, replacing the second predictor sample that is OOB with a second substitute predictor sample; generating a final predicted sample by averaging sample values ​​of the first predictor sample and the second predictor sample; A method comprising:

5. Replacing the first predictor sample that is OOB with the first substitute predictor sample comprises: The method for video coding of claim 4 , further comprising setting the first substitute predictor sample as the second predictor sample.

6. Replacing the second predictor samples that are OOB with the second substitute predictor samples comprises: The method for video coding of claim 4 , further comprising setting the second alternative predictor sample as the first predictor sample.

7. 1. A method for decoding a unidirectionally predictively coded block, comprising the steps of: a decoder deriving a first reference picture for a current coding block; deriving, by the decoder, a predictor sample from the first reference picture using a motion compensation process based on a motion vector associated with the first reference picture; the decoder determining whether the predictor samples are out of bounds (OOB); and in response to the decoder determining that the predictor samples are OOB, replacing the predictor samples that are OOB with substitute predictor samples. A method comprising:

8. The step of replacing the predictor samples that are OOB with the substitute predictor samples comprises: The method for decoding a unidirectionally predictively coded block of claim 7 , further comprising: deriving the substitute predictor sample based on a second predictor sample in a second reference picture.

9. Deriving the substitute predictor sample based on the second predictor sample in the second reference picture includes:

9. The method for decoding a unidirectional predictively coded block of claim 8, comprising: deriving the second predictor sample in the second reference picture by scaling a motion vector of the current coding block according to distances between a current picture, the first reference picture, and the second reference picture.

10. 1. A method for coding a unidirectional predictively coded block, comprising: an encoder deriving a first reference picture for a current coding block; the encoder deriving, using a motion compensation process, a predictor sample from the first reference picture based on a motion vector associated with the first reference picture; the encoder determining whether the predictor samples are out of bounds (OOB), and in response to the encoder determining that the predictor samples are OOB, replacing the predictor samples that are OOB with substitute predictor samples. A method comprising:

11. The step of replacing the predictor samples that are OOB with the substitute predictor samples comprises: The method for video coding of claim 10 , further comprising deriving the alternative predictor sample based on a second predictor sample in a second reference picture.

12. Deriving the substitute predictor sample based on the second predictor sample in the second reference picture includes:

12. The method for video coding of claim 11, comprising: deriving the second predictor sample in the second reference picture by scaling a motion vector of the current coding block according to distances between a current picture and the first and second reference pictures.

13. one or more processors; a memory configured to store instructions executable by the one or more processors; 13. An apparatus comprising: the one or more processors, upon execution of the instructions, configured to perform a method according to any of claims 1 to 12.

14. A non-transitory computer-readable storage medium for storing computer-executable instructions that, when executed by one or more computer processors, cause the one or more computer processors to receive a bitstream and perform a method according to any of claims 1 to 3 or 7 to 9 based on the bitstream.

15. 13. A non-transitory computer-readable storage medium for storing computer-executable instructions that, when executed by one or more computer processors, cause the one or more computer processors to perform a method according to any one of claims 4 to 6 or 10 to 12 to encode a current video block into a bitstream and transmit the bitstream.

Images