Motion compensation considering out-of-boundary conditions in video coding.
By determining and managing out-of-boundary conditions in video encoding through motion compensation, the method enhances compression efficiency and video quality in video encoding technologies.
Patent Information
- Application Number
- JP2024529696
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2021-11-30
- Filing Date
- 2022-11-29
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2042-11-29
AI Technical Summary
Existing video encoding technologies face challenges in handling predictor samples that fall outside the boundaries of reference pictures, leading to out-of-boundary conditions which affect video quality and compression efficiency.
The method involves determining whether predictor samples are outside a certain margin of the reference picture and handling out-of-boundary conditions by appropriate motion compensation processes at both the encoder and decoder sides, using techniques such as motion vector prediction and interpolation to manage these samples effectively.
This approach enhances video encoding by improving compression efficiency and maintaining video quality by addressing out-of-boundary conditions, thereby optimizing the use of video encoding standards like VVC and HEVC.
Smart Images

Figure 2024540760000001_ABST
Abstract
Description
[Technical field]
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is based on and claims priority to Provisional Application No. 63 / 284,613, filed November 30, 2021, the entire contents of which are incorporated herein by reference for all purposes. FIELD OF THE DISCLOSURE This disclosure relates to video encoding and compression. More particularly, this disclosure relates to methods and apparatus for inter prediction in video encoding. [Background technology]
[0002] Digital video is supported by a variety of electronic devices, such as digital televisions, laptop or desktop computers, tablet computers, digital cameras, digital recording devices, digital media players, video gaming consoles, smart phones, video teleconferencing devices, video streaming devices, and the like. The electronic devices transmit and receive or otherwise communicate digital video data over communication networks and / or store the digital video data in storage devices. Due to limited bandwidth capacity of communication networks and limited memory resources of storage devices, video coding may be used to compress the video data according to one or more video coding standards before the video data is communicated or stored. For example, video coding standards include Versatile Video Coding (VVC), Joint Exploration test Model (JEM), High-Efficiency Video Coding (HEVC / H.265), Advanced Video Coding (AVC / H.264), Moving Picture Expert Group (MPEG) coding, and the like. Video coding generally employs prediction methods (e.g., inter-prediction, intra-prediction, etc.) that exploit the redundancy inherent in video data. Video coding aims to compress video data into a format that uses a lower bitrate while avoiding or minimizing degradation of video quality. Summary of the Invention [Problem to be solved by the invention]
[0003] Examples of this disclosure provide methods and apparatus for inter prediction in video encoding. [Means for solving the problem]
[0004] According to a first aspect of the present disclosure, a video encoding method is provided. The method may include: deriving, by a decoder, a reference picture of a current coding block; deriving, by the decoder, a predictor sample based on a motion vector associated with the reference picture using a motion compensation process from the reference picture; determining, by the decoder, whether the predictor sample is located outside the reference picture by a certain margin; and determining, by the decoder, in response to determining that the predictor sample is located outside the reference picture by the certain margin, that the predictor sample is out of bounds (OOB).
[0005] According to a second aspect of the present disclosure, a video encoding method is provided. The method may include: deriving, by an encoder, a reference picture of a current coding block; deriving, by the encoder, a predictor sample from the reference picture using a motion compensation process and based on a motion vector associated with the reference picture; determining, by the encoder, whether the predictor sample is located outside the reference picture by a certain margin; and determining, by the encoder, in response to determining that the predictor sample is located outside the reference picture by the certain margin, that the predictor sample is out of bounds (OOB).
[0006] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not intended to be restrictive of the present disclosure.
[0007] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate examples consistent with the present disclosure and, together with the description, serve to explain the principles of the disclosure. [Brief description of the drawings]
[0008] [Figure 1] FIG. 1 is a block diagram illustrating an example system for encoding and decoding video blocks in accordance with some embodiments of this disclosure. [Diagram 2]1 is a block diagram illustrating an example video encoder according to some embodiments of the present disclosure. [Diagram 3] 1 is a block diagram illustrating an exemplary video decoder according to some embodiments of the present disclosure. [Figure 4A-4E] 1 is a block diagram illustrating how a frame is recursively partitioned into multiple video blocks of different sizes and shapes in accordance with some embodiments of this disclosure. [Diagram 5] 1A-1C are diagrams illustrating example locations of spatial merge candidates according to some embodiments of the present disclosure. [Figure 6] 1A-1C are diagrams illustrating examples of candidate pairs considered for redundancy check of spatial merge candidates according to some embodiments of the present disclosure. [Figure 7] 4A-4C are diagrams illustrating example motion vector scaling of temporal merge candidates according to some embodiments of the present disclosure. [Figure 8] FIG. 2 illustrates candidate positions for temporal merge candidates according to some embodiments of the present disclosure. [Figure 9A] FIG. 13 illustrates an example of an MMVD search point according to some embodiments of the present disclosure. [Figure 9B] FIG. 13 illustrates an example of an MMVD search point according to some embodiments of the present disclosure. [Figure 10A] FIG. 2 illustrates an example of a control point-based affine motion model according to some embodiments of the present disclosure. [Figure 10B] FIG. 2 illustrates an example of a control point-based affine motion model according to some embodiments of the present disclosure. [Figure 11] FIG. 2 illustrates an example of a sub-block-wise affine MVF according to some embodiments of the present disclosure. [Figure 12] FIG. 2 illustrates the location of inherited affine motion predictors according to some embodiments of the present disclosure. [Figure 13] FIG. 13 illustrates locations of candidate positions for constructed affine merge modes according to some embodiments of the present disclosure. [Figure 14]FIG. 2 illustrates an example of sub-blocks MV and pixels according to some embodiments of the present disclosure. [Figure 15A] A diagram showing an example of spatially adjacent blocks used by the ATVMP according to some embodiments of the present disclosure. [Figure 15B] FIG. 1 illustrates an example of an SbTMVP process, according to some embodiments of the present disclosure. [Figure 16] FIG. 2 illustrates an example of an extended CU region used in BDOF according to some embodiments of the present disclosure. [Figure 17] FIG. 2 illustrates an example of decoding side motion vector refinement according to some embodiments of the present disclosure. [Figure 18] FIG. 13 illustrates an example of GPM splits grouped by the same angle, according to some embodiments of the present disclosure. [Figure 19] FIG. 1 illustrates an example of uni-predictive MV selection for geometric partitioning mode, according to some embodiments of the present disclosure. [Figure 20] FIG. 2 illustrates examples of upper and left neighboring blocks used in CIIP weight derivation according to some embodiments of the present disclosure. [Figure 21] FIG. 2 illustrates an example of spatially adjacent blocks used to derive spatial merging candidates according to some embodiments of the present disclosure. [Figure 22] FIG. 13 illustrates an example of template matching performed on a search area around an initial MV according to some embodiments of the present disclosure. [Diagram 23] FIG. 1 illustrates an example of a diamond region within a search area according to some embodiments of the present disclosure. [Figure 24] FIG. 2 illustrates frequency responses of an interpolation filter at half pixel phase and a VVC interpolation filter according to some embodiments of the present disclosure. [Diagram 25] FIG. 2 is a diagram illustrating an example of a template and a reference sample for the template in a reference picture according to some embodiments of the present disclosure. [Figure 26]FIG. 2 illustrates an example of a template and reference samples for a block with sub-block motion using motion information of the sub-blocks of a current block according to some embodiments of the present disclosure. [Figure 27] FIG. 2 is a diagram illustrating an example of a padded reference picture for motion compensation according to some embodiments of the present disclosure. [Figure 28] FIG. 2 illustrates an example of fractional interpolation according to some embodiments of the present disclosure. [Figure 29] FIG. 1 is a block diagram illustrating a computing environment coupled with a user interface according to some embodiments of the present disclosure. [Diagram 30] 1 is a block diagram illustrating a video encoding process according to some embodiments of the present disclosure. [Diagram 31] FIG. 2 illustrates an example of a picture divided into CTUs, according to some embodiments of the present disclosure. [Diagram 32] FIG. 2 illustrates an example of a picture partitioned into tiles and raster scan slices, according to some embodiments of the present disclosure. [Diagram 33] FIG. 2 illustrates an example of a picture partitioned into tiles and rectangular slices according to some embodiments of the present disclosure. [Diagram 34] FIG. 2 illustrates an example of a picture partitioned into four tiles and four rectangular slices, according to some embodiments of the present disclosure. [Diagram 35] FIG. 2 illustrates an example of a picture partitioned into 28 sub-pictures according to some embodiments of the present disclosure. [Diagram 36] FIG. 2 illustrates an example of horizontal wraparound motion compensation in VVC according to some embodiments of the present disclosure. [Figure 37] 1 is a flowchart illustrating a video encoding method according to some embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0009] Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. The following description refers to the accompanying drawings, in which the same numbers in different drawings represent the same or similar elements unless otherwise stated. The implementations described in the following description of the exemplary embodiments do not represent all implementations consistent with the present disclosure. Rather, the implementations are merely examples of apparatuses and methods consistent with aspects related to the present disclosure as described in the appended claims.
[0010] The terms used in this disclosure are for the purpose of describing particular embodiments only and are not intended to be limiting of the disclosure. As used in this disclosure and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. It is also to be understood that the term "and / or" as used herein is intended to mean and include any and all possible combinations of one or more of the associated listed items.
[0011] It should be understood that terms such as "first", "second", and "third" may be used herein to describe various information, but the information should not be limited by these terms. These terms are used only to distinguish one category of information from another. For example, the first information may be referred to as the second information, and similarly, the second information may be referred to as the first information, without departing from the scope of this disclosure. The term "if" as used herein may be understood to mean "when", "in the event of", or "at the discretion of", depending on the context.
[0012] To compress video data, various video coding techniques may be used. Video coding is performed according to one or more video coding standards. For example, today, well-known video coding standards include Versatile Video Coding (VVC), High Efficiency Video Coding (HEVC, also known as H.265 or MPEG-H Part2), and Advanced Video Coding (AVC, also known as H.264 or MPEG-4 Part10), which are jointly developed by ISO / IEC MPEG and ITU-T VECG. AOMedia Video 1 (AV1) was developed by the Alliance for Open Media (AOM) as a successor to its previous standard VP9. Audio Video Coding (AVS), which refers to Digital Audio and Digital Video Compression Standard, is another video compression standard series developed by the Chinese Audio and Video Coding Standards Workgroup. Most of the existing video coding standards are built on the well-known hybrid video coding framework, i.e., using block-based prediction methods (e.g., inter-prediction, intra-prediction) to reduce the redundancy present in a video image or sequence, and transform coding to compact the energy of the prediction error. An important goal of video coding techniques is to compress video data into a format that uses a lower bitrate while avoiding or minimizing the loss of video quality.
[0013] The first generation AVS standard includes the Chinese national standard "Information Technology, Advanced Audio-Video Coding, Part 2: Video" (known as AVS1) and "Information Technology, Advanced Audio-Video Coding, Part 16: Radio-Television-Video" (known as AVS+). It can achieve about 50% bitrate savings with the same perceptual quality compared with the MPEG-2 standard. The video part of the AVS1 standard was promulgated as the Chinese national standard in February 2006. The second generation AVS standard includes a series of Chinese national standards "Information Technology, Efficient Multimedia Coding" (known as AVS2), which are mainly aimed at the transmission of additional HD TV programs. The coding efficiency of AVS2 is twice that of AVS+. In May 2016, AVS2 was issued as the Chinese national standard. Meanwhile, the video part of the AVS2 standard was put forward by the Institute of Electrical and Electronics Engineers (IEEE) as one of the international standards for applications. The AVS3 standard is one of the new generation video coding standards for UHD video applications, aiming to exceed the coding efficiency of the latest international standard HEVC. In March 2019, the 68th AVS Conference finalized the AVS3-P2 baseline, which achieves approximately 30% bitrate savings compared to the HEVC standard. Currently, there is one reference software, called the High Performance Model (HPM), maintained by the AVS group to demonstrate a reference implementation of the AVS3 standard.
[0014] 1 is a block diagram illustrating an example system 10 for encoding and decoding video blocks in parallel, according to some embodiments of the present disclosure. As shown in FIG. 1, system 10 includes a source device 12 that generates and encodes video data that is subsequently decoded by a destination device 14. Source device 12 and destination device 14 may include any of a wide variety of electronic devices, including desktop or laptop computers, tablet computers, smart phones, set-top boxes, digital televisions, cameras, display devices, digital media players, video gaming consoles, video streaming devices, and the like. In some implementations, source device 12 and destination device 14 are equipped with wireless communication capabilities.
[0015] In some implementations, destination device 14 may receive the encoded video data to be decoded via link 16. Link 16 may comprise any type of communication medium or device capable of moving the encoded video data from source device 12 to destination device 14. In one example, link 16 may comprise a communication medium that allows source device 12 to transmit the encoded video data directly to destination device 14 in real time. The encoded video data may be modulated according to a communication standard, such as a wireless communication protocol, and transmitted to destination device 14. The communication medium may include any wireless or wired communication medium, such as the radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form part of a packet-based network, such as a local area network, a wide area network, or a global network such as the Internet. The communication medium may include routers, switches, base stations, or any other equipment that may be useful in facilitating communication from source device 12 to destination device 14.
[0016] In some other implementations, the encoded video data may be transmitted from output interface 22 to storage device 32. The encoded video data in storage device 32 may then be accessed by destination device 14 via input interface 28. Storage device 32 may include any of a variety of distributed or locally accessed data storage media, such as a hard drive, a Blu-ray disc, a digital versatile disc (DVD), a compact disc read only memory (CD-ROM), flash memory, volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded video data. In a further example, storage device 32 may correspond to a file server or another intermediate storage device capable of holding the encoded video data generated by source device 12. Destination device 14 may access the stored video data from storage device 32 via streaming or download. The file server may be any type of computer capable of storing encoded video data and transmitting the encoded video data to destination device 14. Exemplary file servers include web servers (e.g., for websites), File Transfer Protocol (FTP) servers, Network Attached Storage (NAS) devices, or local disk drives. Destination device 14 may access the encoded video data through any standard data connection, including a wireless channel (e.g., a Wireless Fidelity (Wi-Fi) connection), a wired connection (e.g., a Digital Subscriber Line (DSL), cable modem, etc.), or a combination of both suitable for accessing encoded video data stored in a file server. Transmission of the encoded video data from storage device 32 may be a streaming transmission, a download transmission, or a combination of both.
[0017] As shown in FIG. 1, source device 12 includes a video source 18, a video encoder 20, and an output interface 22. Video source 18 may include sources such as a video capture device, e.g., a video camera, a video archive containing previously captured video, a video feed interface for receiving video from a video content provider, and / or a computer graphics system for generating computer graphics data as the source video, or a combination of such sources. As an example, if video source 18 is a video camera of a security surveillance system, source device 12 and destination device 14 may form a camera phone or a video phone. However, implementations described in this application may be applicable to video encoding generally and may be applicable to wireless and / or wired applications.
[0018] The captured, pre-captured, or computer-generated video may be encoded by video encoder 20. The encoded video data may be transmitted directly to destination device 14 via output interface 22 of source device 12. The encoded video data may also (or alternatively) be stored in storage device 32 for later access by destination device 14 or other devices for decoding and / or playback. Output interface 22 may further include a modem and / or a transmitter.
[0019] Destination device 14 includes an input interface 28, a video decoder 30, and a display device 34. Input interface 28 may include a receiver and / or modem and receive encoded video data over link 16. The encoded video data communicated over link 16 or provided on storage device 32 may include various syntax elements generated by video encoder 20 for use by video decoder 30 in decoding the video data. Such syntax elements may be included within the encoded video data transmitted over a communication medium, stored on a storage medium, or stored on a file server.
[0020] In some implementations, destination device 14 may include a display device 34, which may be an integrated display device as well as an external display device configured to communicate with destination device 14. Display device 34 displays the decoded video data to a user and may comprise any of a variety of display devices, such as a Liquid Crystal Display (LCD), a plasma display, an Organic Light Emitting Diode (OLED) display, or another type of display device.
[0021] Video encoder 20 and video decoder 30 may operate according to a proprietary or industry standard, such as VVC, HEVC, MPEG-4, Part 10, AVC, or extensions of such standards. It should be understood that the present application is not limited to a particular video encoding / decoding standard, but is applicable to other video encoding / decoding standards. It is generally contemplated that video encoder 20 of source device 12 may be configured to encode video data according to any of these current or future standards. Similarly, it is generally contemplated that video decoder 30 of destination device 14 may be configured to decode video data according to any of these current or future standards.
[0022] Each of the video encoder 20 and the video decoder 30 may be implemented as any of a variety of suitable encoder and / or decoder circuits, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combination thereof. If implemented partially in software, the electronic device may store instructions for the software on a suitable non-transitory computer readable medium and execute the instructions in hardware using one or more processors to perform the video encoding / decoding operations disclosed in this disclosure. Each of the video encoder 20 and the video decoder 30 may be included in one or more encoders or decoders, any of which may be integrated as part of a combined encoder / decoder (CODEC) within the respective device.
[0023] 2 is a block diagram illustrating an example video encoder 20 according to some embodiments described in the present application. Video encoder 20 may perform intra-predictive and inter-predictive coding of video blocks within video frames. Intra-predictive coding relies on spatial prediction to reduce or remove spatial redundancy in video data within a given video frame or picture. Inter-predictive coding relies on temporal prediction to reduce or remove temporal redundancy in video data within adjacent video frames or pictures of a video sequence. It should be noted that the term "frame" is sometimes used synonymously with the terms "image" or "picture" in the field of video coding.
[0024] As shown in FIG. 2, the video encoder 20 includes a video data memory 40, a prediction processing unit 41, a decoded picture buffer (DPB) 64, an adder 50, a transform processing unit 52, a quantization unit 54, and an entropy coding unit 56. The prediction processing unit 41 further includes a motion estimation unit 42, a motion compensation unit 44, a partition unit 45, an intra prediction processing unit 46, and an intra block copy (BC) unit 48. In some implementations, the video encoder 20 also includes an inverse quantization unit 58 for video block reconstruction, an inverse transform processing unit 60, and an adder 62. An in-loop filter 63, such as a deblocking filter, may be disposed between the adder 62 and the DPB 64 to filter block boundaries to remove blocky artifacts from the reconstructed video. In addition to the deblocking filter, another in-loop filter, such as a Sample Adaptive Offset (SAO) filter and / or an adaptive in-loop filter (ALF), may also be used to filter the output of summer 62. In some examples, the in-loop filter may be omitted and the decoded video block may be provided directly to DPB 64 by summer 62. Video encoder 20 may take the form of a fixed or programmable hardware unit, or may be divided into one or more of the illustrated fixed or programmable hardware units.
[0025] Video data memory 40 may store video data to be encoded by components of video encoder 20. The video data in video data memory 40 may be obtained, for example, from video source 18 shown in FIG. 1. DPB 64 is a buffer that stores reference video data (e.g., reference frames or pictures) for use in encoding video data by video encoder 20 (e.g., in an intra-predictive coding mode or an inter-predictive coding mode). Video data memory 40 and DPB 64 may be formed by any of a variety of memory devices. In various examples, video data memory 40 may be on-chip with other components of video encoder 20 or may be off-chip relative to those components.
[0026] As shown in FIG. 2, after receiving the video data, partition unit 45 in prediction processing unit 41 partitions the video data into video blocks. This partitioning may also include partitioning the video frame into slices, tiles (e.g., a set of video blocks), or other larger coding units (CUs) according to a predefined division structure, such as a Quad-Tree (QT) structure associated with the video data. A video frame may be or be considered as a two-dimensional array or matrix of samples having sample values. The samples in the array may also be referred to as pixels or pels. The number of samples in the horizontal and vertical directions (or axes) of the array or picture defines the size and / or resolution of the video frame. The video frame may be divided into multiple video blocks, for example, by using QT partitioning. A video block may also be or be considered as a two-dimensional array or matrix of samples having sample values, although with smaller dimensions than a video frame. The number of samples in the horizontal and vertical directions (or axes) of a video block defines the size of the video block. A video block may be further partitioned into one or more block partitions or sub-blocks (which may again form blocks), e.g., by repeatedly using QT partitions, Binary-Tree (BT) partitions, or Triple-Tree (TT) partitions, or any combination thereof. Note that the term "block" or "video block" as used herein may be a portion of a frame or picture, in particular a rectangular (square or non-square) portion. For example, with reference to HEVC and VVC, a block or video block may be or correspond to a coding tree unit (CTU), a CU, a prediction unit (PU), or a transform unit (TU), and / or may be or correspond to a corresponding block, e.g., a coding tree block (CTB), a coding block (CB), a prediction block (PB), or a transform block (TB), and / or a sub-block.
[0027] Prediction processing unit 41 may select one of a number of possible predictive coding modes, such as one of a number of intra-predictive coding modes or one of a number of inter-predictive coding modes, for the current video block based on the error result (e.g., the coding rate and level of distortion). Prediction processing unit 41 may provide the resulting intra-predictively coded block or inter-predictively coded block to summer 50 to generate a residual block and to summer 62 to reconstruct the coded block for later use as part of a reference frame. Prediction processing unit 41 also provides syntax elements, such as motion vectors, intra-mode indicators, partition information, and other such syntax information to entropy coding unit 56.
[0028] To select an appropriate intra-prediction coding mode for a current video block, intra-prediction processing unit 46 within prediction processing unit 41 may perform intra-prediction coding of the current video block relative to one or more neighboring blocks in the same frame as the current block to be coded to provide spatial prediction. Motion estimation unit 42 and motion compensation unit 44 within prediction processing unit 41 may perform inter-prediction coding of the current video block relative to one or more predictive blocks in one or more reference frames to provide temporal prediction. Video encoder 20 may perform multiple coding passes to select an appropriate coding mode for each block of video data, for example.
[0029] In some implementations, motion estimation unit 42 determines the inter prediction mode for the current video frame by generating motion vectors that indicate the displacement of video blocks in the current video frame relative to a predictive block in a reference video frame according to a predetermined pattern in the sequence of video frames. Motion estimation performed by motion estimation unit 42 is the process of generating motion vectors that estimate the motion of video blocks. The motion vectors may indicate, for example, the displacement of video blocks in a current video frame or picture relative to a predictive block in a reference frame relative to a current block being coded in the current frame. The predetermined pattern may designate a video frame in the sequence as a P frame or a B frame. Intra BC unit 48 may determine vectors, e.g., block vectors, for intra BC coding in a manner similar to the determination of motion vectors by motion estimation unit 42 for inter prediction, or may utilize motion estimation unit 42 to determine the block vectors.
[0030] A prediction block of a video block may be or correspond to a block of a reference frame or reference block that is deemed to closely match the video block to be encoded in terms of pixel differences, which may be determined by sum of absolute difference (SAD), sum of square difference (SSD), or other difference metric. In some implementations, video encoder 20 may calculate values of sub-integer pixel locations of the reference frame stored in DPB 64. For example, video encoder 20 may interpolate values of quarter-pixel, eighth-pixel, or other fractional pixel locations of the reference frame. Thus, motion estimation unit 42 may perform motion searches for whole and fractional pixel locations and output motion vectors with fractional pixel precision.
[0031] Motion estimation unit 42 calculates a motion vector for a video block in an inter-predictively coded frame by comparing the position of the video block to the position of a predictive block of a reference frame selected from a first reference frame list (List 0) or a second reference frame list (List 1), each of which identifies one or more reference frames stored in DPB 64. Motion estimation unit 42 sends the calculated motion vector to motion compensation unit 44 and then to entropy coding unit 56.
[0032] The motion compensation performed by motion compensation unit 44 may include fetching or generating a predictive block based on the motion vector determined by motion estimation unit 42. Upon receiving the motion vector for the current video block, motion compensation unit 44 may locate the predictive block to which the motion vector points in one of the reference frame lists, obtain the predictive block from DPB 64, and forward the predictive block to summer 50. Summer 50 then forms a residual video block of pixel difference values by subtracting pixel values of the predictive block provided by motion compensation unit 44 from pixel values of the current video block being coded. The pixel difference values forming the residual video block may include luma or chroma difference components, or both. Motion compensation unit 44 may also generate syntax elements associated with the video blocks of the video frames for use by video decoder 30 in decoding the video blocks of the video frames. The syntax elements may include, for example, syntax elements defining the motion vector used to identify the predictive block, any flags indicating a prediction mode, or any other syntax information described herein. It should be noted that motion estimation unit 42 and motion compensation unit 44 may be highly integrated, but are illustrated separately for conceptual purposes.
[0033] In some implementations, intra BC unit 48 may generate a vector and fetch a predictive block in a manner similar to that described above in connection with motion estimation unit 42 and motion compensation unit 44, except that the predictive block is in the same frame as the current block being coded, and the vector is referred to as a block vector rather than a motion vector. Specifically, intra BC unit 48 may determine an intra prediction mode to use to code the current block. In some examples, intra BC unit 48 may code the current block using various intra prediction modes, e.g., during separate coding passes, and test their performance through rate-distortion analysis. Intra BC unit 48 may then select an appropriate intra prediction mode to use from among the various tested intra prediction modes and generate an intra mode indicator accordingly. For example, intra BC unit 48 may calculate rate-distortion values using the rate-distortion analysis for the various tested intra prediction modes, and select the intra prediction mode with the best rate-distortion characteristics from among the tested modes as the appropriate intra prediction mode to use. The rate-distortion analysis generally determines the amount of distortion (or error) between a coded block and the original uncoded block that was coded to create the coded block, and the bitrate (i.e., number of bits) used to create the coded block. Intra BC unit 48 may calculate ratios from the distortions and rates of the various coded blocks to determine which intra prediction mode exhibits the best rate-distortion value for the block.
[0034] In other examples, intra BC unit 48 may use, in whole or in part, motion estimation unit 42 and motion compensation unit 44 to perform such functions for intra BC prediction according to embodiments described herein. In either case, for intra block copying, the predictive block may be a block that is deemed to closely match the block to be coded in terms of pixel differences, which may be determined by SAD, SSD, or other difference metrics, and identification of the predictive block may include calculation of values of sub-integer pixel positions.
[0035] Regardless of whether the predictive block is a block from the same frame via intra prediction or a block from a different frame via inter prediction, video encoder 20 may form a residual video block by subtracting pixel values of the predictive block from pixel values of the current video block being encoded to form pixel difference values. The pixel difference values that form the residual video block may include both luma and chroma component differences.
[0036] Intra-prediction processing unit 46 may intra-predict the current video block as an alternative to inter-prediction performed by motion estimation unit 42 and motion compensation unit 44, or intra-block copy prediction performed by intra BC unit 48, as described above. Specifically, intra-prediction processing unit 46 may determine an intra-prediction mode to use to encode the current block. To do so, intra-prediction processing unit 46 may encode the current block using various intra-prediction modes, e.g., during separate encoding passes, and intra-prediction processing unit 46 (or, in some examples, a mode selection unit) may select an appropriate intra-prediction mode to use from the tested intra-prediction modes. Intra-prediction processing unit 46 may provide information indicating the selected intra-prediction mode for the block to entropy coding unit 56. Entropy coding unit 56 may encode the information indicating the selected intra-prediction mode into the bitstream.
[0037] After prediction processing unit 41 determines the predictive block for the current video block by inter- or intra-prediction, adder 50 forms a residual video block by subtracting the predictive block from the current video block. The residual video data in the residual block, which may be included in one or more TUs, is provided to transform processing unit 52. Transform processing unit 52 converts the residual video data into residual transform coefficients using a transform, such as a Discrete Cosine Transform (DCT) or a conceptually similar transform.
[0038] Transform processing unit 52 may send the resulting transform coefficients to quantization unit 54, which quantizes the transform coefficients to further reduce the bit rate. The quantization process may also reduce the bit depth associated with some or all of the coefficients. The degree of quantization may be modified by adjusting a quantization parameter. In some examples, quantization unit 54 may then perform a scan of a matrix including the quantized transform coefficients. Alternatively, entropy coding unit 56 may perform the scan.
[0039] Following quantization, entropy coding unit 56 entropy codes the quantized transform coefficients into a video bitstream using, for example, Context Adaptive Variable Length Coding (CAVLC), Context Adaptive Binary Arithmetic Coding (CABAC), Syntax-based context-adaptive Binary Arithmetic Coding (SBAC), Probability Interval Partitioning Entropy (PIPE) coding, or another entropy coding methodology or technique. The coded bitstream may then be transmitted to video decoder 30 as shown in FIG. 1 or archived to storage device 32 as shown in FIG. 1 for later transmission to or retrieval by video decoder 30. Entropy coding unit 56 may also entropy code motion vectors and other syntax elements of the current video frame being coded.
[0040] Inverse quantization unit 58 and inverse transform processing unit 60 apply inverse quantization and inverse transform, respectively, to reconstruct the residual video block in the pixel domain to generate reference blocks for prediction of other video blocks. As described above, motion compensation unit 44 may generate motion compensated prediction blocks from one or more reference blocks of frames stored in DPB 64. Motion compensation unit 44 may also apply one or more interpolation filters to the prediction block to calculate sub-integer pixel values for use in motion estimation.
[0041] Summer 62 adds the reconstructed residual block to the motion compensated predictive block produced by motion compensation unit 44 to create a reference block for storage in DPB 64. The reference block may then be used by intra BC unit 48, motion estimation unit 42, and motion compensation unit 44 as a predictive block to inter predict another video block in a subsequent video frame.
[0042] 3 is a block diagram illustrating an example video decoder 30 according to some embodiments of the present application. The video decoder 30 includes a video data memory 79, an entropy decoding unit 80, a prediction processing unit 81, an inverse quantization unit 86, an inverse transform processing unit 88, an adder 90, and a DPB 92. The prediction processing unit 81 further includes a motion compensation unit 82, an intra prediction unit 84, and an intra BC unit 85. The video decoder 30 may perform a decoding process that is generally inverse to the encoding process described above for the video encoder 20 in connection with FIG. 2. For example, the motion compensation unit 82 may generate prediction data based on a motion vector received from the entropy decoding unit 80, while the intra prediction unit 84 may generate prediction data based on an intra prediction mode indicator received from the entropy decoding unit 80.
[0043] In some examples, the units of the video decoder 30 may be tasked with performing implementations of the present application. Also, in some examples, implementations of the present disclosure may be divided into one or more of the units of the video decoder 30. For example, the intra BC unit 85 may perform implementations of the present application alone or in combination with other units of the video decoder 30, such as the motion compensation unit 82, the intra prediction unit 84, and the entropy decoding unit 80. In some examples, the video decoder 30 may not include the intra BC unit 85, and the functions of the intra BC unit 85 may be performed by other components of the prediction processing unit 81, such as the motion compensation unit 82.
[0044] Video data memory 79 may store video data, such as an encoded video bitstream, that is decoded by other components of video decoder 30. The video data stored in video data memory 79 may be obtained, for example, from storage device 32, from a local video source such as a camera, via wired or wireless network communication of video data, or by accessing a physical data storage medium (e.g., a flash drive or hard disk). Video data memory 79 may include a coded picture buffer (CPB) that stores coded video data from the coded video bitstream. DPB 92 of video decoder 30 stores reference video data for use in decoding video data by video decoder 30 (e.g., in an intra-predictive coding mode or an inter-predictive coding mode). Video data memory 79 and DPB 92 may be formed by any of a variety of memory devices, such as dynamic random access memory (DRAM), including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. 3, video data memory 79 and DPB 92 are depicted as two separate components of video decoder 30. However, it will be apparent to one skilled in the art that video data memory 79 and DPB 92 may be provided by the same memory device or separate memory devices. In some examples, video data memory 79 may be on-chip with other components of video decoder 30 or may be off-chip with respect to those components.
[0045] During the decoding process, video decoder 30 receives an encoded video bitstream representing video blocks and associated syntax elements of encoded video frames. Video decoder 30 may receive the syntax elements at the video frame level and / or the video block level. Entropy decoding unit 80 of video decoder 30 entropy decodes the bitstream to generate quantized coefficients, motion vectors or intra-prediction mode indicators, and other syntax elements. Entropy decoding unit 80 then forwards the motion vectors or intra-prediction mode indicators and other syntax elements to prediction processing unit 81.
[0046] When a video frame is coded as an intra-predictive coded (I) frame or for intra-coded predictive blocks in other types of frames, intra prediction unit 84 of prediction processing unit 81 may generate predictive data for video blocks of the current video frame based on a signaled intra-prediction mode and reference data from previously decoded blocks of the current frame.
[0047] When a video frame is coded as an inter-predictive (i.e., B or P) frame, motion compensation unit 82 of prediction processing unit 81 creates one or more prediction blocks for video blocks of the current video frame based on the motion vectors and other syntax elements received from entropy decoding unit 80. Each of the prediction blocks may be created from a reference frame in one of the reference frame lists. Video decoder 30 may construct the reference frame lists, List 0 and List 1, using a default construction technique based on the reference frames stored in DPB 92.
[0048] In some examples, when a video block is encoded according to the intra BC mode described herein, intra BC unit 85 of prediction processing unit 81 creates a predictive block of the current video block based on the block vectors and other syntax elements received from entropy decoding unit 80. The predictive block may be within the same reconstructed region of the picture as the current video block defined by video encoder 20.
[0049] Motion compensation unit 82 and / or intra BC unit 85 determine prediction information for video blocks of the current video frame by analyzing the motion vectors and other syntax elements, and then use the prediction information to create a predictive block for the current video block being decoded. For example, motion compensation unit 82 uses some of the received syntax elements to determine a prediction mode (e.g., intra prediction or inter prediction) used to encode the video blocks of the video frame, an inter prediction frame type (e.g., B or P), construction information regarding one or more of the reference frame lists for the frame, a motion vector for each inter predictive coded video block of the frame, an inter prediction status for each inter predictive coded video block of the frame, and other information for decoding video blocks in the current video frame.
[0050] Similarly, intra BC unit 85 may use some of the received syntax elements, such as flags, to determine that the current video block was predicted using intra BC mode, construction information regarding which video blocks of the frame are within the reconstruction region and should be stored in DPB 92, block vectors for each intra BC predicted video block of the frame, intra BC prediction status for each intra BC predicted video block of the frame, and other information for decoding video blocks in the current video frame.
[0051] Motion compensation unit 82 may also perform interpolation using an interpolation filter used by video encoder 20 during encoding of the video block to calculate sub-integer pixel interpolated values of the reference block. In this case, motion compensation unit 82 may determine the interpolation filter used by video encoder 20 from the received syntax element and use the interpolation filter to create the predictive block.
[0052] Inverse quantization unit 86 inverse quantizes the quantized transform coefficients provided in the bitstream and entropy decoded by entropy decoding unit 80 using the same quantization parameters calculated by video encoder 20 for each video block in a video frame to determine the degree of quantization. Inverse transform processing unit 88 applies an inverse transform, e.g., an inverse DCT, an inverse integer transform, or a conceptually similar inverse transform process, to the transform coefficients in order to reconstruct residual blocks in the pixel domain.
[0053] After motion compensation unit 82 or intra BC unit 85 generates a predictive block for the current video block based on the vectors and other syntax elements, summer 90 reconstructs a decoded video block for the current video block by adding the residual block from inverse transform processing unit 88 and the corresponding predictive block generated by motion compensation unit 82 and intra BC unit 85. An in-loop filter 91, such as a deblocking filter, an SAO filter, and / or an ALF, may be disposed between summer 90 and the DPB to further process the decoded video block. In some examples, in-loop filter 91 may be omitted and the decoded video block may be provided directly to DPB 92 by summer 90. The decoded video block in a given frame is then stored in DPB 92, which stores reference frames used for subsequent motion compensation of the next video block. DPB 92 or a memory device separate from DPB 92 may store the decoded video for later presentation on a display device, such as display device 34 of FIG. 1.
[0054] In a typical video encoding process, a video sequence typically includes an ordered set of frames or pictures. Each frame may include three sample arrays, denoted SL, SCb, and SCr. SL is a two-dimensional array of luma samples. SCb is a two-dimensional array of Cb chroma samples. SCr is a two-dimensional array of Cr chroma samples. In another example, a frame may be monochromatic and thus include only one two-dimensional array of luma samples.
[0055] As shown in FIG. 4A, the video encoder 20 (or, more specifically, the partitioning unit 45) generates an encoded representation of a frame by first partitioning the frame into a set of CTUs. A video frame may include an integer number of CTUs ordered consecutively from left to right and from top to bottom in raster scan order. Each CTU is the largest logical coding unit, and the width and height of the CTUs are signaled by the video encoder 20 in a sequence parameter set such that all CTUs in a video sequence have the same size, which is any one of 128×128, 64×64, 32×32, and 16×16. However, it should be noted that the present application is not necessarily limited to a particular size. As shown in FIG. 4B, each CTU may include one CTB of luma samples, two corresponding coding tree blocks of chroma samples, and syntax elements used to code the samples of the coding tree blocks. The syntax elements describe how a video sequence may be reconstructed in the video decoder 30, including characteristics of different types of units of coded blocks of pixels, as well as inter or intra prediction, intra prediction modes, motion vectors, and other parameters. For a monochromatic picture or a picture with three separate color planes, a CTU may include a single coding tree block and syntax elements used to code samples of the coding tree block. A coding tree block may be an N×N block of samples.
[0056] To achieve better performance, video encoder 20 may recursively perform tree partitioning, such as binary tree partitioning, ternary tree partitioning, quad tree partitioning, or a combination thereof, on the coding tree blocks of the CTU to partition the CTU into smaller CUs. As depicted in FIG. 4C, 64×64 CTU 400 is first partitioned into four smaller CUs, each with a block size of 32×32. Of the four smaller CUs, CU 410 and CU 420 are each partitioned by block size into four 16×16 CUs. The two 16×16 CUs 430 and CU 440 are each further partitioned by block size into four 8×8 CUs. FIG. 4D depicts a quad tree data structure showing the final result of the partitioning process of CTU 400 depicted in FIG. 4C, where each leaf node of the quad tree corresponds to one CU of a respective size ranging from 32×32 to 8×8. Similar to the CTU depicted in FIG. 4B, each CU may include a CB of luma samples, two corresponding coding blocks of chroma samples for the same size frame, and syntax elements used to code the samples of the coding blocks. In a monochromatic picture or a picture with three separate color planes, a CU may include a single coding block and syntax structures used to code the samples of the coding blocks. It should be noted that the quad-tree partition depicted in FIG. 4C and FIG. 4D is for illustration purposes only, and one CTU may be split into CUs based on quad-tree / ternary-tree / binary-tree partitions to accommodate various local characteristics. In a multi-type tree structure, one CTU is partitioned by a quad-tree structure, and each quad-tree leaf CU may be further partitioned by a binary tree structure and a ternary tree structure. As shown in FIG. 4E, there are five possible partition types of a coding block with width W and height H, namely, 4-partition, horizontal 2-partition, vertical 2-partition, horizontal 3-partition, and vertical 3-partition.
[0057] In some implementations, video encoder 20 may further partition the coding blocks of a CU into one or more M×N PBs. A PB is a rectangular (square or non-square) block of samples to which the same prediction, inter- or intra-prediction, is applied. A PU of a CU may include a PB of luma samples, two corresponding PBs of chroma samples, and syntax elements used to predict the PB. In a monochromatic picture or a picture with three separate color planes, a PU may include a single PB and syntax structures used to predict the PB. Video encoder 20 may generate predictive luma blocks, predictive Cb blocks, and predictive Cr blocks for the luma, Cb, and Cr PBs of each PU of a CU.
[0058] Video encoder 20 may generate the predictive blocks for a PU using intra prediction or inter prediction. If video encoder 20 generates the predictive blocks for a PU using intra prediction, video encoder 20 may generate the predictive blocks for the PU based on decoded samples of a frame associated with the PU. If video encoder 20 generates the predictive blocks for a PU using inter prediction, video encoder 20 may generate the predictive blocks for the PU based on decoded samples of one or more frames other than the frame associated with the PU.
[0059] After video encoder 20 generates the predictive luma block, the predictive Cb block, and the predictive Cr block for one or more PUs of a CU, video encoder 20 may generate a luma residual block for the CU by subtracting the predictive luma block of the CU from its original luma coding block, such that each sample in the luma residual block of the CU indicates a difference between a luma sample in one of the predictive luma blocks of the CU and a corresponding sample in the original luma coding block of the CU. Similarly, video encoder 20 may generate a Cb residual block and a Cr residual block for the CU, such that each sample in the Cb residual block of the CU indicates a difference between a Cb sample in one of the predictive Cb blocks of the CU and a corresponding sample in the original Cb coding block of the CU, and such that each sample in the Cr residual block of the CU indicates a difference between a Cr sample in one of the predictive Cr blocks of the CU and a corresponding sample in the original Cr coding block of the CU, respectively.
[0060] Further, as shown in FIG. 4C, video encoder 20 may use quadtree partitioning to decompose the luma residual block, Cb residual block, and Cr residual block of a CU into one or more luma transform blocks, Cb transform blocks, and Cr transform blocks, respectively. A transform block is a rectangular (square or non-square) block of samples to which the same transform is applied. A TU of a CU may include a transform block of luma samples and two corresponding transform blocks of chroma samples, and syntax elements used to transform the transform block samples. Thus, each TU of a CU may be associated with a luma transform block, a Cb transform block, and a Cr transform block. In some examples, the luma transform block associated with a TU may be a sub-block of the luma residual block of the CU. The Cb transform block may be a sub-block of the Cb residual block of the CU. The Cr transform block may be a sub-block of the Cr residual block of the CU. In a monochromatic picture or a picture with three separate color planes, a TU may include a single transform block and syntax structures used to transform samples of the transform block.
[0061] Video encoder 20 may apply one or more transforms to a luma transform block of the TU to generate a luma coefficient block of the TU. The coefficient block may be a two-dimensional array of transform coefficients. The transform coefficients may be scalar quantities. Video encoder 20 may apply one or more transforms to a Cb transform block of the TU to generate a Cb coefficient block of the TU. Video encoder 20 may apply one or more transforms to a Cr transform block of the TU to generate a Cr coefficient block of the TU.
[0062] After generating a coefficient block (e.g., a luma coefficient block, a Cb coefficient block, or a Cr coefficient block), the video encoder 20 may quantize the coefficient block. Quantization generally refers to a process in which transform coefficients are quantized to potentially reduce the amount of data used to represent the transform coefficients to achieve further compression. After the video encoder 20 quantizes the coefficient block, the video encoder 20 may entropy code syntax elements indicating the quantized transform coefficients. For example, the video encoder 20 may perform CABAC on the syntax elements indicating the quantized transform coefficients. Finally, the video encoder 20 may output a bitstream including a sequence of bits forming a representation of the encoded frame and associated data, which is stored in the storage device 32 or transmitted to the destination device 14.
[0063] After receiving the bitstream generated by video encoder 20, video decoder 30 may parse the bitstream to obtain syntax elements from the bitstream. Video decoder 30 may reconstruct a frame of video data based at least in part on the syntax elements obtained from the bitstream. The process of reconstructing the video data is approximately the reverse of the encoding process performed by video encoder 20. For example, video decoder 30 may perform an inverse transform on coefficient blocks associated with TUs of the current CU to reconstruct residual blocks associated with TUs of the current CU. Video decoder 30 also reconstructs coding blocks of the current CU by adding samples of the predictive blocks of the PUs of the current CU to corresponding samples of the transform blocks of the TUs of the current CU. Video decoder 30 may reconstruct the frame after reconstructing the coding blocks for each CU of the frame.
[0064] As mentioned above, video coding achieves video compression mainly using two modes: intraframe prediction (or intra prediction) and interframe prediction (or inter prediction). It should be noted that IBC can be considered as intraframe prediction or a third mode. Of the two modes, interframe prediction contributes more to coding efficiency than intraframe prediction because it uses motion vectors to predict a current video block from a reference video block.
[0065] However, as video data capture technology is constantly improving and video block sizes are becoming finer to preserve the details of the video data, the amount of data required to represent the motion vectors of the current frame is also increasing significantly. One way to overcome this challenge is to benefit from the fact that a group of adjacent CUs in both spatial and temporal domains not only have similar video data for prediction purposes, but also the motion vectors between these adjacent CUs are similar. Therefore, the motion information of spatially adjacent CUs and / or temporally co-located CUs can be used as an approximation of the motion information (e.g., motion vector) of the current CU by investigating their spatial and temporal correlations, which is also called the "Motion Vector Predictor (MVP)" of the current CU.
[0066] Instead of encoding the actual motion vector of the current CU determined by motion estimation unit 42 into the video bitstream as described above in connection with Figure 2, the motion vector predictor of the current CU is subtracted from the actual motion vector of the current CU to create a Motion Vector Difference (MVD) for the current CU. By doing so, the motion vectors determined by motion estimation unit 42 for each CU of a frame do not need to be encoded into the video bitstream, and the amount of data used to represent motion information in the video bitstream may be significantly reduced.
[0067] Similar to the process of choosing a predictive block in a reference frame during inter-frame prediction of a codeblock, a set of rules needs to be adopted by both video encoder 20 and video decoder 30 to build a motion vector candidate list (also called a "merge list") for a current CU using potential candidate motion vectors associated with spatially neighboring CUs and / or temporally co-located CUs of the current CU, and then select one element from the motion vector candidate list as the motion vector predictor for the current CU. By doing so, the motion vector candidate list itself does not need to be transmitted from video encoder 20 to video decoder 30, and the index of the selected motion vector predictor in the motion vector candidate list is sufficient for video encoder 20 and video decoder 30 to use the same motion vector predictor in the motion vector candidate list to encode and decode the current CU.
[0068] For each inter-predicted CU, the motion parameters consist of a motion vector, a reference picture index and a reference picture list usage index, as well as additional information required for the new coding features of VVC used for inter-prediction sample generation. The motion parameters can be signaled in an explicit or implicit manner. If a CU is coded in skip mode, it is associated with one PU and has no significant residual coefficients, no coded motion vector deltas, and no reference picture indexes. A merge mode is specified, whereby the motion parameters of the current CU, including spatial and temporal candidates and additional schedules introduced in VVC, are obtained from neighboring CUs. The merge mode can be applied to any inter-predicted CU, not just skip mode. An alternative to the merge mode is the explicit transmission of motion parameters, where the motion vector, the corresponding reference picture index of each reference picture list, the reference picture list usage flag, and other required information are explicitly signaled for each CU.
[0069] Additionally, ITU-T VCEG (Q6 / 16) and ISO / IEC MPEG (JTC1 / SC29 / WG11) are studying the potential need for standardization of future video coding technologies with compression capabilities significantly beyond those of the current VVC standard. Such future standardization actions could take the form of additional extensions to VVC or entirely new standards. These groups are working together on this search in a collaborative effort known as the Joint Video Exploration Team (JVET) to evaluate compression technology designs proposed by experts in the field. The first exploration experiment (EE) has been established at the JVET meeting on January 6-15, 2021, and this exploration software model has been named the Enhanced Compression Model (ECM), with ECM version 2 (ECM2) being published in August 2021. The newly developed inter-prediction scheme in ECM2 is described in detail in the next section.
[0070] The following sections provide details of the VVC and its inter-prediction methodology as specified in the ECM model under development.
[0071] Enhanced Merge Prediction in VVC
[0072] In VVC, a merge candidate list is constructed by including, in order, the following five types of candidates: 1) Spatial MVP from spatially adjacent CUs 2) Temporal MVP from collocated CUs 3) History-based MVP from FIFO tables 4) Average MVP per pair 5) Zero's music video
[0073] The size of the merge list is signaled in the Sequence Parameter Set header, and the maximum allowed size of the merge list is 6. For each CU code in the merge mode, the index of the best merge candidate is encoded using truncated unary binarization (TU). The first bin of the merge index is encoded with the context, and bypass encoding is used for the other bins.
[0074] In this section, the derivation process for each category of merge candidates is provided. As is done in HEVC, VVC also supports parallel derivation of merge candidate lists for all CUs within an area of a certain size.
[0075] Spatial Candidate Derivation in VVC
[0076] The derivation of spatial merge candidates in VVC is the same as that in HEVC, except that the positions of the first two merge candidates are swapped. Up to four merge candidates are selected from among the candidates at the positions depicted in FIG. 5. The order of derivation is B0, A0, B1, A1, and B2. Position B2 is considered only if one or more CUs at positions B0, A0, B1, A1 are unavailable (e.g., because they belong to another slice or tile) or are intra-coded. After the candidate at position A1 is added, the addition of the remaining candidates is subject to a redundancy check that ensures that candidates with the same motion information are removed from the list so that the coding efficiency is improved. To reduce computational complexity, the aforementioned redundancy check does not consider all possible candidate pairs. Instead, only the pairs linked by arrows in FIG. 6 are considered, and a candidate is added to the list only if the corresponding candidates used for the redundancy check do not have the same motion information.
[0077] Temporal candidate derivation in VVC
[0078] In this step, only one candidate is added to the list. In particular, in this derivation of the temporal merge candidate, a scaled motion vector is derived based on the array CU belonging to the co-located reference picture. The reference picture list and reference index used to derive the array CU are explicitly signaled in the slice header. The scaled motion vector of the temporal merge candidate is obtained as shown by the dotted line in FIG. 7, and this motion vector is scaled from the motion vector of the array CU using POC distances tb and td, where tb is defined to be the POC difference between the reference picture of the current picture and the current picture, and td is defined to be the POC difference between the reference picture of the array picture and the array picture. The reference picture index of the temporal merge candidate is set equal to zero.
[0079] The location of the temporal candidate is selected between candidates C0 and C1, as depicted in Figure 8. If the CU at location C0 is unavailable, is intra-coded, or is outside the current row of the CTU, location C1 is used. Otherwise, location C0 is used to derive the temporal merge candidate.
[0080] History-Based Merge Candidate Derivation in VVC
[0081] History-based MVP (HMVP) merge candidates are added to the merge list after spatial MVP and TMVP. In this method, the motion information of a previously coded block is stored in a table and used as the MVP of the current CU. During the encoding / decoding process, a table with multiple HMVP candidates is maintained. When a new CTU row occurs, the table is reset (emptied). Whenever there is a non-subblock inter-coded CU, the associated motion information is added to the last entry of the table as a new HMVP candidate.
[0082] The HMVP table size S is set to 6, which indicates that up to five History-Based MVP (HMVP) candidates can be added to the table. When inserting a new motion candidate into the table, a constrained First-In-First-Out (FIFO) rule is utilized, and a redundancy check is first applied to find if there is an identical HMVP in the table. If found, the identical HMVP is removed from the table, all subsequent HMVP candidates are moved forward, and the identical HMVP is inserted into the last entry of the table.
[0083] HMVP candidates may be used in the merge candidate list construction process. The most recent few HMVP candidates in the table are checked in order and inserted after the TMVP candidate in the candidate list. Redundancy checks are applied to spatial or temporal merge candidates in the HMVP candidates.
[0084] To reduce the number of redundancy checking operations, the following simplifications are introduced.
[0085] 1. The last two entries in the table are redundancy checked against the A1 and B1 space candidates, respectively.
[0086] 2. The process of building a merge candidate list from HMVP ends when the total number of available merge candidates reaches the maximum allowed merge candidates minus one.
[0087] Pairwise Average Merging Candidate Derivation in VVC
[0088] The pairwise average candidate is generated by averaging the predefined candidate pairs in the existing merge candidate list using the first two merge candidates. Respectively, the first merge candidate may be defined as p0Cand and the second merge candidate as p1Cand. The averaged motion vector is calculated separately for each reference list depending on the availability of motion vectors in p0Cand and p1Cand. If both motion vectors are available in one list, these two motion vectors are averaged even if they point to different reference pictures, and the reference picture is set to the one in p0Cand; if only one motion vector is available, it is used directly and if no motion vector is available, this list is left invalid. Also, if the half pixel interpolation filter index of p0Cand and p1Cand is different, it is set to 0.
[0089] If the merge list is not full after the pairwise average merge candidates are added, MVPs of zeros are inserted at the end until the maximum number of merge candidates is reached.
[0090] High-precision (1 / 16 pixel) motion compensation and storage of motion vectors in VVC.
[0091] VVC enhances the MV accuracy up to 1 / 16 luma sample to improve prediction efficiency for slow-motion videos. This higher motion accuracy is especially useful for video content with locally varying non-translational motion, such as in the case of affine mode. For higher MV accuracy fractional position sample generation, HEVC's 8-tap luma and 4-tap chroma interpolation filters are extended to 16-phase for luma and 32-phase for chroma. This extended filter set is applied to the MC process for inter-coded CUs except for affine mode CUs. For affine mode, a set of 6-tap luma interpolation filters with 16-phase is used to reduce computational complexity and save memory bandwidth.
[0092] In VVC, the highest precision of explicitly signaled motion vectors for non-affine CUs is 1 / 4 luma sample. In some inter prediction modes, such as affine mode, motion vectors may be signaled at 1 / 16 luma sample precision. In all inter coded CUs with implicitly inferred MVs, the MVs are derived at 1 / 16 luma sample precision and motion compensation prediction is performed at 1 / 16 sample precision. For storage of the intra motion field, all motion vectors are stored at 1 / 16 luma sample precision.
[0093] For the temporal motion field storage used by TMVP and SbTVMP, the motion field compression is performed at a granularity of 8x8 size, as opposed to the granularity of 16x16 size in HEVC.
[0094] Merge Mode with MVD in VVC (MMVD)
[0095] In addition to the merge mode where the implicitly derived motion information is directly used to generate the prediction samples of the current CU, the merge mode with motion vector difference (MMVD) is introduced in VVC. Immediately after sending the normal merge flag, the MMVD flag is signaled to the CU to specify whether the MMVD mode is used.
[0096] In MMVD, after a merge candidate is selected, the merge candidate is further refined by signaled MVD information. The further information includes a merge candidate flag, an index for specifying the magnitude of motion, and an index for motion direction indication. In MMVD mode, one of the first two candidates in the merge list is selected to be used as the MV basis. An mmvd candidate flag is signaled to specify which of the first and second merge candidates is used.
[0097] The distance index specifies the information of the magnitude of the movement and indicates a predefined offset from the starting point. As shown in Figures 9A and 9B, the offset is added to either the horizontal or vertical component of the starting MV. Table 1 specifies the relationship between the distance index and the predefined offset.
[0098] [Table 1]
[0099] The direction index represents the direction of the MVD relative to the starting point. The direction index can represent four directions shown in Table 2. Note that the meaning of the MVD code may change according to the information of the starting MV. If the starting MV is a uni-predictive MV or a bi-predictive MV and both lists point to the same side of the current picture (i.e., the POCs of the two references are both greater than the POC of the current picture or both less than the POC of the current picture), the code in Table 2 specifies the sign of the MV offset added to the starting MV. If the starting MV is a bi-predictive MV and the two MVs point to different sides of the current picture (i.e., the POC of one reference is greater than the POC of the current picture and the POC of the other reference is less than the POC of the current picture), if the difference of the POC in list 0 is greater than the difference of the POC in list 1, the code in Table 2 specifies the sign of the MV offset added to the MV component of the starting MV of list 0, and the sign of the MV in list 1 has an opposite value. Otherwise, if the difference in POCs in List 1 is greater than the difference in POCs in List 0, the sign in Table 2 specifies the sign of the MV offset added to the MV component of the starting MV of List 1, and the sign of the MV of List 0 has the opposite value.
[0100] The MVD is scaled according to the POC difference in each direction. If the POC difference in both lists is the same, no scaling is required. Otherwise, if the POC difference in list 0 is larger than the POC difference in list 1, the MVD of list 1 is scaled by defining the POC difference of L0 as td and the POC difference of L1 as tb as described in Figure 7. If the POC difference of L1 is larger than L0, the MVD of list 0 is also scaled in the same way. If the starting MV is uni-predicted, the MVD is added to the available MV.
[0101] [Table 2]
[0102] Symmetric MVD coding in VVC
[0103] In VVC, besides the usual MVD signaling for uni-predictive and bi-predictive modes, a symmetric MVD mode for bi-predictive MVD signaling is applied. In the symmetric MVD mode, motion information including reference picture indices for both list 0 and list 1 and the MVD for list 1 are derived without being signaled.
[0104] The decoding process for symmetric MVD mode is as follows:
[0105] 1. At the slice level, the variables BiDirPredFlag, RefIdxSymL0, and RefIdxSymL1 are derived as follows: - If mvd_l1_zero_flag is 1, then BiDirPredFlag is set equal to 0. - Otherwise, if the closest reference picture in list 0 and the closest reference picture in list 1 form a forward-backward or backward-forward reference picture pair, BiDirPredFlag is set to 1 and the reference pictures in both list 0 and list 1 are short-term reference pictures. Otherwise, BiDirPredFlag is set to 0.
[0106] 2. At the CU level, if the CU is bi-predictively coded and BiDirPredFlag is equal to 1, then a symmetric mode flag is explicitly signaled indicating whether symmetric mode is used or not.
[0107] If the symmetric mode flag is true, then only mvp_l0_flag, mvp_l1_flag, and MVD0 are explicitly signaled. The reference indexes of list0 and list1 are set equal to their respective reference picture pairs. MVD1 is set equal to (-MVD0). The final motion vector is given in the following equation:
number
[0108] At the encoder, symmetric MVD motion estimation starts with initial MV estimation. The set of initial MV candidates consists of the MVs obtained from the uni-predictive search, the MVs obtained from the bi-predictive search, and the MVs from the AMVP list. The one with the lowest rate-distortion cost is selected as the initial MV for the symmetric MVD motion search.
[0109] Affine motion compensation prediction in VVC
[0110] In HEVC, only translational motion model is applied for motion compensation prediction (MCP). Meanwhile, in the real world, there exist many kinds of motion, such as zoom, rotation, perspective motion, and other irregular motion. In VVC, block-based affine transformation motion compensation prediction is applied. As shown in Fig. 10A and Fig. 10B, the affine motion field of a block is described by the motion information of two control points (4 parameters) as shown in Fig. 10A or three control points motion vectors (6 parameters) as shown in Fig. 10B.
[0111] For the four-parameter affine motion model, the motion vector at sample position (x,y) in a block is
number
[0112] For the six-parameter affine motion model, the motion vector at sample position (x,y) in a block is
number
[0113] Here, (mv 0x ,mv 0y ) is the motion vector of the upper left corner control point, and (mv 1x ,mv 1y ) is the motion vector of the control point in the upper right corner, and (mv 2x ,mv 2y ) is the motion vector of the bottom-left corner control point.
[0114] To simplify the motion compensated prediction, block-based affine transformation prediction is applied. To derive the motion vector for each 4x4 luma subblock, the motion vector of the center sample of each subblock as shown in Figure 11 is calculated according to the above equation and rounded to 1 / 16 fractional accuracy. Then, a motion compensated interpolation filter is applied to generate the prediction of each subblock using the derived motion vector. The subblock size of the chroma components is also set to 4x4. The MV of a 4x4 chroma subblock is calculated as the average of the MVs of the top-left and bottom-right luma subblocks in the co-located 8x8 luma region.
[0115] As with translational motion inter prediction, there are also two affine motion inter prediction modes: affine merge mode and affine AMVP mode.
[0116] Affine Merge Prediction
[0117] The AF_MERGE mode may be applied to CUs whose width and height are both equal to or greater than 8. In this mode, the CPVM of the current CU is generated based on the motion information of spatially neighboring CUs. There may be up to five CPMVP candidates, and an index is signaled to indicate the CPMVP candidate used for the current CU. The following three types of CPVM candidates are used to form the affine merge candidate list: - Inherited affine merge candidates extrapolated from the CPMVs of neighboring CUs - Constructed affine merge candidate CPMVP derived using translation MVs of neighboring CUs - Zero's MV
[0118] In VVC, there are up to two inherited affine candidates, which are derived from the affine motion models of the neighboring blocks, one from the left neighboring CU and the other from the top neighboring CU. The candidate blocks are shown in Figure 12. For the left predictor, the scan order is A0->A1, and for the top predictor, the scan order is B0->B1->B2. Only the first inherited candidate from each side is selected. No pruning check is performed between the two inherited candidates. Once a neighboring affine CU is identified, its control point motion vector is used to derive the CPMVP candidate in the affine merge list of the current CU. If the neighboring bottom-left block A is coded in affine mode, the motion vectors v_2, v_3, and v_4 for the top-left, top-right, and bottom-left corners of the CU containing block A are obtained. If block A is coded with a four-parameter affine model, the two CPMVs of the current CU are calculated according to v_2 and v_3. When block A is coded with a six-parameter affine model, the three CPMVs of the current CU are calculated according to v_2, v_3, and v_4.
[0119] Constructed affine candidates means that candidates are constructed by combining neighboring translational motion information of each control point. The motion information of the control points is derived from the specified spatial and temporal neighbors shown in Figure 13. CPMVk (k=1, 2, 3, 4) represents the kth control point. For CPMV1, the block B2->B3->A2 is checked and the MV of the first available block is used. For CPMV2, the block B1->B0 is checked and for CPMV3, the block A1->A0 is checked. TMVP is used as CPMV4 if available.
[0120] After the MVs of the four control points are obtained, an affine merge candidate is constructed based on their motion information. The following combinations of control point MVs are used to construct in order: {CPMV1, CPMV2, CPMV3}, {CPMV1, CPMV2, CPMV4}, {CPMV1, CPMV3, CPMV4}, {CPMV2, CPMV3, CPMV4}, {CPMV1, CPMV2}, {CPMV1, CPMV3}
[0121] A combination of three CPMVs constructs a 6-parameter affine merge candidate, and a combination of two CPMVs constructs a 4-parameter affine merge candidate. To avoid the motion scaling process, if the reference indices of the control points are different, the associated control point MV combination is discarded.
[0122] After the inherited and constructed affine merge candidates are checked, if the list is not yet filled, a MV of zero is inserted at the end of the list.
[0123] Affine AMVP Prediction in VVC
[0124] Affine AMVP mode can be applied to CUs whose width and height are both 16 or greater. A CU-level affine flag is signaled in the bitstream to indicate whether affine AMVP mode is used, and then another flag is signaled to indicate whether it is 4-parameter affine or 6-parameter affine. In this mode, the difference between the CPMV of the current CU and its predictor CPMVP is signaled in the bitstream. The size of the affine AVMP candidate list is 2, and this candidate list is generated by using the following four types of CPVM candidates in order: - Inherited affine AMVP candidates extrapolated from the CPMVs of neighboring CUs - Constructed affine AMVP candidate CPMVP derived using translational MVs of neighboring CUs - Translational MV from adjacent CU - Zero's MV
[0125] The checking order of inherited affine AMVP candidates is the same as the checking order of inherited affine merge candidates. The only difference is that for AVMP candidates, only affine CUs that have the same reference picture as in the current block are considered. No pruning process is applied when inserting inherited affine motion predictors into the candidate list.
[0126] The constructed AMVP candidate is derived from the specified spatial neighborhood shown in Figure 13. The same check order is used as in the construction of the affine merge candidate. In addition, the reference picture indexes of the neighboring blocks are also checked. The first block in the check order that is inter-coded and has the same reference picture as in the current CU is used. There is only one. If the current CU is coded in 4-parameter affine mode and both mv0 and mv1 are available, they are added as one candidate in the affine AMVP list. If the current CU is coded in 6-parameter affine mode and all three CPMVs are available, they are added as one candidate in the affine AMVP list. Otherwise, the constructed AMVP candidate is set as unavailable.
[0127] After inserting the valid inherited and constructed affine AMVP candidates, if the number of affine AMVP list candidates is still less than 2, then mv0, mv1, and mv2 are added in order as translation MVs to predict all control point MVs of the current CU, when available. Finally, if the affine AMVP list is not yet filled, MVs of zero are used to fill the affine AMVP list.
[0128] Prediction refinement using optical flow of affine modes in VVC
[0129] Sub-block-based affine motion compensation can save memory access bandwidth and reduce computational complexity compared to pixel-based motion compensation, but at the expense of prediction accuracy. To achieve finer granularity of motion compensation, prediction refinement with optical flow (PROF) is used to refine sub-block-based affine motion compensation prediction without increasing memory access bandwidth for motion compensation. In VVC, after sub-block-based affine motion compensation is performed, luma prediction samples are refined by adding the difference derived by the optical flow equation. PROF is described as four steps:
[0130] Step 1) Sub-block based affine motion compensation is performed to generate the sub-block prediction I(i,j).
[0131] Step 2) Spatial gradients of subblock predictions g x (i,j) and g y (i,j) is computed at each sample position using a 3-tap filter [-1,0,1]. The gradient computation is exactly the same as the BDOF gradient computation.
number
number
[0132] shift1 is used to control the precision of the gradient. Sub-block (i.e., 4x4) prediction is extended by one sample on each side for gradient calculation. To avoid additional memory bandwidth and additional interpolation calculations, the extended samples on the extension boundary are copied from the nearest integer pixel location in the reference picture.
[0133] Step 3) Luma prediction refinement is performed using the following optical flow equation:
number
[0134] Since the affine model parameters and sample positions relative to the center of the subblock do not change for each subblock, Δv(i,j) can be calculated for the first subblock and reused for other subblocks in the same CU. dx(i,j) and dy(i,j) are calculated from sample position (i,j) to the subblock (x SB ,y SB ), Δv(x,y) is given by the following equation:
number
number
[0135] To maintain accuracy, the subblock (x SB ,y SB The input for ((W SB -1) / 2,(H SB -1) / 2), where W SB and H SB are the width and height of the sub-block, respectively.
[0136] For a four-parameter affine model,
number
[0137] For a six-parameter affine model,
number
[0138] Step 4) Finally, the luma prediction refinement ΔI(i,j) is added to the sub-block prediction I(i,j). The final prediction I′ is generated as the following equation: JPEG2024540760000014.jpg8161
[0139] PROF is not applied in two cases for affine coded CUs: 1) all control points MV are the same, which indicates that the CU has only translational motion, and 2) the affine motion parameters are larger than the specified limit, since sub-block based affine MC is downgraded to CU based MC to avoid large memory access bandwidth requirements.
[0140] To reduce the coding complexity of affine motion estimation with PROF, a fast coding method is applied. PROF is not applied in the affine motion estimation stage in the following two situations: a) if this CU is not a root block and its parent block does not select affine mode as its best mode, PROF is not applied since the current CU is unlikely to select affine mode as its best mode; b) if the magnitudes of the four affine parameters (C, D, E, F) are all smaller than a predefined threshold and the current picture is not a low-latency picture, PROF is not applied since the improvement brought by PROF in this case is small. In this way, affine motion estimation with PROF can be accelerated.
[0141] Sub-block-based Temporal Motion Vector Prediction (SbTMVP) in VVC
[0142] VVC supports the sub-block-based temporal motion vector prediction (SbTMVP) method. Similar to temporal motion vector prediction (TMVP) in HEVC, SbTMVP uses motion fields in aligned pictures to improve the motion vector prediction and merge mode of CUs in the current picture. The same aligned pictures used by TMVP are also used for SbTVMP. SbTMVP differs from TMVP in two main ways:
[0143] - TMVP predicts motion at the CU level, while SbTMVP predicts motion at the sub-CU level.
[0144] - TMVP fetches the temporal motion vector from an alignment block in the alignment picture (the alignment block is the bottom right or center block with respect to the current CU), while SbTMVP applies a motion shift before fetching the temporal motion information from the alignment picture, and the motion shift is obtained from a motion vector from one of the spatially adjacent blocks of the current CU.
[0145] The SbTVMP process is shown in Figure 15A and Figure 15B. SbTMVP predicts the motion vectors of sub-CUs in a current CU in two steps. In the first step, the spatial neighbor A1 in Figure 15A is examined. If A1 has a motion vector that uses an alignment picture as a reference picture, this motion vector is selected as the motion shift to be applied. If no such motion is identified, the motion shift is set to (0,0).
[0146] In the second step, the motion shift identified in step 1 is applied (i.e., added to the coordinates of the current block) to obtain sub-CU level motion information (motion vector and reference index) from the alignment picture, as shown in FIG. 15B. The example of FIG. 15B assumes that the motion shift is set to the motion of block A1. Then, for each sub-CU, the motion information of its corresponding block in the alignment picture (the smallest motion grid covering the center sample) is used to derive the motion information of the sub-CU. After the motion information of the co-located sub-CU is identified, this motion information is converted to the motion vector and reference index of the current sub-CU in a manner similar to the TMVP process of HEVC, and temporal motion scaling is applied to align the reference picture of the temporal motion vector to the picture of the current CU.
[0147] In VVC, a combined subblock-based merge list containing both SbTVMP and affine merge candidates is used to signal the subblock-based merge mode. The SbTVMP mode is enabled / disabled by a sequence parameter set (SPS) flag. If the SbTMVP mode is enabled, the SbTMVP predictor is added as the first entry in the list of subblock-based merge candidates, followed by the affine merge candidates. The size of the subblock-based merge list is signaled in the SPS, and the maximum allowed size of the subblock-based merge list is 5 in VVC.
[0148] The sub-CU size used in SbTMVP is fixed to 8x8, and similar to the affine merge mode, the SbTMVP mode only applies to CUs whose width and height are both 8 or greater.
[0149] The encoding logic of the additional SbTMVP merge candidate is the same as other merge candidates, i.e., for each CU of a P slice or a B slice, an additional RD check is performed to determine whether to use the SbTMVP candidate.
[0150] Adaptive Motion Vector Resolution (AMVR) for VVC
[0151] In HEVC, when use_integer_mv_flag in the slice header is equal to 0, the motion vector differential (MVD) (between the CU's motion vector and the predicted motion vector) is signaled in units of quarter luma samples. In VVC, a CU-level adaptive motion vector resolution (AMVR) scheme is introduced. AMVR allows the MVD of a CU to be coded with different precisions. Depending on the mode of the current CU (normal AMVP mode or affine AVMP mode), the MVD of the current CU can be adaptively selected as follows: - Normal AMVP mode: quarter luma samples, half luma samples, integer luma samples, or 4 luma samples - Affine AMVP mode: quarter luma samples, integer luma samples, or 1 / 16 luma samples
[0152] If the current CU has at least one non-zero MVD component, a CU-level MVD resolution indication is conditionally signaled. If all MVD components (i.e., both horizontal and vertical MVDs in reference list L0 and reference list L1) are zero, a quarter luma sample MVD resolution is inferred.
[0153] For a CU with at least one non-zero MVD component, a first flag is signaled to indicate whether quarter luma sample MVD precision is used for that CU. If the first flag is 0, no further signaling is required and quarter luma sample MVD precision is used for the current CU. Otherwise, a second flag is signaled to indicate that half luma sample or other MVD precision (integer or 4 luma sample) is used for the regular AMVP CU. In the case of half luma sample, a 6-tap interpolation filter is used instead of the default 8-tap interpolation filter for the half luma sample positions. Otherwise, a third flag is signaled to indicate whether integer luma sample or 4 luma sample MVD precision is used for the regular AMVP CU. For affine AMVP CUs, the second flag is used to indicate whether integer luma sample or 1 / 16 luma sample MVD precision is used. To ensure that the reconstructed MV has the intended precision (quarter luma samples, half luma samples, integer luma samples, or 4 luma samples), the motion vector predictor of a CU may be rounded to the same precision as that of the MVD before being added with the MVD. The motion vector predictor is rounded towards zero (i.e., negative motion vector predictors are rounded towards positive infinity and positive motion vector predictors are rounded towards negative infinity).
[0154] The encoder uses the RD check to determine the motion vector resolution of the current CU. To avoid always performing CU-level RD checks four times for each MVD resolution, in VTM14, the RD check for MVD precisions other than quarter luma sample is only conditionally called. For normal AVMP mode, the RD costs of quarter luma sample MVD precision and integer luma sample MV precision are calculated first. Then, the RD cost of integer luma sample MVD precision is compared with the RD cost of quarter luma sample MVD precision to determine whether the RD cost of four luma sample MVD precision needs to be further checked. If the RD cost of quarter luma sample MVD precision is much smaller than the RD cost of integer luma sample MVD precision, the RD check of four luma sample MVD precision is skipped. Then, if the RD cost of integer luma sample MVD precision is significantly larger than the best RD cost of the previously tested MVD precision, the check of half luma sample MVD precision is skipped. For affine AMVP mode, if no affine inter mode is selected after checking the rate-distortion costs of affine merge / skip mode, merge / skip mode, normal AMVP mode with quarter luma sample MVD precision, and affine AMVP mode with quarter luma sample MVD precision, then affine inter modes with 1 / 16 luma sample MV precision and 1 pixel MV precision are not checked. Additionally, the affine parameters obtained in affine inter mode with quarter luma sample MV precision are used as starting search points for affine inter modes with 1 / 16 luma sample and quarter luma sample MV precision.
[0155] Bi-prediction with CU-level weight (BCW) in VVC
[0156] In HEVC, a bi-predictive signal is generated by averaging two prediction signals obtained from two different reference pictures and / or by using two different motion vectors. In VVC, the bi-predictive mode is extended beyond the scope of a simple average to allow a weighted average of the two prediction signals.
number
[0157] Weighted average bi-prediction allows five weights w∈{-2,3,4,5,10}. For each bi-predictive CU, the weight w is determined in one of two ways: 1) for non-merged CUs, the weight index is signaled after the motion vector differential; 2) for merged CUs, the weight index is inferred from the neighboring blocks based on the merge candidate index; BCW is applied only to CUs with 256 or more luma samples (i.e., CU width × CU height is 256 or more); for low latency pictures, all five weights are used; for non-low latency pictures, only three weights (w∈{3,4,5}) are used.
[0158] - In the encoder, fast search algorithms are applied to find the weight index without significantly increasing the encoder complexity. These algorithms are summarized as follows. For more details, see VTM software and document JVET-L0646. When combined with AMVR, for 1-pixel and 4-pixel motion vector precision, unequal weights are only conditionally checked if the current picture is a low-latency picture.
[0159] When combined with affine, affine ME may be performed for unequal weights if and only if the affine mode is selected as the current best mode.
[0160] - When the two reference pictures of bi-prediction are the same, unequal weights are only conditionally checked.
[0161] - Depending on the POC distance between the current picture and its reference pictures, the coding QP and the temporal level, unequal weights are not searched if certain conditions are met.
[0162] The BCW weight index is coded using one context coding bin followed by a bypass coding bin: the first context coding bin indicates whether equal weights are used, and if unequal weights are used, an additional bin is signaled using bypass coding to indicate which unequal weights are used.
[0163] Weighted prediction (WP) is an encoding tool for efficiently encoding video content with fading, supported by H.264 / AVC and HEVC standards. Support for WP was also added to the VVC standard. WP allows for signaling weighting parameters (weight and offset) for each reference picture in each of the reference picture lists L0 and L1. Then, during motion compensation, the weight and offset of the corresponding reference picture are applied. WP and BCW are designed for various types of video content. To avoid interactions between WP and BCW that may complicate VVC decoder design, when a CU uses WP, the BCW weight index is not signaled and w is inferred to be 4 (i.e., equal weights are applied). For merge CUs, the weight index is inferred from neighboring blocks based on the merge candidate index. This can be applied to both normal merge mode and inherited affine merge mode. For constructed affine merge mode, the affine motion information is constructed based on the motion information of up to three blocks. The BCW index of a CU using the constructed affine merge mode is simply set equal to the BCW index of the first control point MV.
[0164] In VVC, CIIP and BCW cannot be applied simultaneously to a CU. If a CU is coded in CIIP mode, the BCW index of the current CU is set to 2, i.e., equal weight.
[0165] Bi-directional optical flow (BDOF) in VVC
[0166] VVC includes a bidirectional optical flow (BDOF) tool. BDOF was previously called BIO and was included in JEM. Compared to the JEM version, BDOF in VVC is a simpler version and requires significantly less computation, especially in terms of the number of multiplications and the size of the multipliers.
[0167] BDOF is used to refine the bi-predictive signal of a CU at the 4×4 sub-block level. BDOF is applied to a CU if the CU satisfies all of the following conditions: - The CU is coded using a "true" bi-prediction mode, i.e., one of the two reference pictures is before the current picture in display order and the other is after the current picture in display order. - The distance from the two reference pictures to the current picture (i.e., the POC difference) is the same - Both reference pictures are short-term reference pictures - The CU is not coded using affine mode or SbTMVP merge mode - CU has more than 64 luma samples - Both CU height and CU width are 8 luma samples or more - BCW Weight Index shows equal weighting - WP is not currently enabled for the CU - CIIP mode is not currently in use for the CU
[0168] BDOF is applied only to the luma component. As the name suggests, the BDOF mode is based on the concept of optical flow, which assumes that object motion is smooth. For each 4x4 subblock, motion refinement (v x ,v y ) is calculated. Then, motion refinement is used to adjust the bi-predictive sample values in the 4x4 sub-block. The following steps are applied in the BDOF process:
[0169] First, the horizontal and vertical gradients of the two prediction signals JPEG2024540760000016.jpg29161 (k=0,1) is between two adjacent samples, i.e.
number
[0170] Then, the autocorrelation and cross-correlation of the gradients S1, S2, S3, S5, and S6 are
number
number
[0171] Next,
number
[0172] Based on the motion refinement and gradients, the following adjustments are calculated for each sample in the 4x4 sub-block:
number
[0173] Finally, the BDOF samples for the CU are calculated by adjusting the bi-predictive samples as follows:
number
[0174] These values are chosen such that the multipliers of the BDOF process do not exceed 15 bits and the maximum bit width of the intermediate parameters of the BDOF process is kept within 32 bits.
[0175] To derive the gradient value, we first select some prediction samples I in list k (k=0, 1) outside the current CU boundary. (k)(i,j) need to be generated. As depicted in FIG. 16, BDOF in VVC uses one extended row / column around the boundary of the CU. To control the computational complexity of generating predicted samples outside the boundary, predicted samples in the extended area (white positions) are generated by directly taking reference samples at nearby integer positions (using floor() operation on the coordinates) without interpolation, and a normal 8-tap motion compensation interpolation filter is used to generate predicted samples within the CU (gray positions). These extended sample values are only used in gradient calculation. In the remaining steps of the BDOF process, if sample and gradient values outside the CU boundary are needed, they are padded (i.e., repeated) from their nearest neighbors.
[0176] If the width and / or height of a CU is greater than 16 luma samples, the CU may be split into sub-blocks with width and / or height equal to 16 luma samples, and the sub-block boundaries are treated as CU boundaries in the BDOF process. The maximum unit size of the BDOF process is limited to 16x16. For each sub-block, the BDOF process may be skipped. If the SAD between the initial L0 predicted sample and the L1 predicted sample is less than a threshold, the BDOF process is not applied to the sub-block. The threshold is set to be equal to (8*W*(H>>1), where W denotes the width of the sub-block and H denotes the height of the sub-block. To avoid further complexity of the SAD calculation, the SAD between the initial L0 predicted sample and the L1 predicted sample calculated in the DVMR process is reused here.
[0177] If BCW is enabled for the current block, i.e., the BCW weight index indicates unequal weights, bidirectional optical flow is disabled. Similarly, if WP is enabled for the current block, i.e., luma_weight_lx_flag is 1 for either of the two reference pictures, BDOF is also disabled. If the CU is coded in symmetric MVD or CIIP mode, BDOF is also disabled.
[0178] Decoder side motion vector refinement (DMVR) in VVC
[0179] To improve the accuracy of MV in merge mode, bilateral matching (BM) based decoder-side motion vector refinement is applied in VVC. In bi-predictive operation, refined MVs around the initial MV in reference picture list L0 and reference picture list L1 are searched. The BM method calculates the distortion between two candidate blocks in reference picture list L0 and list L1. As shown in Figure 17, the SAD between red blocks based on each MV candidate around the initial MV is calculated. The MV candidate with the lowest SAD becomes the refined MV and is used to generate the bi-predictive signal.
[0180] In VVC, the application of DMVR is restricted and applies only to CUs coded with the following modes and features: - CU level merge mode with bi-predictive MV - one reference picture is in the past and the other in the future relative to the current picture - The distance from the two reference pictures to the current picture (i.e., the POC difference) is the same - Both reference pictures are short-term reference pictures - CU has more than 64 luma samples - Both CU height and CU width are 8 luma samples or more - BCW Weight Index shows equal weighting - WP is not currently enabled for the block - CIIP mode is not currently used for the block
[0181] The refined MVs derived by the DMVR process are used to generate inter prediction samples and are also used for temporal motion vector prediction for future picture coding, while the original MVs are used in the deblocking process and are also used for spatial motion vector prediction for future CU coding.
[0182] In the next section, additional features of DMVR are mentioned.
[0183] Search Method
[0184] In DVMR, the search points are around the initial MV, and the MV offset follows the mirroring rule of MV difference. In other words, any point checked by DMVR, indicated by a candidate MV pair (MV0,MV1), follows the following two equations:
number
number
[0185] where MV_offset represents the refinement offset between the initial MV and the refined MV in one of the reference pictures. The refinement search range is two integer luma samples from the initial MV. The search includes an integer sample offset search stage and a fractional sample refinement stage.
[0186] A full search of 25 points is applied to the integer sample offset search. First, the SAD of the initial MV pair is calculated. If the SAD of the initial MV pair is less than the threshold, the integer sample stage of DMVR is terminated. Otherwise, the SAD of the remaining 24 points is calculated and checked in raster scan order. The point with the smallest SAD is selected as the output of the integer sample offset search stage. To mitigate the penalty due to the uncertainty of DMVR refinement, it is proposed to prioritize the original MV during the DMVR process. The SAD between the reference blocks referenced by the initial MV candidates is reduced by ¼ of the SAD value.
[0187] The integer sample search is followed by fractional sample refinement. To reduce computational complexity, the fractional sample refinement is derived by using a parametric error surface equation instead of an additional search with SAD comparison. The fractional sample refinement is conditionally invoked based on the output of the integer sample search stage. When the integer sample search stage ends with the center with the smallest SAD in either the first iteration search or the second iteration search, the fractional sample refinement is further applied.
[0188] In parametric error surface based sub-pixel offset estimation, the cost of the center location and the costs of the four neighboring locations from the center are used to solve a 2D parabolic error surface equation of the form:
number
number
number
[0189] Since all cost values are positive and the smallest value is E(0,0), x min and y min The value of is automatically constrained between -8 and 8. This corresponds to a half pixel offset with 1 / 16 pixel MV accuracy in VVC. The calculated fraction (x min ,y min ) is added to the integer distance refinement MV to obtain the sub-pixel accurate refinement delta MV.
[0190] Bilinear Interpolation and Sample Padding
[0191] In VVC, the resolution of the MV is 1 / 16 luma samples. The samples at fractional positions are interpolated using an 8-tap interpolation filter. In DMVR, the search points surround the initial fractional pixel MV with integer sample offsets, so the samples at these fractional positions need to be interpolated in the DMVR search process. To reduce the computational complexity, a bilinear interpolation filter is used to generate fractional samples for the search process in DMVR. Another important effect is that by using a bilinear filter, the search range of 2 samples means that DVMR does not access more reference samples compared to the normal motion compensation process. After the refined MV is obtained in the DMVR search process, a normal 8-tap interpolation filter is applied to generate the final prediction. To avoid accessing more reference samples for the normal MC process, samples that are not needed for the interpolation process based on the original MV but are needed for the interpolation process based on the refined MV are padded from the available samples.
[0192] Geometric partitioning mode (GPM) in VVC
[0193] VVC supports geometric partitioning mode for inter prediction. The geometric partitioning mode is signaled using a CU-level flag as a type of merge mode, other merge modes include normal merge mode, MMVD mode, CIIP mode, and sub-block merge mode. The geometric partitioning mode allows all possible CU sizes w×h=2 except for 8×64 and 64×8. m ×2 n , m, n∈{3···6}, for which a total of 64 partitions are supported.
[0194] When this mode is used, a CU is split into two parts by a geometrically positioned straight line (Figure 18). The location of the split line is mathematically derived from the angle and offset parameters of the particular partition. Each part of a geometric partition within a CU is inter-predicted using its own motion, and only uni-prediction is allowed for each partition, i.e., each part has one motion vector and one reference index. As with traditional bi-prediction, a uni-prediction motion constraint is applied to ensure that only two motion-compensated predictions are required for each CU. The uni-prediction motion for each partition is derived using the process described in 3.4.11.1.
[0195] If the geometric partition mode is used for the current CU, a geometric partition index indicating the partition mode (angle and offset) of the geometric partition and two merge indices (one for each partition) are further signaled. The number of maximum GPM candidate sizes is explicitly signaled in the SPS to specify the syntactic binarization of the GPM merge index. After predicting each part of the geometric partition, the sample values along the edges of the geometric partition are adjusted using a blending process with adaptive weights as in 3.4.11.2. This is the prediction signal for the entire CU, and similar to other prediction modes, the transformation and quantization process may be applied to the entire CU. Finally, the motion field of the CU predicted using the geometric partition mode is stored as in 3.4.11.3.
[0196] Building a single prediction candidate list
[0197] The uni-prediction candidate list is directly derived from the merge candidate list constructed according to the extended merge prediction process in 3.4.1. Let n be the index of the uni-prediction motion in the geometric uni-prediction candidate list. The LX motion vector (X is equal to the parity of n) of the n-th extended merge candidate is used as the n-th uni-prediction motion vector for the geometric partitioning mode. In FIG. 19, these motion vectors are marked with "x". If the corresponding LX motion vector of the n-th extended merge candidate does not exist, the L(1-X) motion vector of the same candidate is used instead as the uni-prediction motion vector for the geometric partitioning mode.
[0198] Combined inter and intra prediction (CIIP) in VVC
[0199] In VVC, when a CU is coded in merge mode, if the CU contains at least 64 luma samples (i.e., CU width × CU height is 64 or more), and if both the CU width and CU height are less than 128 luma samples, an additional flag is signaled to indicate whether a combined inter / intra prediction (CIIP) mode is currently applied to the CU. As the name suggests, CIIP prediction combines an inter prediction signal and an intra prediction signal. The inter prediction signal P of the CIIP mode inter is derived using the same inter prediction process as applied in the normal merge mode, resulting in an intra prediction signal P intra is derived according to the normal intra prediction process with planar mode. The intra prediction signal and the inter prediction signal are then combined using a weighted average, with the weight value calculated according to the coding modes of the upper and left neighboring blocks (depicted in FIG. 20) as follows: - If the top neighbor is available and is intra-coded, set isIntraTop to 1, otherwise set isIntraTop to 0. - If the left neighbor is available and is intra-coded, set isIntraLeft to 1, otherwise set isIntraLeft to 0. - If (isIntraLeft+isIntraTop) is equal to 2, then wt is set to 3. - Otherwise, if (isIntraLeft+isIntraTop) is equal to 1, then wt is set to 2. - Else, set wt to 1.
[0200] The CIIP forecast is formed as follows:
number
[0201] Intra Block Copy (IBC) in VVC
[0202] Intra Block Copy (IBC) is a tool employed in the HEVC extension on SCC. It is well known that IBC significantly improves the coding efficiency of screen content material. Since IBC mode is implemented as a block-level coding mode, block matching (BM) is performed in the encoder to find the best block vector (or motion vector) for each CU. Here, the block vector is used to indicate the displacement from the current block to a reference block already reconstructed in the current picture. The luma block vectors of IBC coded CUs are in integer precision. The chroma block vectors are also rounded to integer precision. When combined with AMVR, IBC mode can switch between 1-pixel and 4-pixel motion vector precision. IBC coded CUs are treated as a third prediction mode other than intra or inter prediction modes. IBC mode is applicable for CUs whose width and height are both less than or equal to 64 luma samples.
[0203] On the encoder side, hash-based motion estimation is performed for IBC. The encoder performs RD check for blocks whose width or height is less than or equal to 16 luma samples. For non-merge modes, a block vector search is performed first using a hash-based search. If the hash search does not return a valid candidate, a block matching based local search is performed.
[0204] In hash-based search, the hash key matching (32-bit CRC) between the current block and the reference block is extended to all allowed block sizes. The hash key calculation for all positions in the current picture is based on 4x4 sub-blocks. For larger sized current blocks, the hash key is determined to match the hash key of the reference block when all the hash keys of all 4x4 sub-blocks match the hash keys of the corresponding reference positions. If the hash keys of multiple reference blocks are found to match the hash key of the current block, the block vector cost of each matched reference is calculated and the one with the minimum cost is selected.
[0205] In a block matching search, the search range is set to cover both the previous CTU and the current CTU.
[0206] At the CU level, the IBC mode is signaled by a flag, signaled as IBC AMVP mode or IBC skip / merge mode as follows:
[0207] IBC skip / merge mode: The merge candidate index is used to indicate which block vector in the list from neighboring candidate IBC coded blocks is used to predict the current block. The merge list consists of spatial candidates, HMVP candidates, and pairwise candidates.
[0208] IBC AMVP mode: Block vector differentials are coded in the same way as motion vector differentials. The block vector prediction method uses two candidates as predictors: one from the left neighbor (for IBC coding) and one from the top neighbor. If neither neighbor is available, a default block vector is used as predictor. A flag is signaled to indicate the block vector predictor index.
[0209] IBC reference area To reduce memory consumption and decoder complexity, IBC in VVC allows only reconstructed parts of a predefined area, including the area of the current CTU and some area of the left CTU. Depending on the position of the current coding CU within the current CTU, the following applies:
[0210] If the current block falls in the top-left 64x64 block of the current CTU, in addition to the already reconstructed samples in the current CTU, the current block can also refer to reference samples in the bottom-right 64x64 block of the left CTU using CPR mode.The current block can also refer to reference samples in the bottom-left 64x64 block of the left CTU and reference samples in the top-right 64x64 block of the left CTU using CPR mode.
[0211] If the current block falls in the top-right 64x64 block of the current CTU, when the luma position (0,64) relative to the current CTU has not yet been reconstructed, the current block can also refer to reference samples in the bottom-left 64x64 block and bottom-right 64x64 block of the left CTU using the CPR mode, in addition to the already reconstructed samples in the current CTU. Otherwise, the current block can also refer to reference samples in the bottom-right 64x64 block of the left CTU.
[0212] If the current block falls in the bottom-left 64x64 block of the current CTU, and the luma position (64,0) relative to the current CTU has not yet been reconstructed, the current block can also refer to reference samples in the top-right 64x64 block and bottom-right 64x64 block of the left CTU using the CPR mode, in addition to the already reconstructed samples in the current CTU. Otherwise, the current block can also refer to reference samples in the bottom-right 64x64 block of the left CTU using the CPR mode.
[0213] If the current block falls in the bottom-right 64x64 block of the current CTU, the CPR mode can be used to refer only to samples already reconstructed within the current CTU.
[0214] This restriction makes it possible to implement the IBC mode using on-chip memory local to the hardware implementation.
[0215] Local illumination compensation (LIC) in ECM
[0216] LIC is an inter-prediction technique for modeling the local illumination variation between a current block and its predicted block as a function of the local illumination variation between the current block template and the reference block template. The parameters of the function can be represented by a scale α and an offset β, forming a linear equation, i.e., α*p[x]+β to compensate for illumination changes, where p[x] is a reference sample pointed to by the MV at position x on the reference picture. Since α and β can be derived based on the current block template and the reference block template, no signaling overhead is required for α and β, except that a LIC flag is signaled for the AMVP mode to indicate the use of LIC.
[0217] Local illumination compensation proposed in JVET-O0066 is used for uni-predictive inter CUs with the following modifications: - Intra adjacent samples may be used in the LIC parameter derivation. - LIC is disabled for blocks with less than 32 luma samples. - For both non-subblock and affine modes, the LIC parameter derivation is performed based on the partial template block samples corresponding to the current CU instead of the partial template block samples corresponding to the first top-left 16x16 unit. - The samples of the reference block template are generated by using MC together with the block MV without rounding to integer pixel precision.
[0218] Non-adjacent spatial candidates in ECM Non-adjacent spatial merge candidates as in JVET-L0399 are inserted after TMVP in the regular merge candidate list. The spatial merge candidate pattern is shown in Figure 21. The distance between the non-adjacent spatial candidate and the current coding block is based on the width and height of the current coding block. No line buffer restrictions are applied.
[0219] Template Matching (TM) in ECM Template matching (TM) is a decoder-side MV derivation method to refine the motion information of the current CU by finding the closest match between a template in the current picture (i.e., a block adjacent to the top and / or left of the current CU) and a block in a reference picture (i.e., the same size as the template). As shown in Figure 22, a better MV is searched for within a [-8, +8] pixel search range centered on the initial motion of the current CU. The template matching method in JVET-J0021 is used with the following modifications: the size of the search step is determined based on the AMVR mode, and TM can be cascaded with the bilateral matching process in merge mode.
[0220] In AMVP mode, the MVP candidate is determined based on the template matching error to select the MVP candidate that achieves the minimum error between the current block template and the reference block template, and then TM is performed only on this particular MVP candidate for MV refinement. TM refines this MVP candidate starting with full-pixel MVD accuracy (or 4 pixels for 4-pixel AMVR mode) in the [-8, +8] pixel search range using an iterative diamond search. The AMVP candidate may be further refined by using a cross search of full-pixel MVD accuracy (or 4 pixels for 4-pixel AMVR mode) followed by half-pixel and quarter-pixel MVD accuracy depending on the AMVR mode, as specified in Table 3. This search process ensures that the MVP candidate maintains the same MV accuracy as exhibited in AMVR mode after the TM process.
[0221] [Table 3]
[0222] In the merge mode, a similar search method is applied for the merge candidates indicated by the merge index. As Table 3 shows, TM may run up to ⅛-pixel MVD accuracy or skip accuracy beyond half-pixel MVD accuracy depending on whether an alternative interpolation filter (used when AMVR is in half-pixel mode) is used according to the merged motion information. Furthermore, when the TM mode is enabled, template matching may function as an independent process or an additional MV refinement process between the block-based bilateral matching (BM) method and the sub-block-based BM method depending on whether BM can be enabled according to the enablement condition check.
[0223] Multi-pass decoder-side motion vector refinement in ECM. A multi-pass decoder-side motion vector refinement is applied. In the first pass, bilateral matching (BM) is applied to the coding block. In the second pass, BM is applied to each 16x16 sub-block in the coding block. In the third pass, the MV in each 8x8 sub-block is refined by applying bidirectional optical flow (BDOF). The refined MV is stored for both spatial and temporal motion vector prediction.
[0224] 1st Pass – Block-Based Bilateral Matching MV Refinement In the first pass, a refined MV is derived by applying the BM to the coding block. Similar to decoder-side motion vector refinement (DMVR), in bi-predictive operation, a refined MV is searched around two initial MVs (MV0 and MV1) in reference picture lists L0 and L1. Based on the minimum bilateral matching cost between the two reference blocks in L0 and L1, a refined MV (MV0_pass1 and MV1_pass1) is derived around the initial MV.
[0225] BM performs a local search to derive the integer sample precision intDeltaMV. The local search applies a 3x3 square search pattern to loop through the horizontal search range [-sHor,sHor] and the vertical search range [-sVer,sVer], where the values of sHor and sVer are determined by the block dimension, and the maximum value of sHor and sVer is 8.
[0226] The bilateral matching cost is calculated as bilCost=mvDistanceCost+sadCost. When the block size cbW*cbH is larger than 64, the MRSAD cost function is applied to remove the DC effect of distortion between reference blocks. When bilCost at the center point of the 3×3 search pattern has the minimum cost, the intDeltaMV local search ends. Otherwise, the current minimum cost search point becomes the new center point of the 3×3 search pattern, and the search for the minimum cost continues until the end of the search range is reached.
[0227] Further existing fractional sample refinement is applied to derive the final deltaMV. The refined MV after the first pass is then - MV0_pass1=MV0+deltaMV - MV1_pass1=MV1-deltaMV It is derived as:
[0228] Second Pass – Subblock-Based Bilateral Matching MV Refinement
[0229] In the second pass, refined MVs are derived by applying BM to the sub-blocks of a 16x16 grid. For each sub-block, refined MVs are searched around the two MVs (MV0_pass1 and MV1_pass1) obtained in the first pass in the reference picture lists L0 and L1. Refined MVs (MV0_pass2(sbIdx2) and MV1_pass2(sbIdx2)) are derived based on the minimum bilateral matching cost between the two reference sub-blocks in L0 and L1.
[0230] For each subblock, the BM performs a full search to derive the integer sample precision intDeltaMV. The full search has a horizontal search range [-sHor,sHor] and a vertical search range [-sVer,sVer], where the values of sHor and sVer are determined by the block dimension, and the maximum value of sHor and sVer is 8.
[0231] The bilateral matching cost is calculated by applying a cost factor to the SATD cost between two reference subblocks as bilCost=satdCost*costFactor. The search area (2*sHor+1)*(2*sVer+1) is divided into up to five diamond-shaped search regions as shown in Figure 23. Each search region is assigned a costFactor determined by the distance (intDeltaMV) between each search point and the starting MV, and each diamond region is processed sequentially from the center of the search area. In each region, the search points are processed in raster scan order starting from the top-left corner of the region toward the bottom-right corner. If the minimum bilCost in the current search region is less than a threshold equal to sbW*sbH, the integer-pixel full search ends, otherwise the integer-pixel full search proceeds to the next search region until all search points have been examined.
[0232] Further existing VVC DMVR fractional sample refinement is applied to derive the final deltaMV(sbIdx2). The refined MV in the second pass is - MV0_pass2(sbIdx2)=MV0_pass1+deltaMV(sbIdx2) - MV1_pass2(sbIdx2)=MV1_pass1-deltaMV(sbIdx2) It is derived as:
[0233] 3rd Pass – Sub-block based bidirectional optical flow MV refinement In the third pass, refined MVs are derived by applying BDOF to 8x8 grid sub-blocks. For each 8x8 sub-block, BDOF refinement is applied to derive scaled Vx and Vy without clipping, starting from the refined MV of the parent sub-block in the second pass. The derived bioMv(Vx,Vy) are rounded to 1 / 16 sample precision and clipped between -32 and 32.
[0234] The refined MVs in the third pass (MV0_pass3(sbIdx3) and MV1_pass3(sbIdx3)) are - MV0_pass3(sbIdx3)=MV0_pass2(sbIdx2)+bioMv - MV1_pass3(sbIdx3)=MV0_pass2(sbIdx2)-bioMv It is derived as:
[0235] OBMCs in ECM When OBMC is applied, the top and left boundary pixels of a CU are refined using motion information of neighboring blocks together with weighted prediction, as described in JVET-L0101.
[0236] The conditions for which OBMC does not apply are as follows: - If OBMC is disabled at the SPS level - If the current block has intra or IBC mode - If the current block applies LIC - If the current Luma Block Area is 32 or less
[0237] Sub-block boundary OBMC is performed by applying the same blending to the top, left, bottom, and right sub-block boundary pixels using the motion information of the neighboring sub-blocks. This is known as a sub-block-based coding tool, - Affine AMVP mode - Affine merge mode and sub-block based temporal motion vector prediction (SbTMVP) - Subblock-based bilateral matching will be valid for.
[0238] Sample-based BDOF in ECM
[0239] In sample-based BDOF, instead of deriving the motion refinement (Vx, Vy) on a block-by-block basis, the derivation is performed sample-by-sample.
[0240] A coding block is divided into 8x8 subblocks. For each subblock, it is decided whether to apply BDOF or not by checking the SAD between two reference subblocks against a threshold. If it is decided to apply BDOF to a subblock, a sliding 5x5 window is used for all samples in the subblock, and the existing BDOF process is applied for all sliding windows to derive Vx and Vy. The derived motion refinement (Vx,Vy) is applied to adjust the bi-predictive sample value of the center sample of the window.
[0241] Interpolation in ECM The 8-tap interpolation filter used in VVC is replaced by a 12-tap filter. The interpolation filter is derived from a sinc function whose frequency response is cut off at the Nyquist frequency and cropped by a cosine window function. Table 4 shows the filter coefficients for all 16 phases. Figure 24 compares the frequency response of the interpolation filter with the VVC interpolation filter, all at half pixel phase.
[0242] [Table 4]
[0243] Multi-hypothesis prediction (MHP) in ECM In the multi-hypothesis inter-prediction mode (JVET-M0425), in addition to the conventional bi-predictive signal, one or more additional motion-compensated prediction signals are signaled. The resulting overall prediction signal is obtained by weighted superposition on a sample-by-sample basis. The bi-predictive signal p bi and by using the first additional inter prediction signal / hypothesis h3, the resulting prediction signal p3 is obtained as follows: JPEG2024540760000032.jpg8166
[0244] The weighting factor α is specified by the new syntax element add_hyp_weight_idx according to the following mapping: [Table 5]
[0245] Similar to above, two or more additional prediction signals may be used, with the resulting overall prediction signal being iteratively accumulated with each additional prediction signal. JPEG2024540760000034.jpg9166
[0246] The resulting overall prediction signal is the final p n (i.e., the p with the largest index n n ) Within this EE, up to two additional prediction signals may be used (i.e., n is limited to 2).
[0247] The motion parameters of each additional prediction hypothesis are signaled either explicitly by specifying a reference index, a motion vector predictor index, and a motion vector differential, or implicitly by specifying a merge index. A separate multi-hypothesis merge flag distinguishes between these two signaling modes.
[0248] For inter AMVP mode, MHP is applied only if unequal weights in BCW are selected in bi-predictive mode.
[0249] A combination of MHP and BDOF is possible, but BDOF is only applied to the bi-predictive signal part of the predicted signal (ie, usually the first two hypotheses).
[0250] Adaptive reordering of merge candidates with template matching in ECM (ARMC-TM) Merge candidates are adaptively reordered by template matching (TM). This reordering method applies to the normal, template matching (TM) and affine merge modes (except for SbTMVP candidates). For the TM merge mode, the merge candidates are reordered before the refinement process.
[0251] After the merge candidate list is constructed, the merge candidates are divided into several subgroups. For normal merge and TM merge modes, the subgroup size is set to 5. For affine merge mode, the subgroup size is set to 3. The merge candidates in each subgroup are sorted in ascending order according to their cost values based on template matching. For simplicity, the merge candidates in the last subgroup but not the first are not sorted.
[0252] The template matching cost of a merge candidate is measured by the sum of absolute differences (SAD) between the template's samples of the current block and their corresponding reference samples. The template includes a set of reconstructed samples that are neighbors of the current block. The template's reference samples are located by the motion information of the merge candidate.
[0253] If the merge candidate utilizes bi-prediction, the reference samples of the merge candidate's template are also generated by bi-prediction, as shown in FIG.
[0254] For a subblock-based merging candidate with a subblock size equal to Wsub×Hsub, the top template contains several subtemplates of size Wsub×1, and the left template contains several subtemplates of size 1×Hsub. As shown in Figure 26, the motion information of the subblocks in the first row and first column of the current block is used to derive the reference samples of each subtemplate.
[0255] Geometric Partitioning Mode (GPM) with Merged Motion Vector Differential (MMVD) in ECM
[0256] GPM in VVC is extended by applying motion vector refinement on top of the existing GPM unidirectional MV. First, a flag is signaled to specify whether this mode is used for the GPM CU. If this mode is used, each geometric partition of the GPM CU can further decide whether to signal MVD or not. If MVD is signaled for the geometric partition, after the GPM merge candidate is selected, the partition motion is further refined by the signaled MVD information. All other procedures are the same as GPM.
[0257] MVD is signaled as a distance and direction pair, similar to MMVD. There are 9 candidate distances (1 / 4 pixel, 1 / 2 pixel, 1 pixel, 2 pixels, 3 pixels, 4 pixels, 6 pixels, 8 pixels, 16 pixels) and 8 candidate directions (4 horizontal / vertical and 4 diagonal) involved in GPM with MMVD (GPM-MMVD). Additionally, if pic_fpel_mmvd_enabled_flag is equal to 1, MVD is left shifted by 2, similar to MMVD.
[0258] Geometric Partitioning Mode (GPM) using Template Matching (TM) in ECM
[0259] Template matching is applied to the GPM. When GPM mode is enabled for a CU, a CU-level flag is signaled to indicate whether TM is applied to both geometric partitions. The motion information of each geometric partition is refined using the TM. Once a TM is chosen, a template is constructed using the left, top, or left and top neighboring samples according to the partition angle, as shown in Table 5. The motion is then refined by minimizing the difference between the current template and the template in the reference picture using the same search pattern in merge mode with the half-pel interpolation filter disabled.
[0260] [Table 6]
[0261] The GPM candidate list is constructed as follows:
[0262] 1. From the normal merge candidate list, the interleaved list 0 MV candidates and list 1 MV candidates are directly derived. The list 0 MV candidates have higher priority than the list 1 MV candidates. A pruning method with an adaptive threshold based on the current CU size is applied to remove redundant MV candidates.
[0263] 2. From the normal merge candidate list, the interleaved list 1 MV candidates and list 0 MV candidates are further derived directly. The list 1 MV candidates have higher priority than the list 0 MV candidates. Similarly, a pruning method with an adaptive threshold is applied to remove redundant MV candidates.
[0264] 3. The GPM candidate list is padded with zero MV candidates until it is filled.
[0265] GPM-MMVD and GPM-TM are enabled exclusively for one GPM CU. This is done by signaling the GPM-MMVD syntax first. If two GPM-MMVD control flags are both equal to false (i.e., GPM-MMVD is disabled for two GPM partitions), then a GPM-TM flag is signaled to indicate whether template matching applies to the two GPM partitions. Otherwise (if at least one GPM-MMVD flag is equal to true), the value of the GPM-TM flag is inferred to be false.
[0266] Division of the picture into CTUs A picture is partitioned into a sequence of coding tree units (CTUs). The concept of a CTU is the same as that of HEVC [6][7]. For a picture with a three-sample ordering, a CTU consists of an NxN block of luma samples and the corresponding two blocks of chroma samples. Figure 31 shows an example of a picture partitioned into CTUs.
[0267] The maximum allowed size of a luma block in a CTU is specified to be 128x128 (however, the maximum size of a luma transform block is 64x64).
[0268] Partitioning a picture into subpictures, slices, and tiles A picture is divided into one or more tile rows and one or more tile columns. A tile is a sequence of CTUs that covers a rectangular area of the picture.
[0269] A slice consists of an integer number of complete tiles, or an integer number of contiguous complete CTU rows within a tile of a picture.
[0270] Two modes of slices are supported: raster scan slice mode and rectangular slice mode. In raster scan slice mode, a slice contains a sequence of complete tiles within the tile raster scan of the picture. In rectangular slice mode, a slice contains either several complete tiles that collectively form a rectangular area of the picture, or several consecutive complete CTU rows of one tile that collectively form a rectangular area of the picture. The tiles in a rectangular slice are scanned in tile raster scan order within the rectangular area corresponding to the slice.
[0271] A subpicture contains one or more slices that collectively cover a rectangular area of the picture.
[0272] FIG. 32 shows an example of raster scan slice partitioning of a picture, where the picture is divided into 12 tiles and 3 raster scan slices.
[0273] FIG. 33 shows an example of rectangular slice partitioning of a picture, where the picture is divided into 24 tiles (6 tile columns and 4 tile rows) and 9 rectangular slices.
[0274] FIG. 34 shows an example of a picture partitioned into tiles and rectangular slices, where the picture is divided into four tiles (two columns of tiles and two rows of tiles) and four rectangular slices.
[0275] Figure 35 shows an example of sub-picture partitioning of a picture, where the picture is divided into 18 tiles, with 12 tiles on the left hand side covering one slice of a 4x4 CTU each, and 6 tiles on the right hand side covering two vertically stacked slices of a 2x2 CTU each, generating in total 24 slices of various dimensions and 24 sub-pictures (each slice is a sub-picture).
[0276] 360° Video Encoding Tool Horizontal wraparound motion compensation Horizontal wraparound motion compensation in VTM10 is a 360-specific coding tool designed to improve the visual quality of 360-degree videos reconstructed in equi-rectangular projection (ERP) format [5]. In traditional motion compensation, when a motion vector references a sample that crosses the picture boundary of the reference picture, iterative padding is applied to derive the value of the out-of-bounds sample by copying from the nearest sample on the corresponding picture boundary. For 360-degree videos, this iterative padding method is not suitable and can result in visual artifacts called "seam artifacts" in the reconstructed viewport video. Because 360-degree videos are captured on a sphere and are essentially "borderless", reference samples that are outside the boundary of the reference picture in the projected region can always be obtained from neighboring samples in the spherical region. For common projection formats, it can be difficult to derive the corresponding neighboring samples in the spherical region, as it requires 2D-to-3D and 3D-to-2D coordinate transformations [5] as well as sample interpolation of fractional sample positions. For the left and right boundaries of the ERP projection format, this problem is much simpler, since the spherical neighborhood outside the left picture boundary can be obtained from samples in the right picture boundary, and vice versa. Considering that the ERP projection format is widely used and relatively easy to implement, horizontal wraparound motion compensation was adopted in VVC to improve the visual quality of 360-degree videos encoded in the ERP projection format.
[0277] The horizontal wraparound motion compensation process is as depicted in Figure 36. If a portion of a reference block is outside the left (or right) boundary of the reference picture in the projected region, instead of repetitive padding, its "out-of-bounds" portion is obtained from the corresponding spherical neighborhood located in the reference picture towards the right (or left) boundary in the projected region. Repetitive padding is used only for the top and bottom picture boundaries. As depicted in Figure 36, horizontal wraparound motion compensation can be combined with non-standard padding methods often used in 360-degree video coding (see padded ERP in [5]). In VVC, this is achieved by signaling a high-level syntax element indicating the wraparound offset, which should be set to the ERP picture width before padding, and this syntax is used to adjust the position of the horizontal wraparound accordingly. This syntax is not affected by specific padding amounts on the left and right picture boundaries, and therefore naturally supports asymmetric padding of ERP pictures, i.e., when the left and right padding are different. Horizontal wraparound motion compensation provides more meaningful information for motion compensation when the reference samples are outside the left and right boundaries of the reference picture. Under the 360° video CTC [4], this tool improves compression performance not only in terms of rate-distortion performance, but also in terms of reducing seam artifacts and improving the subjective quality of the reconstructed 360° video. Horizontal wraparound motion compensation can also be used for other single-plane projection formats with constant sampling density in the horizontal direction, such as the adjusted equal-area projection in 360Lib [5].
[0278] [Table 7]
[0279] In existing video codecs such as HEVC and VVC, a reference block of an inter-mode coded CU (hereinafter referred to as inter CU) may be located partially or entirely outside a reference picture because an iterative padding process is applied to the reference pictures to generate reference pixels around each reference picture.
[0280] In the VVC specification, the padding process is implemented by modifying the integer reference sample fetching process (as shown in Table 6): Whenever an integer reference sample to be fetched is located outside the reference picture, the nearest integer reference sample in the reference picture is used instead.
[0281] With padded reference pictures, it is valid for inter CUs to have reference blocks that are partially or entirely outside the reference picture as shown in Figure 27, where bidirectional motion compensation is performed to generate inter predicted blocks for the current block. In this example, the reference blocks in list 0 are partially out of bounds (OOB), while the reference blocks in list 1 are entirely inside the reference picture.
[0282] When an inter block is bidirectionally or multidirectionally predicted, the final predictor is simply the average of two or more motion compensated prediction blocks, and when BWC is further applied, different weights are applied to the MC predictors in list 0 and list 1 to generate the final predictor, respectively. However, the out-of-bounds (OOB) part of a motion compensated block has a low prediction effect because the OOB part is simply a repeating pattern generated by the boundary pixels in the reference picture. However, in existing video codecs, the low-effect OOB part of an MC block is not specially considered in inter prediction. Similarly, in VVC, wrap-around motion compensation may also be used. The same problem may be observed in the wrap-around MC scheme.
[0283] In this disclosure, some methods are proposed to improve inter prediction by considering the less effective OOB part of MC block. Furthermore, "block" is used to explain the concept of this disclosure, and "block" may be easily replaced by any concrete definition used in existing codecs. For example, "block" may be coding unit (CU), prediction unit (PU), transform unit (TU), coding tree unit (CTU), coding block (CB), prediction block (PB), transform block (TB), coding tree block (CTB), sub-CU, sub-PU, or any other well-defined term in existing codecs.
[0284] The following methods may be applied alone or in combination.
[0285] In one embodiment according to the present disclosure, when combining two or more prediction blocks generated by a motion compensation process, an additional weighting is applied to the predictor, and the additional weighting is derived based on whether the prediction sample is OOB or not. The basic concept is that when performing a weighted average of two predictor blocks, the OOB predictor sample is given less weight because it has a lower effect.
[0286] JPEG2024540760000037.jpg102167
[0287] As shown in Figure 28, the predictor sample of a block is derived by performing motion compensation from a reference picture. The motion vector of the current block is used to locate the reference block. It should be noted that if the reference block is located at a fractional position between integer reference samples, a fractional sample interpolation process is further applied to derive the predicted sample. As described in VVC subclause 8.5.6.3.2 "Luma Sample Interpolation Filtering Process", fractional sample interpolation first performs a vertical one-dimensional interpolation that accesses eight vertically adjacent integer samples, followed by a horizontal one-dimensional interpolation that accesses the eight vertically interpolated horizontally adjacent samples. In ECM, the 8-tap interpolation filter used in VVC is further replaced by a 12-tap filter.
[0288] JPEG2024540760000038.jpg34167
[0289] The weights may be derived using the following method.
[0290] JPEG2024540760000039.jpg85162JPEG2024540760000040.jpg93162JPEG2024540760000041.jpg91166JPEG2024540760000042.jpg91166
[0291] FIG. 30 shows an example of an encoding method according to the present disclosure. In step 3001, the decoder derives a first reference picture and a second reference picture of a current coding block. In step 3002, the decoder derives a first predictor sample from the first reference picture based on a first motion vector associated with the first reference picture using a motion compensation process. In step 3003, the decoder derives a second predictor sample from the second reference picture based on a second motion vector associated with the second reference picture using a motion compensation process. In step 3004, the decoder obtains a final predictor sample in the current coding block based on at least one of the first predictor sample or the second predictor sample and an out-of-bounds (OOB) condition. This example method may be performed by an encoder as well.
[0292] JPEG2024540760000043.jpg131166
[0293] JPEG2024540760000044.jpg85166JPEG2024540760000045.jpg92166JPEG2024540760000046.jpg92166JPEG2024540760000047.jpg92166
[0294] JPEG2024540760000048.jpg120167
[0295] JPEG2024540760000049.jpg86165JPEG2024540760000050.jpg93165JPEG2024540760000051.jpg92166JPEG2024540760000052.jpg92166
[0296] JPEG2024540760000053.jpg118166
[0297] JPEG2024540760000054.jpg90163JPEG2024540760000055.jpg97163JPEG2024540760000056.jpg92165JPEG2024540760000057.jpg91165
[0298] Based on the motion compensation scheme described above, if at least one of the integer reference samples used to generate a predictor sample (through the interpolation process) is located outside the reference picture, the predictor sample is defined as OOB.
[0299] However, in some cases, only a few OOB reference integer samples are used to generate a predictor sample through an interpolation process, in which case the predictor sample may still provide an efficient prediction (e.g., only two of the eight reference integer samples are OOB). Therefore, in order to provide a tolerance for the OOB decision, various schemes have been proposed for determining whether a predictor sample is OOB or not.
[0300] In another embodiment of the present disclosure, if at least one of the vertically closest integer reference samples is OOB or at least one of the horizontally closest integer reference samples is OOB, the predictor sample is determined to be OOB. For example, as shown in Figure 28, the horizontally closest integer reference samples of predictor P_0,0^Lx are samples d and e.
[0301] In another embodiment of the present disclosure, a predictor sample is determined to be OOB if at least N of the integer samples used to generate the predictor sample are OOB, where N is any integer.
[0302] In another embodiment of the present disclosure, when performing a weighted average of two or more predictor blocks, predictor samples that use OOB integer reference samples to perform interpolation are given less weight. Furthermore, the weight is inversely proportional to the number of OOB integer reference samples.
[0303] In another embodiment of the present disclosure, when performing weighted averaging of two or more predictor blocks, weights are given to predictor samples that use OOB integer reference samples to perform interpolation. The weights are signaled in the bitstream at different levels, such as sequence level (sequence parameter set), picture level (picture parameter set), slice level (slice header), or block level.
[0304] JPEG2024540760000058.jpg171167
[0305] Note that in this disclosure, it is assumed that the position index starts from the top-left of the picture, e.g., the position index of the top-left sample of the picture is denoted as (0,0), and the position index increases from top to bottom and left to right. In one exemplary scheme, one predictor sample is determined to be OOB using the following condition checks:
[0306] In one exemplary scheme, one predictor sample is determined to be OOB using the following condition checks (M, N, O, P are any integers):
[0307] JPEG2024540760000059.jpg77166
[0308] JPEG2024540760000060.jpg23166
[0309] FIG. 37 shows a flowchart of a video encoding process according to an example of this disclosure. In step 3701, the encoder or decoder may derive a reference picture of a current coding block. In step 3702, the encoder or decoder may use a motion compensation process to derive a predictor sample from the reference picture based on a motion vector associated with the reference picture. In step 3703, the encoder or decoder may determine whether the predictor sample is located outside the reference picture by a certain margin. In step 3704, the encoder or decoder may determine that the predictor sample is OOB in response to determining that the predictor sample is located outside the reference picture by a certain margin.
[0310] In yet another embodiment, the OOB decision of a predictor sample may be made based on a group of predictor samples. In one example, a block of M×N predictor samples, where M is the block width and N is the block height, share the same OOB status (determined as OOB or not OOB) according to the OOB decision process described above (but not limited to the proposed scheme).
[0311] DMVR considering OOB conditions In order to improve the accuracy of MV in merge mode, bilateral matching (BM) based decoder-side motion vector refinement is applied in VVC. In bi-predictive operation, refined MVs around the initial MV in reference picture list L0 and reference picture list L1 are searched. The BM method calculates the distortion between two candidate blocks in reference picture list L0 and list L1. As shown in FIG. 17, the SAD between red blocks based on each MV candidate around the initial MV is calculated. The MV candidate with the lowest SAD becomes the refined MV and is used to generate the bi-predictive signal. In another embodiment, if a sample of a candidate block in reference picture list L0 or list L1 is determined to be OOB, the sample and its related samples of other corresponding candidate blocks are not used to calculate the matching cost (e.g., sample absolute difference sum or sample squared difference sum).
[0312] Template matching considering OOB conditions Template matching (TM) is a decoder-side MV derivation method for refining the motion information of a current CU by finding the closest match between a template in a current picture (i.e., a neighboring block above and / or to the left of the current CU) and a reference block in a reference picture (i.e., the same size as the template). In another embodiment, if a sample of a reference block in a reference picture is determined to be OOB, that sample and its associated sample of the template in the current picture are not used to calculate a matching cost (e.g., sample sum of absolute difference or sample sum of squared difference).
[0313] MHP considering OOB conditions In the multi-hypothesis inter-prediction mode (JVET-M0425), in addition to the conventional bi-predictive signal, one or more additional motion-compensated prediction signals are signaled. The resulting overall prediction signal is obtained by weighted superposition on a sample-by-sample basis. The bi-predictive signal p biand by using the first additional inter prediction signal / hypothesis h3, the resulting prediction signal p3 is obtained as follows: JPEG2024540760000061.jpg7166
[0314] In another embodiment, if a sample of the additional motion compensated prediction is determined to be OOB, that sample is not used to generate the resulting overall prediction signal for the MHP mode.
[0315] LIC considering OOB conditions LIC is an inter-prediction technique for modeling the local illumination variation between a current block and its predicted block as a function derived from pixels between the current block template and the reference block of the current block template. The parameters of the function can be represented by a scale α and an offset β, forming a linear equation, i.e., α*p[x]+β to compensate for illumination changes, where p[x] is a reference sample pointed to by the MV at position x on the reference picture. Since α and β can be derived based on the current block template and the reference block of the current block template, no signaling overhead is required for α and β, except that a LIC flag is signaled to indicate the use of LIC. In another embodiment, if a sample of the reference block of the current block template is determined to be OOB, the sample and its associated sample of the current block template are not used to derive the LIC parameters.
[0316] Affine PROF In another embodiment, when performing PROF, if the samples used to derive the horizontal / vertical gradient values include the OOB reference samples, the PROF offset is set as zero.
[0317] In another embodiment, when running PROF, if the samples used to derive the horizontal / vertical gradient values include OOB reference samples, the horizontal / vertical gradient values are set as zero.
[0318] High-level syntax signaling In another embodiment, one or more syntax elements may be signaled in the bitstream to indicate the enabling / disabling of the proposed MC considering the OOB conditions. Furthermore, one or more syntax elements may be signaled in the bitstream to indicate the parameters or thresholds to be used in the OOB-considered MC. The proposed syntax elements may be signaled at the sequence level (e.g., sequence parameter set), picture level (e.g., picture parameter set), slice level (e.g., slice header), CTU / CTB level, CU / CB level, PU / PB level. Furthermore, the control of the enabling / disabling may be hierarchical. For example, one syntax element is signaled at the sequence level to indicate the enabling / disabling of the proposed OOB scheme. If the syntax element indicates that the OOB scheme is disabled, no further syntax elements are signaled in the bitstream, and if the syntax element indicates that the OOB scheme is enabled, more syntax elements may be signaled at lower levels to indicate the enabling / disabling of OOB at lower levels. For example, if a sequence-level syntax element indicates that OOB is enabled, then one slice-level syntax element is signaled to indicate enabling / disabling of the OOB scheme at the slice level.
[0319] In motion compensation (MC) schemes of existing video coding standards, when a reference sample pointed to by a motion vector is located outside a predefined boundary, either padding of boundary samples or horizontal / vertical wraparound is utilized to enable the MC process to use reference samples located outside the boundary. The predefined boundaries include, but are not limited to, coding picture boundaries, sub-picture boundaries, slice boundaries, and / or tile boundaries. In another embodiment, the proposed MC scheme may be applied to any scheme when motion compensation may be performed based on reference samples located outside the boundary as described above. It is also noted that the proposed scheme may be applied when reference picture resampling (RPR) is used.
[0320] It should be noted that the proposed motion compensation (MC) scheme taking into account the OOB condition is not limited to be applied to the encoding method in the proposed embodiment, but can also be applied to all inter-tools that perform MC to generate predictor samples, such as sub-block MC (sub-block TMVP, affine mode, etc.), OBMC, IBC, SMVD, DMVR, etc., as described in the previous section.
[0321] The above methods may be implemented using an apparatus including one or more circuits including an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a controller, a microcontroller, a microprocessor, or other electronic components. The apparatus may use the circuits in combination with other hardware or software components to perform the above described methods. Each module, sub-module, unit, or sub-unit disclosed above may be at least partially implemented using one or more circuits.
[0322] 29 illustrates a computing environment 1610 coupled with a user interface 1650. The computing environment 1610 may be part of a data processing server. The computing environment 1610 includes a processor 1620, a memory 1630, and an input / output (I / O) interface 1640.
[0323] The processor 1620 typically controls the overall operation of the computing environment 1610, such as operations related to display, data acquisition, data communication, and image processing. The processor 1620 may include one or more processors for executing instructions called for performing all or a portion of the steps in the methods described above. Additionally, the processor 1620 may include one or more modules that facilitate interaction between the processor 1620 and other components. The processor may be a central processing unit (CPU), a microprocessor, a single chip machine, a graphics processing unit (GPU), or the like.
[0324] The memory 1630 is configured to store various types of data to support the operation of the computing environment 1610. The memory 1630 may include predefined software 1632. Examples of such data include instructions for any application or method operated on the computing environment 1610, video data sets, image data, etc. The memory 1630 may be implemented by using any type of volatile or non-volatile memory device or combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic memory, a flash memory, a magnetic disk, or an optical disk.
[0325] The I / O interface 1640 provides an interface between the processor 1620 and a peripheral interface module, such as a keyboard, a click wheel, buttons, etc. The buttons may include, but are not limited to, a home button, a start scan button, and a stop scan button. The I / O interface 1640 may be coupled to an encoder and a decoder.
[0326] In one embodiment, a non-transitory computer-readable storage medium is also provided that includes, for example in memory 1630, a number of programs executable by processor 1620 in computing environment 1610 for performing the methods described above. In one example, the programs may be executed by processor 1620 in computing environment 1610 to receive (e.g., from video encoder 20 of FIG. 2 ) a bitstream or datastream including encoded video information (e.g., video blocks representing encoded video frames, and / or one or more associated syntax elements, etc.) and to perform the above-described decoding method according to the received bitstream or datastream. In another example, the programs may be executed by the processor 1620 in the computing environment 1610 to perform the encoding method described above to encode video information (e.g., video blocks representing video frames, and / or one or more associated syntax elements, etc.) into a bitstream or data stream, and may also be executed by the processor 1620 in the computing environment 1610 to transmit the bitstream or data stream (e.g., to the video decoder 30 of FIG. 3). Alternatively, a non-transitory computer-readable storage medium may store a bitstream or data stream including encoded video information (e.g., video blocks representing encoded video frames, and / or one or more associated syntax elements, etc.) generated by an encoder (e.g., the video encoder 20 of FIG. 2) using, for example, the encoding method described above, for use by a decoder (e.g., the video decoder 30 of FIG. 3) in decoding video data. The non-transitory computer-readable storage medium may be, for example, a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.
[0327] In one embodiment, a computing device is also provided that includes one or more processors (e.g., processor 1620) and a non-transitory computer-readable storage medium or memory 1630 storing a plurality of programs executable by the one or more processors, wherein the one or more processors are configured, upon execution of the plurality of programs, to perform the methods described above.
[0328] In one embodiment, a computer program product is also provided that includes a number of programs, e.g., in memory 1630, executable by the processor 1620 in the computing environment 1610 to perform the methods described above. For example, the computer program product may include a non-transitory computer-readable storage medium.
[0329] In one embodiment, the computing environment 1610 may be implemented using one or more ASICs, DSPs, digital signal processing devices (DSPDs), programmable logic devices (PLDs), FPGAs, GPUs, controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.
[0330] The description of the present disclosure has been presented for purposes of illustration and is not intended to be exhaustive or limiting of the disclosure. Many modifications, variations and alternative implementations will be apparent to one skilled in the art having the benefit of the teachings presented in the foregoing descriptions and the associated drawings.
[0331] Unless otherwise specified, the order of steps of the method according to the present disclosure is intended to be illustrative only, and the steps of the method according to the present disclosure are not limited to the order specifically described above, and may be changed according to actual conditions. Moreover, at least one of the steps of the method according to the present disclosure may be adjusted, combined, or deleted according to actual requirements.
[0332] The examples have been chosen and described in order to explain the principles of the disclosure, to enable those skilled in the art to understand the disclosure in its various implementations, and to make best use of the underlying principles and various implementations with various modifications as appropriate for the particular use contemplated. Thus, it should be understood that the scope of the disclosure is not limited to the particular examples of implementations disclosed, and that modifications and other implementations are intended to be included within the scope of the disclosure.
Claims
1. deriving, by a decoder, a reference picture for a current coding block; deriving, by the decoder, a predictor sample based on a motion vector associated with the reference picture using a motion compensation process from the reference picture; determining, by the decoder, whether the predictor samples are located outside the reference picture by a certain margin; determining, by the decoder, in response to the predictor sample being located outside the reference picture by a margin, that the predictor sample is out of bounds (OOB); A video coding method comprising:
2. determining that the predictor samples are OOB; 2. The video coding method of claim 1, further comprising: determining that the predictor sample is OOB based on whether the predictor sample is located outside the reference picture by a fractional sample distance.
3. determining that the predictor samples are OOB; determining that the predictor sample is OOB in response to determining that a sum of a horizontal position index of the predictor sample and a horizontal component of a predicted motion vector of the current block is greater than or equal to a position index of a right boundary of the reference picture plus a first offset M; or determining that the predictor sample is OOB in response to determining that a sum of the horizontal position index of the predictor sample and the horizontal component of the predictor motion vector of the current block is less than or equal to a position index of a left boundary of the reference picture minus a second offset N; or determining that the predictor sample is OOB in response to determining that a sum of a vertical position index of the predictor sample and a vertical component of the predictor motion vector of the current block is greater than or equal to a position index of a bottom boundary of the reference picture plus a third offset O; or determining that the predictor sample is OOB in response to determining that a sum of the vertical position index of the predictor sample and the vertical component of the predictor motion vector of the current block is less than or equal to a position index of a top boundary of the reference picture minus a fourth offset P; 3. The video coding method of claim 2, wherein the first offset M, the second offset N, the third offset O, and the fourth offset P are integers.
4. deriving a block of predictor samples for the current coding block; in response to determining that one predictor sample in the block of predictor samples is OOB, determining that all predictor samples in the block of predictor samples are OOB; The video coding method of claim 1 , further comprising:
5. In response to determining that a decoder-side motion vector refinement (DMVR) is applied to the current coding block, determining whether a predictor sample of a candidate block in a first reference picture list or a second reference picture list is OOB; determining, in response to determining that the predictor sample of the candidate block is OOB, that the predictor sample of the candidate block in the first reference picture list or the second reference picture list and a corresponding predictor sample of the candidate block in the second reference picture list or the first reference picture list are not used to calculate a matching cost, wherein the matching cost comprises a sample sum of absolute differences or a sample sum of squared differences; The video coding method of claim 1 , further comprising:
6. In response to determining that template matching (TM) is applied to the current coding block, determining whether a predictor sample of a reference block in a reference picture for a current block template is OOB; in response to determining that the predictor samples of the reference block in the reference picture for a current block template are OOB, determining that the predictor samples of the reference block in the reference picture and associated samples of a current block template in a current picture are not used to calculate a matching cost, wherein the matching cost comprises a sample sum of absolute differences or a sample sum of squared differences; The video coding method of claim 1 , further comprising:
7. determining whether a predictor sample of an additional motion compensated prediction is OOB in response to determining that a multi-hypothesis inter prediction (MHP) mode is applied to the current coding block; in response to determining that the predictor sample of the additional motion compensated prediction is OOB, determining that the predictor sample of the additional motion compensated prediction is not used to generate an overall prediction signal of the MHP; The video coding method of claim 1 , further comprising:
8. determining whether a predictor sample of a reference block for a current block template is OOB in response to determining that local illumination compensation (LIC) is applied to the current coding block; in response to determining that the predictor samples of the reference block are OOB, determining that the predictor samples of the reference block and associated samples of a current block template are not used to derive LIC parameters; The video coding method of claim 1 , further comprising:
9. in response to determining that a prediction refinement using optical flow (PROF) is to be performed for the current coding block, determining whether a predictor sample used to drive a horizontal gradient value or a vertical gradient value is OOB; in response to determining that the predictor sample used to drive the horizontal gradient value or the vertical gradient value is OOB, setting the PROF offset associated with the predictor sample as zero; The video coding method of claim 1 , further comprising:
10. in response to determining that a prediction refinement using optical flow (PROF) is to be performed for the current coding block, determining whether a predictor sample used to drive a horizontal gradient value or a vertical gradient value is OOB; in response to determining that the predictor sample used to drive the horizontal gradient value or the vertical gradient value is OOB, setting the horizontal gradient value or the vertical gradient value associated with the predictor sample as zero; The video coding method of claim 1 , further comprising:
11. 2. The video coding method of claim 1, further comprising receiving, by the decoder, from a bitstream a signaled syntax element for indicating whether a motion compensation process that takes OOB conditions into account is enabled, wherein one or more syntax elements are signaled within the bitstream to indicate parameters or thresholds to be used in the motion compensation process that takes OOB conditions into account.
12. 12. The video encoding method of claim 11, wherein the one or more syntax elements are signaled at a sequence level, a picture level, a slice level, a coding tree unit (CTU) or coding tree block (CTB) level, a coding unit (CU) or coding block (CB) level, or a prediction unit (PU) or prediction block (PB) level.
13. receiving, by the decoder, a syntax element signaled at a sequence level to indicate whether the motion compensation process is enabled to take into account OOB conditions; in response to determining that the signaled syntax elements indicate that the motion compensation process that takes into account OOB conditions is disabled, not signaling any further OOB-related syntax elements in the bitstream; in response to determining that the signaled syntax element indicates that the motion compensation process that takes into account OOB conditions is enabled, signaling a syntax element at a lower level to indicate whether the motion compensation process that takes into account OOB conditions is enabled at the lower level; The video coding method of claim 11 , further comprising:
14. 12. The video coding method of claim 11, further comprising signaling a slice-level syntax element to indicate whether the motion compensation process that takes OOB conditions into account is enabled at a slice level in response to determining that a sequence-level syntax element that indicates the motion compensation process that takes OOB conditions into account is enabled.
15. deriving, by an encoder, a reference picture for a current coding block; deriving, by the encoder, a predictor sample from the reference picture based on a motion vector associated with the reference picture using a motion compensation process; determining, by the encoder, whether the predictor samples are located outside the reference picture by a certain margin; determining, by the encoder, in response to the predictor sample being located outside the reference picture by a margin, that the predictor sample is out of bounds (OOB); A video coding method comprising:
16. determining that the predictor samples are OOB; 16. The video coding method of claim 15, further comprising determining that the predictor sample is OOB based on whether the predictor sample is located outside the reference picture by a fractional sample distance.
17. determining that the predictor samples are OOB; determining that the predictor sample is OOB in response to determining that a sum of a horizontal position index of the predictor sample and a horizontal component of a predicted motion vector of the current block is greater than or equal to a position index of a right boundary of the reference picture plus a first offset M; or determining that the predictor sample is OOB in response to determining that a sum of the horizontal position index of the predictor sample and the horizontal component of the predictor motion vector of the current block is less than or equal to a position index of a left boundary of the reference picture minus a second offset N; or determining that the predictor sample is OOB in response to determining that a sum of a vertical position index of the predictor sample and a vertical component of the predictor motion vector of the current block is greater than or equal to a position index of a bottom boundary of the reference picture plus a third offset O; or determining that the predictor sample is OOB in response to determining that a sum of the vertical position index of the predictor sample and the vertical component of the predictor motion vector of the current block is less than or equal to a position index of a top boundary of the reference picture minus a fourth offset P; 17. The video coding method of claim 16, wherein the first offset M, the second offset N, the third offset O, and the fourth offset P are integers.
18. deriving a block of predictor samples for the current coding block; in response to determining that one predictor sample in the block of predictor samples is OOB, determining that all predictor samples in the block of predictor samples are OOB; The video coding method of claim 15, further comprising:
19. In response to determining that a decoder-side motion vector refinement (DMVR) is applied to the current coding block, determining whether a predictor sample of a candidate block in a first reference picture list or a second reference picture list is OOB; determining, in response to determining that the predictor sample of the candidate block is OOB, that the predictor sample of the candidate block in the first reference picture list or the second reference picture list and a corresponding predictor sample of the candidate block in the second reference picture list or the first reference picture list are not used to calculate a matching cost, wherein the matching cost comprises a sample sum of absolute differences or a sample sum of squared differences; The video coding method of claim 15, further comprising:
20. In response to determining that template matching (TM) is applied to the current coding block, determining whether a predictor sample of a reference block in a reference picture for a current block template is OOB; in response to determining that the predictor samples of the reference block in the reference picture for a current block template are OOB, determining that the predictor samples of the reference block in the reference picture and associated samples of a current block template in a current picture are not used to calculate a matching cost, wherein the matching cost comprises a sample sum of absolute differences or a sample sum of squared differences; The video coding method of claim 15, further comprising:
21. determining whether a predictor sample of an additional motion compensated prediction is OOB in response to determining that a multi-hypothesis inter prediction (MHP) mode is applied to the current coding block; in response to determining that the predictor sample of the additional motion compensated prediction is OOB, determining that the predictor sample of the additional motion compensated prediction is not used to generate an overall prediction signal of the MHP; The video coding method of claim 15, further comprising:
22. determining whether a predictor sample of a reference block for a current block template is OOB in response to determining that local illumination compensation (LIC) is applied to the current coding block; in response to determining that the predictor samples of the reference block are OOB, determining that the predictor samples of the reference block and associated samples of a current block template are not used to derive LIC parameters; The video coding method of claim 15, further comprising:
23. in response to determining that a prediction refinement using optical flow (PROF) is to be performed for the current coding block, determining whether a predictor sample used to drive a horizontal gradient value or a vertical gradient value is OOB; setting the PROF offset as zero in response to determining that the predictor sample used to drive the horizontal gradient value or the vertical gradient value is OOB; The video coding method of claim 15, further comprising:
24. in response to determining that a prediction refinement using optical flow (PROF) is to be performed for the current coding block, determining whether a predictor sample used to drive a horizontal gradient value or a vertical gradient value is OOB; in response to determining that the predictor sample used to drive the horizontal gradient value or the vertical gradient value is OOB, setting the horizontal gradient value or the vertical gradient value associated with the predictor sample as zero; The video coding method of claim 15, further comprising:
25. 16. The video coding method of claim 15, further comprising receiving, by the encoder, from a bitstream a signaled syntax element for indicating whether a motion compensation process that takes into account OOB conditions is enabled, wherein one or more syntax elements are signaled within the bitstream to indicate parameters or thresholds to be used in the motion compensation process that takes into account OOB conditions.
26. 26. The video coding method of claim 25, wherein the one or more syntax elements are signaled at a sequence level, a picture level, a slice level, a coding tree unit (CTU) or coding tree block (CTB) level, a coding unit (CU) or coding block (CB) level, or a prediction unit (PU) or predictive block (PB) level.
27. receiving, by the encoder, a syntax element signaled at a sequence level to indicate whether the motion compensation process is enabled to take into account OOB conditions; in response to determining that the signaled syntax elements indicate that the motion compensation process that takes into account OOB conditions is disabled, not signaling any further OOB-related syntax elements in the bitstream; in response to determining that the signaled syntax element indicates that the motion compensation process that takes into account OOB conditions is enabled, signaling a syntax element at a lower level to indicate whether the motion compensation process that takes into account OOB conditions is enabled at the lower level; The video coding method of claim 25, further comprising:
28. 26. The video coding method of claim 25, further comprising signaling a slice-level syntax element to indicate whether the motion compensation process that takes OOB conditions into account is enabled at a slice level in response to determining that a sequence-level syntax element that indicates the motion compensation process that takes OOB conditions into account is enabled.
29. one or more processors; a memory configured to store instructions executable by the one or more processors; 29. A video encoding apparatus comprising: the one or more processors configured, upon execution of the instructions, to perform the method of any of claims 1 to 28.
30. 1. A non-transitory computer-readable storage medium for video encoding, comprising:
29. A non-transitory computer readable storage medium storing computer executable instructions that, when executed by one or more computer processors, cause the one or more computer processors to perform the method of any of claims 1 to 28.
Citation Information
Patent Citations
Sample Derivation for 360-Degree Video Coding
JP2022501905A
Inter-prediction method and image decoding device
US20210266559A1
Clipping in reference picture resampling
WO2021036977A1
Methods for signaling video coding data
WO2021195546A1
Cited By
Multiple Hypothesis Prediction for Video Coding
JP2024501154A
Multi-hypothesis prediction for video coding
JP7795538B2
