Overlapping Block Motion Compensation

Abstract

[0009] The present invention provides a system and techniques for overlapped block motion compensation (OBMC). The method may include determining whether a first condition, a second condition, and a third condition are satisfied for a neighboring sub-block(s) adjacent to the current sub-block, the first condition comprising all reference picture lists for predicting the current sub-block being used to predict the neighboring sub-block, the second condition comprising the same reference picture being used to determine motion vectors associated with the current sub-block and the neighboring sub-block, and the third condition comprising a difference between a motion vector of the current sub-block and a motion vector of the neighboring sub-block not exceeding a threshold, and determining not to use motion information of the neighboring sub-block for motion compensation of the current sub-block based on determining that the OBMC mode is enabled and the first condition, the second condition, and the third condition are satisfied.

Description

[Technical Field]

[0001]

[0001] This application relates generally to video encoding and decoding. For example, aspects of this disclosure relate to systems and techniques for performing overlapped block motion compensation. [Background technology]

[0002] Digital video capabilities can be incorporated into a wide range of devices, including digital televisions, digital direct broadcast systems, wireless broadcast systems, personal digital assistants (PDAs), laptop or desktop computers, tablet computers, e-book readers, digital cameras, digital recording devices, digital media players, video gaming devices, video game consoles, cellular or satellite radiotelephones, so-called "smartphones," video teleconferencing devices, video streaming devices, and the like. Such devices enable video data to be processed and output for consumption. Digital video data comprises a large amount of data to meet the demands of consumers and video providers. For example, video data consumers desire the highest quality video, with high fidelity, resolution, frame rates, and the like. As a result, the large amount of video data required to meet these demands places a strain on communication networks and devices that process and store the video data.

[0003]

[0003] Digital video devices can implement video coding techniques for compressing video data. The video coding may be performed according to one or more video coding standards or formats. For example, video coding standards or formats include, among others, Generic Video Coding (VVC), High Efficiency Video Coding (HEVC), Advanced Video Coding (AVC), MPEG2 Part 2 coding (MPEG stands for Moving Picture Experts Group), and proprietary video codecs / formats such as AOMedia Video1 (AV1) developed by the Alliance for Open Media. Video coding generally utilizes prediction methods (e.g., inter-prediction, intra-prediction, etc.) that exploit redundancy present in a video image or sequence. The goal of video coding techniques is to compress video data into a format that uses a lower bitrate while avoiding or minimizing degradation to video quality. As ever-evolving video services become available, coding techniques with better coding efficiency are needed. Summary of the Invention

[0004]

[0004] Systems, methods, and computer-readable media for implementing overlapped block motion compensation (OBMC) are disclosed. According to at least one example, a method for implementing OBMC is provided.An exemplary method includes determining that an overlapped block motion compensation (OBMC) mode is enabled for a current subblock of a block of video data; and determining, for at least one neighboring subblock adjacent to the current subblock, whether a first condition, a second condition, and a third condition are satisfied, wherein the first condition comprises all of one or more reference picture lists for predicting the current subblock being used to predict the neighboring subblock, the second condition comprises the same one or more reference pictures being used to determine motion vectors associated with the current subblock and the neighboring subblock, and the third condition comprises a first difference between a horizontal motion vector of the current subblock and the horizontal motion vector of the neighboring subblock, and a second difference between a vertical motion vector of the current subblock and the vertical motion vector of the neighboring subblock. and determining, based on determining that the first condition, the second condition, and the third condition are satisfied, not to use motion information of the neighboring sub-blocks for motion compensation of the current sub-block.

[0005] According to at least one example, a non-transitory computer-readable medium for OBMC is provided. The exemplary non-transitory computer-readable medium, when executed by one or more processors, comprises: determining that an overlapped block motion compensation (OBMC) mode is enabled for a current sub-block of a block of video data; and determining, for at least one neighboring sub-block adjacent to the current sub-block, whether a first condition, a second condition, and a third condition are satisfied, wherein the first condition is that all of one or more reference picture lists for predicting the current sub-block are used to predict the neighboring sub-block, and the second condition is that the same one or more reference pictures are used for predicting both the current sub-block and the neighboring sub-block. and a third condition comprising a first difference between the horizontal motion vector of the current sub-block and the horizontal motion vector of the neighboring sub-block and a second difference between the vertical motion vector of the current sub-block and the vertical motion vector of the neighboring sub-block not exceeding a motion vector difference threshold, wherein the motion vector difference threshold is greater than 0. The method may include instructions to: determine to use the OBMC mode for the current sub-block; and determine not to use motion information of the neighboring sub-block for motion compensation of the current sub-block based on determining that the first condition, the second condition, and the third condition are satisfied.

[0006] According to at least one example, an apparatus for OBMC is provided. The example apparatus can include a memory and one or more processors coupled to the memory, where the one or more processors determine that an overlapped block motion compensation (OBMC) mode is enabled for a current sub-block of the block of video data, and determine, for at least one neighboring sub-block adjacent to the current sub-block, whether a first condition, a second condition, and a third condition are satisfied, where the first condition comprises all of one or more reference picture lists for predicting the current sub-block being used to predict the neighboring sub-block, and the second condition comprises all of one or more reference picture lists for predicting the current sub-block being used to predict the neighboring sub-block, and the same one or more reference pictures are used for predicting the current sub-block and the neighboring sub-block. the first condition, the second condition, and the third condition are used to determine a motion vector associated with the current sub-block and the neighboring sub-block, and a third condition is that a first difference between the horizontal motion vector of the current sub-block and the horizontal motion vector of the neighboring sub-block and a second difference between the vertical motion vector of the current sub-block and the vertical motion vector of the neighboring sub-block do not exceed a motion vector difference threshold, wherein the motion vector difference threshold is greater than 0; and based on determining that the first condition, the second condition, and the third condition are satisfied, determine to use the OBMC mode for the current sub-block; and determine not to use motion information of the neighboring sub-block for motion compensation of the current sub-block.

[0007] According to at least one example, another apparatus for OBMC is provided. The example apparatus includes: determining that an overlapped block motion compensation (OBMC) mode is enabled for a current sub-block of a block of video data; and determining, for at least one neighboring sub-block adjacent to the current sub-block, whether a first condition, a second condition, and a third condition are satisfied, the first condition comprising: all of one or more reference picture lists for predicting the current sub-block being used to predict the neighboring sub-block; the second condition comprising the same one or more reference pictures being used to determine a motion vector associated with the current sub-block and the neighboring sub-block; and The third condition may comprise a first difference between the horizontal motion vector of the current sub-block and the horizontal motion vector of the neighboring sub-block and a second difference between the vertical motion vector of the current sub-block and the vertical motion vector of the neighboring sub-block not exceeding a motion vector difference threshold, wherein the motion vector difference threshold is greater than 0, and the method may include means for determining to use the OBMC mode for the current sub-block and determining not to use motion information of the neighboring sub-block for motion compensation of the current sub-block based on determining that the first condition, the second condition, and the third condition are satisfied.

[0008]

[0008] In some aspects, the methods, non-transitory computer-readable media, and devices described above may include determining to implement a subblock-boundary OBMC mode for the current subblock based on a decision to use a decoder side motion vector refinement (DMVR) mode, a subblock-based temporal motion vector prediction (SbTMVP) mode, or an affine motion compensation prediction mode for the current subblock.

[0009] In some cases, implementing the sub-block boundary OBMC mode for a current sub-block includes determining a first prediction associated with the current sub-block, a second prediction associated with a first OBMC block adjacent to a top border of the current sub-block, a third prediction associated with a second OBMC block adjacent to a left border of the current sub-block, a fourth prediction associated with a third OBMC block adjacent to a bottom border of the current sub-block, and a fifth prediction associated with a fourth OBMC block adjacent to a right border of the current sub-block, applying a first weight to the first prediction, a second weight to the second prediction, a third weight to the third prediction, and a fifth weight to the fourth prediction. determining a sixth prediction based on a result of applying a fifth weight to the fourth prediction and a sixth weight to the fifth prediction; and generating a blended subblock corresponding to the current subblock based on the sixth prediction.

[0010] In some examples, the second weight, the third weight, the fourth weight, and the fifth weight may each include one or more weight values ​​associated with one or more samples from a corresponding sub-block of the current sub-block. In some cases, a sum of the weight values ​​of the corner samples of the current sub-block is greater than a sum of the weight values ​​of the other boundary samples of the current sub-block. In some examples, a sum of the weight values ​​of the other boundary samples of the current sub-block is greater than a sum of the weight values ​​of the non-boundary samples of the current sub-block.

[0011]

[0011] In some aspects, the methods, non-transitory computer-readable media, and devices described above may include determining to use a local illumination compensation (LIC) mode for an additional block of video data, and skipping signaling of information related to an OBMC mode for the additional block based on the determination to use the LIC mode for the additional block.

[0012]

[0012] In some cases, skipping signaling of information related to the OBMC mode for additional blocks may include signaling a syntax flag with an empty value, the syntax flag being related to the OBMC mode.

[0013] In some aspects, the methods, non-transitory computer-readable media, and apparatus described above can include receiving a signal including a syntax flag with an empty value, the syntax flag associated with an OBMC mode for additional blocks of video data. In some aspects, the methods, non-transitory computer-readable media, and apparatus described above can include determining not to use the OBMC mode for the additional blocks based on the syntax flag with an empty value.

[0014]

[0014] In some examples, skipping signaling of information related to the OBMC mode for the additional blocks may include deciding not to use or enable the OBMC mode for the additional blocks based on a decision to use the LIC mode for the additional blocks, and skipping signaling values ​​related to the OBMC mode for the additional blocks.

[0015]

[0015] In some aspects, the methods, non-transitory computer-readable media, and devices described above may include determining whether OBMC mode is enabled for additional blocks, and determining whether OBMC mode is enabled for additional blocks, and based on a determination to use LIC mode for the additional blocks, determining to skip signaling information related to OBMC mode for the additional blocks.

[0016]

[0016] In some aspects, the methods, non-transitory computer-readable media, and devices described above may include determining to use a coding unit (CU)-boundary OBMC mode for a current sub-block of a block of video data, and determining a final prediction for the current sub-block based on a sum of a first result of applying a weight associated with the current sub-block to a respective prediction associated with the current sub-block and a second result of applying one or more respective weights to one or more respective predictions associated with one or more sub-blocks adjacent to the current sub-block.

[0017]

[0017] In some examples, deciding not to use motion information of neighboring sub-blocks for motion compensation of the current sub-block may include skipping the use of motion information of neighboring sub-blocks for motion compensation of the current sub-block.

[0018] In some examples, the OBMC mode may include a sub-block boundary OBMC mode.

[0019] In some aspects, one or more of the devices described above are part of, can be part of, or can include a mobile device, a camera device, an encoder, a decoder, an Internet of Things (IoT) device, and / or an extended reality (XR) device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device). In some aspects, the device includes a camera device. In some examples, the device may include or be part of a vehicle, a mobile device (e.g., a mobile phone or so-called “smartphone” or other mobile device), a wearable device, a personal computer, a laptop computer, a tablet computer, a server computer, a robotics device or system, an aviation system, or other device. In some aspects, the device includes an image sensor (e.g., a camera) or multiple image sensors (e.g., multiple cameras) for capturing one or more images. In some aspects, the device includes one or more displays for displaying one or more images, notifications, and / or other displayable data. In some embodiments, the device includes one or more speakers, one or more lighting devices, and / or one or more microphones. In some embodiments, the devices described above may include one or more sensors.

[0020]

[0020] This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used separately to determine the scope of the claimed subject matter, which subject matter should be understood by reference to the entire specification of this patent, any or all drawings, and appropriate portions of each claim.

[0021]

[0021] The foregoing, together with other features and embodiments, will become more apparent with reference to the following specification, claims, and accompanying drawings.

[0022]

[0022] In order to explain the manner in which the various advantages and features of the present disclosure may be obtained, a more particular description of the principles described above will be rendered with reference to specific embodiments thereof that are illustrated in the accompanying drawings. With the understanding that these drawings merely illustrate exemplary embodiments of the present disclosure and should not be considered as limiting its scope, the principles herein will be explained and explained with additional specificity and detail through the use of drawings. [Brief explanation of the drawings]

[0023] [Figure 1]

[0023] A block diagram illustrating an example of an encoding device and a decoding device in accordance with some examples of the present disclosure. [Figure 2A]

[0024] 1 is a conceptual diagram illustrating exemplary spatially neighboring motion vector candidates for merge mode, in accordance with some examples of this disclosure. [Figure 2B]

[0025] 1 is a conceptual diagram illustrating example spatially neighboring motion vector candidates for advanced motion vector prediction (AMVP) mode, in accordance with some examples of the present disclosure. [Figure 3A]

[0026] 1 is a conceptual diagram illustrating exemplary temporal motion vector predictor (TMVP) candidates, in accordance with some examples of this disclosure. [Figure 3B]

[0027] 1 is a conceptual diagram illustrating an example of motion vector scaling, according to some examples of the present disclosure. [Figure 4A]

[0028] 1 is a conceptual diagram illustrating an example of neighboring samples of a current coding unit used to estimate motion compensation parameters for the current coding unit, in accordance with some examples of this disclosure. [Figure 4B]

[0029] 1 is a conceptual diagram illustrating an example of neighboring samples of a reference block used to estimate motion compensation parameters for a current coding unit, in accordance with some examples of this disclosure. [Figure 5]

[0030] 10A-10C illustrate an example of coding unit boundary overlapped block motion compensation (OBMC) blending for OBMC mode, in accordance with some examples of this disclosure. [Figure 6]

[0031] 10A and 10B are diagrams illustrating an example of sub-block boundary overlapped block motion compensation (OBMC) blending for OBMC mode, in accordance with some examples of this disclosure. [Figure 7]

[0032] 10 is a table illustrating an example of sums of weighting factors from overlapping block motion compensation sub-blocks used for overlapping block motion compensation, according to some examples of this disclosure. [Figure 8] 10 is a table illustrating an example of sums of weighting factors from overlapping block motion compensation sub-blocks used for overlapping block motion compensation, in accordance with some examples of this disclosure. [Figure 9]

[0033] 1A and 1B are diagrams illustrating example coding units with sub-blocks in a block of video data, in accordance with some examples of this disclosure. [Figure 10]

[0034] 1 is a flowchart illustrating an example process for performing overlapped block motion compensation, in accordance with some examples of this disclosure. [Figure 11]

[0035] 10 is a flowchart illustrating another example process for performing overlapped block motion compensation, in accordance with some examples of this disclosure. [Figure 12]

[0036] 1 is a block diagram illustrating an example video encoding device, in accordance with some examples of this disclosure. [Figure 13]

[0037] 1 is a block diagram illustrating an example video decoding device, in accordance with some examples of this disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0024]

[0038]

[0023] Several aspects and embodiments of the present disclosure are provided below. As will be apparent to those skilled in the art, some of these aspects and embodiments may be applied independently, and some of them may be applied in combination. In the following description, for purposes of explanation, specific details are set forth to provide a thorough understanding of the embodiments of the present application. However, it will be apparent that various embodiments may be practiced without these specific details. The figures and descriptions are not limiting.

[0025]

[0039] The following description merely provides exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. Rather, the following description of exemplary embodiments provides those skilled in the art with an enabling description for implementing the exemplary embodiments. It should be understood that various changes may be made in the function and arrangement of elements without departing from the scope of the present application, as set forth in the appended claims.

[0026]

[0040] Video compression techniques used in video coding may include applying different prediction modes, including spatial prediction (e.g., intra-frame prediction or intra-prediction), temporal prediction (e.g., inter-frame prediction or inter-prediction), inter-layer prediction (across different layers of video data), and / or other prediction techniques, to reduce or remove redundancy inherent in video sequences. A video encoder may partition each picture of an original video sequence into rectangular regions called video blocks or coding units (described in more detail below). These video blocks may be encoded using particular prediction modes.

[0027]

[0041] Motion compensation is generally used in coding video data for video compression. In some examples, motion compensation may include and / or implement algorithmic techniques used to predict frames in a video based on previous and / or future frames of the video by considering the movement of a camera and / or elements (e.g., objects, etc.) in the video. Motion compensation may describe a picture in terms of transforming a reference picture to a current picture. The reference picture may be a picture that is temporally earlier or even from the future. In some examples, motion compensation may improve compression efficiency by allowing images to be accurately synthesized from previously transmitted and / or stored images.

[0028]

[0042] One example of a motion compensation technique includes block motion compensation (BMC), also known as motion-compensated discrete cosine transform (MC DCT), in which a frame is partitioned into non-overlapping blocks of pixels, and each block is predicted from one or more blocks in one or more reference frames. In BMC, blocks are shifted to the position of the predicted block. Such shifts are represented by motion vectors (MVs) or motion compensation vectors. To exploit redundancy between neighboring block vectors, BMC may be used to encode only the difference between the current and previous motion vectors in a video bitstream. In some cases, BMC may introduce discontinuities at block boundaries (e.g., blocking artifacts). Such artifacts generally appear in the form of sharp horizontal and vertical edges that are perceptible by the human eye and can result in false edges and ringing effects (e.g., large coefficients in high-frequency subbands) due to the quantization of coefficients of the Fourier-related transform used for transform coding of the residual frame.

[0029]

[0043] Generally, in BMC, a current reconstructed block is composed of a predicted block from a previous frame (e.g., referenced by a motion vector) and residual data transmitted in the bitstream for the current block. Another example of a motion compensation technique includes overlapped block motion compensation (OBMC). OBMC can increase prediction accuracy and avoid blocking artifacts. In OBMC, a prediction can be or include a weighted sum of multiple predictions. In some cases, a block can be larger in each dimension and can overlap with neighboring blocks. In such cases, each pixel can belong to multiple blocks. For example, in some illustrative examples, each pixel can belong to four different blocks. In such a scheme, OBMC can implement four predictions for each pixel, which are summed to calculate a weighted average.

[0030]

[0044] In some cases, OBMC may be switched on and off using specific syntax (e.g., one or more specific syntax elements) at the CU level. In some examples, there are two direction modes in OBMC (e.g., top, left, right, bottom, or bottom side), including CU boundary OBMC mode and sub-block boundary OBMC mode. When CU boundary OBMC mode is used, an original prediction block using the current CU MV is blended with another prediction block (e.g., an "OBMC block") using a neighboring CU MV. In some examples, the top-left sub-block in a CU (e.g., the first or left-most sub-block on the first / top row of a CU) may have both top and left OBMC blocks, and other top-most sub-blocks (e.g., other sub-blocks on the first / top row of a CU) may only have top OBMC blocks. Other left-most sub-blocks (e.g., sub-blocks on the first column of a CU to the left of the CU) may only have left OBMC blocks.

[0031]

[0045] Sub-block boundary OBMC mode may be enabled when a sub-CU coding tool is enabled in the current CU (e.g., affine motion compensation prediction, advanced temporal motion vector prediction (ATMVP), etc.). In sub-block boundary mode, a separate OBMC block using the MV of a connected neighboring sub-block may be successively blended with the original predicted block using the MV of the current sub-block. In some cases, CU boundary OBMC mode may be implemented before sub-block boundary OBMC mode, and a predefined blending order for sub-block boundary OBMC mode may include top, left, bottom, and right.

[0032]

[0046] Prediction based on the MV of neighboring sub-block N (e.g., sub-blocks above the current sub-block, to the left of the current sub-block, below the current sub-block, and to the right of the current sub-block) is P N The MV-based prediction of the current sub-block can be denoted as P C When sub-block N contains the same motion information as the current sub-block, the original prediction block may not be blended with the MV-based prediction block of sub-block N. In some cases, P N The four rows / columns of samples in P C In some instances, P N For P, weighting factors of 1 / 4, 1 / 8, 1 / 16, 1 / 32 may be used. C For P, the corresponding weighting factors 3 / 4, 7 / 8, 15 / 16, 31 / 32 may be used. In some cases, if the coding block height or width is equal to 4 or if the CU is coded in sub-CU mode, P N Only two rows and / or columns in may be enabled for OBMC blending.

[0033]

[0047] Described herein are systems, apparatuses, methods, and computer-readable media (hereinafter collectively referred to as "systems and techniques") for implementing improved video coding. In some aspects, the systems and techniques described herein may be used to implement overlapped block motion compensation (OBMC). For example, local illumination compensation (LIC) is a coding tool for modifying the illumination of a current prediction block based on a reference block with a linear model using a scaling factor and an offset. In some aspects, because OBMC and LIC both adjust prediction, the systems and techniques described herein can disable OBMC when LIC is enabled, or can disable LIC when OBMC is enabled. Alternatively, in some aspects, the systems and techniques described herein can skip OBMC signaling when LIC is enabled, or can skip LIC signaling when OBMC is enabled.

[0034]

[0048] In some aspects, the systems and techniques described herein may implement multi-hypothesis prediction (MHP) to improve inter-prediction modes, such as advanced motion vector prediction (AMVP) mode, skip and merge mode, and intra-prediction modes. In some examples, the systems and techniques described herein may combine prediction modes with extra merge indexed prediction. Merge indexed prediction may be performed as in merge mode, where a merge index is signaled to collect motion information for motion-compensated prediction. Because OBMC and MHP generally require access to different reference pictures for prediction, the decoder may utilize large buffers for processing. To reduce memory buffers, the systems and techniques described herein may disable OBMC when MHP is enabled, or disable MHP when OBMC is enabled. In other examples, the systems and techniques described herein may instead skip OBMC signaling when MHP is enabled, or skip MHP signaling when OBMC is enabled. In some cases, the systems and techniques described herein may allow MHP and OBMC to be enabled simultaneously when the current slice is an inter-B slice.

[0035]

[0049] Some video coding standards, such as VVC, support a geometric partitioning mode (GEO) for inter prediction. When this mode is used, a CU may be split into two parts by a geometrically located line. The location of the splitting line may be mathematically derived from the angle and offset parameters of a particular partition. Because OBMC and GEO generally need to access different reference pictures for prediction, a decoder may utilize a large buffer for processing. In some cases, to reduce the memory buffer, the systems and techniques described herein may disable OBMC when GEO is enabled, disable GEO when OBMC is enabled, skip OBMC signaling when GEO is enabled, or skip GEO signaling when OBMC is enabled. In some cases, GEO and OBMC may be allowed to be enabled simultaneously when the current slice is an inter B-slice.

[0036]

[0050] In some video coding standards, such as VVC, affine motion compensation prediction, sub-block-based temporal motion vector prediction (SbTMVP), and decoder-side motion vector refinement (DMVR) may be supported for inter prediction. These coding tools generate different MVs for sub-blocks in a CU. SbTMVP mode may be one of the affine merge candidates. Therefore, in some examples, the systems and techniques described herein may enable the sub-block boundary OBMC mode to be enabled when the current CU uses the affine motion compensation prediction mode, when the current CU enables SbTMVP, or when the current CU enables DMVR. In some cases, the systems and techniques described herein may infer that the sub-block boundary OBMC mode is enabled when the current CU enables DMVR.

[0037]

[0051] In some cases, the CU boundary OBMC mode and / or the sub-block boundary OBMC mode may apply different weighting factors. In other cases, the CU boundary OBMC mode and the sub-block boundary OBMC mode may share the same weighting factor. For example, in JEM, the CU boundary OBMC mode and the sub-block boundary OBMC mode may share the same weighting factor, i.e., the final prediction for blending is P=W C *P C +W N *P N where P N represents the MV-based prediction of the neighboring sub-block N (e.g., the upper, left, lower, and right sub-blocks), and P C is the MV-based prediction of the current sub-block, and the difference between the CU boundary OBMC mode and the sub-block boundary OBMC mode is C and W N Use the same value of the weighting factor W N may be set as 1 / 4, 1 / 8, 1 / 16, 1 / 32 for the sample row / column of the current sub-block that is first, second, third, and fourth closest to the neighboring sub-block N, respectively. The sub-block may have a size of 4x4. The first element 1 / 4 is for the sample row or column that is closest to the neighboring sub-block N, and the last element 1 / 32 is for the sample row or column that is farthest to the neighboring sub-block N. The weight W of the current sub-block C is 1-W N (weight of neighboring sub-block). Because sub-blocks in a CU for sub-CU mode may have more connections to neighboring blocks, the weighting factor for sub-block boundary OBMC mode may be different from that for CU boundary OBMC mode. Therefore, the systems and techniques described herein can provide different weighting factors.

[0038]

[0052] In some examples, the weighting factor may be: In CU boundary OBMC mode, WN can be set as {a1, b1, c1, d1}. Otherwise, W N may be set as {a2, b2, c2, d2}, where {a1, b1, c1, d1} is different from {a2, b2, c2, d2}. In examples, a2 may be less than a1, b2 may be less than b1, c2 may be less than c1, and / or d2 may be less than d1.

[0039]

[0053] In JEM, the predefined blending order for sub-block boundary OBMC mode is top, left, bottom, and right. In some cases, this order may increase computation complexity, decrease performance, result in unequal weighting, and / or result in inconsistencies. In some examples, this sequential order may cause problems because sequential computation is not amenable to parallel hardware design. In some cases, this may result in unequal weighting. For example, during the blending process, OBMC blocks of neighboring sub-blocks in later sub-block blending may contribute more to the final sample prediction value than in earlier sub-block blending. The systems and techniques described herein can blend the prediction value of the current sub-block with four OBMC sub-blocks in one formula and fix the weighting factor without favoring a particular neighboring sub-block. For example, the final prediction may be calculated as P = w * P c +w2*P top +w3*P left +w4*P below +w5*P right where P top is the prediction based on the MV of the upper neighboring sub-block, and P left is the prediction based on the MV of the left neighboring subblock, and P below is the prediction based on the MV of the lower neighboring sub-block, and P rightis a prediction based on the MV of the right neighboring sub-block, and w1, w2, w3, w4, and w5 are weighting factors. In some cases, the weight w1 may be equal to 1 - w2 - w3 - w4 - w5. The prediction based on the MV of the neighboring sub-block N can add / include / introduce noise to the samples in the row / column that is farthest from the sub-block N. Therefore, the systems and techniques described herein can set the values for each of the weights w2, w3, w4, and w5 to {a, b, c, 0} for the sample row / column of the current sub-block that is {first, second, third, fourth} closest to the neighboring sub-block N, respectively. For example, the first element a can be for the sample row or column of the current sub-block that is closest to the neighboring sub-block N, e.g., adjacent to it, and the last element 0 can be for the sample row or column of the current sub-block that is farthest from the neighboring sub-block N. For illustration purposes, using the positions (0,0), (0,1), and (1,1) for the top-left sample of a current sub-block having a size of 4×4 samples as an example, the final prediction P(x,y) can be derived as follows.

[0040]

Number

[0041]

[0054] An example of the sum of the weighting factors (e.g., w2 + w3 + w4 + w5) from the neighboring OBMC sub-blocks for a 4×4 current sub-block can be as shown in Table 1 below. In some cases, the weighting factors can be left-shifted to avoid division operations. For example, {a’, b’, c’, 0} can be set to {a<<shift, b<<shift, c<<shift, 0}, where shift is a positive integer. In this example, the weight w1 may be equal to (1<<shift) - a’ - b’ - c’, and P is (w1*P c +w2*P top +w3*P left +w4*P below +w5*P right+(1<<(shift-1)))>>shift. An example for setting {a', b', c', 0} is {15, 8, 3, 0}, where the values ​​are the original values ​​left-shifted by 6, and w1 is equal to (1<<6)-abc. P=(w1*P c +w2*P top +w3*P left +w4*P below +w5*P right +(1<<5))>>6.

[0042] [Table 1]

[0043]

[0055] In some aspects, the values ​​of w2, w3, w4, and w5 may be set to {a, b, 0, 0} for the sample row / column of the current sub-block that is {first, second, third, fourth} closest to neighboring sub-block N, respectively. For illustrative purposes, using positions (0,0), (0,1), and (1,1) for the top-left sample of the current sub-block having a size of 4x4 samples as an example, the final prediction P(x,y) may be derived as follows:

[0044]

number

[0045]

[0056] An exemplary sum of weighting factors (eg, w2+w3+w4+w5) from neighboring OBMC sub-blocks for a 4x4 current sub-block is shown in Table 2 below.

[0046] [Table 2]

[0047]

[0057] In some examples, the weights may be selected such that the sum of w2 + w3 + w4 + w5 at corner samples (e.g., samples at (0,0), (0,3), (3,0), and (3,3)) is greater than the sum of w2 + w3 + w4 + w5 at other boundary samples (e.g., samples at (0,1), (0,2), (1,0), (2,0), (3,1), (3,2), (1,3), and (2,3)), and / or the sum of w2 + w3 + w4 + w5 at boundary samples is greater than the value at intermediate samples (e.g., samples at (1,1), (1,2), (2,1), and (2,2)).

[0048]

[0058] In some cases, some motion compensations are skipped during the OBMC process based on the similarity between the MV of the current sub-block and the MVs of its spatial neighboring blocks / sub-blocks (e.g., above, left, below, and right). For example, every time motion compensation is invoked using motion information from a given neighboring block / sub-block, the MV(s) of the neighboring block(s) may be compared with the MV(s) of the current sub-block based on one or more of the following conditions: The one or more conditions may include, for example, a first condition that all prediction lists used by neighboring blocks / sub-blocks (e.g., either list L0 or list L1 in uni-prediction, or both L0 and L1 in bi-prediction) are also used for predicting the current sub-block; a second condition that the same reference picture(s) are used by the MV(s) of the neighboring sub-block(s) and the MV(s) of the current sub-block; and / or a third condition that the absolute value of the horizontal MV difference between the neighboring MV(s) and the current MV(s) is not greater than (or does not exceed) a predefined MV difference threshold T, and the absolute value of the vertical MV difference between the neighboring MV(s) and the current MV(s) is not greater than a predefined MV difference threshold T (if bi-prediction is used, both L0 MV and L1 MV may be checked).

[0049]

[0059] In some examples, if the first condition, the second condition, and the third condition are met, motion compensation using a given neighboring block / sub-block is not performed, and an OBMC sub-block using the MV of the given neighboring block / sub-block N is disabled and not blended with the original sub-block. In some cases, the CU boundary OBMC mode and the sub-block boundary OBMC mode may have different values ​​of the threshold T. When the mode is the CU boundary OBMC mode, T is set to T1; otherwise, T is set to T2, where T1 and T2 are greater than 0. In some cases, when the conditions are met, a lossy algorithm for skipping neighboring blocks / sub-blocks may be applied only to the sub-block boundary OBMC mode. The CU boundary OBMC mode may instead apply a lossless algorithm for skipping neighboring blocks / sub-blocks when one or more conditions are met, such as a fourth condition that all prediction lists used by neighboring blocks / sub-blocks (e.g., either L0 or L1 in uni-prediction, or both L0 and L1 in bi-prediction) are also used for predicting the current sub-block; a fifth condition that the same reference picture(s) are used by neighboring MV(s) and current MV(s); and a sixth condition that neighboring MV(s) and current MV(s) are the same (if bi-prediction is used, both L0 MV and L1 MV may be checked).

[0050]

[0060] In some cases, when the first, second, and third conditions are met, the lossy algorithm for skipping neighboring blocks / sub-blocks is applied only to the CU boundary OBMC mode. In some cases, the sub-block boundary OBMC mode can apply the lossless algorithm for skipping neighboring blocks / sub-blocks when the fourth, fifth, and sixth conditions are met.

[0051]

[0061] In some aspects, in the CU boundary OBMC mode, a lossy fast algorithm may be implemented to save encoding and decoding time. For example, a first OBMC block and an adjacent OBMC block may be merged and jointly generated into a larger OBMC block if one or more conditions are met. The one or more conditions may include, for example, a condition that all prediction lists (e.g., either L0 or L1 in uni-prediction, or both L0 and L1 in bi-prediction) used by the first neighboring block of the current CU are also used for predicting the second neighboring block of the current CU (which is in the same direction as the first neighboring block); a condition that the same reference picture(s) are used by the MV of the first neighboring block and the MV of the second neighboring block; and a condition that the absolute value of the horizontal MV difference between the MV of the first neighboring block and the MV of the second neighboring block is not greater than a predefined MV difference threshold T3, and the absolute value of the vertical MV difference between the MV of the first neighboring block and the MV of the second neighboring block is not greater than a predefined MV difference threshold T3 (if bi-prediction is used, both L0 MV and L1 MV may be checked).

[0052]

[0062] In some aspects, in the sub-block boundary OBMC mode, a lossy fast algorithm may be implemented to save encoding and decoding time. In some examples, the SbTMVP mode and DMVR are implemented on an 8x8 basis, and affine motion compensation is implemented on a 4x4 basis. The systems and techniques described herein can implement the sub-block boundary OBMC mode on an 8x8 basis. In some cases, the systems and techniques described herein can perform a similarity test on every 8x8 sub-block to determine whether the 8x8 sub-block should be split into four 4x4 sub-blocks, and if so, OBMC is performed on a 4x4 basis. In some examples, the algorithm may include, for each 8x8 sub-block, four 4x4 OBMC sub-blocks (e.g., P, Q, R, and S), and may satisfy the following conditions: a first condition that the prediction list(s) used by sub-blocks P, Q, R, and S (e.g., either L0 or L1 in uni-prediction, or both L0 and L1 in bi-prediction) are the same; a second condition that the same reference picture(s) are used by the MVs of sub-blocks P, Q, R, and S; and a third condition that the absolute value of the horizontal MV difference between the MVs of any two sub-blocks (e.g., P and Q, P and R, P and S, Q and R, Q and S, and R and S) is not greater than a predefined MV difference threshold T4, and the absolute value of the vertical MV difference between the MVs of any two sub-blocks (e.g., P and Q, P and R, P and S, Q and R, Q and S, and R and S) is not greater than a predefined MV difference threshold T4 (if bi-prediction is used, the L0 MV and L1 MV are not greater than the L1 MV). MV) is not satisfied.

[0053]

[0063] If all of the above conditions are met, the systems and techniques described herein can perform 8x8 sub-block OBMC, where 8x8 OBMC sub-blocks from the top, left, bottom, and right MVs are generated using OBMC blending for the sub-block boundary OBMC mode. Otherwise, when at least one of the above conditions is not met, OBMC is performed on a 4x4 basis in this 8x8 sub-block, where every 4x4 sub-block in the 8x8 sub-block generates four OBMC sub-blocks from the top, left, bottom, and right MVs.

[0054]

[0064] In some aspects, when a CU is coded in merge mode, the OBMC flag is copied from a neighboring block in a manner similar to motion information copying in merge mode. Alternatively, when a CU is not coded in merge mode, the OBMC flag may be signaled to indicate whether OBMC is applied for the CU.

[0055]

[0065] The systems and techniques described herein may be applied to any existing video codec (e.g., High Efficiency Video Coding (HEVC), Advanced Video Coding (AVC), or other suitable existing video codec) and / or may be efficient coding tools for any developing and / or future video coding standard, such as, for example, Generic Video Coding (VVC), Joint Exploration Model (JEM), VP9, ​​AV1 format / codec, and / or other video coding standards under development or to be developed.

[0056]

[0066] Further details regarding the systems and techniques are described with respect to the figures.

[0057]

[0067] 1 is a block diagram illustrating an example of a system 100 including an encoding device 104 and a decoding device 112. The encoding device 104 may be part of a source device, and the decoding device 112 may be part of a receiving device. The source device and / or the receiving device may include an electronic device such as a mobile or landline telephone handset (e.g., a smartphone, a cellular telephone, etc.), a desktop computer, a laptop or notebook computer, a tablet computer, a set-top box, a television, a camera, a display device, a digital media player, a video gaming console, a video streaming device, an Internet Protocol (IP) camera, or any other suitable electronic device. In some examples, the source device and the receiving device may include one or more wireless transceivers for wireless communication. The coding techniques described herein are applicable to video coding in various multimedia applications, including streaming video transmission (e.g., over the Internet), television broadcasting or transmission, encoding digital video for storage on a data storage medium, decoding digital video stored on a data storage medium, or other applications. As used herein, the term coding may refer to encoding and / or decoding. In some examples, the system 100 may support one-way or two-way video transmission to support applications such as video conferencing, video streaming, video playback, video broadcasting, gaming, and / or video telephony.

[0058]

[0068] The encoding device 104 (or encoder) may be used to encode video data using a video coding standard, format, codec, or protocol to generate an encoded video bitstream. Examples of video coding standards and formats / codecs include ITU-T H.261, ISO / IEC MPEG-1 Visual, ITU-T H.262 or ISO / IEC MPEG-2 Visual, ITU-T H.263, ISO / IEC MPEG-4 Visual, ITU-T H.264 (also known as ISO / IEC MPEG-4 AVC) including its Scalable Video Coding (SVC) extension and Multiview Video Coding (MVC) extension, High Efficiency Video Coding (HEVC) or ITU-T H.265, and Generic Video Coding (VVC) or ITU-T H.266. There are various extensions to HEVC that address multi-layer video coding, including range and screen content coding extensions, 3D video coding (3D-HEVC) and multiview extensions (MV-HEVC), and scalable extensions (SHVC). HEVC and its extensions were developed by the ITU-T Video Coding Experts Group (VCEG) and the ISO / IEC Motion Picture Experts Group (MPEG) Joint Collaboration Team on Video Coding (JCT-VC) and the Joint Collaboration Team on 3D Video Coding Extensions Development (JCT-3V). VP9, ​​AOMedia Video1 (AV1) developed by the Alliance for Open Media Alliance of Open Media (AOMedia), and Essential Video Coding (EVC) are other video coding standards to which the techniques described herein may be applied.

[0059]

[0069] The techniques described herein may be applied to any existing video codec (e.g., High Efficiency Video Coding (HEVC), Advanced Video Coding (AVC), or other suitable existing video codec) and / or may be efficient coding tools for any developing and / or future video coding standard, such as, for example, VVC and / or other video coding standards under development or to be developed. For example, examples described herein may be implemented using video codecs such as VVC, HEVC, AVC, and / or extensions thereof. However, the techniques and systems described herein may also be applicable to other coding standards, codecs, or formats, such as MPEG, JPEG (or other coding standards for still images), VP9, ​​AV1, extensions thereof, or other suitable coding standards that are already available or not yet available or developed. For example, in some examples, encoding device 104 and / or decoding device 112 may operate according to a proprietary video codec / format, such as AV1, an extension of AVI, and / or a successor version of AV1 (e.g., AV2), or other proprietary format or industry standard. Thus, while the techniques and systems described herein may be described with reference to a particular video coding standard, those skilled in the art will appreciate that the description should not be construed as applying only to that particular standard.

[0060]

[0070] 1, video source 102 may provide video data to encoding device 104. Video source 102 may be part of a source device or part of a device other than the source device. Video source 102 may include a video capture device (e.g., a video camera, a camera phone, a video phone, etc.), a video archive containing stored video, a video server or content provider providing video data, a video feed interface receiving video from a video server or content provider, a computer graphics system for generating computer graphics video data, a combination of such sources, or any other suitable video source.

[0061]

[0071] The video data from the video source 102 may include one or more input pictures or frames. A picture or frame, in some cases, is a still image that is part of a video. In some examples, the data from the video source 102 may be a still image that is not part of a video. In HEVC, VVC, and other video coding specifications, a video sequence may include a series of pictures. A picture may include three sample arrays, denoted SL, SCb, and SCr. SL is a two-dimensional array of luma samples, SCb is a two-dimensional array of Cb chrominance samples, and SCr is a two-dimensional array of Cr chrominance samples. Chrominance samples are sometimes referred to herein as “chroma” samples. A pixel may refer to all three components (luma and chroma samples) for a given location in the picture's array. In other cases, a picture may be monochrome and include only an array of luma samples, in which case the terms pixel and sample may be used interchangeably. For example techniques described herein that refer to individual samples for purposes of explanation, the same techniques may be applied to pixels (e.g., all three sample components for a given location in an array of pictures). For example techniques described herein that refer to pixels (e.g., all three sample components for a given location in an array of pictures) for purposes of explanation, the same techniques may be applied to individual samples.

[0062]

[0072] The encoder engine 106 (or encoder) of the encoding device 104 encodes video data to generate a coded video bitstream. In some examples, a coded video bitstream (or "video bitstream" or "bitstream") is a series of one or more coded video sequences. A coded video sequence (CVS) includes a series of AUs (access units) starting with an access unit (AU) in a base layer that has a random access point picture with some properties, up to but not including the next AU in the base layer that has a random access point picture with some properties. For example, some properties of the random access point picture that starts the CVS may include a RASL flag equal to 1 (e.g., NoRaslOutputFlag). In other cases, the random access point picture (with a RASL flag equal to 0) does not start a CVS. An access unit (AU) includes one or more coded pictures and control information corresponding to coded pictures that share the same output time. At the bitstream level, coded slices of a picture are encapsulated in data units called network abstraction layer (NAL) units. For example, an HEVC video bitstream may contain one or more CVSs containing NAL units. Each NAL unit has a NAL unit header. In one example, the header is one byte in H.264 / AVC (excluding multi-layer extensions) and two bytes in HEVC. Syntax elements in the NAL unit header take designated bits and are therefore visible to all types of systems and transport layers, such as transport streams, real-time transport (RTP) protocols, file formats, among others.

[0063]

[0073] Two classes of NAL units exist in the HEVC standard, including video coding layer (VCL) NAL units and non-VCL NAL units. VCL NAL units contain coded picture data that form a coded video bitstream. For example, a sequence of bits that form a coded video bitstream resides in a VCL NAL unit. A VCL NAL unit may contain one slice or slice segment (described below) of coded picture data, while a non-VCL NAL unit contains control information related to one or more coded pictures. In some cases, NAL units may be referred to as packets. An HEVC AU includes VCL NAL units that contain coded picture data and non-VCL NAL units that correspond to the coded picture data (if any). Non-VCL NAL units may contain, in addition to other information, parameter sets with high-level information related to the coded video bitstream. For example, parameter sets may include a video parameter set (VPS), a sequence parameter set (SPS), and a picture parameter set (PPS). In some cases, each slice or other portion of the bitstream may reference a single active PPS, SPS, and / or VPS to enable decoding device 112 to access information that can be used to decode the slice or other portion of the bitstream.

[0064]

[0074] An NAL unit may contain a sequence of bits (e.g., an encoded video bitstream, a CVS of a bitstream, etc.) that form a coded representation of video data, such as a coded representation of a picture in a video. The encoder engine 106 generates coded representations of pictures by partitioning each picture into multiple slices. Slices are independent of other slices such that information in a slice is coded without dependency on data from other slices within the same picture. A slice includes one or more slice segments, including independent slice segments, and one or more dependent slice segments, if present, that depend on previous slice segments.

[0065]

[0075] In HEVC, slices are then partitioned into coding tree blocks (CTBs) of luma samples and chroma samples. A CTB of luma samples and one or more CTBs of chroma samples, together with the syntax for the samples, are called a coding tree unit (CTU). A CTU is sometimes called a "tree block" or "largest coding unit" (LCU). A CTU is the basic processing unit for HEVC encoding. A CTU can be split into multiple coding units (CUs) of various sizes. A CU contains luma and chroma sample arrays called coding blocks (CBs).

[0066]

[0076] The luma and chroma CBs may be further split into prediction blocks (PBs). A PB is a block of luma or chroma component samples that uses the same motion parameters for inter prediction or intra block copy (IBC) prediction (when available or enabled for use). A luma PB and one or more chroma PBs, together with associated syntax, form a prediction unit (PU). For inter prediction, a set of motion parameters (e.g., one or more motion vectors, reference indexes, etc.) is signaled in the bitstream for each PU and used for inter prediction of the luma PB and one or more chroma PBs. The motion parameters are sometimes referred to as motion information. A CB may also be partitioned into one or more transform blocks (TBs). A TB represents a square block of color component samples to which a residual transform (e.g., the same two-dimensional transform in some cases) is applied to code the prediction residual signal. A transform unit (TU) represents a TB of luma and chroma samples and corresponding syntax elements. Transform coding is described in more detail below.

[0067]

[0077] The size of a CU corresponds to the size of a coding mode and may be square in shape. For example, the size of a CU may be 8x8 samples, 16x16 samples, 32x32 samples, 64x64 samples, or any other suitable size up to the size of the corresponding CTU. The phrase "NxN" is used herein to refer to the pixel dimensions of a video block in vertical and horizontal dimensions (e.g., 8 pixels x 8 pixels). The pixels in a block may be arranged in rows and columns. In some implementations, a block may not have the same number of pixels in the horizontal direction as in the vertical direction. Syntax data associated with a CU may, for example, represent the partitioning of the CU into one or more PUs. The partitioning mode may differ between whether the CU is coded in intra-prediction mode or inter-prediction mode. The PU may be partitioned to be non-square in shape. Syntax data associated with a CU may also, for example, represent the partitioning of a CU into one or more TUs according to a CTU. The TUs may be square or non-square in shape.

[0068]

[0078] According to the HEVC standard, transforms may be performed using transform units (TUs). TUs may be different for different CUs. TUs may be sized based on the size of the PUs within a given CU. TUs may be the same size as or smaller than the PUs. In some examples, residual samples corresponding to a CU may be subdivided into smaller units using a quad tree structure known as a residual quad tree (RQT). Leaf nodes of the RQT may correspond to TUs. Pixel difference values ​​associated with the TUs may be transformed to produce transform coefficients. The transform coefficients may then be quantized by the encoder engine 106.

[0069]

[0079] Once a picture of video data is partitioned into CUs, the encoder engine 106 predicts each PU using a prediction mode. The prediction unit or prediction block is then subtracted from the original video data to obtain a residual (described below). For each CU, a prediction mode may be signaled in the bitstream using syntax data. The prediction mode may include intra-prediction (or intra-picture prediction) or inter-prediction (or inter-picture prediction). Intra-prediction exploits the correlation between spatially neighboring samples within a picture. For example, using intra-prediction, each PU is predicted from neighboring image data in the same picture using, for example, DC prediction to find the average value for the PU, planar prediction to fit a flat surface to the PU, directional prediction to extrapolate from neighboring data, or any other suitable type of prediction. Inter-prediction uses temporal correlation between pictures to derive motion-compensated predictions for blocks of image samples. For example, using inter-prediction, each PU is predicted using motion-compensated prediction from image data in one or more reference pictures (before or after the current picture in output order). The decision of whether to code a picture area using inter-picture prediction or intra-picture prediction may be made, for example, at the CU level.

[0070]

[0080] The encoder engine 106 and the decoder engine 116 (described in more detail below) may be configured to operate according to VVC. According to VVC, a video coder (such as the encoder engine 106 and / or the decoder engine 116) partitions a picture into multiple coding tree units (CTUs) (where a CTB for luma samples and one or more CTBs for chroma samples, together with syntax for the samples, are referred to as a CTU). The video coder may partition the CTUs according to a tree structure, such as a quad-tree binary tree (QTBT) structure or a multi-type tree (MTT) structure. The QTBT structure eliminates the concept of multiple partition types, such as the separation between CUs, PUs, and TUs in HEVC. The QTBT structure includes two levels, including a first level partitioned according to quad-tree partitioning and a second level partitioned according to binary tree partitioning. The root node of the QTBT structure corresponds to a CTU. The leaf nodes of the binary tree correspond to coding units (CUs).

[0071]

[0081] In the MTT partitioning structure, blocks may be partitioned using quadtree partitioning, binary tree partitioning, and one or more types of tripletree partitioning. Tripletree partitioning is a partition in which a block is split into three sub-blocks. In some examples, tripletree partitioning splits a block into three sub-blocks without splitting the original block through the center. Partition types in MTT (e.g., quadtree, binary tree, and tripletree) can be symmetric or asymmetric.

[0072]

[0082] When operating according to the AV1 codec, encoding device 104 and decoding device 112 may be configured to code video data in blocks. In AV1, the largest coding block that can be processed is called a superblock. In AV1, a superblock can be either 128x128 luma samples or 64x64 luma samples. However, in successor video coding formats (e.g., AV2), a superblock can be defined by a different (e.g., larger) luma sample size. In some examples, a superblock is the top level of a block quadtree. Encoding device 104 may further partition the superblock into smaller coding blocks. Encoding device 104 may partition the superblock and other coding blocks into smaller blocks using square or non-square partitions. Non-square blocks may include N / 2xN, NxN / 2, N / 4xN, and NxN / 4 blocks. Encoding device 104 and decoding device 112 may perform separate prediction and transform processes for each of the coding blocks.

[0073]

[0083] AV1 also defines tiles of video data. A tile is a rectangular array of superblocks that may be coded independently of other tiles. That is, encoding device 104 and decoding device 112 may encode and decode coding blocks within a tile, respectively, without using video data from other tiles. However, encoding device 104 and decoding device 112 may perform filtering across tile boundaries. Tiles may be uniform or non-uniform in size. Tile-based coding may enable parallel processing and / or multithreading for encoder and decoder implementations.

[0074]

[0084] In some examples, the video coder may use a single QTBT or MTT structure to represent each of the luminance and chrominance components, and in other examples, the video coder may use two or more QTBT or MTT structures, such as one QTBT or MTT structure for the luminance component and another QTBT or MTT structure for both chrominance components (or two QTBT and / or MTT structures for each chrominance component).

[0075]

[0085] The video coder may be configured to use quadtree partitioning, QTBT partitioning, MTT partitioning, superblock partitioning, or other partitioning structures.

[0076]

[0086] In some examples, one or more slices of a picture are assigned a slice type. Slice types include an intra-coded slice (I slice), an inter-coded P slice, and an inter-coded B slice. An I slice (intra-coded frame, independently decodable) is a slice of a picture that is coded only by intra prediction. Therefore, an I slice is independently decodable because it requires only data within the frame to predict any prediction unit or predictive block of the slice. A P slice (unidirectionally predicted frame) is a slice of a picture that can be coded using both intra prediction and unidirectional inter prediction. Each prediction unit or predictive block in a P slice is coded using either intra prediction or inter prediction. When inter prediction is applied, a prediction unit or predictive block is predicted by only one reference picture, and therefore, the reference samples are from only one reference region of a frame. A B slice (bidirectionally predicted frame) is a slice of a picture that can be coded using both intra prediction and inter prediction (e.g., either bi-predictive or uni-predictive). A prediction unit or prediction block of a B slice may be bidirectionally predicted from two reference pictures, where each picture contributes one reference region, and the sample sets of the two reference regions are weighted (e.g., with equal weights or with different weights) to produce a prediction signal for the bidirectionally predicted block. As described above, the slices of a picture are coded independently. In some cases, a picture may be coded as just one slice.

[0077]

[0087] As mentioned above, intra-picture prediction of a picture exploits correlation between spatially neighboring samples within a picture. There are multiple intra-prediction modes (also referred to as "intra modes"). In some examples, intra-prediction of luma blocks includes 35 modes, including planar mode, DC mode, and 33 angular modes (e.g., diagonal intra-prediction mode and angular modes adjacent to the diagonal intra-prediction mode). The 35 modes of intra-prediction are indexed as shown below in Table 1. In other examples, more intra-modes may be defined, including prediction angles that may not yet be represented by the 33 angular modes. In other examples, the prediction angles associated with the angular modes may differ from those used in HEVC.

[0078] [Table 3]

[0079]

[0088] Inter-picture prediction uses temporal correlation between pictures to derive motion-compensated predictions for blocks of image samples. Using a translational motion model, the position of a block in a previously decoded picture (reference picture) is indicated by a motion vector (Δx, Δy), where Δx specifies the horizontal displacement of the reference block relative to the position of the current block and Δy specifies its vertical displacement. In some cases, the motion vector (Δx, Δy) may be integer sample accurate (also called integer accuracy), in which case the motion vector points to an integer pel grid (or integer pixel sampling grid) of the reference frame. In some cases, the motion vector (Δx, Δy) may be fractional sample accurate (also called fractional pel accuracy or non-integer accuracy) to more accurately capture the movement of underlying objects without being restricted to the integer pel grid of the reference frame. The accuracy of the motion vector may be represented by the quantization level of the motion vector. For example, the quantization level may be integer accuracy (e.g., 1 pixel) or fractional pel accuracy (e.g., ¼ pixel, ½ pixel, or other sub-pixel value). When the corresponding motion vector has fractional sample accuracy, interpolation is applied to the reference picture to derive the prediction signal. For example, samples available at integer positions may be filtered (e.g., using one or more interpolation filters) to estimate values ​​at fractional positions. A previously decoded reference picture is indicated by a reference index (refIdx) into a reference picture list. The motion vector and the reference index may be referred to as motion parameters. Two types of inter-picture prediction may be implemented, including uni-prediction and bi-prediction.

[0080]

[0089] In the case of inter-prediction using bi-prediction (also called bidirectional inter-prediction), two sets of motion parameters (Δx0, y0, refIdx0 and Δx1, y1, refIdx1) are used to generate two motion-compensated predictions (from the same reference picture or possibly from different reference pictures). For example, in the case of bi-prediction, each prediction block uses two motion-compensated prediction signals to generate a B prediction unit. The two motion-compensated predictions are then combined to obtain a final motion-compensated prediction. For example, the two motion-compensated predictions may be combined by averaging. In another example, weighted prediction may be used, in which case different weights may be applied to each motion-compensated prediction. Reference pictures that may be used in bi-prediction are stored in two separate lists, denoted as List 0 and List 1. The motion parameters may be derived in the encoder using a motion estimation process.

[0081]

[0090] For inter-prediction using uni-prediction (also called unidirectional inter-prediction), one set of motion parameters (Δx0, y0, refIdx0) is used to generate a motion-compensated prediction from a reference picture. For example, for uni-prediction, each prediction block uses at most one motion-compensated prediction signal to generate a P prediction unit.

[0082]

[0091] A PU may include data related to the prediction process (e.g., motion parameters or other suitable data). For example, when a PU is encoded using intra prediction, the PU may include data representing an intra prediction mode for the PU. As another example, when a PU is encoded using inter prediction, the PU may include data defining a motion vector for the PU. The data defining a motion vector for the PU may represent, for example, a horizontal component (Δx) of the motion vector, a vertical component (Δy) of the motion vector, a resolution of the motion vector (e.g., integer precision, ¼-pixel precision, or ⅛-pixel precision), a reference picture to which the motion vector points, a reference index, a reference picture list for the motion vector (e.g., List 0, List 1, or List C), or any combination thereof.

[0083]

[0092] AV1 includes two general techniques for encoding and decoding coding blocks of video data. The two general techniques are intra-prediction (e.g., intra-frame prediction or spatial prediction) and inter-prediction (e.g., inter-frame prediction or temporal prediction). In the context of AV1, when predicting a block of a current frame of video data using an intra-prediction mode, encoding device 104 and decoding device 112 do not use video data from other frames of the video data. In most intra-prediction modes, video encoding device 104 encodes a block of the current frame based on a difference between sample values ​​in the current block and predicted values ​​generated from reference samples in the same frame. Video encoding device 104 determines the predicted values ​​generated from the reference samples based on the intra-prediction mode.

[0084]

[0093] After performing prediction using intra prediction and / or inter prediction, the encoding device 104 may perform transform and quantization. For example, after prediction, the encoder engine 106 may calculate a residual value corresponding to the PU. The residual value may comprise pixel difference values ​​between the current block of pixels being coded (PU) and a predictive block (e.g., a predicted version of the current block) used to predict the current block. For example, after generating a predictive block (e.g., issuing inter prediction or intra prediction), the encoder engine 106 may generate a residual block by subtracting the predictive block produced by the prediction unit from the current block. The residual block includes a set of pixel difference values ​​that quantify differences between pixel values ​​of the current block and pixel values ​​of the predictive block. In some examples, the residual block may be represented in a two-dimensional block format (e.g., a two-dimensional matrix or array of pixel values). In such examples, the residual block is a two-dimensional representation of pixel values.

[0085]

[0094] Any residual data that may remain after prediction is performed is transformed using a block transform, which may be based on a discrete cosine transform, a discrete sine transform, an integer transform, a wavelet transform, other suitable transform functions, or any combination thereof. In some cases, one or more block transforms (e.g., of size 32x32, 16x16, 8x8, 4x4, or other suitable sizes) may be applied to the residual data in each CU. In some embodiments, TUs may be used for the transform and quantization process implemented by the encoder engine 106. A given CU having one or more PUs may also include one or more TUs. As described in more detail below, residual values ​​may be transformed into transform coefficients using a block transform, and then quantized and scanned using the TUs to produce serialized transform coefficients for entropy coding.

[0086]

[0095] In some embodiments, after intra-predictive coding or inter-predictive coding using the PU of a CU, encoder engine 106 may calculate residual data for the TUs of the CU. The PU may comprise pixel data in the spatial domain (or pixel domain). The TUs may comprise coefficients in the transform domain after application of a block transform. As mentioned above, the residual data may correspond to pixel difference values ​​between pixels of the uncoded picture and predicted values ​​corresponding to the PU. Encoder engine 106 may form TUs including the residual data for the CU and then transform the TUs to produce transform coefficients for the CU.

[0087]

[0096] The encoder engine 106 may perform quantization of the transform coefficients. Quantization provides further compression by quantizing the transform coefficients to reduce the amount of data used to represent the coefficients. For example, quantization may reduce the bit depth associated with some or all of the coefficients. In one example, a coefficient with an n-bit value may be truncated to an m-bit value during quantization, where n is greater than m.

[0088]

[0097] Once quantization is performed, the coded video bitstream includes the quantized transform coefficients, prediction information (e.g., prediction modes, motion vectors, block vectors, etc.), partition information, and any other suitable data, such as other syntax data. Different elements of the coded video bitstream may then be entropy coded by the encoder engine 106. In some examples, the encoder engine 106 may utilize a predefined scan order to scan the quantized transform coefficients to produce serialized vectors that can be entropy coded. In some examples, the encoder engine 106 may perform adaptive scanning. After scanning the quantized transform coefficients to form vectors (e.g., one-dimensional vectors), the encoder engine 106 may entropy code the vectors. For example, the encoder engine 106 may use context-adaptive variable-length coding, context-adaptive binary arithmetic coding, syntax-based context-adaptive binary arithmetic coding, probability interval partition entropy coding, or another suitable entropy coding technique.

[0089]

[0098] The output 110 of the encoding device 104 may send the NAL units constituting the encoded video bitstream data to a decoding device 112 of a receiving device via a communication link 120. The input 114 of the decoding device 112 may receive the NAL units. The communication link 120 may include channels provided by a wireless network, a wired network, or a combination of wired and wireless networks. The wireless network may include any wireless interface or combination of wireless interfaces, including any suitable wireless network (e.g., the Internet or other wide area network, a packet-based network, WiFi, radio frequency (RF), ultra-wideband (UWB), WiFi-Direct, cellular, Long Term Evolution (LTE), WiMax, etc.). The wired network may include any wired interface (e.g., fiber, Ethernet, powerline Ethernet, Ethernet over coaxial cable, digital signal line (DSL), etc.). Wired and / or wireless networks may be implemented using a variety of equipment, such as base stations, routers, access points, bridges, gateways, switches, etc. The encoded video bitstream data may be modulated according to a communication standard, such as a wireless communication protocol, and transmitted to a receiving device.

[0090]

[0099] In some examples, encoding device 104 may store the encoded video bitstream data in storage 108. Output unit 110 may retrieve the encoded video bitstream data from encoder engine 106 or from storage 108. Storage 108 may include any of a variety of distributed or locally accessed data storage media. For example, storage 108 may include a hard drive, a storage disk, flash memory, volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded video data. Storage 108 may also include a decoded picture buffer (DPB) for storing reference pictures for use in inter-prediction. In further examples, storage 108 may correspond to a file server or another intermediate storage device that may store encoded video generated by a source device. In such cases, receiving devices, including decoding device 112, can access the stored video data from the storage device via streaming or download. The file server may be any type of server capable of storing encoded video data and transmitting the encoded video data to receiving devices. Exemplary file servers include web servers (e.g., for websites), FTP servers, network-attached storage (NAS) devices, or local disk drives. Receiving devices may access the encoded video data through any standard data connection, including an Internet connection. This may include wireless channels (e.g., Wi-Fi® connections), wired connections (e.g., DSL, cable modems, etc.), or a combination of both that is suitable for accessing encoded video data stored on a file server. Transmission of the encoded video data from storage 108 may be a streaming transmission, a download transmission, or a combination thereof.

[0091]

[0100] The input 114 of the decoding device 112 may receive encoded video bitstream data and provide the video bitstream data to the decoder engine 116 or to the storage 118 for later use by the decoder engine 116. For example, the storage 118 may include a DPB for storing reference pictures for use in inter-prediction. A receiving device including the decoding device 112 may receive encoded video data to be decoded via the storage 108. The encoded video data may be modulated according to a communication standard, such as a wireless communication protocol, and transmitted to the receiving device. The communication medium for transmitting the encoded video data may comprise any wireless or wired communication medium, such as a radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form part of a packet-based network, such as a local area network, a wide area network, or a global network such as the Internet. The communication medium may include routers, switches, base stations, or any other equipment that may be useful for enabling communication from a source device to a receiving device.

[0092]

[0101] The decoder engine 116 may decode the encoded video bitstream data by entropy decoding (e.g., using an entropy decoder) to extract elements of one or more coded video sequences that make up the encoded video data. The decoder engine 116 may then rescale the encoded video bitstream data and perform an inverse transform on the encoded video bitstream data. The residual data is then passed to a prediction stage of the decoder engine 116. The decoder engine 116 then predicts a block of pixels (e.g., a PU). In some examples, the prediction is added to the output of the inverse transform (the residual data).

[0093]

[0102] Video decoding device 112 may output decoded video to video destination device 122, which may include a display or other output device for displaying the decoded video data to a content consumer. In some aspects, video destination device 122 may be part of a receiving device that includes decoding device 112. In some aspects, video destination device 122 may be part of a separate device other than the receiving device.

[0094]

[0103] In some embodiments, the video encoding device 104 and / or the video decoding device 112 may be integrated with an audio encoding device and an audio decoding device, respectively. The video encoding device 104 and / or the video decoding device 112 may also include other hardware or software necessary to implement the coding techniques described above, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combination thereof. The video encoding device 104 and the video decoding device 112 may be integrated as part of a combined encoder / decoder (CODEC) in their respective devices.

[0095]

[0104] The exemplary system shown in FIG. 1 is one illustrative example that may be used herein. Techniques for processing video data using the techniques described herein may be implemented by any digital video encoding and / or decoding device. Generally, the techniques of this disclosure are implemented by a video encoding device or a video decoding device, although the techniques may also be implemented by a composite video encoder-decoder, commonly referred to as a "codec." Moreover, the techniques of this disclosure may also be implemented by a video preprocessor. The source device and the receiving device are merely examples of coding devices, such that the source device generates coded video data for transmission to the receiving device. In some examples, the source device and the receiving device may operate substantially symmetrically, such that each device includes a video encoding component and a video decoding component. Thus, the exemplary system may support one-way or two-way video transmission between video devices, for example, for video streaming, video playback, video broadcasting, or video telephony.

[0096]

[0105] Extensions to the HEVC standard include a multiview video coding extension called MV-HEVC and a scalable video coding extension called SHVC. MV-HEVC and SHVC extensions share the concept of layered coding, in which different layers are included in a coded video bitstream. Each layer in a coded video sequence is addressed by a unique layer identifier (ID). The layer ID may be present in the header of an NAL unit to identify the layer to which the NAL unit is associated. In MV-HEVC, different layers typically represent different views of the same scene in a video bitstream. In SHVC, different scalable layers are provided that represent a video bitstream at different spatial resolutions (or picture resolutions) or with different reconstruction fidelity. The scalable layers may include a base layer (with layer ID = 0) and one or more enhancement layers (with layer ID = 1, 2, ... n). The base layer may conform to the HEVC first version profile and represent the lowest available layer in the bitstream. Enhancement layers have increased spatial resolution, temporal resolution or frame rate, and / or reconstruction fidelity (or quality) compared to the base layer. Enhancement layers are organized hierarchically and may or may not depend on lower layers. In some examples, different layers may be coded using a single-standard codec (e.g., all layers are encoded using HEVC, SHVC, or other coding standard). In some examples, different layers may be coded using a multi-standard codec. For example, the base layer may be coded using AVC, while one or more enhancement layers may be coded using SHVC and / or MV-HEVC extensions to the HEVC standard.

[0097]

[0106] Generally, a layer includes a set of VCL NAL units and a corresponding set of non-VCL NAL units. NAL units are assigned specific layer ID values. Layers may be hierarchical, in the sense that a layer may depend on a lower layer. A layer set refers to a set of layers represented in a bitstream that is self-contained, meaning that a layer in a layer set may depend on other layers in the layer set in the decoding process, but does not depend on any other layers for decoding. Thus, layers in a layer set can form an independent bitstream that can represent video content. A set of layers in a layer set can be obtained from another bitstream by the operation of a sub-bitstream extraction process. A layer set may correspond to a set of layers that should be decoded when a decoder wishes to operate according to certain parameters.

[0098]

[0107] As previously described, an HEVC bitstream includes a group of NAL units, including VCL NAL units and non-VCL NAL units. VCL NAL units include coded picture data that form a coded video bitstream. For example, a sequence of bits that form a coded video bitstream resides in a VCL NAL unit. Non-VCL NAL units may include, among other information, parameter sets with high-level information related to the coded video bitstream. For example, parameter sets may include a video parameter set (VPS), a sequence parameter set (SPS), and a picture parameter set (PPS). Example parameter set goals include bitrate efficiency, error resiliency, and providing a system layer interface. Each slice references a single active PPS, SPS, and VPS to access information that the decoding device 112 can use to decode that slice. An ID, including a VPS identifier (ID), an SPS ID, and a PPS ID, may be coded for each parameter set. The SPS includes an SPS ID and a VPS ID. The PPS includes a PPS ID and an SPS ID. Each slice header includes a PPS ID. The IDs can be used to identify the active parameter set for a given slice.

[0099]

[0108] The PPS contains information that applies to all slices in a given picture. Thus, all slices in a picture point to the same PPS. Slices in different pictures may also point to the same PPS. The SPS contains information that applies to all pictures in the same coded video sequence (CVS) or bitstream. As previously described, a coded video sequence is a series of AUs (access units) starting with a random access point picture with some properties (e.g., an instantaneous decode reference (IDR) picture or a broken link access (BLA) picture, or other suitable random access point picture) in the base layer (described above) up to and including the next access unit (AU) in the base layer that has a random access point picture with some properties (or the end of the bitstream). The information in the SPS may not change from picture to picture within a coded video sequence. Pictures in a coded video sequence may use the same SPS. The VPS contains information that applies to all layers in a coded video sequence or bitstream. The VPS contains a syntax structure with syntax elements that apply to the entire coded video sequence. In some embodiments, the VPS, SPS, or PPS may be transmitted in-band with the coded bitstream. In some embodiments, the VPS, SPS, or PPS may be transmitted out-of-band in a transmission separate from the NAL units containing the coded video data.

[0100]

[0109] This disclosure may generally refer to “signaling” certain information, such as syntax elements. The term “signaling” may generally refer to communication of values ​​for syntax elements and / or other data used to decode encoded video data. For example, video encoding device 104 may signal values ​​for syntax elements in a bitstream. Generally, signaling refers to generating values ​​in the bitstream. As mentioned above, video source 102 may transport the bitstream to video destination device 122 in substantially real time, or may transport the bitstream to video destination device 122 in non-real time, such as may be done when storing syntax elements in storage 108 for later retrieval by video destination device 122.

[0101]

[0110] A video bitstream may also include supplemental enhancement information (SEI) messages. For example, an SEI NAL unit may be part of the video bitstream. In some cases, the SEI message may contain information that is not required by the decoding process. For example, the information in the SEI message may not be essential for a decoder to decode video pictures of the bitstream, but the decoder may use the information to improve the display or processing of the pictures (e.g., the decoded output). The information in the SEI message may be embedded metadata. In one illustrative example, the information in the SEI message may be used by a decoder-side entity to improve the viewability of the content. In some cases, some application standards may specify the presence of such SEI messages in the bitstream so that quality improvements can be provided to all devices that comply with the application standard (e.g., the carriage of a frame packing SEI message for a frame-compatible planar stereoscopic 3DTV video format, in which a frame packing SEI message is carried for every frame of video, handling of a recovery point SEI message, the use of a pan-scan scan rectangle SEI message in DVB, among many other examples).

[0102]

[0111] As described above, for each block, a set of motion information (also referred to herein as motion parameters) may be available. The set of motion information includes motion information for a forward prediction direction and a backward prediction direction. The forward prediction direction and the backward prediction direction are two prediction directions of a bidirectional prediction mode, in which case the terms "forward" and "backward" do not necessarily have a geometric meaning. Instead, "forward" and "backward" correspond to reference picture list 0 (RefPicList0 or L0) and reference picture list 1 (RefPicList1 or L1) of the current picture. In some examples, when only one reference picture list is available for a picture or slice, only RefPicList0 is available, and the motion information of each block of the slice is always forward.

[0103]

[0112] In some cases, a motion vector, along with its reference index, is used in the coding process (e.g., motion compensation). Such motion vectors with associated reference indexes are denoted as a uni-predictive set of motion information. For each prediction direction, the motion information may include a reference index and a motion vector. In some cases, for simplicity, the motion vector itself may be referred to as being assumed to have an associated reference index. The reference index is used to identify a reference picture in the current reference picture list (RefPicList0 or RefPicList1). The motion vector has horizontal and vertical components that provide an offset from a coordinate location in the current picture to a coordinate in the reference picture identified by the reference index. For example, the reference index may indicate a particular reference picture to be used for a block in the current picture, and the motion vector may indicate where in the reference picture the best matched block (the block that best matches the current block) is located.

[0104]

[0113] In video coding standards, a picture order count (POC) may be used to identify the display order of pictures. Although there are cases where two pictures in one coded video sequence may have the same POC value, this does not generally occur within a coded video sequence. When multiple coded video sequences are present in a bitstream, pictures with the same value of POC may be close to each other in terms of decoding order. The POC value of a picture may be used for reference picture list construction, derivation of reference picture sets as in HEVC, and motion vector scaling.

[0105]

[0114] In H.264 / AVC, each inter macroblock (MB) can be partitioned in four different ways, including one 16x16MB partition, two 16x8MB partitions, two 8x16MB partitions, and four 8x8MB partitions. Different MB partitions within one MB may have different reference index values ​​(RefPicList0 or RefPicList1) for each direction. In some cases, when an MB is not partitioned into four 8x8MB partitions, the MB may have only one motion vector in each direction for each MB partition. In some cases, when an MB is partitioned into four 8x8MB partitions, each 8x8MB partition may be further partitioned into sub-blocks, in which case each sub-block may have a different motion vector in each direction. In some examples, there are four different ways to obtain sub-blocks from an 8x8MB partition, including one 8x8 sub-block, two 8x4 sub-blocks, two 4x8 sub-blocks, and four 4x4 sub-blocks. Each sub-block can have a different motion vector in each direction, so the motion vectors are at levels equal to higher than the sub-block.

[0106]

[0115] In AVC, temporal direct mode can be enabled at either the MB level or the MB partition level for skip and / or direct mode in B slices. For each MB partition, the motion vectors of the blocks co-located with the current MB partition in the current block's RefPicList1[0] are used to derive a motion vector. Each motion vector in the co-located blocks is scaled based on the POC distance.

[0107]

[0116] Spatial direct mode may also be implemented in AVC, for example, in AVC, where direct mode can also predict motion information from spatial neighbors.

[0108]

[0117] As mentioned above, in HEVC, the largest coding unit in a slice is called a coding tree block (CTB). The CTB contains a quadtree, and its nodes are coding units. The size of the CTB can range from 16x16 to 64x64 in the HEVC Main Profile. In some cases, an 8x8 CTB size may be supported. A coding unit (CU) is the same size as the CTB and can be as small as 8x8. In some cases, each coding unit is coded in one mode. When a CU is inter-coded, it can be further partitioned into two or four prediction units (PUs), or it can be just one PU when no further partitioning is applied. When there are two PUs in one CU, they can be half-sized rectangles, or two rectangles with 1 / 4 or 3 / 4 the size of the CU.

[0109]

[0118] When a CU is inter-coded, there is one set of motion information for each PU, and each PU is coded with a unique inter-prediction mode to derive the set of motion information.

[0110]

[0119] For example, for motion prediction in HEVC, there are two inter-prediction modes, including merge mode and advanced motion vector prediction (AMVP) mode for a prediction unit (PU). Skip is considered a special case of merge. In either AMVP mode or merge mode, a motion vector (MV) candidate list is maintained for multiple motion vector predictors. The motion vector(s) of the current PU, as well as the reference index in merge mode, are generated by taking one candidate from the MV candidate list. In some examples, one or more scaling window offsets may be included in the MV candidate list along with the stored motion vector.

[0111]

[0120] In an example in which an MV candidate list is used for motion prediction of a block, the MV candidate list may be constructed separately by the encoding device and the decoding device. For example, the MV candidate list may be generated by the encoding device when encoding a block and may be generated by the decoding device when decoding the block. Information related to motion information candidates in the MV candidate list (e.g., information related to one or more motion vectors, information related to one or more LIC flags that may possibly be stored in the MV candidate list, and / or other information) may be signaled between the encoding device and the decoding device. For example, in merge mode, index values ​​to the stored motion information candidates may be signaled from the encoding device to the decoding device (e.g., in a syntax structure such as a picture parameter set (PPS), a sequence parameter set (SPS), a video parameter set (VPS), a slice header, a supplemental enhancement information (SEI) message sent in the video bitstream or separately from the video bitstream, and / or other signaling). The decoding device may use the signaled reference or index to construct an MV candidate list and obtain one or more motion information candidates from the constructed MV candidate list to use for motion-compensated prediction. For example, decoding device 112 may construct an MV candidate list and use the motion vector (and, in some cases, the LIC flag) from the indexed location for motion prediction of the block. In the case of AMVP mode, in addition to the reference or index, a difference or residual value may also be signaled as a delta. For example, in the case of AMVP mode, the decoding device may construct one or more MV candidate lists and apply the delta value to one or more motion information candidates obtained using the signaled index value when performing motion-compensated prediction of the block.

[0112]

[0121] In some examples, the MV candidate list includes up to five candidates for merge mode and two candidates for AMVP mode. In other examples, a different number of candidates may be included in the MV candidate list for merge mode and / or AMVP mode. A merge candidate may include a set of motion information. For example, the set of motion information may include motion vectors corresponding to both reference picture lists (list 0 and list 1) and a reference index. When a merge candidate is identified by the merge index, the reference picture is used for prediction of the current block, and the associated motion vector is determined. However, under AMVP mode, for each potential prediction direction from either list 0 or list 1, the reference index needs to be explicitly signaled along with the MVP index to the MV candidate list, because the AMVP candidate includes only a motion vector. In AMVP mode, the predicted motion vector may be further refined.

[0113]

[0122] As can be seen above, merge candidates correspond to the full set of motion information, while AMVP candidates contain only one motion vector for a particular prediction direction and reference index. Candidates for both modes are derived similarly from the same spatial and temporal neighboring blocks.

[0114]

[0123] In some examples, merge mode allows an inter-predicted PU to inherit one or more of the same motion vectors, prediction directions, and one or more reference picture indices from an inter-predicted PU that includes motion data locations selected from a group of spatially neighboring motion data locations and one of two temporally co-located motion data locations. In the case of AMVP mode, one or more motion vectors of the PU may be predicatively coded with respect to one or more motion vector predictors (MVPs) from an AMVP candidate list constructed by the encoder and / or decoder. In some cases, in the case of unidirectional inter-prediction of the PU, the encoder and / or decoder may generate a single AMVP candidate list. In some cases, in the case of bidirectional prediction of the PU, the encoder and / or decoder may generate two AMVP candidate lists, one using motion data of spatial and temporal neighboring PUs from the forward prediction direction and one using motion data of spatial and temporal neighboring PUs from the backward prediction direction.

[0115]

[0124] Candidates for both modes may be derived from spatial and / or temporal neighboring blocks. For example, Figures 2A and 2B include conceptual diagrams illustrating spatial neighboring candidates. Figure 2A illustrates spatial neighboring motion vector (MV) candidates for merge mode. Figure 2B illustrates spatial neighboring motion vector (MV) candidates for AMVP mode. Although spatial MV candidates are derived from neighboring blocks for a particular PU (PU0), the methods for generating candidates from blocks differ for merge mode and AMVP mode.

[0116]

[0125] In merge mode, an encoder may form a merging candidate list by considering merging candidates from various motion data positions. For example, as shown in FIG. 2A, up to five spatial MV candidates may be derived for spatially neighboring motion data positions, indicated by numbers 0 through 4 in FIG. 2A. The MV candidates may be ordered in the merging candidate list in the order indicated by the numbers 0 through 4. For example, the positions and orders may include a left position (0), an upper position (1), a top-right position (2), a bottom-left position (3), and a top-left position (4). In FIG. 2A, block 200 includes PU0 202 and PU1 204. In some examples, when a video coder is to code motion information for PU0 202 using merge mode, the video coder may add motion information from spatial neighboring block 210, spatial neighboring block 212, spatial neighboring block 214, spatial neighboring block 216, and spatial neighboring block 218 to the candidate list in the order described above.

[0117]

[0126] In the AVMP mode shown in FIG. 2B, neighboring blocks are divided into two groups: a left group including blocks 0 and 1 and an upper group including blocks 2, 3, and 4. In FIG. 2B, blocks 0, 1, 2, 3, and 4 are labeled as blocks 230, 232, 234, 236, and 238, respectively. Here, block 220 includes PU0 222 and PU1 224, and blocks 230, 232, 234, 236, and 238 represent spatial neighbors to PU0 222. For each group, potential candidates among neighboring blocks that reference the same reference picture as the reference picture indicated by the signaled reference index have the highest priority to be selected to form the final candidate of the group. It is possible that not all neighboring blocks contain motion vectors pointing to the same reference picture. Therefore, if such a candidate cannot be found, the first available candidate will be scaled to form the final candidate, thereby compensating for the temporal distance difference.

[0118]

[0127] 3A and 3B include conceptual diagrams illustrating temporal motion vector prediction. FIG. 3A shows an example CU 300 including PU0 302 and PU1 304. PU0 302 includes a center block 310 for PU0 302 and a bottom-right block 306 for PU0 302. FIG. 3A also shows an outer block 308 for which motion information can be predicted from the motion information of PU0 302, as described below. FIG. 3B shows a current picture 342 including a current block 326 for which motion information should be predicted. FIG. 3B also shows a co-located picture 330 for the current picture 342 (including a co-located block 324 for the current block 326), a current reference picture 340, and a co-located reference picture 332. The co-located block 324 is predicted using the co-located motion vector 320 , which is used as a temporal motion vector predictor (TMVP) candidate 322 for the motion information of block 326 .

[0119]

[0128] A video coder can add a temporal motion vector predictor (TMVP) candidate (e.g., TMVP candidate 322) to the MV candidate list after any spatial motion vector candidate if it is enabled and available. The process of motion vector derivation for a TMVP candidate is the same for both merge mode and AMVP mode. However, in some cases, the target reference index for a TMVP candidate in merge mode is always set to 0.

[0120]

[0129] The primary block location for TMVP candidate derivation is the outer, bottom-right block 306 of the co-located PU 304, as shown in Figure 3A, to compensate for the bias toward the upper and left blocks used to generate spatial neighboring candidates. However, if block 306 is located outside the current CTB (or LCU) row (e.g., as shown by block 308 in Figure 3A), or if motion information for block 306 is not available, the block is replaced with the center block 310 of PU 302.

[0121]

[0130] 3B, a motion vector for a TMVP candidate 322 may be derived from a collocated block 324 of a collocated picture 330, shown at the slice level. Similar to the temporal direct mode in AVC, the motion vector of the TMVP candidate may undergo motion vector scaling, which is performed to compensate for the distance difference between the current picture 342 and the current reference picture 340, and between the collocated picture 330 and the collocated reference picture 332. That is, the motion vector 320 may be scaled to produce the TMVP candidate 322 based on the distance difference between the current picture (e.g., current picture 342) and the current reference picture (e.g., current reference picture 340), and between the collocated picture (e.g., collocated picture 330) and the collocated reference picture (e.g., collocated reference picture 332).

[0122]

[0131] Other aspects of motion prediction are covered in the HEVC standard and / or other standards, formats, or codecs. For example, some other aspects of merge mode and AMVP mode are covered. One aspect includes motion vector scaling. With regard to motion vector scaling, the value of a motion vector may be assumed to be proportional to the distance of pictures in presentation time. A motion vector relates two pictures: a reference picture and a picture that contains the motion vector (i.e., a containing picture). When a motion vector is used to predict another motion vector, the distance between the containing picture and the reference picture is calculated based on a Picture Order Count (POC) value.

[0123]

[0132] For a motion vector to be predicted, both its associated containing picture and reference picture may be different. Therefore, a new distance (based on POC) is calculated. Furthermore, the motion vector may be scaled based on these two POC distances. For spatial neighbor candidates, the containing pictures for two motion vectors are the same, but the reference pictures are different. In HEVC, motion vector scaling is applied to both TMVP and AMVP for spatial and temporal neighbor candidates.

[0124]

[0133] Another aspect of motion prediction involves pseudo motion vector candidate generation. For example, if the motion vector candidate list is not complete, pseudo motion vector candidates are generated and inserted at the end of the list until all candidates are obtained. In merge mode, there are two types of pseudo MV candidates: combined candidates derived only for B slices, and 0 candidates used only for AMVP when the first type does not provide enough pseudo candidates. For each pair of candidates already in the candidate list and with the required motion information, a bidirectional combined motion vector candidate is derived by combining the motion vector of the first candidate that references a picture in list 0 and the motion vector of the second candidate that references a picture in list 1.

[0125]

[0134] In some implementations, a pruning process may be performed when adding or inserting a new candidate into the MV candidate list. For example, in some cases, MV candidates from different blocks may contain the same information. In such cases, storing duplicate motion information of multiple MV candidates in the MV candidate list may lead to redundancy and reduced efficiency of the MV candidate list. In some examples, the pruning process may eliminate or minimize redundancy in the MV candidate list. For example, the pruning process may include comparing a potential MV candidate to be added to the MV candidate list with MV candidates already stored in the MV candidate list. In one illustrative example, the horizontal displacement (Δx) and vertical displacement (Δy) (indicating the position of the reference block relative to the position of the current block) of a stored motion vector may be compared with the horizontal displacement (Δx) and vertical displacement (Δy) of the motion vector of the potential candidate. If the comparison reveals that the motion vector of the potential candidate does not match any of the one or more stored motion vectors, the potential candidate may not be considered a candidate to be pruned and may be added to the MV candidate list. If a match is found based on this comparison, the potential MV candidate is not added to the MV candidate list, which avoids the insertion of an identical candidate. In some cases, to reduce complexity, instead of comparing each potential MV candidate with all existing candidates, only a limited number of comparisons are performed during the pruning process.

[0126]

[0135] Some coding schemes, such as HEVC, support weighted prediction (WP), in which a scaling factor (denoted by a), a shift number (denoted by s), and an offset (denoted by b) are used in motion compensation. Assuming that a pixel value at a position (x, y) in a reference picture is p(x, y), p'(x, y) = ((a * p(x, y) + (1 < < (s - 1))) >> s) + b is used as the predicted value in motion compensation instead of p(x, y).

[0127]

[0136] When WP is enabled, for each reference picture of the current slice, a flag is signaled to indicate whether WP is applied for the reference picture. If WP is applied for a reference picture, a set of WP parameters (i.e., a, s, and b) is sent to the decoder and used for motion compensation from the reference picture. In some examples, to flexibly turn on / off WP for luma and chroma components, the WP flag and WP parameters are signaled separately for the luma component and the chroma component. In WP, one and the same set of WP parameters is used for all pixels in one reference picture.

[0128]

[0137] FIG. 4A illustrates an example of reconstructed samples of neighbors of a current block 402 and neighbor samples of a reference block 404 used for unidirectional inter prediction. A motion vector MV 410 may be coded for the current block 402, where the MV 410 may include a reference index into a reference picture list and / or other motion information to identify the reference block 404. For example, the MV may include horizontal and vertical components that provide an offset from a coordinate location in the current picture to a coordinate in the reference picture identified by the reference index. FIG. 4B illustrates an example of reconstructed samples of neighbors of a current block 422 and neighbor samples of a first reference block 424 and a second reference block 426 used for bidirectional inter prediction. In this case, two motion vectors MV0 and MV1 may be coded for the current block 422 to identify the first reference block 424 and the second reference block 426, respectively.

[0129]

[0138] As previously described, OBMC is an exemplary motion compensation technique that can be implemented for motion compensation. OBMC can increase prediction accuracy and avoid blocking artifacts. In OBMC, a prediction can be or include a weighted sum of multiple predictions. In some cases, a block can be larger in each dimension and can quadrantly overlap with neighboring blocks. Thus, each pixel can belong to multiple blocks. For example, in some exemplary cases, each pixel can belong to four blocks. In such a scheme, OBMC can implement four predictions for each pixel, which are summed up to a weighted average.

[0130]

[0139] In some cases, OBMC may be switched on and off using specific syntax at the CU level. In some examples, there are two direction modes in OBMC (e.g., top, left, right, bottom, or lower side), including CU boundary OBMC mode and sub-block boundary OBMC mode. When CU boundary OBMC mode is used, an original predicted block using the current CU MV is blended with another predicted block (e.g., an "OBMC block") using a neighboring CU MV. In some examples, the top-left sub-block in a CU (e.g., the first or leftmost sub-block on the first / top row of a CU) may have both top and left OBMC blocks, and other top-most sub-blocks (e.g., other sub-blocks on the first / top row of a CU) may only have top OBMC blocks. Other left-most sub-blocks (e.g., sub-blocks on the first column of a CU to the left of a CU) may only have left OBMC blocks.

[0131]

[0140] The sub-block boundary OBMC mode may be enabled when a sub-CU coding tool is enabled in the current CU (e.g., affine motion compensation prediction, advanced temporal motion vector prediction (ATMVP), etc.), which allows different MVs on a sub-block basis. In the sub-block boundary OBMC mode, separate OBMC blocks using MVs of connected neighboring sub-blocks may be blended with an original predicted block using an MV of the current sub-block. In some examples, in the sub-block boundary OBMC mode, separate OBMC blocks using MVs of connected neighboring sub-blocks may be blended in parallel with an original predicted block using an MV of the current sub-block, as further described herein. In other examples, in the sub-block boundary mode, separate OBMC blocks using MVs of connected neighboring sub-blocks may be blended successively with an original predicted block using an MV of the current sub-block. In some cases, the CU boundary OBMC mode may be implemented before the sub-block boundary OBMC mode, and a predefined blending order for the sub-block boundary OBMC mode may include top, left, bottom, and right.

[0132]

[0141] Prediction based on the MV of neighboring sub-block N (e.g., sub-blocks above the current sub-block, to the left of the current sub-block, below the current sub-block, and to the right of the current sub-block) is P N and the MV-based prediction of the current sub-block can be denoted as P C When sub-block N contains the same motion information as the current sub-block, the original prediction block may not be blended with the MV-based prediction block of sub-block N. In some cases, P N The four rows / columns of samples in P C In some instances, P N For P, weighting factors of 1 / 4, 1 / 8, 1 / 16, 1 / 32 may be used. CFor P, the corresponding weighting factors 3 / 4, 7 / 8, 15 / 16, 31 / 32 may be used. In some cases, if the coding block height or width is equal to 4 or if the CU is coded in sub-CU mode, P N Only two rows / columns in can be enabled for OBMC blending.

[0133]

[0142] FIG. 5 is a diagram illustrating an example of OBMC blending for CU boundary OBMC mode. As shown in FIG. 5, when the CU boundary OBMC mode is used, an original predicted block using a current CU motion vector (MV) (denoted as "original block" in FIG. 5) is blended with another predicted block using a neighboring CU MV (denoted as "OBMC block" in FIG. 5). The top-leftmost sub-block of CU 530 may have an upper and a left OBMC block, which may be used to generate a blended block as described herein. The other topmost sub-block of CU 530 has only an upper OBMC block, which may be used to generate a blended block as described herein. For example, sub-block 502 located above CU 530 has only an upper OBMC block, which is shown as OBMC sub-block 504 in FIG. 5. OBMC sub-block 504 may be a sub-block of an upper neighboring CU, which may include one or more sub-blocks. The other leftmost sub-block of CU 530 has only a left OBMC block, which may be used to generate a blended block as described herein. For example, sub-block 506 of CU 530 has only a left OBMC block, shown in Figure 5 as OBMC sub-block 508. OBMC sub-block 508 may be a sub-block of a left neighboring CU, which may include one or more sub-blocks.

[0134]

[0143] In the example shown in FIG. 5, sub-block 502 and OBMC sub-block 504 may be used to generate blended block 515. For example, to generate a first prediction result for sub-block 502, a sample of CU 530 at the location of sub-block 502 may be predicted using the MV of sub-block 502 and then multiplied by weight factor 510. Similarly, to generate a second prediction result for sub-block 502, a sample of CU 530 at the location of sub-block 502 may be predicted using the MV of OBMC sub-block 504 and then multiplied by weight factor 512. The first prediction result generated for sub-block 502 may be summed with the second prediction result generated for sub-block 502 to derive blended block 515. Weight factor 510 may be the same as or different from weight factor 512. In some examples, weight factor 510 may be different from weight factor 512. In some cases, weight factor 510 may depend on the distance from sub-block 502 to the CU boundary of the image data and / or samples being blended (e.g., to the boundary of CU 530), and weight factor 512 may depend on the distance from sub-block 502 to the CU boundary of the image data and / or samples being blended (e.g., to the boundary of CU 530). Weight factor 510 and weight factor 512 may sum to 1.

[0135]

[0144] The sub-block 506 and the OBMC sub-block 508 may be used to generate a blended block 520. For example, to generate a first prediction result for the sub-block 506, a sample of the CU 530 at the location of the sub-block 506 may be predicted using the MV of the sub-block 506 and then multiplied by the weight factor 516. Similarly, to generate a second prediction result for the sub-block 506, a sample of the CU 530 at the location of the sub-block 506 may be predicted using the MV of the OBMC sub-block 508 and then multiplied by the weight factor 518. The first prediction result generated for the sub-block 506 may be added with the second prediction result generated for the sub-block 506 to derive the blended block 520. The weight factor 516 may be the same as or different from the weight factor 518. In some examples, the weight factor 516 may be different from the weight factor 518. In some cases, the weight factor 516 may depend on the distance from the sub-block 506 to the CU boundary of the image data and / or samples being blended (e.g., to the boundary of the CU 530), and the weight factor 518 may depend on the distance from the sub-block 506 to the CU boundary of the image data and / or samples being blended (e.g., to the boundary of the CU 530).

[0136]

[0145] FIG. 6 is a diagram illustrating an example of OBMC blending for sub-block boundary OBMC mode. In some examples, sub-block boundary OBMC mode may be enabled when a sub-CU coding tool, such as an affine mode or tool, an advanced temporal motion vector prediction (ATMVP) mode or tool, or the like, is enabled for the current CU. As shown in FIG. 6, four separate OBMC blocks using MVs of four connected neighboring sub-blocks are blended with an original prediction block using the current sub-block MV. In other words, the MVs from the four separate OBMC blocks are used to generate four predictions of samples of the current sub-block 602 in addition to the original prediction using the current sub-block MV, which are then combined with the original prediction to form a blended block 625. For example, sub-block 602 of CU 630 may be blended with neighboring OBMC blocks 604-610. In some cases, sub-block 602 may be blended with OBMC blocks 604-610 according to the blending order for sub-block boundary OBMC mode. In some examples, the blending order may include a top OBMC block (e.g., OBMC block 604), a left OBMC block (e.g., OBMC block 606), a bottom OBMC block (e.g., OBMC block 608), and finally a right OBMC block (e.g., OBMC block 610). In some cases, the sub-block 602 may be blended in parallel with the OBMC blocks 604-610, as described further herein.

[0137]

[0146] 6, sub-block 602 may be blended with each OBMC block 620 according to equation 622. Equation 622 may be performed once for each of OBMC blocks 604-610, and the respective results may be added to generate blended block 625. For example, OBMC block 620 in equation 622 may represent the OBMC block used in equation 622 from OBMC blocks 604-610. In some examples, weighting factor 612 may depend on the location of image data and / or samples within sub-block 602 being blended. In some examples, weighting factor 612 may depend on the distance of image data and / or samples from each OBMC block being blended (e.g., OBMC block 604, OBMC block 606, OBMC block 608, OBMC block 610).

[0138]

[0147] To illustrate, OBMC block 620 may represent OBMC block 604 when the MV-using prediction of OBMC block 604 is blended with the MV-using prediction of sub-block 602 according to equation 622. Here, the original prediction of sub-block 602 may be multiplied by weight measurement factor 612, and the result may be added to the result of multiplying the MV-using prediction of OBMC block 604 by weight measurement factor 614. OBMC block 620 may also represent OBMC block 606 when the MV-using prediction of OBMC block 606 is blended with the MV-using prediction of sub-block 602 according to equation 622. Here, the original prediction of sub-block 602 may be multiplied by weight measurement factor 612, and the result may be added to the result of multiplying the MV-using prediction of OBMC block 606 by weight measurement factor 614. OBMC block 620 may further represent OBMC block 608 when the MV-using prediction of OBMC block 608 is blended with the MV-using prediction of sub-block 602 according to equation 622. The original prediction of sub-block 602 may be multiplied by weight measurement factor 612, and the result may be added with the result of multiplying the MV-using prediction of OBMC block 608 by weight measurement factor 614. Finally, OBMC block 620 may represent OBMC block 610 when the MV-using prediction of OBMC block 610 is blended with the MV-using prediction of sub-block 602 according to equation 622. The original prediction of sub-block 602 may be multiplied by weight measurement factor 612, and the result may be added with the result of multiplying the MV-using prediction of OBMC block 610 by weight measurement factor 614. The results from equation 622 for each of OBMC blocks 604-610 may be added to derive blended block 625.

[0139]

[0148] Parallel blending according to Equation 622 is amenable to parallel hardware computation design and can avoid or limit unequal weighting, avoid inconsistencies, etc. For example, in JEM, the predefined sequential blending order for the sub-block boundary OBMC mode is top, left, bottom, and right. This order may increase computation complexity, decrease performance, result in unequal weighting, and / or result in inconsistencies. In some examples, this sequential order may cause problems because sequential computation is not amenable to parallel hardware design. Moreover, this sequential order may result in unequal weighting. For example, during the blending process, OBMC blocks of neighboring sub-blocks in later sub-block blending may contribute more to the final sample prediction value than in earlier sub-block blending.

[0140]

[0149] In contrast, the systems and techniques described herein can blend the predictions of the current sub-block with the four OBMC sub-blocks in one equation implementing parallel blending as shown in Figure 6, and fix the weighting factors without favoring a particular neighboring sub-block. For example, using the equation implementing parallel blending, the final prediction P can be expressed as P = w1 * P c +w2*P top +w3*P left +w4*P below +w5*P right where P top is the prediction based on the MV of the upper neighboring sub-block, and P left is the prediction based on the MV of the left neighboring subblock, and P below is the prediction based on the MV of the lower neighboring sub-block, and P rightis a prediction based on the MV of the right neighboring sub-block, and w1, w2, w3, w4, and w5 are respective weighting factors. In some cases, the weight w1 may be equal to 1 - w2 - w3 - w4 - w5. The prediction based on the MV of the neighboring sub-block N may add / include / introduce noise to the samples in the row / column that is farthest from sub-block N. Thus, the systems and techniques described herein may, respectively, set the values for each of the weights w2, w3, w4, and w5 to {a, b, c, 0} for the sample row / column of the current sub-block that is {first, second, third, fourth} closest to the neighboring sub-block N.

[0141]

[0150] For example, the first element a (e.g., the weighting factor a) may be for the sample row or column that is closest to each neighboring sub-block N, and the last element 0 may be for the sample row or column that is farthest from each neighboring sub-block N. For purposes of illustration, using the positions (0,0), (0,1), and (1,1) relative to the top-left sample of a current sub-block having a size of 4×4 samples as an example, the final prediction P(x,y) may be derived as follows.

[0142]

Number

[0143]

[0151] An exemplary sum of the weighting factors (e.g., w2 + w3 + w4 + w5) from the neighboring OBMC sub-blocks for a 4×4 current sub-block may be as shown in Table 700 shown in FIG. 7. In some cases, the weighting factors may be left-shifted to avoid division operations, which may increase the computational complexity / load and / or result in inconsistencies. For example, {a’, b’, c’, 0} may be set such that {a<<shift, b<<shift, c<<shift, 0}, where shift is a positive integer. In this example, the weight w1 may be equal to (1<<shift) - a’ - b’ - c’, and P may be (w1 * Pc +w2*P top +w3*P left +w4*P below +w5*P right +(1<<(shift-1)))>>shift. An illustrative example for setting {a', b', c', 0} is {15, 8, 3, 0}, where those values ​​are the result of left-shifting the original values ​​by 6, and w1 is equal to (1<<6)-abc. P=(w1*P c +w2*P top +w3*P left +w4*P below +w5*P right +(1<<5))>>6.

[0144]

[0152] In some aspects, the values ​​of w2, w3, w4, and w5 may be set to {a, b, 0, 0} for the sample row / column of the current sub-block that is {first, second, third, fourth} closest to neighboring sub-block N, respectively. For illustrative purposes, using positions (0,0), (0,1), and (1,1) for the top-left sample of the current sub-block having a size of 4x4 samples as an example, the final prediction P(x,y) may be derived as follows:

[0145]

number

[0146]

[0153] Exemplary sums of weighting factors (e.g., w2+w3+w4+w5) from neighboring OBMC sub-blocks for a 4x4 current sub-block are shown in table 800 shown in Figure 8. As shown, in some examples, the weighting factors may be selected such that the sum of w2+w3+w4+w5 at corner samples (e.g., samples at (0,0), (0,3), (3,0), and (3,3)) is greater than the sum of w2+w3+w4+w5 at other boundary samples (e.g., samples at (0,1), (0,2), (1,0), (2,0), (3,1), (3,2), (1,3), and (2,3)) and / or such that the sum of w2+w3+w4+w5 at boundary samples is greater than the value at intermediate samples (e.g., samples at (1,1), (1,2), (2,1), and (2,2)).

[0147]

[0154] In some cases, some motion compensation may be skipped during the OBMC process based on the similarity between the MV of the current sub-block and the MV of its spatial neighboring blocks / sub-blocks (e.g., above, left, below, and right). For example, every time motion compensation is invoked using motion information from a given neighboring block / sub-block, the MV(s) of the neighboring block(s) may be compared with the MV(s) of the current sub-block based on one or more of the following conditions: The one or more conditions may include, for example, a first condition that all prediction lists used by neighboring blocks / sub-blocks (e.g., list L0 or list L1 in uni-prediction, or both L0 and L1 in bi-prediction) are also used for predicting the current sub-block; a second condition that the same reference picture(s) are used by the MV(s) of the neighboring blocks / sub-blocks and the MV(s) of the current sub-block; and / or a third condition that the absolute value of the horizontal MV difference between the neighboring MV(s) and the current MV(s) is not greater than a predefined MV difference threshold T, and the absolute value of the vertical MV difference between the neighboring MV(s) and the current MV(s) is not greater than a predefined MV difference threshold T (if bi-prediction is used, both L0 MV and L1 MV may be checked).

[0148]

[0155] In some examples, if the first condition, the second condition, and the third condition are met, motion compensation using a given neighboring block / sub-block is not performed, and an OBMC sub-block using the MV of the given neighboring block / sub-block N is disabled and not blended with the original sub-block. In some cases, the CU boundary OBMC mode and the sub-block boundary OBMC mode may have different values ​​of the threshold T. When the mode is the CU boundary OBMC mode, T is set to T1; otherwise, T is set to T2, where T1 and T2 are greater than 0. In some cases, when the conditions are met, a lossy algorithm for skipping neighboring blocks / sub-blocks may be applied only to the sub-block boundary OBMC mode. The CU boundary OBMC mode may instead apply a lossless algorithm for skipping neighboring blocks / sub-blocks when one or more conditions are met, such as a fourth condition that all prediction lists used by neighboring blocks / sub-blocks (e.g., either L0 or L1 in uni-prediction, or both L0 and L1 in bi-prediction) are also used for predicting the current sub-block; a fifth condition that the same reference picture(s) are used by neighboring MV(s) and current MV(s); and a sixth condition that neighboring MV(s) and current MV(s) are the same (if bi-prediction is used, both L0 MV and L1 MV may be checked).

[0149]

[0156] In some cases, when the first, second, and third conditions are met, the lossy algorithm for skipping neighboring blocks / sub-blocks is applied only to the CU boundary OBMC mode. In some cases, the sub-block boundary OBMC mode can apply the lossless algorithm for skipping neighboring blocks / sub-blocks when the fourth, fifth, and sixth conditions are met.

[0150]

[0157] In some aspects, in the CU boundary OBMC mode, a lossy fast algorithm may be implemented to save encoding and decoding time. For example, a first OBMC block and an adjacent OBMC block may be merged and jointly generated into a larger OBMC block if one or more conditions are met. The one or more conditions may include, for example, a condition that all prediction lists (e.g., either L0 or L1 in uni-prediction, or both L0 and L1 in bi-prediction) used by the first neighboring block of the current CU are also used for predicting the second neighboring block of the current CU (which is in the same direction as the first neighboring block); a condition that the same reference picture(s) are used by the MV of the first neighboring block and the MV of the second neighboring block; and a condition that the absolute value of the horizontal MV difference between the MV of the first neighboring block and the MV of the second neighboring block is not greater than a predefined MV difference threshold T3, and the absolute value of the vertical MV difference between the MV of the first neighboring block and the MV of the second neighboring block is not greater than a predefined MV difference threshold T3 (if bi-prediction is used, both L0 MV and L1 MV may be checked).

[0151]

[0158] In some aspects, in the sub-block boundary OBMC mode, a lossy fast algorithm may be implemented to save encoding and decoding time. In some examples, the SbTMVP mode and DMVR are implemented on an 8x8 basis, and affine motion compensation is implemented on a 4x4 basis. The systems and techniques described herein can implement the sub-block boundary OBMC mode on an 8x8 basis. In some cases, the systems and techniques described herein can perform a similarity test on every 8x8 sub-block to determine whether the 8x8 sub-block should be split into four 4x4 sub-blocks, and if so, OBMC is performed on a 4x4 basis.

[0152]

[0159] 9 is a diagram illustrating an example CU 910 with sub-blocks 902-908 in an 8×8 block. In some examples, the lossy fast algorithm in sub-block boundary OBMC mode may include four 4×4 OBMC sub-blocks (e.g., OBMC sub-block 902(P), OBMC sub-block 904(Q), OBMC sub-block 906(R), and OBMC sub-block 908(S)) for each 8×8 sub-block. The OBMC sub-blocks 902-908 satisfy the following conditions: a first condition that the prediction list(s) (e.g., either L0 or L1 in uni-prediction, or both L0 and L1 in bi-prediction) used by sub-blocks 902(P), 904(Q), 906(R), and 908(S) are the same; a second condition that the same reference picture(s) are used by the MVs of sub-blocks 902(P), 904(Q), 906(R), and 908(S); and a second condition that any two sub-blocks (e.g., 902(P) and 904(Q), 902(P) and 906(R), 902(P) and 908(S)) are the same. 908(S), 904(Q) and 906(R), 904(Q) and 908(S), and 906(R) and 908(S)) is not greater than a predefined MV difference threshold T4, and the absolute value of the vertical MV difference between the MVs of any two sub-blocks (e.g., 902(P) and 904(Q), 902(P) and 906(R), 902(P) and 908(S), 904(Q) and 906(R), 904(Q) and 908(S), and 906(R) and 908(S)) is not greater than a predefined MV difference threshold T4. When at least one of the third conditions (when bi-prediction is used, both L0 MV and L1 MV may be examined) is not met, OBMC blending may be enabled.

[0153]

[0160] If all of the above conditions are met, the systems and techniques described herein can perform 8x8 sub-block OBMC, where 8x8 OBMC sub-blocks from the top, left, bottom, and right MVs are generated using OBMC blending for the sub-block boundary OBMC mode. Otherwise, when at least one of the above conditions is not met, OBMC is performed on a 4x4 basis in this 8x8 sub-block, where every 4x4 sub-block in the 8x8 sub-block generates four OBMC sub-blocks from the top, left, bottom, and right MVs.

[0154]

[0161] In some aspects, when a CU is coded in merge mode, the OBMC flag is copied from a neighboring block in a manner similar to motion information copying in merge mode. Alternatively, when a CU is not coded in merge mode, the OBMC flag may be signaled to indicate whether OBMC is applied for the CU.

[0155]

[0162] 10 is a flowchart illustrating an example process 1000 for implementing OBMC. At block 1002, the process 1000 may include determining that an OBMC mode is enabled for a current sub-block of a block of video data. In some examples, the OBMC mode may include a sub-block boundary OBMC mode.

[0156]

[0163] At block 1004, process 1000 may include determining a first prediction associated with the current sub-block, a second prediction associated with a first OBMC block adjacent to the top border of the current sub-block, a third prediction associated with a second OBMC block adjacent to the left border of the current sub-block, a fourth prediction associated with a third OBMC block adjacent to the bottom border of the current sub-block, and a fifth prediction associated with a fourth OBMC block adjacent to the right border of the current sub-block.

[0157]

[0164] At block 1006, process 1000 may include determining a sixth prediction based on results of applying a first weight to the first prediction, a second weight to the second prediction, a third weight to the third prediction, a fourth weight to the fourth prediction, and a fifth weight to the fifth prediction. In some cases, a sum of weight values ​​of corner samples of a corresponding sub-block (e.g., a current sub-block, a first OBMC block, a second OBMC block, a third OBMC block, a fourth OBMC block) may be greater than a sum of weight values ​​of other boundary samples of the corresponding sub-block. In some cases, a sum of weight values ​​of other boundary samples may be greater than a sum of weight values ​​of non-boundary samples of the corresponding sub-block (e.g., samples that do not border a sub-block boundary).

[0158]

[0165] For example, in some cases, each of the first weight, the second weight, the third weight, and the fourth weight may include one or more weight values ​​associated with one or more samples from a corresponding sub-block of the current sub-block, the first OBMC block, the second OBMC block, the third OBMC block, or the fourth OBMC block. Moreover, the sum of the weight values ​​of the corner samples of the corresponding sub-block may be greater than the sum of the weight values ​​of the other boundary samples of the corresponding sub-block, and the sum of the weight values ​​of the other boundary samples of the corresponding sub-block may be greater than the sum of the weight values ​​of the non-boundary samples of the corresponding sub-block.

[0159]

[0166] At block 1008, the process 1000 may include generating a blended sub-block corresponding to the current sub-block of the block of video data based on the sixth prediction.

[0160]

[0167] 11 is a flowchart illustrating another example process 1100 for implementing OBMC. At block 1102, the process 1100 may include determining that an OBMC mode is enabled for a current sub-block of a block of video data. In some examples, the OBMC mode may include a sub-block boundary OBMC mode.

[0161]

[0168] At block 1104, the process 1100 may include determining whether a first condition, a second condition, and a third condition are satisfied for at least one neighboring sub-block adjacent to the current sub-block. In some examples, the first condition may include that all of one or more reference picture lists for predicting the current sub-block are used to predict the neighboring sub-block.

[0162]

[0169] In some examples, the second condition may include that the same reference picture or pictures are used to determine the motion vectors associated with the current sub-block and the neighboring sub-blocks.

[0163]

[0170] In some examples, the third condition may include a first difference between the horizontal motion vector of the current sub-block and the horizontal motion vector of the neighboring sub-block and a second difference between the vertical motion vector of the current sub-block and the vertical motion vector of the neighboring sub-block not exceeding a motion vector difference threshold. In some examples, the motion vector difference threshold is greater than 0.

[0164]

[0171] At block 1106, process 1100 may include determining not to use motion information of neighboring sub-blocks for motion compensation of the current sub-block based on determining that OBMC mode is enabled for the current sub-block and determining that the first condition, the second condition, and the third condition are satisfied.

[0165]

[0172] In some aspects, the process 1100 may include determining to implement a sub-block boundary OBMC mode for the current sub-block based on a decision to use a decoder-side motion vector refinement (DMVR) mode, a sub-block-based temporal motion vector prediction (SbTMVP) mode, or an affine motion compensation prediction mode for the current sub-block.

[0166]

[0173] In some aspects, process 1100 may include performing a sub-block boundary OBMC mode for the sub-block. In some cases, performing the sub-block boundary OBMC mode for the sub-block may include determining a first prediction associated with a current sub-block, a second prediction associated with a first OBMC block adjacent to a top border of the current sub-block, a third prediction associated with a second OBMC block adjacent to a left border of the current sub-block, a fourth prediction associated with a third OBMC block adjacent to a bottom border of the current sub-block, and a fifth prediction associated with a fourth OBMC block adjacent to a right border of the current sub-block, determining a sixth prediction based on results of applying the first weight to the first prediction, applying the second weight to the second prediction, applying the third weight to the third prediction, applying the fourth weight to the fourth prediction, and applying the fifth weight to the fifth prediction, and generating a blended sub-block corresponding to the current sub-block based on the sixth prediction.

[0167]

[0174] In some cases, the sum of the weight values ​​of the corner samples of the corresponding sub-block (e.g., the current sub-block, the first OBMC block, the second OBMC block, the third OBMC block, the fourth OBMC block) may be greater than the sum of the weight values ​​of the other boundary samples of the corresponding sub-block. In some cases, the sum of the weight values ​​of the other boundary samples may be greater than the sum of the weight values ​​of the non-boundary samples of the corresponding sub-block (e.g., samples that do not border the boundary of the current sub-block).

[0168]

[0175] For example, in some cases, each of the second weight, the third weight, the fourth weight, and the fifth weight may include one or more weight values ​​associated with one or more samples from a corresponding sub-block of the current sub-block. Moreover, the sum of the weight values ​​of the corner samples of the current sub-block may be greater than the sum of the weight values ​​of the other boundary samples of the current sub-block, and the sum of the weight values ​​of the other boundary samples of the current sub-block may be greater than the sum of the weight values ​​of the non-boundary samples of the current sub-block.

[0169]

[0176] In some aspects, process 1100 may include determining to use a local illumination compensation (LIC) mode for additional blocks of video data and, based on the determination to use the LIC mode for the additional blocks, skipping signaling of information related to an OBMC mode for the additional blocks. In some examples, skipping signaling of information related to an OBMC mode for the additional blocks may include signaling a syntax flag with an empty value (e.g., no value included for the flag), where the syntax flag is associated with the OBMC mode. In some aspects, process 1100 may include receiving a signal including a syntax flag with an empty value, where the syntax flag is associated with the OBMC mode for the additional blocks of video data. In some aspects, process 1100 may include determining not to use the OBMC mode for the additional blocks based on the syntax flag with an empty value.

[0170]

[0177] In some cases, skipping signaling of information related to the OBMC mode for the additional blocks may include determining not to use or enable the OBMC mode for the additional blocks based on a decision to use the LIC mode for the additional blocks, and skipping signaling values ​​related to the OBMC mode for the additional blocks.

[0171]

[0178] In some aspects, process 1100 may include determining whether OBMC mode is enabled for the additional block, and determining whether OBMC mode is enabled for the additional block and, based on the determination to use LIC mode for the additional block, determining to skip signaling information related to the OBMC mode for the additional block.

[0172]

[0179] In some aspects, process 1100 may include determining to use a coding unit (CU) boundary OBMC mode for a current sub-block of a block of video data, and determining a final prediction for the current sub-block based on a sum of a first result of applying a weight associated with the current sub-block to a respective prediction associated with the current sub-block and a second result of applying one or more respective weights to one or more respective predictions associated with one or more sub-blocks neighboring the current sub-block.

[0173]

[0180] In some examples, determining not to use motion information of neighboring sub-blocks for motion compensation of the current sub-block may include skipping the use of motion information of neighboring sub-blocks for motion compensation of the current sub-block.

[0174]

[0181] In some cases, process 1000 and / or process 1100 may be implemented by an encoder and / or a decoder.

[0175]

[0182] In some implementations, the processes (or methods) described herein (including process 1000 and process 1100) may be performed by a computing device or apparatus, such as system 100 shown in Figure 1. For example, the processes may be performed by encoding device 104 shown in Figures 1 and 12, by another video source side device or video transmission device, by decoding device 112 shown in Figures 1 and 13, and / or by another client side device, such as a player device, a display, or any other client side device. In some cases, the computing device or apparatus may include one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, and / or other component(s) configured to perform the steps of process 1000 and / or process 1100.

[0176]

[0183] In some examples, the computing device may include a mobile device, a desktop computer, a server computer and / or server system, or other type of computing device. Components of a computing device (e.g., one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, and / or other components) may be implemented in circuitry. For example, the components may include and / or be implemented using electronic circuitry or other electronic hardware, which may include one or more programmable electronic circuits (e.g., a microprocessor, a graphics processing unit (GPU), a digital signal processor (DSP), a central processing unit (CPU), and / or other suitable electronic circuitry), and / or may include and / or be implemented using computer software, firmware, or any combination thereof, to perform various operations described herein. In some examples, a computing device or apparatus may include a camera configured to capture video data (e.g., a video sequence) including video frames. In some examples, the camera or other capture device that captures the video data is separate from the computing device, in which case the computing device receives or acquires the captured video data. The computing device may include a network interface configured to communicate the video data. The network interface may be configured to communicate Internet Protocol (IP)-based data or other types of data. In some examples, the computing device or apparatus may include a display for displaying output video content, such as sample pictures of a video bitstream.

[0177]

[0184] A process may be described in terms of logical flow diagrams, where the operations represent sequences of actions that may be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, the operations represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform particular functions or implement particular data types. The order in which the operations are described is not intended to be limiting, and any number of the described operations may be combined in any order and / or in parallel to implement a process.

[0178]

[0185] Furthermore, the process may be performed under the control of one or more computer systems configured of executable instructions and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that collectively execute on one or more processors, by hardware, or a combination thereof. As mentioned above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program comprising a plurality of instructions executable by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0179]

[0186] The coding techniques described herein may be implemented in an exemplary video encoding and decoding system (e.g., system 100). In some examples, the system includes a source device that provides encoded video data to be subsequently decoded by a destination device. In particular, the source device provides the video data to the destination device via a computer-readable medium. The source and destination devices may comprise any of a wide range of devices, including desktop computers, notebook (i.e., laptop) computers, tablet computers, set-top boxes, telephone handsets such as so-called "smart" phones, so-called "smart" pads, televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, etc. In some cases, the source and destination devices may be equipped for wireless communication.

[0180]

[0187] The destination device may receive the encoded video data to be decoded via a computer-readable medium. The computer-readable medium may comprise any type of medium or device capable of moving encoded video data from a source device to a destination device. In one example, the computer-readable medium may comprise a communication medium for enabling the source device to transmit the encoded video data directly to the destination device in real time. The encoded video data may be modulated according to a communication standard, such as a wireless communication protocol, and transmitted to the destination device. The communication medium may comprise any wireless or wired communication medium, such as a radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form part of a packet-based network, such as a local area network, a wide area network, or a global network such as the Internet. The communication medium may include routers, switches, base stations, or any other equipment that may be useful for enabling communication from the source device to the destination device.

[0181]

[0188] In some examples, the encoded data may be output from the output interface to a storage device. Similarly, the encoded data may be accessed from the storage device by the input interface. The storage device may include any of a variety of distributed or locally accessed data storage media, such as a hard drive, Blu-ray disc, DVD, CD-ROM, flash memory, volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded video data. In a further example, the storage device may correspond to a file server or another intermediate storage device that may store the encoded video generated by the source device. The destination device may access the stored video data from the storage device via streaming or download. The file server may be any type of server capable of storing the encoded video data and transmitting the encoded video data to the destination device. Exemplary file servers include a web server (e.g., for a website), an FTP server, a network-attached storage (NAS) device, or a local disk drive. The destination device may access the encoded video data through any standard data connection, including an Internet connection. This may include a wireless channel (e.g., a Wi-Fi connection), a wired connection (e.g., DSL, cable modem, etc.), or a combination of both, that is suitable for accessing the encoded video data stored on the file server. The transmission of the encoded video data from the storage device may be a streaming transmission, a download transmission, or a combination thereof.

[0182]

[0189] The techniques of this disclosure are not necessarily limited to wireless applications or settings. The techniques may be applied to video coding supporting any of a variety of multimedia applications, such as over-the-air television broadcast, cable television transmission, satellite television transmission, Internet streaming video transmission such as Dynamic Adaptive Streaming over HTTP (DASH), digital video encoded on a data storage medium, decoding of digital video stored on a data storage medium, or other applications. In some examples, a system may be configured to support one-way or two-way video transmission to support applications such as video streaming, video playback, video broadcasting, and / or video telephony.

[0183]

[0190] In one example, the source device includes a video source, a video encoder, and an output interface. The destination device may include an input interface, a video decoder, and a display device. The video encoder of the source device may be configured to apply the techniques disclosed herein. In other examples, the source device and the destination device may include other components or configurations. For example, the source device may receive video data from an external video source, such as an external camera. Similarly, the destination device may interface with an external display device rather than including an integrated display device.

[0184]

[0191] The above exemplary system is merely an example. Techniques for processing video data in parallel may be implemented by any digital video encoding and / or decoding device. Generally, the techniques of this disclosure are implemented by a video encoding device, but the techniques may also be implemented by a video encoder / decoder, commonly referred to as a "codec." Moreover, the techniques of this disclosure may also be implemented by a video preprocessor. The source device and destination device are merely examples of coding devices, such that the source device generates coded video data for transmission to the destination device. In some examples, the source device and destination device may operate substantially symmetrically, such that each device includes a video encoding component and a video decoding component. Thus, the exemplary system may support one-way or two-way video transmission between video devices, for example, for video streaming, video playback, video broadcasting, or video telephony.

[0185]

[0192] The video source may include a video capture device such as a video camera, a video archive containing previously captured video, and / or a video feed interface for receiving video from a video content provider. As a further alternative, the video source may generate computer graphics-based data as the source video, or a combination of live, archived, and computer-generated video. In some cases, when the video source is a video camera, the source and destination devices may form a so-called camera phone or video phone. However, as mentioned above, the techniques described in this disclosure may be applicable to video coding generally and may be applied to wireless and / or wired applications. In each case, the captured video, pre-captured video, or computer-generated video may be encoded by a video encoder. The encoded video information may then be output onto a computer-readable medium by an output interface.

[0186]

[0193] As mentioned, computer-readable media may include transient media, such as wireless broadcast or wired network transmissions, or storage media (i.e., non-transitory storage media), such as hard disks, flash drives, compact discs, digital video discs, Blu-ray discs, or other computer-readable media. In some examples, a network server (not shown) may receive encoded video data from a source device, for example, via a network transmission, and provide the encoded video data to a destination device. Similarly, a computing device at a media production facility, such as a disc stamping facility, may receive encoded video data from a source device and produce a disc containing the encoded video data. Thus, computer-readable media may be understood to include one or more computer-readable media of various forms in various examples.

[0187]

[0194] An input interface of the destination device receives information from a computer-readable medium. The information on the computer-readable medium may include syntax information, including syntax elements that describe characteristics and / or processing of blocks and other coded units, e.g., groups of pictures (GOPs), defined by a video encoder and also used by a video decoder. The display device displays the decoded video data to a user and may comprise any of a variety of display devices, such as a cathode ray tube (CRT), a liquid crystal display (LCD), a plasma display, an organic light-emitting diode (OLED) display, or another type of display device. Various embodiments of the present application have been described.

[0188]

[0195] Specific details of encoding device 104 and decoding device 112 are shown in Figures 12 and 13, respectively. Figure 12 is a block diagram illustrating an example encoding device 104 that may implement one or more of the techniques described in this disclosure. Encoding device 104 may, for example, generate a syntax structure described herein (e.g., a syntax structure for a VPS, SPS, PPS, or other syntax element). Encoding device 104 may perform intra-predictive and inter-predictive coding of video blocks within video slices. As previously described, intra-coding relies at least in part on spatial prediction to reduce or remove spatial redundancy within a given video frame or picture. Inter-coding relies at least in part on temporal prediction to reduce or remove temporal redundancy within adjacent or surrounding frames of a video sequence. Intra-mode (I-mode) may refer to any of several spatial-based compression modes. Inter-mode, such as unidirectional prediction (P-mode) or bi-prediction (B-mode), may refer to any of several temporal-based compression modes.

[0189]

[0196] Encoding device 104 includes partition unit 35, prediction processing unit 41, filter unit 63, picture memory 64, summer 50, transform processing unit 52, quantization unit 54, and entropy coding unit 56. Prediction processing unit 41 includes motion estimation unit 42, motion compensation unit 44, and intra-prediction processing unit 46. For video block reconstruction, encoding device 104 also includes inverse quantization unit 58, inverse transform processing unit 60, and summer 62. Filter unit 63 is intended to represent one or more loop filters, such as a deblocking filter, an adaptive loop filter (ALF), and a sample adaptive offset (SAO) filter. Although filter unit 63 is shown in FIG. 12 as being an in-loop filter, in other configurations, filter unit 63 may be implemented as a post-loop filter. Post-processing device 57 may perform additional processing on the encoded video data generated by encoding device 104. The techniques of this disclosure may, in some cases, be implemented by encoding device 104. However, in other cases, one or more of the techniques of this disclosure may be implemented by post-processing device 57.

[0190]

[0197] As shown in FIG. 12, encoding device 104 receives video data, and partitioning unit 35 partitions the data into video blocks. Partitioning may also include partitioning into slices, slice segments, tiles, or other larger units, as well as video block partitioning, for example, according to a quadtree structure of LCUs and CUs. Encoding device 104 generally refers to components that encode video blocks within a video slice to be coded. A slice may be divided into multiple video blocks (and possibly into sets of video blocks called tiles). Prediction processing unit 41 may select one of multiple possible coding modes, such as one of multiple intra-predictive coding modes or one of multiple inter-predictive coding modes, for a current video block based on error results (e.g., coding rate, distortion level, etc.). Prediction processing unit 41 may provide the resulting intra-coded or inter-coded block to adder 50 to generate residual block data and to adder 62 to reconstruct a coded block for use as a reference picture.

[0191]

[0198] Intra-prediction processing unit 46 within prediction processing unit 41 may perform intra-predictive coding of the current video block relative to one or more neighboring blocks in the same frame or slice as the current block to be coded to provide spatial compression. Motion estimation unit 42 and motion compensation unit 44 within prediction processing unit 41 perform inter-predictive coding of the current video block relative to one or more predictive blocks in one or more reference pictures to provide temporal compression.

[0192]

[0199] Motion estimation unit 42 may be configured to determine an inter-prediction mode for a video slice according to a predetermined pattern for the video sequence. The predetermined pattern may designate the video slices in the sequence as P slices, B slices, or GPB slices. Motion estimation unit 42 and motion compensation unit 44 may be highly integrated but are shown separately for conceptual purposes. Motion estimation performed by motion estimation unit 42 is the process of generating motion vectors that estimate motion for video blocks. A motion vector may indicate, for example, the displacement of a prediction unit (PU) of a video block in a current video frame or picture relative to a predictive block in a reference picture.

[0193]

[0200] The predictive block is a block that is found to closely match the PU of the video block to be coded, in terms of pixel differences, which may be determined by sum of absolute differences (SAD), sum of squared differences (SSD), or other difference metrics. In some examples, encoding device 104 may calculate values ​​for sub-integer pixel locations of reference pictures stored in picture memory 64. For example, encoding device 104 may interpolate values ​​at quarter-pixel locations, eighth-pixel locations, or other fractional-pixel locations of the reference pictures. Thus, motion estimation unit 42 may perform motion searches for full-pixel and fractional-pixel locations and output motion vectors with fractional-pixel precision.

[0194]

[0201] Motion estimation unit 42 calculates a motion vector for a PU of a video block in an inter-coded slice by comparing the position of the PU with the position of a predictive block of a reference picture. The reference picture may be selected from a first reference picture list (List 0) or a second reference picture list (List 1), each of which identifies one or more reference pictures stored in picture memory 64. Motion estimation unit 42 sends the calculated motion vector to entropy coding unit 56 and motion compensation unit 44.

[0195]

[0202] Motion compensation performed by motion compensation unit 44 may involve fetching or generating a predictive block based on a motion vector determined by motion estimation, possibly performing interpolation to sub-pixel precision. Upon receiving a motion vector for a PU of a current video block, motion compensation unit 44 may locate the predictive block to which the motion vector points in a reference picture list. Encoding device 104 forms a residual video block by subtracting pixel values ​​of the predictive block from pixel values ​​of the current video block being coded to form pixel difference values. The pixel difference values ​​form residual data for the block and may include both luma and chroma difference components. Adder 50 represents one or more components that perform this subtraction operation. Motion compensation unit 44 may also generate syntax elements associated with the video block and the video slice for use by decoding device 112 in decoding the video block of the video slice.

[0196]

[0203] The intra-prediction processing unit 46 may intra-predict the current block as an alternative to the inter-prediction performed by the motion estimation unit 42 and the motion compensation unit 44, as described above. In particular, the intra-prediction processing unit 46 may determine an intra-prediction mode to use to encode the current block. In some examples, the intra-prediction processing unit 46 may encode the current block using various intra-prediction modes, e.g., during separate encoding passes, and the intra-prediction processing unit 46 may select an appropriate intra-prediction mode to use from the tested modes. For example, the intra-prediction processing unit 46 may calculate rate-distortion values ​​using a rate-distortion analysis for the various tested intra-prediction modes and select the intra-prediction mode with the best rate-distortion characteristics among the tested modes. The rate-distortion analysis generally determines the amount of distortion (or error) between the coded block and the original uncoded block that was coded to produce the coded block, as well as the bitrate (i.e., number of bits) used to produce the coded block. Intra-prediction processing unit 46 may calculate ratios from the distortions and rates for the various coded blocks to determine which intra-prediction mode exhibits the best rate-distortion value for the block.

[0197]

[0204] In either case, after selecting an intra-prediction mode for the block, intra-prediction processing unit 46 may provide information indicating the selected intra-prediction mode for the block to entropy coding unit 56. Entropy coding unit 56 may encode the information indicating the selected intra-prediction mode. Encoding device 104 may include in the transmitted bitstream configuration data definitions of coding contexts for various blocks, as well as an indication of the most probable intra-prediction mode, intra-prediction mode index table, and modified intra-prediction mode index table to use for each of the contexts. The bitstream configuration data may include multiple intra-prediction mode index tables and multiple modified intra-prediction mode index tables (also referred to as codeword mapping tables).

[0198]

[0205] After prediction processing unit 41 generates a predictive block for a current video block via either inter-prediction or intra-prediction, encoding device 104 forms a residual video block by subtracting the predictive block from the current video block. The residual video data in the residual block may be included in one or more TUs and applied to transform processing unit 52. Transform processing unit 52 converts the residual video data into residual transform coefficients using a transform, such as a discrete cosine transform (DCT) or a conceptually similar transform. Transform processing unit 52 may convert the residual video data from the pixel domain to a transform domain, such as the frequency domain.

[0199]

[0206] Transform processing unit 52 may send the resulting transform coefficients to quantization unit 54. Quantization unit 54 quantizes the transform coefficients to further reduce the bit rate. The quantization process may reduce the bit depth associated with some or all of the coefficients. The degree of quantization may be modified by adjusting a quantization parameter. In some examples, quantization unit 54 may then perform a scan of a matrix including the quantized transform coefficients. Alternatively, entropy coding unit 56 may perform the scan.

[0200]

[0207] After quantization, entropy coding unit 56 entropy codes the quantized transform coefficients. For example, entropy coding unit 56 may implement context-adaptive variable length coding (CAVLC), context-adaptive binary arithmetic coding (CABAC), syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioned entropy (PIPE) coding, or another entropy coding technique. After entropy coding by entropy coding unit 56, the coded bitstream may be transmitted to decoding device 112 or archived for later transmission or retrieval by decoding device 112. Entropy coding unit 56 may also entropy code motion vectors and other syntax elements for the current video slice being coded.

[0201]

[0208] Inverse quantization unit 58 and inverse transform processing unit 60 apply inverse quantization and inverse transform, respectively, to reconstruct the residual block in the pixel domain for later use as a reference block of a reference picture. Motion compensation unit 44 may calculate a reference block by adding the residual block to a predictive block of one of the reference pictures in the reference picture list. Motion compensation unit 44 may also apply one or more interpolation filters to the reconstructed residual block to calculate sub-integer pixel values ​​for use in motion estimation. Adder 62 adds the reconstructed residual block to the motion-compensated predictive block produced by motion compensation unit 44 to produce a reference block for storage in picture memory 64. The reference block may be used as a reference block by motion estimation unit 42 and motion compensation unit 44 to inter-predict blocks in subsequent video frames or pictures.

[0202]

[0209] In this manner, encoding device 104 of Figure 12 represents an example of a video encoder configured to perform any of the techniques described herein, including the process described above with respect to Figure 10 and / or the process described above with respect to Figure 11. In some cases, some of the techniques of this disclosure may also be implemented by post-processing device 57.

[0203]

[0210] 13 is a block diagram illustrating an exemplary decoding device 112. Decoding device 112 includes entropy decoding unit 80, prediction processing unit 81, inverse quantization unit 86, inverse transform processing unit 88, adder 90, filter unit 91, and picture memory 92. Prediction processing unit 81 includes motion compensation unit 82 and intra-prediction processing unit 84. Decoding device 112 may, in some examples, perform a decoding path that is generally inverse to the encoding path described with respect to encoding device 104 from FIG.

[0204]

[0211] During the decoding process, decode device 112 receives an encoded video bitstream representing video blocks of encoded video slices and associated syntax elements sent by encoding device 104. In some embodiments, decode device 112 may receive the encoded video bitstream from encoding device 104. In some embodiments, decode device 112 may receive the encoded video bitstream from a network entity 79, such as a server, a media-aware network element (MANE), a video editor / splitter, or other such device configured to implement one or more of the techniques described above. Network entity 79 may or may not include encoding device 104. Some of the techniques described in this disclosure may be implemented by network entity 79 before network entity 79 sends the encoded video bitstream to decode device 112. In some video decoding systems, network entity 79 and decode device 112 may be part of separate devices, while in other cases, the functionality described with respect to network entity 79 may be performed by the same device that comprises decode device 112.

[0205]

[0212] Entropy decoding unit 80 of decoding device 112 entropy decodes the bitstream to generate quantized coefficients, motion vectors, and other syntax elements. Entropy decoding unit 80 forwards the motion vectors and other syntax elements to prediction processing unit 81. Decoding device 112 may receive video slice-level and / or video block-level syntax elements. Entropy decoding unit 80 may process and parse both fixed-length and variable-length syntax elements, such as VPS, SPS, and PPS, or in multiple parameter sets.

[0206]

[0213] When a video slice is coded as an intra-coded (I) slice, intra-prediction processing unit 84 of prediction processing unit 81 may generate predictive data for video blocks of the current video slice based on the signaled intra-prediction mode and data from previously decoded blocks of the current frame or picture. When a video frame is coded as an inter-coded (i.e., B, P, or GPB) slice, motion compensation unit 82 of prediction processing unit 81 produces predictive blocks for video blocks of the current video slice based on motion vectors received from entropy decoding unit 80 and other syntax elements. The predictive blocks may be produced from one of the reference pictures in a reference picture list. Decoding device 112 may construct the reference frame lists, i.e., List 0 and List 1, using a default construction technique based on the reference pictures stored in picture memory 92.

[0207]

[0214] Motion compensation unit 82 determines prediction information for video blocks of the current video slice by parsing the motion vectors and other syntax elements and uses the prediction information to produce predictive blocks for the current video block being decoded. For example, motion compensation unit 82 may use one or more syntax elements in a parameter set to determine a prediction mode (e.g., intra or inter prediction) used to code the video blocks of the video slice, an inter-prediction slice type (e.g., B slice, P slice, or GPB slice), construction information for one or more reference picture lists for the slice, a motion vector for each inter-coded video block of the slice, an inter-prediction status for each inter-coded video block of the slice, and other information for decoding video blocks in the current video slice.

[0208]

[0215] Motion compensation unit 82 may also perform interpolation based on an interpolation filter. Motion compensation unit 82 may use the interpolation filter used by encoding device 104 during encoding of the video block to calculate interpolated values ​​for sub-integer pixels of the reference block. In this case, motion compensation unit 82 may determine the interpolation filter used by encoding device 104 from the received syntax element and may use that interpolation filter to produce the predictive block.

[0209]

[0216] Inverse quantization unit 86 inverse quantizes, i.e., dequantizes, the quantized transform coefficients provided in the bitstream and decoded by entropy decoding unit 80. The inverse quantization process may involve the use of quantization parameters calculated by encoding device 104 for each video block in a video slice to determine the degree of quantization, and similarly, the degree of inverse quantization that should be applied. Inverse transform processing unit 88 applies an inverse transform (e.g., an inverse DCT or other suitable inverse transform), an inverse integer transform, or a conceptually similar inverse transform process to the transform coefficients to produce residual blocks in the pixel domain.

[0210]

[0217] After motion compensation unit 82 generates a predictive block for the current video block based on the motion vector and other syntax elements, decoding device 112 forms a decoded video block by adding a residual block from inverse transform processing unit 88 with the corresponding predictive block generated by motion compensation unit 82. Adder 90 represents one or more components that perform this addition operation. If desired, a loop filter (either within the coding loop or after the coding loop) may also be used to smooth pixel transitions or otherwise improve video quality. Filter unit 91 shall represent one or more loop filters, such as a deblocking filter, an adaptive loop filter (ALF), and a sample adaptive offset (SAO) filter. Although filter unit 91 is shown in FIG. 13 as being an in-loop filter, in other configurations, filter unit 91 may be implemented as a post-loop filter. The decoded video blocks in a given frame or picture are then stored in picture memory 92, which stores reference pictures used for subsequent motion compensation. Picture memory 92 also stores decoded video for later presentation on a display device, such as video destination device 122 shown in FIG.

[0211]

[0218] In this manner, the decoding device 112 of FIG. 13 represents an example of a video decoder configured to implement any of the techniques described herein, including the process described above with respect to FIG. 10 and the process described above with respect to FIG. 11.

[0212]

[0219] The term "computer-readable medium" as used herein includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or carrying instruction(s) and / or data. Computer-readable media may include non-transitory media on which data may be stored and which do not include carrier waves and / or transitory electronic signals propagating wirelessly or over wired connections. Examples of non-transitory media may include, but are not limited to, magnetic disks or tapes, optical storage media such as compact discs (CDs) or digital versatile discs (DVDs), flash memory, memory, or memory devices. A computer-readable medium may have code and / or machine-executable instructions stored thereon, which may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, etc.

[0213]

[0220] In some embodiments, computer-readable storage devices, media, and memories may include cable or wireless signals containing bitstreams, etc. However, when stated, non-transitory computer-readable storage media specifically excludes media such as energy, carrier signals, electromagnetic waves, and the signals themselves.

[0214]

[0221] Specific details are provided in the above description to provide a thorough understanding of the embodiments and examples provided herein. However, those skilled in the art will understand that the embodiments may be practiced without these specific details. For clarity of explanation, in some instances, the technology may be presented as including individual functional blocks, including devices, device components, method steps or routines implemented in software, or functional blocks comprising a combination of hardware and software. Additional components other than those shown in the figures and / or described herein may be used. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form so as not to obscure the embodiments in unnecessary detail. In other instances, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail to avoid obscuring the embodiments.

[0215]

[0222] Individual embodiments may be described above as a process or method that is depicted as a flowchart, flow diagram, data flow diagram, structure diagram, or block diagram. While a flowchart may describe operations as a sequential process, many of the operations may be performed in parallel or simultaneously. Moreover, the order of operations may be rearranged. A process is terminated when the operations of a process are completed, but may have additional steps not included in the diagram. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination may correspond to a return of the function to the calling function or the main function.

[0216]

[0223] The processes and methods according to the examples described above may be implemented using computer-executable instructions stored on or otherwise available from a computer-readable medium. Such instructions may include, for example, instructions and data that cause or otherwise configure a general-purpose computer, a special-purpose computer, or a processing device to perform a certain function or group of functions. Portions of the computer resources used may be accessible over a network. The computer-executable instructions may be, for example, binaries, intermediate format instructions such as assembly language, firmware, source code, etc. Examples of computer-readable media that may be used to store instructions, information used, and / or information created during methods according to the described examples include magnetic or optical disks, flash memory, USB devices with non-volatile memory, networked storage devices, etc.

[0217]

[0224] Devices implementing processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., a computer program product) to perform the necessary tasks may be stored on a computer-readable or machine-readable medium. Processor(s) may perform the necessary tasks. Typical examples of form factors include laptops, smartphones, mobile phones, tablet devices or other small form factor personal computers, personal digital assistants, rack-mounted devices, standalone devices, etc. The functionality described herein may also be embodied in peripheral devices or add-in cards. Such functionality may also be implemented on a circuit board, in different chips, or different processes executing in a single device, as further examples.

[0218]

[0225] The instructions, media for carrying such instructions, computing resources for executing them, and other structures for supporting such computing resources are exemplary means for providing the functionality described in this disclosure.

[0219]

[0226] In the foregoing description, aspects of the present application have been described with reference to specific embodiments thereof, but those skilled in the art will recognize that the present application is not limited thereto. Accordingly, while exemplary embodiments of the present application have been described in detail herein, it should be understood that the inventive concepts may, in some cases, be embodied and employed in various ways, and that the appended claims are intended to include such variations, except as limited by the prior art. Various features and aspects of the applications described above may be used individually or together. Furthermore, the embodiments may be utilized in any number of environments and applications other than those described herein without departing from the broader spirit and scope of the present specification. Accordingly, the specification and drawings should be considered illustrative and not restrictive. For illustrative purposes, methods have been described in a particular order. It should be appreciated that in alternative embodiments, methods may be performed in an order different from that described.

[0220]

[0227] Those skilled in the art will appreciate that the less than ("<") and greater than (">") symbols or terminology used herein may be replaced with the less than or equal to ("≦") and greater than or equal to ("≧") symbols, respectively, without departing from the scope of this description.

[0221]

[0228] When a component is described as being "configured to" perform some operations, such configuration may be achieved, for example, by designing electronic circuitry or other hardware to perform the operations, by programming programmable electronic circuitry (e.g., a microprocessor or other suitable electronic circuitry) to perform the operations, or any combination thereof.

[0222]

[0229] The phrase "coupled to" refers to any component that is physically connected, either directly or indirectly, to another component and / or that is in communication, either directly or indirectly, with another component (e.g., connected to another component via a wired or wireless connection and / or other suitable communication interface).

[0223]

[0230] Claim language or other language in this disclosure reciting "at least one of" a set and / or "one or more" of a set indicates that one member of the set or multiple members of the set (in any combination) satisfies the claim. For example, claim language reciting "at least one of A and B" means A, B, or A and B. As another example, claim language reciting "at least one of A, B, and C" means A, B, C, or A and B, or A and C, or B and C, or A, B, and C. The language "at least one of" a set and / or "one or more" of a set does not limit the set to the items listed in the set. For example, claim language reciting "at least one of A and B" can mean A, B, or A and B, and can further include items not listed in the set of A and B.

[0224]

[0231] The various illustrative logic blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, firmware, or combinations thereof. To clearly illustrate this interchangeability of hardware and software, the various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.

[0225]

[0232] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as a general-purpose computer, a wireless communication device handset, or an integrated circuit device with multiple uses, including applications in wireless communication device handsets and other devices. Features described as modules or components may be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, the techniques may be realized at least in part by a computer-readable data storage medium comprising program code including instructions that, when executed, perform one or more of the methods described above. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may comprise a memory or data storage medium, such as a random access memory (RAM) such as synchronous dynamic random access memory (SDRAM), a read-only memory (ROM), a nonvolatile random access memory (NVRAM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic or optical data storage medium, etc. The techniques may additionally or alternatively be realized at least in part by a computer-readable communications medium, such as a propagated signal or radio waves, that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer.

[0226]

[0233] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Such a processor may be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor, but alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Accordingly, the term “processor,” as used herein, may refer to any of the above structures, any combination of the above structures, or any other structure or apparatus suitable for implementing the techniques described herein. Furthermore, in some aspects, the functionality described herein may be provided within dedicated software or hardware modules configured for encoding and decoding, or may be incorporated into a composite video encoder / decoder (codec).

[0227]

[0234] Illustrative examples of the present disclosure include the following:

[0228]

[0235] Aspect 1. An apparatus for processing video data, comprising: a memory; and one or more processors coupled to the memory, wherein the one or more processors determine that an overlapped block motion compensation (OBMC) mode is enabled for a current sub-block of the block of video data; and determine, for at least one neighboring sub-block neighboring the current sub-block, whether a first condition, a second condition, and a third condition are satisfied; the first condition comprising all of one or more reference picture lists for predicting the current sub-block being used to predict the neighboring sub-block; and the second condition comprising all of one or more reference picture lists for predicting the current sub-block being used to predict the neighboring sub-block; and the second condition comprising all of the one or more reference picture lists for predicting the current sub-block being used to predict the neighboring sub-block; and a third condition comprises a first difference between a horizontal motion vector of the current sub-block and the horizontal motion vector of the neighboring sub-block and a second difference between a vertical motion vector of the current sub-block and the vertical motion vector of the neighboring sub-block not exceeding a motion vector difference threshold, wherein the motion vector difference threshold is greater than 0. The apparatus is configured to: determine that an OBMC mode is enabled for the current sub-block; and determine not to use motion information of the neighboring sub-block for motion compensation of the current sub-block based on determining that the first condition, the second condition, and the third condition are satisfied.

[0229]

[0236] Aspect 2. The apparatus described in aspect 1, wherein one or more processors are configured to determine to implement sub-block boundary OBMC mode for a current sub-block based on a decision to use decoder-side motion vector refinement (DMVR) mode, sub-block-based temporal motion vector prediction (SbTMVP) mode, or affine motion compensation prediction mode for the current sub-block.

[0230]

[0237] Aspect 3. The apparatus of Aspect 2, wherein, to implement a sub-block boundary OBMC mode for a current sub-block, the one or more processors are configured to: determine a first prediction associated with the current sub-block, a second prediction associated with a first OBMC block adjacent to a top border of the current sub-block, a third prediction associated with a second OBMC block adjacent to a left border of the current sub-block, a fourth prediction associated with a third OBMC block adjacent to a bottom border of the current sub-block, and a fifth prediction associated with a fourth OBMC block adjacent to a right border of the current sub-block; determine a sixth prediction based on results of applying the first weight to the first prediction, the second weight to the second prediction, the third weight to the third prediction, the fourth weight to the fourth prediction, and the fifth weight to the fifth prediction; and generate a blended sub-block corresponding to the current sub-block based on the sixth prediction.

[0231]

[0238] Embodiment 4. The apparatus of embodiment 3, wherein the second weight, the third weight, the fourth weight, and the fifth weight each comprise one or more weight values ​​associated with one or more samples from a corresponding sub-block of the current sub-block, and wherein a sum of the weight values ​​of the corner samples of the current sub-block is greater than a sum of the weight values ​​of other boundary samples of the current sub-block.

[0232]

[0239] Embodiment 5. The apparatus of embodiment 4, wherein a sum of weight values ​​of other boundary samples of the current sub-block is greater than a sum of weight values ​​of non-boundary samples of the current sub-block.

[0233]

[0240] Aspect 6. The apparatus of any of aspects 1 to 5, wherein the one or more processors are configured to determine to use a local illumination compensation (LIC) mode for additional blocks of video data, and, based on the determination to use the LIC mode for the additional blocks, skip signaling of information related to an OBMC mode for the additional blocks.

[0234]

[0241] Aspect 7. The apparatus of aspect 6, wherein the one or more processors are configured to signal a syntax flag with an empty value to skip signaling of information related to the OBMC mode for additional blocks, and the syntax flag is associated with the OBMC mode.

[0235]

[0242] Embodiment 8. The apparatus of any of embodiments 6 to 7, wherein the one or more processors are configured to receive a signal including a syntax flag with an empty value, the syntax flag associated with an OBMC mode for additional blocks of video data.

[0236]

[0243] Embodiment 9. The apparatus of any of embodiments 7 to 8, wherein the one or more processors are configured to determine, based on a syntax flag having an empty value, not to use the OBMC mode for additional blocks.

[0237]

[0244] Embodiment 10. The apparatus of any of embodiments 6 to 9, wherein, to skip signaling of information related to an OBMC mode for additional blocks, the one or more processors are configured to: determine not to use or enable the OBMC mode for the additional blocks based on a determination to use the LIC mode for the additional blocks; and skip signaling a value related to the OBMC mode for the additional blocks.

[0238]

[0245] Embodiment 11. The apparatus of any of embodiments 1 to 10, wherein the one or more processors are configured to: determine whether OBMC mode is enabled for additional blocks; determine whether OBMC mode is enabled for the additional blocks; and, based on a determination to use LIC mode for the additional blocks, determine to skip signaling information related to the OBMC mode for the additional blocks.

[0239]

[0246] Aspect 12. An apparatus described in any of aspects 1 to 11, wherein one or more processors are configured to determine to use coding unit (CU) boundary OBMC mode for a current sub-block of a block of video data, and to determine a final prediction for the current sub-block based on a sum of a first result of applying a weight associated with the current sub-block to each prediction associated with the current sub-block and a second result of applying one or more respective weights to one or more respective predictions associated with one or more sub-blocks neighboring the current sub-block.

[0240]

[0247] Aspect 13. The apparatus of any one of aspects 1 to 12, wherein, to determine not to use motion information of neighboring sub-blocks for motion compensation of the current sub-block, one or more processors are configured to skip using motion information of neighboring sub-blocks for motion compensation of the current sub-block.

[0241]

[0248] Embodiment 14. The apparatus of any of embodiments 1 to 13, wherein the apparatus includes a decoder.

[0242]

[0249] Embodiment 15. The apparatus of any of embodiments 1 to 14, further comprising a display configured to display one or more output pictures associated with the video data.

[0243]

[0250] Embodiment 16. The apparatus of any of embodiments 1 to 15, wherein the OBMC mode comprises a sub-block boundary OBMC mode.

[0244]

[0251] Embodiment 17. The apparatus of any of embodiments 1 to 16, wherein the apparatus includes an encoder.

[0245]

[0252] Embodiment 18. The apparatus of any of embodiments 1 to 17, further comprising a camera configured to capture pictures associated with the video data.

[0246]

[0253] Embodiment 19. The apparatus of any of embodiments 1 to 18, wherein the apparatus is a mobile device.

[0247]

[0254] Aspect 20. A method for processing video data, comprising: determining that an overlapped block motion compensation (OBMC) mode is enabled for a current sub-block of a block of the video data; and determining, for at least one neighboring sub-block adjacent to the current sub-block, whether a first condition, a second condition, and a third condition are satisfied; wherein the first condition is that all of one or more reference picture lists for predicting the current sub-block are used to predict the neighboring sub-block; and the second condition is that the same one or more reference pictures are used to determine motion vectors associated with the current sub-block and the neighboring sub-block. and a third condition comprising a first difference between a horizontal motion vector of the current sub-block and the horizontal motion vector of the neighboring sub-block and a second difference between a vertical motion vector of the current sub-block and the vertical motion vector of the neighboring sub-block not exceeding a motion vector difference threshold, wherein the motion vector difference threshold is greater than 0, and determining not to use motion information of the neighboring sub-block for motion compensation of the current sub-block based on determining that the first condition, the second condition, and the third condition are satisfied.

[0248]

[0255] Aspect 21. The method of aspect 20, further comprising determining to implement a sub-block boundary OBMC mode for the current sub-block based on a decision to use a decoder-side motion vector refinement (DMVR) mode, a sub-block-based temporal motion vector prediction (SbTMVP) mode, or an affine motion compensation prediction mode for the current sub-block.

[0249]

[0256] Embodiment 22. The method of embodiment 21, wherein performing a sub-block boundary OBMC mode for a current sub-block comprises determining a first prediction associated with the current sub-block, a second prediction associated with a first OBMC block adjacent to a top border of the current sub-block, a third prediction associated with a second OBMC block adjacent to a left border of the current sub-block, a fourth prediction associated with a third OBMC block adjacent to a bottom border of the current sub-block, and a fifth prediction associated with a fourth OBMC block adjacent to a right border of the current sub-block; determining a sixth prediction based on results of applying the first weight to the first prediction, applying the second weight to the second prediction, applying the third weight to the third prediction, applying the fourth weight to the fourth prediction, and applying the fifth weight to the fifth prediction; and generating a blended sub-block corresponding to the current sub-block based on the sixth prediction.

[0250]

[0257] Aspect 23. The method of aspect 22, wherein the second weight, the third weight, the fourth weight, and the fifth weight each comprise one or more weight values ​​associated with one or more samples from a corresponding sub-block of the current sub-block, and wherein a sum of the weight values ​​of the corner samples of the current sub-block is greater than a sum of the weight values ​​of other boundary samples of the current sub-block.

[0251]

[0258] Aspect 24. The method of aspect 23, wherein a sum of weight values ​​of other boundary samples of the current sub-block is greater than a sum of weight values ​​of non-boundary samples of the current sub-block.

[0252]

[0259] Embodiment 25. The method of any of embodiments 20 to 24, further comprising: determining to use a local illumination compensation (LIC) mode for an additional block of video data; and skipping signaling of information related to an OBMC mode for the additional block based on the determination to use the LIC mode for the additional block.

[0253]

[0260] Aspect 26. The method of aspect 25, wherein skipping signaling of information related to the OBMC mode for the additional block comprises signaling a syntax flag with an empty value, the syntax flag being related to the OBMC mode.

[0254]

[0261] Embodiment 27. The method of any one of embodiments 25 to 26, further comprising receiving a signal including a syntax flag with an empty value, the syntax flag associated with an OBMC mode for the additional block of video data.

[0255]

[0262] Embodiment 28. The method of any of embodiments 26 to 27, further comprising determining not to use the OBMC mode for the additional block based on a syntax flag having an empty value.

[0256]

[0263] Embodiment 29. The method of any of embodiments 25 to 28, wherein skipping signaling of information related to the OBMC mode for the additional block comprises: determining not to use or enable the OBMC mode for the additional block based on a decision to use the LIC mode for the additional block; and skipping signaling a value related to the OBMC mode for the additional block.

[0257]

[0264] Embodiment 30. The method of any of embodiments 25 to 29, further comprising: determining whether an OBMC mode is enabled for an additional block; and determining whether the OBMC mode is enabled for the additional block and, based on the determination to use the LIC mode for the additional block, determining to skip signaling information related to the OBMC mode for the additional block.

[0258]

[0265] Aspect 31. The method of any of aspects 20 to 30, further comprising: determining to use a coding unit (CU) boundary OBMC mode for a current sub-block of the block of video data; and determining a final prediction for the current sub-block based on a sum of a first result of applying a weight associated with the current sub-block to each prediction associated with the current sub-block and a second result of applying one or more respective weights to one or more respective predictions associated with one or more sub-blocks neighboring the current sub-block.

[0259]

[0266] Embodiment 32. The method of any one of embodiments 20 to 31, wherein determining not to use motion information of neighboring sub-blocks for motion compensation of the current sub-block comprises skipping use of motion information of neighboring sub-blocks for motion compensation of the current sub-block.

[0260]

[0267] Aspect 33. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform the method of any of aspects 20-32.

[0261]

[0268] Embodiment 34. An apparatus comprising means for carrying out the method of any of embodiments 20 to 32. The inventions described in the claims of the present application as originally filed are set forth below. [C1] 1. An apparatus for processing video data, comprising: Memory and one or more processors coupled to the memory; wherein the one or more processors: determining that an overlapped block motion compensation (OBMC) mode is enabled for a current sub-block of the block of video data; For at least one neighboring sub-block adjacent to the current sub-block, determining whether a first condition, a second condition, and a third condition are satisfied; the first condition comprises that all of one or more reference picture lists for predicting the current sub-block are used to predict the neighboring sub-blocks; the second condition comprises that the same one or more reference pictures are used to determine motion vectors associated with the current sub-block and the neighboring sub-block; the third condition comprises a first difference between the horizontal motion vector of the current sub-block and the horizontal motion vector of the neighboring sub-block and a second difference between the vertical motion vector of the current sub-block and the vertical motion vector of the neighboring sub-block do not exceed a motion vector difference threshold, wherein the motion vector difference threshold is greater than 0; determining not to use motion information of the neighboring sub-block for motion compensation of the current sub-block based on determining that the OBMC mode is enabled for the current sub-block and determining that the first condition, the second condition, and the third condition are satisfied; An apparatus configured to: [C2] the one or more processors: determining, based on a decision to use a decoder-side motion vector refinement (DMVR) mode, a sub-block-based temporal motion vector prediction (SbTMVP) mode, or an affine motion compensation prediction mode for the current sub-block, to implement a sub-block boundary OBMC mode for the current sub-block; The apparatus of C1, configured to perform the following: [C3] To implement the sub-block boundary OBMC mode for the current sub-block, the one or more processors: determining a first prediction associated with the current sub-block, a second prediction associated with a first OBMC block adjacent to a top border of the current sub-block, a third prediction associated with a second OBMC block adjacent to a left border of the current sub-block, a fourth prediction associated with a third OBMC block adjacent to a bottom border of the current sub-block, and a fifth prediction associated with a fourth OBMC block adjacent to a right border of the current sub-block; determining a sixth prediction based on results of applying a first weight to the first prediction, a second weight to the second prediction, a third weight to the third prediction, a fourth weight to the fourth prediction, and a fifth weight to the fifth prediction; generating a blended sub-block corresponding to the current sub-block based on the sixth prediction; and 3. The apparatus of claim 2, configured to: [C4] The apparatus of C3, wherein each of the second weight, the third weight, the fourth weight, and the fifth weight comprises one or more weight values ​​associated with one or more samples from a corresponding sub-block of the current sub-block, and wherein a sum of weight values ​​of corner samples of the current sub-block is greater than a sum of weight values ​​of other boundary samples of the current sub-block. [C5] The apparatus of C4, wherein the sum of weight values ​​of the other boundary samples of the current sub-block is greater than the sum of weight values ​​of non-boundary samples of the current sub-block. [C6] the one or more processors: determining to use a local illumination compensation (LIC) mode for an additional block of video data; skipping signaling of information related to an OBMC mode for the additional block based on a decision to use the LIC mode for the additional block; and The apparatus of C1, configured to perform the following: [C7] To skip signaling of information related to the OBMC mode for the additional block, the one or more processors: Signaling a syntax flag with an empty value and wherein the syntax flag is associated with the OBMC mode. [C8] the one or more processors: Receiving a signal containing a syntax flag with an empty value and wherein the syntax flag is associated with an OBMC mode for the additional block of video data. [C9] the one or more processors: determining not to use the OBMC mode for the additional block based on the syntax flag having the empty value; 9. The apparatus of claim 8, configured to: [C10] To skip signaling of information related to the OBMC mode for the additional block, the one or more processors: determining not to use or enable an OBMC mode for the additional block based on the decision to use the LIC mode for the additional block; and skipping signaling a value related to the OBMC mode for the additional block; 10. The apparatus of claim 6, configured to: [C11] the one or more processors: determining whether the OBMC mode is enabled for the additional block; and determining whether the OBMC mode is enabled for the additional block, and determining, based on the determination to use the LIC mode for the additional block, to skip signaling information related to the OBMC mode for the additional block; 10. The apparatus of claim 6, configured to: [C12] the one or more processors: determining to use a coding unit (CU) boundary OBMC mode for the current sub-block of the block of video data; and determining a final prediction for the current sub-block based on a sum of a first result of applying a weight associated with the current sub-block to each prediction associated with the current sub-block and a second result of applying one or more respective weights to one or more respective predictions associated with one or more sub-blocks neighboring the current sub-block; The apparatus of C1, configured to perform the following: [C13] To determine not to use motion information of the neighboring sub-blocks for motion compensation of the current sub-block, the one or more processors: Skipping use of motion information of the neighboring sub-blocks for motion compensation of the current sub-block. The apparatus of C1, configured to perform the following: [C14] The apparatus of C1, wherein the apparatus includes a decoder. [C15] The apparatus of C14, further comprising a display configured to display one or more output pictures associated with the video data. [C16] 3. The apparatus of claim 1, wherein the OBMC mode comprises a sub-block boundary OBMC mode. [C17] The apparatus of C1, wherein the apparatus includes an encoder. [C18] The apparatus of C17, further comprising a camera configured to capture pictures associated with the video data. [C19] The apparatus of C1, wherein the apparatus is a mobile device. [C20] 1. A method for processing video data, comprising: determining that an overlapped block motion compensation (OBMC) mode is enabled for a current sub-block of the block of video data; determining whether a first condition, a second condition, and a third condition are satisfied for at least one neighboring sub-block adjacent to the current sub-block; the first condition comprises that all of one or more reference picture lists for predicting the current sub-block are used to predict the neighboring sub-blocks; the second condition comprises that the same one or more reference pictures are used to determine motion vectors associated with the current sub-block and the neighboring sub-block; the third condition comprises a first difference between the horizontal motion vector of the current sub-block and the horizontal motion vector of the neighboring sub-block and a second difference between the vertical motion vector of the current sub-block and the vertical motion vector of the neighboring sub-block do not exceed a motion vector difference threshold, wherein the motion vector difference threshold is greater than 0; determining not to use motion information of the neighboring sub-block for motion compensation of the current sub-block based on determining to use the OBMC mode for the current sub-block and determining that the first condition, the second condition, and the third condition are satisfied; A method comprising: [C21] determining, based on a decision to use a decoder-side motion vector refinement (DMVR) mode, a sub-block-based temporal motion vector prediction (SbTMVP) mode, or an affine motion compensation prediction mode for the current sub-block, to implement a sub-block boundary OBMC mode for the current sub-block; The method of C20, further comprising: [C22] performing the sub-block boundary OBMC mode for the current sub-block; determining a first prediction associated with the current sub-block, a second prediction associated with a first OBMC block adjacent to a top border of the current sub-block, a third prediction associated with a second OBMC block adjacent to a left border of the current sub-block, a fourth prediction associated with a third OBMC block adjacent to a bottom border of the current sub-block, and a fifth prediction associated with a fourth OBMC block adjacent to a right border of the current sub-block; determining a sixth prediction based on results of applying a first weight to the first prediction, a second weight to the second prediction, a third weight to the third prediction, a fourth weight to the fourth prediction, and a fifth weight to the fifth prediction; generating a blended sub-block corresponding to the current sub-block based on the sixth prediction; and The method of claim C21, comprising: [C23] The method of C22, wherein each of the second weight, the third weight, the fourth weight, and the fifth weight comprises one or more weight values ​​associated with one or more samples from a corresponding sub-block of the current sub-block, and wherein a sum of weight values ​​of corner samples of the current sub-block is greater than a sum of weight values ​​of other boundary samples of the current sub-block. [C24] The method of C23, wherein the sum of weight values ​​of the other boundary samples of the current sub-block is greater than the sum of weight values ​​of non-boundary samples of the current sub-block. [C25] determining to use a local illumination compensation (LIC) mode for an additional block of video data; skipping signaling of information related to an OBMC mode for the additional block based on a decision to use the LIC mode for the additional block; and The method of C20, further comprising: [C26] skipping signaling of information related to the OBMC mode for the additional block; Signaling a syntax flag with an empty value and wherein the syntax flag is associated with the OBMC mode. [C27] Receiving a signal containing a syntax flag with an empty value and wherein the syntax flag is associated with an OBMC mode for additional blocks of video data. [C28] determining not to use the OBMC mode for the additional block based on the syntax flag having the empty value; The method of C27, further comprising: [C29] skipping signaling of information related to the OBMC mode for the additional block; determining not to use or enable an OBMC mode for the additional block based on the decision to use the LIC mode for the additional block; and skipping signaling a value related to the OBMC mode for the additional block; The method of C25, comprising: [C30] determining whether the OBMC mode is enabled for the additional block; and determining whether the OBMC mode is enabled for the additional block, and determining, based on the determination to use the LIC mode for the additional block, to skip signaling information related to the OBMC mode for the additional block; The method of C25, further comprising: [C31] determining to use a coding unit (CU) boundary OBMC mode for the current sub-block of the block of video data; and determining a final prediction for the current sub-block based on a sum of a first result of applying a weight associated with the current sub-block to each prediction associated with the current sub-block and a second result of applying one or more respective weights to one or more respective predictions associated with one or more sub-blocks neighboring the current sub-block; The method of C20, further comprising: [C32] determining not to use motion information of the neighboring sub-block for motion compensation of the current sub-block; Skipping use of motion information of the neighboring sub-blocks for motion compensation of the current sub-block. The method of claim C20, comprising: [C33] The method of C20, wherein the OBMC mode comprises a sub-block boundary OBMC mode.

Claims

1. 1. An apparatus for processing video data, comprising: Memory and one or more processors coupled to the memory; wherein the one or more processors: determining that an overlapped block motion compensation (OBMC) mode is enabled for a current sub-block of a block within a bidirectional slice (B slice) of video data; For at least one neighboring sub-block adjacent to the current sub-block, determining that the first condition, the second condition, and the third condition are satisfied; the first condition comprises that a reference picture list for uni-prediction of the current sub-block has been used to predict the at least one neighboring sub-block, the reference picture list including an indication of a reference picture that can be used for the uni-prediction of the current sub-block; the second condition comprises that the same one or more reference pictures are used to determine motion vectors associated with the current sub-block and the at least one neighboring sub-block; the third condition comprises a first difference between the horizontal motion vector of the current sub-block and the horizontal motion vector of the at least one neighboring sub-block and a second difference between the vertical motion vector of the current sub-block and the vertical motion vector of the at least one neighboring sub-block do not exceed a motion vector difference threshold, wherein the motion vector difference threshold is greater than 0; determining not to use motion information of the at least one neighboring sub-block for motion compensation of the current sub-block based on the determination that the OBMC mode is enabled for the current sub-block and the determination that the first condition, the second condition, and the third condition are satisfied; An apparatus configured to:

2. the one or more processors: determining, based on a decision to use a decoder-side motion vector refinement (DMVR) mode, a sub-block-based temporal motion vector prediction (SbTMVP) mode, or an affine motion compensation prediction mode for the current sub-block, to implement a sub-block boundary OBMC mode for the current sub-block; The apparatus of claim 1 configured to:

3. To implement the sub-block boundary OBMC mode for the current sub-block, the one or more processors: determining a first prediction associated with the current sub-block, a second prediction associated with a first OBMC block adjacent to a top border of the current sub-block, a third prediction associated with a second OBMC block adjacent to a left border of the current sub-block, a fourth prediction associated with a third OBMC block adjacent to a bottom border of the current sub-block, and a fifth prediction associated with a fourth OBMC block adjacent to a right border of the current sub-block; determining a sixth prediction based on a result of applying a first weight to the first prediction, a second weight to the second prediction, a third weight to the third prediction, a fourth weight to the fourth prediction, and a fifth weight to the fifth prediction; generating a blended sub-block corresponding to the current sub-block based on the sixth prediction; and The apparatus of claim 2 configured to:

4. 4. The apparatus of claim 3, wherein each of the second weight, the third weight, the fourth weight, and the fifth weight comprises one or more weight values ​​associated with one or more samples from a corresponding sub-block of the current sub-block, and wherein a sum of weight values ​​of corner samples of the current sub-block is greater than a sum of weight values ​​of other boundary samples of the current sub-block.

5. The apparatus of claim 4 , wherein the sum of weight values ​​of the other boundary samples of the current sub-block is greater than the sum of weight values ​​of non-boundary samples of the current sub-block.

6. the one or more processors: determining to use a local illumination compensation (LIC) mode for the additional block of video data; skipping signaling of information related to an OBMC mode for the additional block based on a decision to use the LIC mode for the additional block; and and, in order to skip signaling of information related to the OBMC mode for the additional block, the one or more processors are configured to: Signaling a syntax flag with an empty value wherein the syntax flag is associated with the OBMC mode; or the one or more processors: Receiving a signal containing a syntax flag with an empty value wherein the syntax flag relates to an OBMC mode for the additional block of video data; the one or more processors: determining not to use the OBMC mode for the additional block based on the syntax flag having the empty value; The apparatus of claim 1 configured to:

7. the one or more processors: determining to use a local illumination compensation (LIC) mode for the additional block of video data; skipping signaling of information related to an OBMC mode for the additional block based on a decision to use the LIC mode for the additional block; and configured to: To skip signaling of information related to the OBMC mode for the additional block, the one or more processors: determining not to use or enable an OBMC mode for the additional block based on the decision to use the LIC mode for the additional block; and skipping signaling the OBMC mode related value for the additional block; The apparatus of claim 1 configured to:

8. the one or more processors: determining to use a local illumination compensation (LIC) mode for the additional block of video data; skipping signaling of information related to an OBMC mode for the additional block based on a decision to use the LIC mode for the additional block; and configured to: the one or more processors: determining whether the OBMC mode is enabled for the additional block; and determining whether the OBMC mode is enabled for the additional block, and determining, based on the determination to use the LIC mode for the additional block, to skip signaling information related to the OBMC mode for the additional block; The apparatus of claim 1 configured to:

9. the one or more processors: determining to use a coding unit (CU) boundary OBMC mode for the current sub-block of the block of video data; determining a final prediction for the current sub-block based on a sum of a first result of applying a weight associated with the current sub-block to each prediction associated with the current sub-block and a second result of applying one or more respective weights to one or more respective predictions associated with one or more sub-blocks neighboring the current sub-block; The apparatus of claim 1 configured to:

10. To determine not to use motion information of the at least one neighboring sub-block for motion compensation of the current sub-block, the one or more processors: Skipping use of motion information of the at least one neighboring sub-block for motion compensation of the current sub-block. The apparatus of claim 1 configured to:

11. The apparatus of claim 1 , wherein the apparatus includes a decoder, the decoder further comprising a display configured to display one or more output pictures associated with the video data.

12. The apparatus of claim 1 , wherein the OBMC mode comprises a sub-block boundary OBMC mode.

13. The apparatus of claim 1 , wherein the apparatus includes an encoder, the encoder further comprising a camera configured to capture pictures associated with the video data.

14. The apparatus of claim 1 , wherein the apparatus is a mobile device.

15. 1. A method for processing video data, comprising: determining that an overlapped block motion compensation (OBMC) mode is enabled for a current sub-block of a block within a bidirectional slice (B slice) of video data; determining that a first condition, a second condition, and a third condition are satisfied for at least one neighboring sub-block adjacent to the current sub-block; the first condition comprises that a reference picture list for uni-prediction of the current sub-block has been used to predict the at least one neighboring sub-block, the reference picture list including an indication of a reference picture that can be used for the uni-prediction of the current sub-block; the second condition comprises that the same one or more reference pictures are used to determine motion vectors associated with the current sub-block and the at least one neighboring sub-block; the third condition comprises a first difference between the horizontal motion vector of the current sub-block and the horizontal motion vector of the at least one neighboring sub-block and a second difference between the vertical motion vector of the current sub-block and the vertical motion vector of the at least one neighboring sub-block do not exceed a motion vector difference threshold, wherein the motion vector difference threshold is greater than 0; determining not to use motion information of the at least one neighboring sub-block for motion compensation of the current sub-block based on the decision to use the OBMC mode for the current sub-block and the decision that the first condition, the second condition, and the third condition are satisfied; and A method comprising:

Images