Using replacement motion information in coding video data

Abstract

An example apparatus for decoding video data includes: a memory configured to store video data; and a processing system comprising one or more processors implemented in circuitry, the processing system configured to: determine that a previously coded block of video data is coded using a unidirectional prediction mode that does not assign a value to a bidirectional prediction syntax element; determining that a current block of the video data is to be coded using a bidirectional prediction mode and motion information of the current block is to be predicted according to a previously coded block, including prediction according to a bidirectional prediction syntax element of the previously coded block; and in response to the bidirectional prediction syntax element of the previously coded block having no assigned value, decoding the current block using the replacement value of the bidirectional prediction syntax element.

Description

[0001] This application claims priority to U.S. Patent Application No. 18 / 748,939, filed June 20, 2024, and U.S. Provisional Application No. 63 / 509,741, filed June 22, 2023, the entire contents of each of which are hereby incorporated by reference herein. U.S. Patent Application No. 18 / 748,939, filed June 20, 2024, claims the benefit of U.S. Provisional Application No. 63 / 509,741, filed June 22, 2023. TECHNICAL FIELD

[0002] This disclosure relates to video coding, including video encoding and video decoding. BACKGROUND

[0003] Digital video capabilities can be incorporated into a wide range of devices, including digital televisions, digital direct broadcast systems, wireless broadcast systems, personal digital assistants (PDAs), laptop or desktop computers, tablet computers, e-book readers, digital cameras, digital recording devices, digital media players, video gaming devices, video game consoles, cellular or satellite radio telephones, so-called “smart phones,” video teleconferencing devices, video streaming devices, and the like. Digital video devices implement video coding techniques to compress video data into a compressed form suitable for transmission, storage, or playback. Video coding techniques include those disclosed in the MPEG-2, MPEG-4, ITU-T H.263, ITU-T H.264 / MPEG-4, Part 10, Advanced Video Coding (AVC), ITU-T H.265 / High Efficiency Video Coding (HEVC), ITU-T H.266 / Versatile Video Coding (VVC), standards and extensions thereof, and proprietary video codecs / formats such as AOMedia Video 1 (AV1) developed by the Alliance for Open Media. By implementing such video coding techniques, video devices can transmit, receive, encode, decode, and / or store digital video information more efficiently.

[0004] Video coding techniques include spatial (intra-picture) prediction and / or temporal (inter-picture) prediction to reduce or remove redundancy in video sequences. For block-based video coding, a video slice (e.g., a video picture or a portion of a video picture) can be partitioned into video blocks, which can also be referred to as coding tree units (CTUs), coding units (CUs), and / or coding nodes. Video blocks in the intra-coded (I) slices of a picture are encoded using spatial prediction with respect to reference samples in neighboring blocks within the same picture. Video blocks in the inter-coded (P or B) slices of a picture can use spatial prediction with respect to reference samples in neighboring blocks within the same picture or temporal prediction with respect to reference samples in other reference pictures. Pictures can be referred to as frames, and reference pictures can be referred to as reference frames. SUMMARY

[0005] Generally, this disclosure describes techniques for substituting values for unavailable values that would otherwise be referenced when coding a current block due to a coding mode of a previous block. The substituted values can be pre-defined preset values, or can be extracted or derived from neighboring blocks of the current block or a previously coded block. For example, certain syntax elements can only be used for bi-prediction. As an example, for bi-predictive with coding unit (CU) level weights (BCW), an index value (e.g., bcwldx) can be coded to indicate the weight value to be used. However, the bcwldx value is not coded for a uni-predicted block. Motion information for the current block can be predicted from neighboring blocks of the current block. Thus, if the current block is predicted using, for example, BCW, and the neighboring blocks of the current block are coded using uni-prediction, no value is assigned to the bcwldx syntax element for the neighboring blocks. According to the techniques of this disclosure, a preset value or a substituted value can be used for such syntax elements, e.g., to determine the motion information for the current block.

[0006] In one example, a method of decoding video data includes determining that a previously coded block of video data was coded using a uni-prediction mode for which a bi-prediction syntax element was not assigned a value; determining that a current block of video data is to be coded using a bi-prediction mode and that motion information for the current block is to be predicted from the previously coded block, including from the bi-prediction syntax element of the previously coded block; and in response to the bi-prediction syntax element of the previously coded block not having an assigned value, decoding the current block using a substituted value for the bi-prediction syntax element.

[0007] In another example, a device for decoding video data includes a memory configured to store video data; and a processing system including one or more processors implemented in circuitry, the processing system configured to: determine that a previously coded block of video data was coded using a uni-prediction mode for which a bi-prediction syntax element was not assigned a value; determine that a current block of video data is to be coded using a bi-prediction mode and that motion information for the current block is to be predicted from the previously coded block, including from the bi-prediction syntax element of the previously coded block; and in response to the bi-prediction syntax element of the previously coded block not having an assigned value, decode the current block using a substituted value for the bi-prediction syntax element.

[0008] The details of one or more examples are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF DRAWINGS

[0009] Figure 1is a block diagram illustrating an example video encoding and decoding system that can perform the techniques of this disclosure.

[0010] Figure 2A and Figure 2B is a conceptual diagram illustrating an example of sub-blocks used in overlapping block motion compensation (OBMC).

[0011] Figure 3 is a conceptual diagram illustrating sub-block template generation for a sub-block based temporal motion vector prediction (SbTMVP) mode.

[0012] Figure 4 is a conceptual diagram illustrating geometric partitioning for a geometric partitioning mode (GPM).

[0013] Figure 5 is a table illustrating how a split mode index maps to an angular offset.

[0014] Figure 6 is a block diagram illustrating an example video encoder that can perform the techniques of this disclosure.

[0015] Figure 7 is a block diagram illustrating an example video decoder that can perform the techniques of this disclosure.

[0016] Figure 8 is a flowchart illustrating an example method for encoding a current block according to the techniques of this disclosure.

[0017] Figure 9 is a flowchart illustrating an example method for decoding a current block according to the techniques of this disclosure.

[0018] Figure 10 is a flowchart illustrating an example method of coding a block of video data according to the techniques of this disclosure. DETAILED DESCRIPTION

[0019] Video coding typically includes partitioning a picture into blocks and coding each block individually. Coding a block can include predicting the block to form a predicted block and coding a residual block representing a difference between the predicted block and the original uncoded block. Prediction can be performed using neighboring data of the current block in the same picture (intra prediction) or using data of previously coded pictures (inter prediction), which can be identified using motion information. Motion information itself can also be coded, for example, using a merge mode or an advanced motion vector prediction (AMVP) mode. Generally, when coding motion information for a current block, the motion information can be predicted from motion information of spatial and / or temporal neighboring blocks of the current block.

[0020] Motion information can be stored using a motion buffer, which can represent motion information using a grid of fixed-size entries (e.g., 4x4 sample entries). Each entry of the grid can store motion information for a corresponding block of video data. In this way, motion information for a neighboring block of a current block can be extracted from an entry of the motion buffer corresponding to the neighboring block.

[0021] In some cases, a block can be further divided into sub-blocks, and each of the sub-blocks can be predicted individually. For example, motion information can be coded for the entire block, followed by refinement for each of the sub-blocks. When sub-blocks of a block are predicted individually, overlapped block motion compensation (OBMC) can be performed, as explained in more detail below.

[0022] Inter prediction of a block can be performed in a single direction (uni-prediction) or in two directions (bi-prediction). Certain motion information syntax elements can only apply to one direction or the other. For example, bi-prediction with CU-level weights (BCW) only applies to bi-prediction. Thus, a BCW index (bcwldx) value, which can indicate a weight value to be applied in BCW, can not have a value assigned in the motion buffer for a uni-predicted block. In this regard, certain syntax elements can not have assigned values if the motion information for a current block, which is BCW predicted, is predicted from motion information of uni-predicted neighboring blocks. According to the techniques of this disclosure, a substitute value can be used for such syntax elements. In this way, prediction of motion information can be harmonized across various scenarios of motion information prediction, which can reduce the complexity of a video coder. In this way, performance of a video coder can be improved.

[0023] Figure 1 FIG. 1 is a block diagram illustrating an example video encoding and decoding system 100 that can perform the techniques of this disclosure. The techniques of this disclosure generally relate to coding (encoding and / or decoding) video data. In general, video data includes any data for processing video. Thus, video data can include raw uncoded video, encoded video, decoded (e.g., reconstructed) video, and video metadata, such as signaling data.

[0024] As Figure 1As shown, in this example, system 100 includes a source device 102 that provides encoded video data to be decoded and displayed by a destination device 116. Specifically, source device 102 provides the video data to destination device 116 via computer-readable medium 110. Source device 102 and destination device 116 can include any of a wide variety of devices, including desktop computers, laptop computers, mobile devices, tablet computers, set-top boxes, mobile phones such as smartphones, televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, broadcast receiver devices, etc. In some cases, source device 102 and destination device 116 may be configured for wireless communication and are therefore referred to as wireless communication devices.

[0025] exist Figure 1 In the example, source device 102 includes a video source 104, memory 106, video encoder 200, and output interface 108. Destination device 116 includes an input interface 122, video decoder 300, memory 120, and display device 118. According to this disclosure, the video encoder 200 of source device 102 and the video decoder 300 of destination device 116 can be configured to apply techniques for using substitution values ​​when decoding video data. Thus, source device 102 represents an example of a video encoding device, while destination device 116 represents an example of a video decoding device. In other examples, the source device and destination device may include other components or arrangements. For example, source device 102 may receive video data from an external video source such as an external camera. Similarly, destination device 116 may interface with an external display device instead of including an integrated display device.

[0026] like Figure 1 The system 100 shown is merely an example. Typically, any digital video encoding and / or decoding device can perform techniques for using substitution values ​​when decoding video data. Source device 102 and destination device 116 are merely examples of such decoding devices, where source device 102 generates decoded video data for transmission to destination device 116. This disclosure refers to a “decoding” device as a device that performs the decoding (e.g., encoding and / or decoding) of data. Thus, video encoder 200 and video decoder 300 represent examples of decoding devices, specifically, a video encoder and a video decoder, respectively. In some examples, source device 102 and destination device 116 may operate in a substantially symmetrical manner, such that each of source device 102 and destination device 116 includes video encoding and decoding components. Therefore, system 100 may support one-way or two-way video transmission between source device 102 and destination device 116, for example, for video streaming, video playback, video broadcasting, or video telephony.

[0027] In general, video source 104 represents a source of video data (i.e., raw, uncoded video data) and provides a sequential series of pictures (also referred to as “frames”) of the video data to video encoder 200, which encodes the data of the pictures. Video source 104 of source device 102 can include a video capture device, such as a video camera, a video archive containing previously captured raw video, and / or a video feed interface to receive video from a video content provider. As a further alternative, video source 104 can generate computer graphics-based data as the source video, or a combination of live video, archived video, and computer-generated video. In each case, video encoder 200 encodes the captured, pre-captured, or computer-generated video data. Video encoder 200 can rearrange the pictures from the received order (sometimes referred to as “display order”) into the coding order for coding. Video encoder 200 can generate a bitstream including encoded video data. Source device 102 can then output the encoded video data via output interface 108 onto computer- readable medium 110 for reception and / or retrieval by, for example, input interface 122 of destination device 116.

[0028] Memory 106 of source device 102 and memory 120 of destination device 116 represent general purpose memories. In some examples, memories 106, 120 can store raw video data, e.g., raw video from video source 104 and raw decoded video data from video decoder 300. Additionally or alternatively, memories 106, 120 can store software instructions, which are executable by, for example, video encoder 200 and video decoder 300, respectively. Although memories 106 and 120 are displayed separately from video encoder 200 and video decoder 300 in this example, it should be understood that video encoder 200 and video decoder 300 can also include internal memories for similar or equivalent purposes. Furthermore, memories 106, 120 can store encoded video data, e.g., from an output of video encoder 200 and an input of video decoder 300. In some examples, portions of memories 106, 120 can be allocated as one or more video buffers, e.g., to store raw decoded and / or encoded video data.

[0029] Computer-readable medium 110 can represent any type of medium or device capable of transporting encoded video data from source device 102 to destination device 116. In one example, computer-readable medium 110 represents a communication medium to enable source device 102 to transmit encoded video data directly to destination device 116 in real-time, e.g., via a radio frequency network or computer-based network. Output interface 108 can modulate the transmission signal including the encoded video data, and input interface 122 can demodulate the received transmission signal, according to a communication standard, such as a wireless communication protocol. The communication medium can comprise any wireless or wired communication medium, such as a radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium can form part of a packet-based network, such as a local area network, a wide-area network, or a global network such as the Internet. The communication medium can include routers, switches, base stations, or any other equipment that can be useful to facilitate communication from source device 102 to destination device 116.

[0030] In some examples, source device 102 can output encoded data from output interface 108 to storage device 112. Similarly, destination device 116 can access encoded data from storage device 112 via input interface 122. Storage device 112 can include any of a variety of distributed or locally accessed data storage media such as a hard drive, Blu-ray discs, DVDs, CD-ROMs, flash memory, volatile or non-volatile memory, or any other suitable digital storage media for storing encoded video data.

[0031] In some examples, source device 102 can output encoded video data to file server 114, or another intermediate storage device, which can store the encoded video data generated by source device 102. Destination device 116 can access stored video data from file server 114 via streaming or download.

[0032] The file server 114 can be any type of server device capable of storing encoded video data and transmitting that encoded video data to the destination device 116. The file server 114 can represent a web server (e.g., for a website), a server configured to provide file delivery protocol services (such as the File Delivery Protocol (FTP) or the File Delivery over Unidirectional Transport (FLUTE) protocol), a content delivery network (CDN) device, a hypertext transfer protocol (HTTP) server, a multimedia broadcast multicast service (MBMS) or enhanced MBMS (eMBMS) server, and / or a network attached storage (NAS) device. The file server 114 can additionally or alternatively implement one or more HTTP streaming protocols, such as Dynamic Adaptive Streaming over HTTP (DASH), HTTP Live Streaming (HLS), Real Time Streaming Protocol (RTSP), HTTP Dynamic Streaming, and the like.

[0033] The destination device 116 can access encoded video data from the file server 114 through any standard data connection, including an Internet connection. This can include a wireless channel (e.g., a Wi-Fi connection), a wired connection (e.g., a digital subscriber line (DSL), a cable modem, or the like), or a combination of both that is suitable for accessing encoded video data stored on the file server 114. The input interface 122 can be configured to operate according to any one or more of various protocols discussed in the preceding sections for retrieving or receiving media data from the file server 114, or other such protocols for retrieving media data.

[0034] The output interface 108 and the input interface 122 can represent wireless transmitters / receivers, modems, wired networking components (e.g., Ethernet cards), wireless communication components operating according to any of a variety of IEEE 802.11 standards, or other physical components. In examples where the output interface 108 and the input interface 122 comprise wireless components, the output interface 108 and the input interface 122 can be configured to transfer data, such as encoded video data, according to a cellular communication standard, such as 4G, 4G-LTE (Long-Term Evolution), LTE Advanced, 5G, or the like. In some examples where the output interface 108 includes a wireless transmitter, the output interface 108 and the input interface 122 can be configured to transfer data according to other wireless standards, such as an IEEE 802.11 specification, an IEEE 802.15 specification (e.g., ZigBee), a Bluetooth ™ ™ ​Standard) to communicate data (such as encoded video data). In some examples, source device 102 and / or destination device 116 can include respective system on a chip (SoC) devices. For example, source device 102 can include SoC devices to perform the functionality of video encoder 200 and / or output interface 108, and destination device 116 can include SoC devices to perform the functionality of video decoder 300 and / or input interface 122.

[0035] The techniques of this disclosure can be applied to video coding in support of any of a variety of multimedia applications, such as over-the-air television broadcasts, cable television transmissions, satellite television transmissions, Internet streaming video transmissions, digital video that is encoded onto a data storage medium, decoding of digital video stored on a data storage medium, or other applications.

[0036] Input interface 122 of destination device 116 receives an encoded video bitstream from computer-readable medium 110 (e.g., a communication medium, storage device 112, file server 114, or the like). The encoded video bitstream can include signaling information defined by video encoder 200, and also used by video decoder 300, such as syntax elements having values

[0037] Although not shown in FIG. 5, in some examples, video encoder 200 and video decoder 300 can each be integrated as components of an integrated circuit (IC) device, such as an SoC device. For example, video encoder 200 can be integrated as a component of an SoC device to perform the functionality of video encoder 200, and video decoder 300 can be integrated as a component of an SoC device to perform the functionality of video decoder 300. Figure 1 Although not shown in FIG. 5, in some examples, video encoder 200 and video decoder 300 can each be integrated as components of an integrated circuit (IC) device, such as an SoC device. For example, video encoder 200 can be integrated as a component of an SoC device to perform the functionality of video encoder 200, and video decoder 300 can be integrated as a component of an SoC device to perform the functionality of video decoder 300.

[0038] Video encoder 200 and video decoder 300 each can be implemented as any of a variety of suitable encoder and / or decoder circuitry, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware or any combinations thereof. When the techniques are implemented partially in software, a device can store instructions for the software in a suitable, non- transitory computer-readable medium and execute the instructions in hardware using one or more processors to perform the techniques of this disclosure. Each of video encoder 200 and video decoder 300 can be included in one or more encoders or decoders, either of which can be integrated as part of a combined encoder / decoder (CODEC) in a respective device. A device including video encoder 200 and / or video decoder 300 can comprise an integrated circuit, a microprocessor, and / or a wireless communication device, such as a cellular telephone.

[0039] Video encoder 200 and video decoder 300 can operate according to a video coding standard, such as ITU-T H.265, also referred to as High Efficiency Video Coding (HEVC), or extensions thereof, such as the multi-view and / or scalable video coding extensions. Alternatively, video encoder 200 and video decoder 300 can operate according to other proprietary or industry standards, such as ITU-T H.266, also referred to as Versatile Video Coding (VVC). In other examples, video encoder 200 and video decoder 300 can operate according to a proprietary video codec / format, such as AOMedia Video 1 (AV1), extensions of AV1, and / or subsequent versions of AV1 (e.g., AV2). In other examples, video encoder 200 and video decoder 300 can operate according to other proprietary formats or industry standards. The techniques of this disclosure, however, are not limited to any particular coding standard or format. In general, video encoder 200 and video decoder 300 can be configured to perform the techniques of this disclosure in connection with any video coding technique that uses substitute values when coding video data.

[0040] In general, video encoder 200 and video decoder 300 can perform block-based coding of pictures. The term “block” generally refers to a structure containing data to be processed (e.g., encoded, decoded, or otherwise used) during encoding and / or decoding. For example, a block can include a two-dimensional matrix of samples of luma and / or chroma data. In general, video encoder 200 and video decoder 300 can code video data represented in a YUV (e.g., Y, Cb, Cr) format. That is, rather than coding red, green, and blue (RGB) data for a sample of a picture, video encoder 200 and video decoder 300 can code a luma component and chroma components, where the chroma components can include both a red hue chroma component and a blue hue chroma component. In some examples, video encoder 200 converts received RGB format data to a YUV representation prior to encoding, and video decoder 300 converts the YUV representation to the RGB format. Alternatively, pre- and post-processing units (not shown) can perform these conversions.

[0041] This disclosure can generally relate to coding (e.g., encoding and decoding) of pictures to include processes of encoding or decoding data of pictures. Similarly, this disclosure can relate to coding of blocks of pictures to include processes of encoding or decoding (e.g., prediction and / or residual coding) data for blocks. A coded video bitstream generally includes a series of values for syntax elements that represent coding decisions (e.g., coding modes) and partitioning of pictures into blocks. Accordingly, references to coding of pictures or blocks generally should be understood to refer to coding values of syntax elements that form the pictures or blocks.

[0042] HEVC defines various blocks, including coding units (CUs), prediction units (PUs), and transform units (TUs). According to HEVC, a video coder (such as video encoder 200) partitions a coding tree unit (CTU) into CUs according to a quadtree structure. That is, the video coder partitions a CTU and CUs into four equal, non overlapping squares, and each node of the quadtree has either zero or four child nodes. Nodes with zero child nodes can be referred to as “leaf nodes,” and CUs of such leaf nodes can include one or more PUs and / or one or more TUs. Video coder can further partition PUs and TUs. For example, in HEVC, a residual quadtree (RQT) represents partitioning of TUs. In HEVC, PUs represent inter-predicted data, while TUs represent residual data. Intra-predicted CUs include intra-prediction information, such as an intra-mode indication.

[0043] As another example, video encoder 200 and video decoder 300 can be configured to operate according to VVC. According to VVC, a video coder (such as video encoder 200) partitions a picture into a plurality of coding tree units (CTUs). Video encoder 200 can partition a CTU according to a tree structure, such as a quad-tree binary-tree (QTBT) structure or a multi-type tree (MTT) structure. The QTBT structure removes the concepts of multiple partition types, such as the separation between CUs, PUs, and TUs of HEVC. The QTBT structure includes two levels: a first level partitioned according to quad-tree partitioning and a second level partitioned according to binary tree partitioning. The root node of the QTBT structure corresponds to a CTU. The leaf nodes of the binary tree correspond to coding units (CUs).

[0044] In the MTT partitioning structure, blocks can be partitioned using quad-tree (QT) partitioning, binary tree (BT) partitioning, and one or more types of ternary tree (TT) (also referred to as tri-tree) partitioning. A ternary tree or tri-tree partitioning is a partitioning in which a block is split into three sub-blocks. In some examples, a ternary tree or tri-tree partitioning divides a block into three sub-blocks without dividing the original block through the center. The partitioning types (e.g., QT, BT, and TT) in the MTT can be symmetric or asymmetric.

[0045] When operating according to the AV1 codec, video encoder 200 and video decoder 300 can be configured to code video data in units of blocks. In AV1, the largest coding block that can be processed is referred to as a superblock. In AV1, a superblock can be 128x128 luma samples or 64x64 luma samples. However, in subsequent video coding formats (e.g., AV2), a superblock can be defined by different (e.g., larger) luma sample sizes. In some examples, a superblock is the top level of a block quadtree. Video encoder 200 can further partition a superblock into smaller coding blocks. Video encoder 200 can partition superblocks and other coding blocks into smaller blocks using square or non-square partitions. Non-square blocks can include N / 2xN blocks, NxN / 2 blocks, N / 4xN blocks, and NxN / 4 blocks. Video encoder 200 and video decoder 300 can perform separate prediction and transform processing for each coding block.

[0046] AV1 also defines tiles of video data. A tile is a rectangular array of superblocks that can be coded independently of other tiles. That is, video encoder 200 and video decoder 300 can encode and decode coding blocks within a tile without using video data from other tiles. However, video encoder 200 and video decoder 300 can perform filtering across tile boundaries. The size of a tile can be uniform or non-uniform. Tile-based coding can enable parallel processing and / or multi-threading of encoder and decoder implementations.

[0047] In some examples, video encoder 200 and video decoder 300 can use a single QTBT or MTT structure to represent each of luma and chroma components, while in other examples, video encoder 200 and video decoder 300 can use two or more QTBT or MTT structures, such as one QTBT / MTT structure for luma components and another QTBT / MTT structure for two chroma components (or two QTBT / MTT structures for respective chroma components).

[0048] Video encoder 200 and video decoder 300 can be configured to use quadtree partitioning, QTBT partitioning, MTT partitioning, superblock partitioning, or other partition structures.

[0049] In some examples, a CTU includes a coding tree block (CTB) of luma samples, two corresponding CTBs of chroma samples of a picture having three sample arrays, or a CTB of samples of a picture coded using three separate color planes and syntax structures for coding samples. A CTB can be an NxN block of samples of some N value, such that one type of partitioning is dividing a component into CTBs. A component can be an array from one of three arrays (luma and two chroma) for 4:2:0, 4:2:2, or 4:4:4 color format pictures or a single sample, or an array or a single sample of an array for monochrome format pictures. In some examples, a coding block is an MxN block of samples of some M value and N value, such that one type of partitioning is dividing a CTB into coding blocks.

[0050] Blocks (e.g., CTUs or CUs) can be grouped in pictures in various ways. As one example, a tile can refer to a rectangular region of CTU rows within a particular tile in a picture. A tile column can refer to a rectangular region of CTUs within a particular tile column and a particular tile row in a picture. A tile column refers to a rectangular region of CTUs having a height equal to a height of a picture and a width specified by a syntax element (e.g., such as in a picture parameter set) and a tile row refers to a rectangular region of CTUs having a height specified by a syntax element (e.g., such as in a picture parameter set) and a width equal to a width of a picture.

[0051] In some examples, a tile can be divided into multiple bricks, each of which can include one or more CTU rows within the tile. A tile that is not divided into multiple bricks can also be referred to as a brick. However, a brick that is a true subset of a tile can not be referred to as a tile. Bricks in a picture can also be arranged in slices. A slice can be an integer number of bricks of a picture, which can be uniquely contained in a single network abstraction layer (NAL) unit. In some examples, a slice includes multiple complete tiles or only a contiguous sequence of complete bricks of one tile.

[0052] The disclosure interchangeably uses "NxN" and "N by N" to refer to the sample dimensions of a block, such as a CU or other video block, in terms of vertical and horizontal dimensions, e.g., 16x16 samples or 16 by 16 samples. In general, a 16x16 CU will have 16 samples in the vertical direction (y = 16) and 16 samples in the horizontal direction (x = 16). Likewise, an NxN CU generally has N samples in the vertical direction and N samples in the horizontal direction, where N represents a non negative integer value. The samples in a CU can be arranged in rows and columns. Moreover, a CU need not necessarily have the same number of samples in the horizontal direction as in the vertical direction. For example, a CU can include N x M samples, where M is not necessarily equal to N.

[0053] Video encoder 200 encodes video data representing prediction and / or residual information for a CU, among other information. Prediction information indicates how to predict the CU in order to form a prediction block for the CU. Residual information generally represents sample- wise differences between the CU prior to encoding and the prediction block.

[0054] To predict a CU, video encoder 200 can generally form a prediction block for the CU through inter-prediction or intra-prediction. Inter-prediction generally refers to predicting the CU from data of a previously coded picture, whereas intra-prediction generally refers to predicting the CU from previously coded data of the same picture. To perform inter-prediction, video encoder 200 can use one or more motion vectors to generate the prediction block. Video encoder 200 can generally perform a motion search to identify a reference block that closely matches the CU, e.g., according to a difference between the CU and the reference block. Video encoder 200 can calculate the difference metric using a sum of absolute difference (SAD), sum of squared difference (SSD), mean absolute difference (MAD), mean squared difference (MSD), or other such difference calculations to determine whether a reference block closely matches a current CU. In some examples, video encoder 200 can use uni -prediction or bi-prediction to predict the current CU.

[0055] Video encoder 200 can use sub-block modes to predict a current block, such as bi-prediction with CU-level weights (BCW). In cases where a previously coded neighboring block is coded in a mode that does not have a corresponding syntax element for motion information assigned a value, such as a bcwldx syntax element, video encoder 200 can determine a substitute value for the bcwldx syntax element. For example, video encoder 200 can determine the substitute value to be equal to a BCW DEFAULT value, which can be equal to BCW_NUM » 1, where ">>" represents a bitwise right shift operator and BCW_NUM represents the number of weight options. Video decoder 300 can perform similar operations when decoding a block.

[0056] Some examples of VVC also provide an affine motion compensation mode, which can be considered an inter-prediction mode. In the affine motion compensation mode, video encoder 200 can determine two or more motion vectors that represent non-translational motion, such as scaling or zooming, rotation, perspective motion, or other irregular types of motion.

[0057] To perform intra-prediction, video encoder 200 can select an intra-prediction mode to generate a prediction block. Some examples of VVC provide sixty-seven intra-prediction modes, including various directional modes, as well as a planar mode and a DC mode. In general, video encoder 200 selects an intra-prediction mode that describes neighboring samples of a current block (e.g., a block of a CU) from which to predict samples of the current block. Such samples can generally be located above, above and to the left, or to the left of the current block in the same picture as the current block, assuming video encoder 200 is coding CTUs and CUs in a raster scan order (left-to-right, top-to-bottom).

[0058] Video encoder 200 encodes data that represents the prediction mode for the current block. For example, for inter-prediction modes, video encoder 200 can encode data that indicates which of various available inter-prediction modes to use, as well as motion information for the corresponding mode. For example, for uni- or bi-prediction, video encoder 200 can encode motion vectors using advanced motion vector prediction (AMVP) or merge mode. Video encoder 200 can use similar modes to encode motion vectors for the affine motion compensation mode.

[0059] AV1 includes two general techniques for encoding and decoding blocks of video data. The two general techniques are intra-prediction (e.g., intra-prediction or spatial prediction) and inter-prediction (e.g., inter-prediction or temporal prediction). In the context of AV1, when a block of a current frame of video data is predicted using an intra-prediction mode, video encoder 200 and video decoder 300 do not use video data from other frames of the video data. For most intra-prediction modes, video encoder 200 encodes a block of a current frame based on differences between sample values in the current block and prediction values generated from reference samples in the same frame. Video encoder 200 determines the prediction values generated from the reference samples based on the intra-prediction mode.

[0060] Following prediction, such as intra prediction or inter prediction of a block, video encoder 200 can calculate residual data for the block. The residual data, such as a residual block, represents sample-by-sample differences between the block and a prediction block for the block, formed using the corresponding prediction mode. Video encoder 200 can apply one or more transforms to the residual block to produce transform data in a transform domain instead of in the sample domain. For example, video encoder 200 can apply a discrete cosine transform (DCT), an integer transform, a wavelet transform, or a conceptually similar transform. Additionally, video encoder 200 can apply a secondary transform, such as a mode-dependent non-separable secondary transform (MDNSST), a signal-dependent transform, a Karhunen-Loeve transform (KLT), and the like, following the first transform. Video encoder 200 produces transform coefficients following application of the one or more transforms.

[0061] As noted above, following any transforms that produce transform coefficients, video encoder 200 can perform quantization of the transform coefficients. Quantization generally refers to a process in which transform coefficients are quantized to possibly reduce the amount of data used to represent the transform coefficients, providing further compression. By performing the quantization process, video encoder 200 can reduce the bit depth associated with some or all of the transform coefficients. For example, video encoder 200 can round n-bit values down to m-bit values during quantization, where n is greater than m. In some examples, to perform quantization, video encoder 200 can perform a bitwise right-shift of the values to be quantized.

[0062] Following quantization, video encoder 200 can scan the transform coefficients, producing a one-dimensional vector from the two-dimensional matrix that includes the quantized transform coefficients. The scan can be designed to place higher energy (and thus lower frequency) transform coefficients earlier in the vector and lower energy (and thus higher frequency) transform coefficients later in the vector. In some examples, video encoder 200 can utilize a predefined scan order to scan the quantized transform coefficients to produce a serialized vector, and then entropy encode the quantized transform coefficients of the vector. In other examples, video encoder 200 can perform an adaptive scan. Following scanning of the quantized transform coefficients to form a one-dimensional vector, video encoder 200 can entropy encode the one-dimensional vector, e.g., according to context adaptive binary arithmetic coding (CABAC). Video encoder 200 can also entropy encode values for syntax elements that describe metadata associated with encoded video data, which is used by video decoder 300 when decoding the video data.

[0063] To perform CABAC, video encoder 200 can assign a context within a context model to a symbol to be transmitted. The context can relate to, for example, whether neighboring values of the symbol are zero-valued or not. The probability determination can be based on a context assigned to the symbol.

[0064] Video encoder 200 can further generate syntax data, such as block-based syntax data, picture-based syntax data, and sequence-based syntax data, for example, in a picture header, a block header, a slice header, or other syntax data such as a sequence parameter set (SPS), a picture parameter set (PPS), or a video parameter set (VPS), to video decoder 300. Video decoder 300 can likewise decode such syntax data to determine how to decode corresponding video data.

[0065] In this way, video encoder 200 can generate a bitstream that includes encoded video data, e.g., syntax elements that describe partitioning of a picture into blocks (e.g., CUs) and prediction and / or residual information for the blocks. Ultimately, video decoder 300 can receive the bitstream and decode the encoded video data.

[0066] In general, video decoder 300 performs a reciprocal process to that performed by video encoder 200 to decode the encoded video data of the bitstream. For example, video decoder 300 can use CABAC in substantially a similar but reciprocal manner to the CABAC encoding process of video encoder 200 to decode values for syntax elements of the bitstream. The syntax elements can define partitioning information for partitioning a picture into CTUs, and each CTU according to a corresponding partition structure such as a QTBT structure, to define CUs of the CTU. The syntax elements can further define prediction and residual information for blocks (e.g., CUs) of video data.

[0067] The residual information can be represented by, for example, quantized transform coefficients. Video decoder 300 can inverse quantize and inverse transform the quantized transform coefficients of a block to reproduce a residual block for the block. Video decoder 300 forms a prediction block for a block using the signaled prediction mode (intra-prediction or inter-prediction) and related prediction information (e.g., motion information for inter-prediction). Video decoder 300 can then combine the prediction block and the residual block (on a sample-by-sample basis) to reproduce the original block. Video decoder 300 can perform additional processing such as performing a deblocking process to reduce visual artifacts along boundaries of the blocks.

[0068] The disclosure can generally relate to "signaling" certain information, such as syntax elements. The term "signaling" can generally refer to the communication of values for syntax elements and / or other data used for decoding encoded video data. That is, video encoder 200 can signal values for syntax elements in a bitstream. Generally, signaling refers to generating the values in the bitstream. As noted above, source device 102 can transmit the bitstream to destination device 116 in substantially real time or not in real time, such as can occur when syntax elements are stored to storage device 112 for later retrieval by destination device 116.

[0069] Figure 2A and Figure 2B is a conceptual diagram illustrating an example of sub-blocks used in overlapped block motion compensation (OBMC). Specifically, Figure 2A depicts block 130, which can be a coding unit (CU), including prediction units 136 and 138. The diagonally shaded sub-blocks of prediction units 136, 138 correspond to sub-blocks for which OBMC can be applied. Motion information of neighboring sub-blocks 132A, 132B, and 132C can be used to perform OBMC for prediction unit 136, while motion information of neighboring sub-block 134 can be used to perform OBMC for prediction unit 138. Figure 2B depicts block 140, which can be a CU. Block 140 includes sub-block 142. Motion information of neighboring block 144 can be used to perform OBMC for sub-block 142.

[0070] The technique of "overlapped block motion compensation" (OBMC) has been used in various video coding standards, including ITU-T H.263. In Joint Exploration Motion (JEM), described in JVET-G1001, OBMC is turned on and off using syntax at the coding unit (CU) level, unlike H.263. When OBMC is used in JEM, OBMC is performed for all motion compensation (MC) block boundaries, except for the right and bottom boundaries of a CU. Furthermore, OBMC is applied to both the luma component and the chroma components. In JEM, a MC block corresponds to a coding block. When a CU is coded using sub-CU modes (including sub-CU merge, affine, and frame rate up conversion (FRUC) modes), each sub-block of the CU is a MC block. To handle CU boundaries in a uniform manner, OBMC is performed at the sub-block level for all MC block boundaries, with the sub-block size being set equal to 4x4, as illustrated in FIG. 2.

[0071] When OBMC is applied to the current subblock, in addition to the current motion vector, the motion vectors of four connected neighboring subblocks (if available and not identical to the current motion vector) are used to derive the prediction block of the current subblock. Such multiple prediction blocks based on multiple motion vectors are combined to produce the final prediction signal of the current subblock. For example, the motion vector of the neighboring block 144 can be used to derive the prediction block of the subblock 142 in Figure 2B

[0072] The prediction block based on the motion vector of the neighboring subblock can be denoted as P N where N indicates the index of the neighboring top, bottom, left, right subblock, and the prediction block based on the motion vector of the current subblock is denoted as P C When P N is based on the motion information of the neighboring subblock containing the same motion information as the current subblock, no OBMC is performed from P N Otherwise, each sample of P N is added to the same sample in P C , i.e., four columns / rows of P N are added to P C . The weighting factors {1 / 4, 1 / 8, 1 / 16, 1 / 32} are used for P N and the weighting factors {3 / 4, 7 / 8, 15 / 16, 31 / 32} are used for P C . The exception is the small MC block (i.e., when the height or width of the coded block is equal to 4 or the CU is coded using sub-CU mode), where only two columns / rows of P N are added to P C . In this case, the weighting factors {1 / 4, 1 / 8} are used for P N and the weighting factors {3 / 4, 7 / 8} are used for P C . For P N generated based on the motion vector of the vertical (horizontal) neighboring subblock, the samples in the same column (row) of P N are added to P C using the same weighting factors.

[0073] BCW (Bi-prediction with CU-level weights) is a mode that allows each CU to select a set of weights when performing the blending of bi-prediction. In ECM-9.0, an index (bcwldx) is used to represent the weights selected for the current CU, and a default value (BCW_DEFAULT = 2) of the BCW mode corresponding to the equal-weight blending case.

[0074] Figure 3 ​is a conceptual diagram illustrating subblock-based temporal motion vector prediction (SbTMVP) mode subblock template generation for the current block 150. In VVC, temporal motion vector prediction (TMVP) for the advanced motion vector prediction (AMVP) and merge modes is derived by taking motion information from the center or bottom-right of the collocated block in the collocated picture. Similarly, for the subblock-based temporal motion vector prediction (SbTMVP) mode, motion information from the left neighboring location is used as a motion shift, which is then adopted to obtain TMVP at the sub-CU level.

[0075] In ECM, to further improve the coding efficiency of TMVP, two aspects are modified. First, two collocated pictures are utilized, which are the two reference frames with the smallest POC distance with respect to the frame to be coded. Second, the motion shift to locate TMVP is adaptively determined from multiple locations according to the template cost. More specifically, two motion shift candidate lists are constructed for the two collocated frames, respectively. The motion shift with the smallest template matching cost is used to derive the SbTMVP or TMVP candidate. At most 4 SbTMVP candidates are included in the subblock-based merge list. The SbTMVP candidate with the smallest template matching cost derived from the first collocated frame is placed in the first entry without reordering, while the other SbTMVP candidates are reordered together with affine candidates. In addition, the prediction direction of each subblock template is determined based on the center subblock. As shown in Figure 3 if the center subblock is uni-predicted, all subblock templates are uni-predicted, and vice versa. If the motion vector of the corresponding neighboring subblock at the determined reference list is not available for the subblock template, a zero MV is used for that subblock template.

[0076] Figure 4 is a conceptual diagram illustrating geometric partition 160 for the geometric partition mode (GPM). In VVC, the geometric partition mode is supported for inter prediction. When this mode is used, a CU is split into two parts by a straight line positioned geometrically as shown in Figure 4 The position of the split line is mathematically derived from the angle and offset parameters of the particular partition. In VVC, there are 64 split modes. They are first ordered by angle (from smaller to larger) and second by offset (from smaller to larger), and each angle-offset setting is assigned an index binarized with a fixed-length code (i.e., 0 to 63), where each bin is bypass coded. The fixed-length code is a full-tree structure, with 6 bins at each leaf node.

[0077] Figure 5 is a table illustrating how the split mode index is mapped to the angle-offset. Figure 5The notation shows how the split mode index maps to the angular offset, where the Nth angular mode (i.e., N = 0,..., 7 or 16,..., 23) has an edge that is physically perpendicular to the edge of the (N+8)th angular mode. Each part of the geometric partition in a CU is inter predicted using its own motion; only uni-prediction is allowed for each partition, that is, each part has one motion vector and one reference index.

[0078] The uni-prediction candidate list of GPM modes is derived directly from the regular merge candidate list. Let n denote the index of the uni-prediction motion in the geometric uni-prediction candidate list. The LX motion vector of the nth merge candidate, where X equals the parity (even or odd) of n, is used as the nth uni-prediction motion vector for the geometric partition mode. Such motion vectors are marked with an "x" in the figures. In the absence of a corresponding LX motion vector of the nth extended merge candidate, the L(l-X) motion vector of the same candidate is used instead as the uni-prediction motion vector for the geometric partition mode.

[0079] Local illumination compensation (LIC) is an inter prediction technique to model the local illumination change between the current block and its prediction block as a function of the current block template and the reference block template. The parameters of the function can be represented by a scale a and an offset b, which form a linear equation, i.e., to compensate for the illumination change, where p[x] is the reference sample pointed to by the MV at position x on the reference picture. When wrapped motion compensation is enabled, the MV can be clipped considering the wrap-around offset. Since a and b can be derived based on the current block template and the reference block template, they do not require signaling overhead except for the LIC flag to be signaled for the AMVP mode to indicate the usage of LIC.

[0080] The local illumination compensation proposed in JVET-O0066 is used for uni-prediction inter CUs with the following modifications:

[0081] Intra neighboring samples can be used for LIC parameter derivation;

[0082] LIC is disabled for blocks with less than 32 luma samples;

[0083] For both non-subblock mode and affine mode, LIC parameter derivation is performed based on the template block samples corresponding to the current CU instead of the partial template block samples corresponding to the first top-left 16x16 unit;

[0084] The samples of the reference block template are generated by using MC and the block MV without rounding them to integer pixel precision.

[0085] In ECM-9.0, the value of a syntax element is unspecified when it is not used in the decoding process. For example, if the current CU is coded using uni-prediction (without the use of BCW), the value of bcwldx of the CU is unspecified. However, when another CU 2 coded after the current CU needs the information of the current CU for parsing / decoding, having a value that is unspecified can bring additional complexity for properly processing CU 2.

[0086] In ECM-9.0, the inter direction of a PU can be modified by the template matching process, which indicates whether uni-prediction (L0 or L1) or bi-prediction is applied, let us denote this direction parameter as interDir. For example, a bi-predicted PU can be modified to uni-prediction. However, the bi-prediction index, which indicates how the weights (using the parameter denoted as bcwldx) of the uni-predictors of the corresponding PU are weighted, is unchanged. This inconsistency can cause incorrect decisions for another block in the encoding / decoding process.

[0087] In ECM-9.0, the motion compensation of OBMC is performed on a subblock basis, and the motion similarity between the current subblock and the neighboring ones is used in the OMBC process. When comparing the MV of the current subblock with the neighboring ones, the subblock level information is used. When comparing bcwldx with the neighboring ones, the PU level interDir and the CU level bcwldx are compared. However, using the PU / CU level information can encounter problems because the subblock level and the PU / CU level information can not always be the same. As an example, in the context of subblock-based temporal motion vector prediction (SbTMVP) and geometric partition mode (GPM), each subblock can have different interDir and / or bcwldx. In this case, using the PU / CU level interDir and / or bcwldx can not be accurate.

[0088] In the OBMC design of ECM-9.0, the motion similarity based OBMC control is only performed for the boundary subblocks. When using subblock-based motion compensation, the motion similarity (as well as other information related to intra / inter prediction, e.g., LIC) is not considered. Ignoring this information can be suboptimal.

[0089] The present disclosure describes various techniques that can overcome various problems related to OBMC as discussed above.

[0090] In some examples, Figure 1Video encoder 200 and video decoder 300 can be configured to use a preset value of the syntax element when the selected syntax element(s) related to the current block is not used for the current block but can be referenced by another block. As an example, for coding units / prediction units / sub-blocks using single prediction (interDir!= 3), a preset value of bcwldx can be assigned even though bcwldx can not be used by the current block because it can be coded in single prediction mode. The preset value can be any predefined value. When this example applies to ECM-9.0, a preset value BCW_DEFAULT that can be derived as (BCW_NUM » 1) = 2 can be assigned. In this example, BCW_NUM is the number of weight options and BCW_DEFAULT corresponds to equal weight mixing.

[0091] In some examples, video encoder 200 and video decoder 300 can be configured such that for each entry (motion information, also referred to as "MI") in the motion buffer, a preset value of bcwldx can be assigned if the MI uses single prediction (interDir!= 3) even though bcwldx is not used by the current block because it can be coded by single prediction mode. The preset value can be any predefined value. When this example applies to ECM-9.0, a preset value BCW_DEFAULT that can be derived as (BCW_NUM » 1) = 2 is assigned, where BCW_NUM is the number of weight options and BCW_DEFAULT corresponds to equal weight mixing.

[0092] The difference between assigning values for coding / prediction units and motion information buffer is that in the first case, the value is assigned to the whole block, while in the second case, the value is assigned at the motion information granularity level, which can be, for example, 4x4 sub-blocks. This means that each sub-block can have a different assigned value.

[0093] As yet another example, in the case of SbTMVP, a preset value (e.g., BCW_DEFAULT value) can be assigned to the motion information at the sub-block level and / or CU / PU level.

[0094] In some examples, video encoder 200 and video decoder 300 can be configured to use the information of the sub-blocks associated with the neighboring locations (instead of using the PU / CU level information) when certain types of information from the neighboring locations are needed. For example, in the case of OBMC, when checking the similarity between the current block / sub-block and the neighboring locations, the interDir information of the current block / sub-block and the sub-blocks associated with the neighboring locations can be used.

[0095] As another example, in the case of OBMC, when checking the similarity between the MI of the current block / subblock and the neighboring location, the motion parameters (e.g., bcwldx information) of the current block / subblock and the subblock associated with the neighboring location are used, instead of the motion parameters (e.g., bcwldx) of the current CU / PU and the neighboring CU / PU associated with the neighboring location.

[0096] Other examples of scenarios using subblock-level motion information instead of PU / CU-level motion information include the boundary strength calculation for the deblocking filter, or other deblocking filter parameters (e.g., filter width, filter strength, etc.).

[0097] Other examples include using any motion information from the neighboring location. When the current block / subblock requires a motion vector, a block vector, a reference picture, LIC information (LIC flag, LIC scaling factor, LIC offset), or other motion information associated with the neighboring location, the corresponding information can alternatively be obtained from the subblock associated with the neighboring location, instead of from the PU / CU associated with the neighboring location.

[0098] In general, the similarity check can be performed by checking some motion parameters (not necessarily all available) that are defined in some way as important parameters.

[0099] Figure 6 FIG. 1 is a block diagram illustrating an example video encoder 200 that can perform the techniques of this disclosure. Figure 6 FIG. 1 is a block diagram illustrating an example video encoder 200 that can perform the techniques of this disclosure.

[0100] In Figure 6In the example of FIG. 2, video encoder 200 includes video data memory 230, mode select unit 202, residual generation unit 204, transform processing unit 206, quantization unit 208, inverse quantization unit 210, inverse transform processing unit 212, reconstruction unit 214, filter unit 216, decoded picture buffer (DPB) 218, and entropy encoding unit 220. Any or all of video data memory 230, mode select unit 202, residual generation unit 204, transform processing unit 206, quantization unit 208, inverse quantization unit 210, inverse transform processing unit 212, reconstruction unit 214, filter unit 216, DPB 218, and entropy encoding unit 220 can be implemented in one or more processors or in processing circuitry. For instance, the units of video encoder 200 can be implemented as part of a hardware circuit, as part of a software program, or as a combination of a hardware circuit and a software program. Additionally, video encoder 200 can include additional or alternative processors or processing circuitry to perform these and other functions.

[0101] Video data memory 230 can store video data to be encoded by the components of video encoder 200. Video encoder 200 can receive the video data stored in video data memory 230 from, for example, video source 104 Figure 1 DPB 218 can function as a reference picture memory that stores reference video data for use in encoding future video data by video encoder 200. Video data memory 230 and DPB 218 can be formed by any of a variety of memory devices, such as dynamic random access memory (DRAM), including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. Video data memory 230 and DPB 218 can be provided by the same memory device or separate memory devices. In various examples, video data memory 230 can be on-chip with other components of video encoder 200, as illustrated, or off-chip relative to those components.

[0102] In this disclosure, reference to video data memory 230 should not be interpreted as being limited to memory internal to video encoder 200 (unless specifically described as such) or memory external to video encoder 200 (unless specifically described as such). Rather, reference to video data memory 230 should be understood as a reference memory that stores video data that video encoder 200 receives for encoding (e.g., video data for a current block to be encoded). Figure 1 Memory 106 of source device 102 can also provide temporary storage of outputs from the various units of video encoder 200.

[0103] FIG. 2 illustrates Figure 6The various units of video encoder 200 are shown to help understand the operations performed by video encoder 200. The units can be implemented as fixed- function circuits, programmable circuits, or a combination thereof. Fixed-function circuits refer to circuits that provide particular functionality, and are preset on the operations that can be performed. Programmable circuits refer to circuits that can be programmed to perform various tasks and provide flexible functionality over the operations that can be performed. For instance, programmable circuits can execute software or firmware that causes the programmable circuits to operate in the manner defined by the instructions of the software or firmware. Fixed-function circuits can execute software instructions (e.g., to receive parameters or output parameters), but the types of operations that the fixed-function circuits perform are generally immutable. In some examples, one or more of the units can be distinct circuit blocks (fixed-function or programmable), and in some examples, one or more of the units can be integrated circuits.

[0104] Video encoder 200 can include arithmetic logic units (ALUs), elementary function units (EFUs), digital circuits, analog circuits, and / or programmable cores formed from programmable circuits. In examples where the operations of video encoder 200 are performed using software executed by the programmable circuits, memory 106 Figure 1 ) can store the instructions (e.g., object code) of the software that video encoder 200 receives and executes, or another memory within video encoder 200 (not shown) can store such instructions.

[0105] Video data memory 230 is configured to store video data that is received for encoding. Video encoder 200 can retrieve pictures of the video data from video data memory 230 and provide the video data to residual generation unit 204 and mode selection unit 202. Video data in video data memory 230 can be raw video data that is awaiting encoding.

[0106] Mode selection unit 202 includes motion estimation unit 222, motion compensation unit 224, and intra-prediction unit 226. Mode selection unit 202 can include additional functional units that perform video prediction according to other prediction modes. As examples, mode selection unit 202 can include a palette unit, an intra block copy unit (which can be part of motion estimation unit 222 and / or motion compensation unit 224), an affine unit, a linear model (LM) unit, etc.

[0107] Mode selection unit 202 generally coordinates multiple encoding passes to test combinations of encoding parameters and resulting rate-distortion values for such combinations. The encoding parameters can include partitioning of CTUs into CUs, prediction modes for CUs, transform types for residual data of CUs, quantization parameters for residual data of CUs, etc. Mode selection unit 202 can ultimately select the combination of encoding parameters that has the best rate-distortion value compared to other tested combinations.

[0108] Video encoder 200 can partition a picture retrieved from video data memory 230 into a series of CTUs, and encapsulate one or more CTUs within a slice. Mode select unit 202 can partition CTUs of a picture according to a tree structure such as the MTT structure, the QTBT structure, the superblock structure, or the quadtree structure described above. As described above, video encoder 200 can form one or more CUs by partitioning a CTU according to the tree structure. Such CUs can also be referred to as “video blocks” or “blocks” generally.

[0109] In general, mode select unit 202 also controls its components (e.g., motion estimation unit 222, motion compensation unit 224, and intra-prediction unit 226) to generate a prediction block for a current block (e.g., a current CU, or in HEVC, an overlapping portion of a PU and a TU). To perform inter-prediction for the current block, motion estimation unit 222 can perform a motion search to identify one or more closely matching reference blocks in one or more reference pictures (e.g., one or more previously coded pictures stored in DPB 218). Specifically, motion estimation unit 222 can calculate a value that represents how similar a potential reference block would be to the current block, e.g., according to a sum of absolute difference (SAD), a sum of squared difference (SSD), a mean absolute difference (MAD), a mean squared difference (MSD), etc. Motion estimation unit 222 can generally perform these calculations using a sample-by-sample difference between the current block and the reference block under consideration. Motion estimation unit 222 can identify the reference block with the lowest value resulting from these calculations, to indicate the reference block that is most matching to the current block.

[0110] Motion estimation unit 222 can form one or more motion vectors (MVs) that define the location of a reference block in a reference picture relative to the location of the current block in the current picture. Motion estimation unit 222 can then provide the motion vector(s) to motion compensation unit 224. For example, motion estimation unit 222 can provide a single motion vector for single -prediction inter-prediction, and two motion vectors for bi-prediction inter-prediction. Motion compensation unit 224 can then generate the prediction block using the motion vector(s). For example, motion compensation unit 224 can use the motion vector(s) to retrieve data for the reference block. As another example, where the motion vector(s) have fractional sample precision, motion compensation unit 224 can interpolate values for the prediction block according to one or more interpolation filters. Further, for bi-prediction, motion compensation unit 224 can retrieve data for two reference blocks identified by the respective motion vectors and combine the retrieved data, e.g., by a sample-wise average or weighted average.

[0111] Motion compensation unit 224 can use sub-block modes to predict a current block, such as bi-prediction with CU-level weights (BCW). In cases where a mode using corresponding syntax elements of motion information are coded for a previously coded neighboring block do not have an assigned value, such as a bcwldx syntax element, motion compensation unit 224 can determine a substitute value for the bcwldx syntax element. For example, motion compensation unit 224 can determine the substitute value to be equal to a BCW DEFAULT value, which can be equal to BCW_NUM » 1, where "»" represents a bitwise right-shift operation and BCW_NUM represents the number of weight options.

[0112] When operating according to the AV1 video coding format, motion estimation unit 222 and motion compensation unit 224 can be configured to use translational motion compensation, affine motion compensation, overlapped block motion compensation (OBMC), and / or compound inter-intra prediction to encode coding blocks (e.g., both luma coding blocks and chroma coding blocks) of video data.

[0113] As another example, for intra prediction or intra prediction coding, intra prediction unit 226 can generate a prediction block from neighboring samples of the current block. For example, for directional modes, intra prediction unit 226 can generally mathematically combine values of neighboring samples and fill these computed values across the current block in a defined direction to produce the prediction block. As another example, for a DC mode, intra prediction unit 226 can compute an average of neighboring samples of the current block and generate the prediction block to include that resulting average for each sample of the prediction block.

[0114] When operating according to the AV1 video coding format, intra prediction unit 226 can be configured to use directional intra prediction, non-directional intra prediction, recursive filter intra prediction, chroma from luma (CFL) prediction, intra block copy (IBC), and / or palette mode to encode coding blocks (e.g., both luma coding blocks and chroma coding blocks) of video data. Mode selection unit 202 can include additional functional units that perform video prediction according to other prediction modes.

[0115] Mode selection unit 202 provides the prediction block to residual generation unit 204. Residual generation unit 204 receives the original uncoded version of the current block from video data memory 230 and the prediction block from mode selection unit 202. Residual generation unit 204 computes the sample-by-sample difference between the current block and the prediction block. The resulting sample-by-sample difference defines a residual block for the current block. In some examples, residual generation unit 204 can also determine differences between sample values in the residual block to generate the residual block using residual differential pulse code modulation (RDPCM). In some examples, residual generation unit 204 can be formed using one or more subtractor circuits that perform binary subtraction.

[0116] In examples in which the mode selection unit 202 partitions a CU into PUs, each PU can be associated with a luma prediction unit and corresponding chroma prediction units. Video encoder 200 and video decoder 300 can support PUs having various sizes. As noted above, the size of a CU can refer to the size of the CU's luma coding block, while the size of a PU can refer to the size of the PU's luma prediction unit. Assuming that a particular CU has a size of 2Nx2N, video encoder 200 can support PU sizes of 2Nx2N or NxN for intra-prediction, and 2Nx2N, 2NxN, Nx2N, NxN, or similar symmetric PU sizes for inter-prediction. Video encoder 200 and video decoder 300 can also support asymmetric partitioning for PU sizes of 2NxnU, 2NxnD, nLx2N, and nRx2N for inter-prediction.

[0117] In examples in which the mode selection unit 202 does not further partition a CU into PUs, each CU can be associated with a luma coding block and corresponding chroma coding blocks. As above, the size of a CU can refer to the size of the CU's luma coding block. Video encoder 200 and video decoder 300 can support CU sizes of 2Nx2N, 2NxN, or Nx2N.

[0118] For other video coding techniques, such as intra block copy mode coding, affine mode coding, and linear model (LM) mode coding, as some examples, the mode selection unit 202 generates, via a respective unit associated with the coding technique, a prediction block for the current block being encoded. In some examples, such as palette mode coding, the mode selection unit 202 can not generate a prediction block, but instead generate syntax elements indicative of the manner in which the block is to be reconstructed based on a selected palette. In such modes, the mode selection unit 202 can provide these syntax elements to the entropy encoding unit 220 for encoding.

[0119] As described above, the residual generation unit 204 receives video data for a current block and a corresponding prediction block. The residual generation unit 204 then generates a residual block for the current block. To generate the residual block, the residual generation unit 204 computes the sample-by-sample difference between the prediction block and the current block.

[0120] Transform processing unit 206 applies one or more transforms to the residual block to produce a block of transform coefficients (referred to herein as a "transform coefficient block"). Transform processing unit 206 can apply various transforms to the residual block to form the transform coefficient block. For example, transform processing unit 206 can apply a discrete cosine transform (DCT), a directional transform, a Karhunen-Loeve transform (KLT), or a conceptually similar transform. In some examples, transform processing unit 206 can perform multiple transforms on the residual block, e.g., a primary transform and a secondary transform such as a rotation transform. In some examples, transform processing unit 206 does not apply a transform to the residual block.

[0121] When operating according to AV1, transform processing unit 206 can apply one or more transforms to the residual block to produce a block of transform coefficients (referred to herein as a "transform coefficient block"). Transform processing unit 206 can apply various transforms to the residual block to form the transform coefficient block. For example, transform processing unit 206 can apply a horizontal / vertical transform combination, which can include a discrete cosine transform (DCT), an asymmetric discrete sine transform (ADST), a flipped ADST (e.g., ADST in reverse order), and an identity transform (IDTX). When the identity transform is used, the transform is skipped in one of the vertical or horizontal directions. In some examples, transform processing can be skipped.

[0122] Quantization unit 208 can quantize the transform coefficients in the transform coefficient block to produce a quantized transform coefficient block. Quantization unit 208 can quantize the transform coefficients of the transform coefficient block according to a quantization parameter (QP) value associated with the current block. Video encoder 200 (e.g., via mode select unit 202) can adjust the degree of quantization applied to the transform coefficient block associated with the current block by adjusting the QP value associated with the CU. Quantization can introduce loss of information, and thus, quantized transform coefficients can have lower precision than the original transform coefficients produced by transform processing unit 206.

[0123] Inverse quantization unit 210 and inverse transform processing unit 212 can apply inverse quantization and inverse transforms, respectively, to the quantized transform coefficient block to reconstruct the residual block from the transform coefficient block. Reconstruction unit 214 can produce a reconstructed block corresponding to the current block (although potentially with some degree of distortion) based on the reconstructed residual block and the prediction block generated by mode select unit 202. For example, reconstruction unit 214 can add samples of the reconstructed residual block to corresponding samples from the prediction block generated by mode select unit 202 to produce the reconstructed block.

[0124] Filter unit 216 can perform one or more filtering operations on the reconstructed block. For example, filter unit 216 can perform a deblocking operation to reduce blocking artifacts along the edges of the CU. In some examples, the operations of filter unit 216 can be skipped.

[0125] When operating according to AV1, filter unit 216 can perform one or more filtering operations on the reconstructed block. For example, filter unit 216 can perform a deblocking operation to reduce blocking artifacts along the edges of the CU. In other examples, filter unit 216 can apply a constrained direction enhancement filter (CDEF), which can be applied after deblocking and can include the application of a non-separable, non-linear, low-pass directional filter based on an estimated edge direction. Filter unit 216 can also include a loop restoration filter applied after CDEF and can include a separable, symmetric, normalized Wiener filter or a double- self-guided filter.

[0126] Video encoder 200 stores the reconstructed block in DPB 218. For example, in examples in which the operations of filter unit 216 are not performed, reconstructed unit 214 can store the reconstructed block to DPB 218. In examples in which the operations of filter unit 216 are performed, filter unit 216 can store the filtered reconstructed block to DPB 218. Motion estimation unit 222 and motion compensation unit 224 can retrieve reference pictures formed from reconstructed (and potentially filtered) blocks from DPB 218 to inter-predict blocks of subsequent coded pictures. In addition, intra-prediction unit 226 can use reconstructed blocks of the current picture in DPB 218 to intra-predict other blocks in the current picture.

[0127] In general, entropy encoding unit 220 can entropy encode syntax elements received from other functional components of video encoder 200. For example, entropy encoding unit 220 can entropy encode quantized transform coefficient blocks from quantization unit 208. As another example, entropy encoding unit 220 can entropy encode prediction syntax elements (e.g., motion information for inter-prediction or intra-mode information for intra-prediction) from mode select unit 202. Entropy encoding unit 220 can perform one or more entropy encoding operations on the syntax elements, which are another example of video data, to generate entropy encoded data. For example, entropy encoding unit 220 can perform a context- adaptive variable length coding (CAVLC) operation, a CABAC operation, a variable- to-variable (V2V) length coding operation, a syntax-based context-adaptive binary arithmetic coding (SBAC) operation, a Probability Interval Partitioning Entropy (PIPE) coding operation, an Exponential-Golomb coding operation, or another type of entropy encoding operation. In some examples, entropy encoding unit 220 can operate in a bypass mode in which syntax elements are not entropy encoded.

[0128] Video encoder 200 can output a bitstream that includes the entropy encoded syntax elements needed to reconstruct blocks of a slice or picture. In particular, entropy encoding unit 220 can output the bitstream.

[0129] According to AV1, entropy encoding unit 220 can be configured as a symbol-to-symbol adaptive multi-symbol arithmetic coder. Syntax elements in AV1 include an alphabet of N elements, and a context (e.g., a probability model) includes a set of N probabilities. Entropy encoding unit 220 can store the probabilities as an n-bit (e.g., 15-bit) cumulative distribution function (CDF). Entropy encoding unit 220 can perform recursive scaling using an update factor based on the alphabet size to update the context.

[0130] The operations described above are described with respect to a block. Such description should be understood as operations for a luma coding block and / or a chroma coding block. As described above, in some examples, the luma coding block and the chroma coding block are luma and chroma components of a CU. In some examples, the luma coding block and the chroma coding block are luma and chroma components of a PU.

[0131] In some examples, operations performed with respect to a luma coding block do not need to be repeated for a chroma coding block. As one example, operations to identify a motion vector (MV) and a reference picture for a luma coding block do not need to be repeated for identifying an MV and a reference picture for a chroma block. Instead, the MV for the luma coding block can be scaled to determine the MV for the chroma block, and the reference picture can be the same. As another example, an intra prediction process can be the same for a luma coding block and a chroma coding block.

[0132] Figure 7 FIG. 3 is a block diagram illustrating an example video decoder 300 that can perform the techniques of this disclosure. Figure 7 FIG. 3 is provided for purposes of explanation and is not limiting on the techniques of this disclosure, as broadly exemplified and described herein. For purposes of explanation, this disclosure describes video decoder 300 according to the techniques of VVC (ITU-T H.266, under development) and HEVC (ITU-T H.265). However, the techniques of this disclosure can be performed by video coding devices configured to other video coding standards.

[0133] In Figure 7In the example of FIG. 3, video decoder 300 includes coded picture buffer (CPB) memory 320, entropy decoding unit 302, prediction processing unit 304, inverse quantization unit 306, inverse transform processing unit 308, reconstruction unit 310, filter unit 312, and decoded picture buffer (DPB) 314. Any or all of CPB memory 320, entropy decoding unit 302, prediction processing unit 304, inverse quantization unit 306, inverse transform processing unit 308, reconstruction unit 310, filter unit 312, and DPB 314 can be implemented in one or more processors or in processing circuitry. For instance, the units of video decoder 300 can be implemented as part of a hardware circuit, as part of a processor, as part of an ASIC or FPGA, or as part of a software program running on a processor, ASIC, FPGA, or other hardware. Also, video decoder 300 can include additional or alternative processors or processing circuitry to perform these and other functions.

[0134] Prediction processing unit 304 includes motion compensation unit 316 and intra-prediction unit 318. Prediction processing unit 304 can include additional units to perform prediction according to other prediction modes. As examples, prediction processing unit 304 can include a palette unit, an intra-block copy unit (which can form part of motion compensation unit 316), an affine unit, a linear model (LM) unit, etc. In other examples, video decoder 300 can include more, fewer, or different functional components.

[0135] Motion compensation unit 316 can use sub-block modes to predict a current block, such as bi-prediction with CU-level weights (BCW). In cases where a previously coded neighboring block is coded in a mode that does not have a corresponding syntax element for motion information assigned a value, such as a bcwldx syntax element, motion compensation unit 316 can determine a substitute value for the bcwldx syntax element. For example, motion compensation unit 316 can determine the substitute value to be equal to a BCW DEFAULT value, which can be equal to BCW_NUM » 1, where “>>” denotes a bitwise right-shift operation and BCW_NUM denotes the number of weight options.

[0136] As described above, when operating according to AV1, compensation unit 316 can be configured to decode coding blocks (e.g., both luma coding blocks and chroma coding blocks) of video data using translational motion compensation, affine motion compensation, OBMC, and / or compound inter-intra prediction. As described above, intra-prediction unit 318 can be configured to decode coding blocks (e.g., both luma coding blocks and chroma coding blocks) of video data using directional intra-prediction, non-directional intra-prediction, recursive filter intra-prediction, CFL, intra-block copy (IBC), and / or color palette mode.

[0137] CPB memory 320 can store video data, such as encoded video bitstreams, to be decoded by the components of video decoder 300. The video data stored in CPB memory 320 can be obtained, for example, from computer-readable medium 110 Figure 1 ). CPB memory 320 can include a CPB that stores encoded video data (e.g., syntax elements) from an encoded video bitstream. Also, CPB memory 320 can store video data other than syntax elements of a coded picture, such as temporary data representing outputs from the various units of video decoder 300. DPB 314 generally stores decoded pictures, which video decoder 300 can output and / or use as reference video data when decoding subsequent data or pictures of the encoded video bitstream. CPB memory 320 and DPB 314 can be formed by any of a variety of memory devices, such as dynamic random access memory (DRAM) including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. CPB memory 320 and DPB 314 can be provided by the same memory devices or separate memory devices. In various examples, CPB memory 320 can be on-chip with other components of video decoder 300 or off-chip relative to those components.

[0138] Additionally or alternatively, in some examples, video decoder 300 can retrieve coded video data from memory 120 Figure 1 ). That is, memory 120 can utilize CPB memory 320 to store data as discussed above. Likewise, when some or all functions of video decoder 300 are implemented in software executed by the processing circuitry of video decoder 300, memory 120 can store the instructions to be executed by video decoder 300.

[0139] The various units shown in FIG. 3 Figure 7 are illustrated to help understand the operations performed by video decoder 300. The units can be implemented as fixed- function circuits, programmable circuits, or a combination thereof. Similar to FIG. 2, FIG. 3 is a high-level architecture diagram of a video decoder. Figure 6Fixed function circuitry refers to circuitry that provides specific functionality and is preset for operations that can be performed. Programmable circuitry refers to circuitry that can be programmed to perform various tasks and provide flexible functionality in terms of operations that can be performed. For example, programmable circuitry can execute software or firmware that causes the programmable circuitry to operate in a manner defined by instructions of the software or firmware. Fixed function circuitry can execute software instructions (e.g., to receive parameters or output parameters), but the type of operations that the fixed function circuitry performs is generally immutable. In some examples, one or more of the units can be distinct circuit blocks (fixed function or programmable), and in some examples, one or more of the units can be integrated circuitry.

[0140] Video decoder 300 can include ALUs, EFUs, digital circuits, analog circuits, and / or programmable cores formed from programmable circuitry. In examples where the operations of video decoder 300 are performed by software executing on the programmable circuitry, on-chip or off-chip memory can store instructions of the software that video decoder 300 receives and executes.

[0141] Entropy decoding unit 302 can receive encoded video data from the CPB and entropy decode the video data to reconstruct syntax elements. Prediction processing unit 304, inverse quantization unit 306, inverse transform processing unit 308, reconstruction unit 310, and filter unit 312 can generate decoded video data based on the syntax elements extracted from the bitstream.

[0142] In general, video decoder 300 reconstructs a picture on a block-by-block basis. Video decoder 300 can perform reconstruction operations individually for each block (where the block that is currently being reconstructed (i.e., decoded) can be referred to as the “current block”).

[0143] Entropy decoding unit 302 can entropy decode syntax elements defining quantized transform coefficient blocks, as well as transform information such as quantization parameters (QPs) and / or transform mode indications. Inverse quantization unit 306 can use a QP associated with a quantized transform coefficient block to determine a degree of quantization and, likewise, a degree of inverse quantization for inverse quantization unit 306 to apply. Inverse quantization unit 306 may, for example, perform a bitwise left-shift operation to inverse quantize a quantized transform coefficient. Inverse quantization unit 306 may

[0144] After inverse quantization unit 306 forms a transform coefficient block, inverse transform processing unit 308 can apply one or more inverse transforms to the transform coefficient block to generate a residual block associated with the current block. For example, inverse transform processing unit 308 can apply an inverse DCT, an inverse integer transform, an inverse Karhunen-Loeve transform (KLT), an inverse rotational transform, an inverse directional transform, or another inverse transform to the transform coefficient block.

[0145] Furthermore, prediction processing unit 304 generates the prediction block from the prediction information syntax elements entropy decoded by entropy decoding unit 302. For example, in the case that the prediction information syntax elements indicate that the current block is inter predicted, motion compensation unit 316 can generate the prediction block. In this case, the prediction information syntax elements can indicate a reference picture in DPB 314 from which to retrieve a reference block, and a motion vector that identifies a location of the reference block in the reference picture relative to a location of the current block in the current picture. Motion compensation unit 316 can generally perform the inter prediction process in a manner substantially similar to that described with respect to motion compensation unit 224 Figure 6 ) described above.

[0146] As another example, in the case that the prediction information syntax elements indicate that the current block is intra predicted, intra prediction unit 318 can generate the prediction block according to an intra prediction mode indicated by the prediction information syntax elements. Again, intra prediction unit 318 can generally perform the intra prediction process in a manner substantially similar to that described with respect to intra prediction unit 226 Figure 6 ) described above. Intra prediction unit 318 can retrieve data for neighboring samples of the current block from DPB 314.

[0147] Reconstruction unit 310 can reconstruct the current block using the prediction block and the residual block. For example, reconstruction unit 310 can add samples of the residual block to corresponding samples of the prediction block to reconstruct the current block.

[0148] Filter unit 312 can perform one or more filtering operations on the reconstructed block. For example, filter unit 312 can perform a deblocking operation to reduce blocking artifact pseudo-ghosts along edges of the reconstructed block. The operations of filter unit 312 are not necessarily performed in all examples.

[0149] Video decoder 300 can store the reconstructed blocks in DPB 314. For example, in examples in which the operations of filter unit 312 are not performed, reconstructed unit 310 can store the reconstructed blocks to DPB 314. In examples in which the operations of filter unit 312 are performed, filter unit 312 can store the filtered reconstructed blocks to DPB 314. As discussed above, DPB 314 can provide reference information, such as samples of a current picture for intra prediction and previously decoded pictures for subsequent motion compensation, to prediction processing unit 304. In addition, video decoder 300 can output decoded pictures (e.g., decoded video) from DPB 314 for subsequent presentation on a display device, such as display device 118. Figure 1

[0150] Figure 8 is a flowchart illustrating an example method for encoding a current block in accordance with the techniques of this disclosure. The current block can include a current CU. Although described with respect to video encoder 200 Figure 1 and Figure 6 ), it should be understood that other devices can be configured to perform methods similar to the methods of Figure 8

[0151] In this example, video encoder 200 initially predicts the current block (350). For example, video encoder 200 can form a prediction block for the current block. When predicting the current block, video encoder 200 can use any of the various techniques of this disclosure, alone or in any combination. For example, when coding the current block, video encoder 200 can use a replacement value (such as a pre-defined default value or a value taken from a neighboring block) to replace values of unavailable syntax elements of a previously coded block. Video encoder 200 can then calculate a residual block for the current block (352). To calculate the residual block, video encoder 200 can calculate the difference between the original uncoded block of the current block and the prediction block. Video encoder 200 can then transform the residual block and quantize transform coefficients of the residual block (354). Next, video encoder 200 can scan the quantized transform coefficients of the residual block (356). During or after the scan, video encoder 200 can entropy encode the transform coefficients (358). For example, video encoder 200 can encode the transform coefficients using CAVLC or CABAC. Video encoder 200 can then output the entropy encoded data of the block (360).

[0152] ​​Video encoder 200 can also decode the current block after encoding the current block to use the decoded version of the current block as reference data for subsequent coded data (e.g., in inter or intra prediction modes). Thus, video encoder 200 can inverse quantize and inverse transform the coefficients to reproduce a residual block (362). Video encoder 200 can combine the residual block with a prediction block to form a decoded block (364). Video encoder 200 can then store the decoded block in DPB 218 (366).

[0153] Figure 9 is a flowchart illustrating an example method for decoding a current block of video data in accordance with the techniques of this disclosure. The current block can include a current CU. Although described with respect to video decoder 300 Figure 1 and Figure 7 ) it should be understood that other devices can be configured to perform methods similar to the method of Figure 9

[0154] Video decoder 300 can receive entropy encoded data for the current block, such as entropy encoded prediction information and entropy encoded data for transform coefficients of a residual block corresponding to the current block (370). Video decoder 300 can entropy decode the entropy encoded data to determine prediction information for the current block and reproduce transform coefficients of the residual block (372). Video decoder 300 can predict the current block, e.g., using an intra prediction mode or an inter prediction mode as indicated by the prediction information for the current block (374), to calculate a prediction block for the current block. Video decoder 300 can use any of the various techniques of this disclosure, alone or in any combination, when predicting the current block. For example, video decoder 300 can use substitute values, such as pre-defined default values or values taken from neighboring blocks, to substitute for values of unavailable syntax elements of previously decoded blocks when coding the current block. Video decoder 300 can then inverse scan the reproduced transform coefficients to create a block of quantized transform coefficients (376). Video decoder 300 can then inverse quantize the transform coefficients and apply inverse transforms to the transform coefficients to produce a residual block (378). Video decoder 300 can finally decode the current block by combining the prediction block and the residual block (380).

[0155] Figure 10 is a flowchart illustrating an example method of coding a block of video data in accordance with the techniques of this disclosure. Figure 10 The method of Figure 10 may be performed, for example, by video encoder 200 or video decoder 300. For purposes of example, the method of

[0156] ​Initially, video decoder 300 can determine that a previously coded block of video data was coded using uni-prediction (400). Because the previously coded block of video data was coded using uni-prediction, one or more syntax elements related to bi-prediction can not have an assigned value in a motion buffer entry corresponding to the previously coded block (e.g., bcwldx).

[0157] Video decoder 300 can then determine that the current block is predicted using a bi-prediction mode, such as BCW (402). In addition, video decoder 300 can determine that the motion information for the current block is to be predicted from the motion information of the previously coded block (e.g., bcwldx of the previously coded block) (404). However, as mentioned above, in this example, the previously coded block is uni-predicted, such that the bcwldx syntax element does not have an assigned value. In response, according to the techniques of this disclosure, video decoder 300 can replace the unavailable value of the motion information with a substitute value (406). Thus, video decoder 300 can use the substitute value to form a prediction block for the current block (408). Video decoder 300 can then use the prediction block to decode the current block (410).

[0158] In this way, Figure 10 The method of Example 1 represents an example of a method of decoding video data, the method comprising: determining that a previously coded block of video data was coded using a first coding mode that did not assign a value to a reference syntax element; determining that a current block of video data is to be coded using a second coding mode that references the reference syntax element; and in response to the reference syntax element not having an assigned value, decoding the current block using a substitute value for the reference syntax element.

[0159] Various examples of the techniques of this disclosure are summarized in the following clauses:

[0160] Clause 1 : A method of decoding video data, the method comprising: determining that a previously coded block of video data was coded using a first coding mode that did not assign a value to a reference syntax element; determining that a current block of the video data is to be coded using a second coding mode that references the reference syntax element; and decoding the current block using a substitute value for the reference syntax element.

[0161] Clause 2: The method of clause 1, wherein the substitute value comprises a pre-defined value.

[0162] Clause 3: The method of clause 2, wherein the first coding mode includes uni-directional inter prediction, and wherein the reference syntax element includes a bi-prediction (bcwldx) syntax element with coding unit (CU) level weight index.

[0163] Clause 4: The method of clause 3, wherein the pre-defined value includes a pre-set value BCW DEFAULT derived as (BCW_NUM » 1) = 2, where BCW_NUM is the number of weight options and BCW DEFAULT corresponds to equal weight mixing.

[0164] Clause 5: The method of clause 1, wherein the substitute value includes a value of a corresponding syntax element of a neighboring block of the current block.

[0165] Clause 6: The method of clause 5, wherein the corresponding syntax element includes an inter prediction direction (interDir) syntax element.

[0166] Clause 7: The method of clause 5, wherein the corresponding syntax element includes a bi-prediction (bcwldx) syntax element with coding unit (CU) level weight index.

[0167] Clause 8: The method of clause 5, wherein the corresponding syntax element includes a boundary strength syntax element.

[0168] Clause 9: The method of any of clauses 1-8, the method further comprising encoding the current block prior to decoding the current block.

[0169] Clause 10: A method comprising any of the techniques of this disclosure, alone or in any combination.

[0170] Clause 11: A device for decoding video data, the device comprising one or more means for performing the method of any of clauses 1-10.

[0171] Clause 12: The device of clause 11, wherein the one or more means comprise a processing system including one or more processors implemented in circuitry.

[0172] Clause 13: The device of any of clauses 11-12, the device further comprising a display configured to display the decoded video data.

[0173] Clause 14: The device of any of clauses 11-13, wherein the device comprises one or more of a camera, a computer, a mobile device, a broadcast receiver device, or a set-top box.

[0174] Clause 15: The device of clauses 11-14, further comprising a memory configured to store the video data.

[0175] Clause 16: A computer-readable storage medium having stored thereon instructions that, when executed, cause a processor of a device for decoding video data to perform the method of any of clauses 1-10.

[0176] Clause 17: A device for decoding video data, the device comprising: means for determining that a previously coded block of video data was coded using a first coding mode for which a reference syntax element was not assigned a value; means for determining that a current block of the video data is to be coded using a second coding mode that references the reference syntax element; and means for coding the current block using a substitute value for the reference syntax element.

[0177] Clause 18: A method of decoding video data, the method comprising: determining that a previously coded block of video data was coded using a uni-prediction mode for which a bi-prediction syntax element was not assigned a value; determining that a current block of the video data is to be coded using a bi-prediction mode and that motion information for the current block is to be predicted from the previously coded block, including from the bi-prediction syntax element of the previously coded block; and in response to the bi-prediction syntax element of the previously coded block not having an assigned value, decoding the current block using a substitute value for the bi-prediction syntax element.

[0178] Clause 19: The method of clause 18, wherein the bi-prediction syntax element comprises a bi-prediction (bcwldx) syntax element having a coding unit (CU) level weight index, wherein the substitute value comprises a preset value (BCW DEFAULT), the method further comprising deriving BCW DEFAULT = (BCW NUM » 1), where BCW NUM is a number of weight options.

[0179] Clause 20: The method of clause 19, wherein BCW NUM is equal to 1.

[0180] Clause 21: The method of clause 19, wherein BCW DEFAULT corresponds to an equal-weighted blending of the BCW mode.

[0181] Clause 22: The method of clause 18, wherein the current block comprises 4x4 sub-blocks of prediction units (PUs).

[0182] Clause 23: The method of clause 22, wherein the 4x4 subblock comprises a first 4x4 subblock of the PU, the method further comprising determining a value of a bi-prediction syntax element of a second, different 4x4 subblock of the PU that is different from the substitution value.

[0183] Clause 24: The method of clause 18, the method further comprising determining the substitution value from a value of a corresponding bi-prediction syntax element of a neighboring block of the current block.

[0184] Clause 25: The method of clause 24, wherein the corresponding bi-prediction syntax element comprises an inter-prediction direction (interDir) syntax element.

[0185] Clause 26: The method of clause 24, wherein the corresponding bi-prediction syntax element comprises a syntax element used to determine a boundary strength of a deblocking filter.

[0186] Clause 27: The method of clause 18, the method further comprising encoding the current block prior to decoding the current block.

[0187] Clause 28: A device for decoding video data, the device comprising: a memory configured to store video data; and a processing system comprising one or more processors implemented in circuitry, the processing system configured to: determine that a previously coded block of video data was coded using a uni-prediction mode for which a value was not assigned to a bi-prediction syntax element; determine that a current block of the video data is to be coded using a bi-prediction mode and that motion information of the current block is to be predicted from the previously coded block, including from the bi-prediction syntax element of the previously coded block; and responsive to the bi-prediction syntax element of the previously coded block not having an assigned value, decode the current block using a substitution value for the bi-prediction syntax element.

[0188] Clause 29: The device of clause 28, wherein the bi-prediction syntax element comprises a bi-prediction with coding unit (CU) level weight index (bcwldx) syntax element, wherein the substitution value comprises a preset value (BCW DEFAULT), and wherein the processing system is further configured to derive BCW DEFAULT = (BCW NUM » 1), where BCW NUM is a number of weight options.

[0189] Clause 30: The device of clause 28, wherein the current block comprises a 4x4 subblock of a prediction unit (PU).

[0190] Clause 31 : The device of clause 30, wherein the 4x4 sub-block comprises a first 4x4 sub-block of the PU, and wherein the processing system is further configured to determine a value of a different bi-prediction syntax element of a second different 4x4 sub-block of the PU that is different from the substitution value.

[0191] Clause 32: The device of clause 28, wherein the processing system is further configured to determine the substitution value from a value of a corresponding bi-prediction syntax element of a neighboring block of the current block.

[0192] Clause 33 : The device of clause 32, wherein the corresponding bi-prediction syntax element comprises an inter-prediction direction (interDir) syntax element.

[0193] Clause 34: The device of clause 32, wherein the corresponding bi-prediction syntax element comprises a syntax element used to determine a boundary strength of a deblocking filter.

[0194] Clause 35 : The device of clause 28, wherein the processing system is further configured to encode the current block prior to decoding the current block.

[0195] Clause 36: The device of clause 28, further comprising a display configured to display the decoded video data.

[0196] Clause 37 : The device of clause 28, wherein the device comprises one or more of a camera, a computer, a mobile device, a broadcast receiver device, or a set-top box.

[0197] Clause 38 : A method of decoding video data, the method comprising: determining that a previously coded block of video data was coded using a uni-prediction mode for which a value was not assigned to a bi-prediction syntax element; determining that a current block of the video data is to be coded using a bi-prediction mode, and that motion information for the current block is to be predicted from the previously coded block, including from the bi-prediction syntax element of the previously coded block; and in response to the bi-prediction syntax element of the previously coded block not having an assigned value, decoding the current block using a substitution value for the bi-prediction syntax element.

[0198] Clause 39 : The method of clause 38, wherein the bi-prediction syntax element comprises a bi-prediction with coding unit (CU) level weight index (bcwldx) syntax element, wherein the substitution value comprises a preset value (BCW DEFAULT), the method further comprising deriving BCW DEFAULT = (BCW NUM » 1), where BCW NUM is a number of weight options.

[0199] Clause 40: The method of clause 39, wherein BCW_NUM is equal to 1.

[0200] Clause 41 : The method of any of clauses 38 and 39, wherein BCW_DEFAULT corresponds to an equal-weighted mix of the BCW modes.

[0201] Clause 42: The method of any of clauses 38-41, wherein the current block comprises 4x4 sub-blocks of a prediction unit (PU).

[0202] Clause 43: The method of clause 42, wherein the 4x4 sub-blocks comprise a first 4x4 sub-block of the PU, the method further comprising determining a different value for a bi-prediction syntax element of a second, different 4x4 sub-block of the PU than the substitution value.

[0203] Clause 44: The method of clause 38, the method further comprising determining the substitution value from a value of a corresponding bi-prediction syntax element of a neighboring block of the current block.

[0204] Clause 45: The method of clause 24, wherein the corresponding bi-prediction syntax element comprises an inter-prediction direction (interDir) syntax element.

[0205] Clause 46: The method of clause 24, wherein the corresponding bi-prediction syntax element comprises a syntax element used to determine a boundary strength of a deblocking filter.

[0206] Clause 47: The method of any of clauses 38-46, the method further comprising encoding the current block prior to decoding the current block.

[0207] Clause 48: A device for decoding video data, the device comprising a memory configured to store video data, and a processing system comprising one or more processors implemented in circuitry, the processing system configured to: determine that a previously coded block of video data was coded using a uni-prediction mode for which a value was not assigned to a bi-prediction syntax element; determine that a current block of the video data is to be coded using a bi-prediction mode and that motion information for the current block is to be predicted from the previously coded block, including from the bi-prediction syntax element of the previously coded block; and responsive to the bi-prediction syntax element of the previously coded block not having an assigned value, decode the current block using a substitution value for the bi-prediction syntax element.

[0208] Clause 49: The device of clause 48, wherein the bi-prediction syntax element comprises a bi-prediction (bcwldx) syntax element with a coding unit (CU)-level weight index, wherein the substitution value comprises a preset value (BCW DEFAULT), and wherein the processing system is further configured to derive BCW DEFAULT = (BCW NUM » 1), where BCW NUM is a number of weight options.

[0209] Clause 50: The device of any of clauses 48 and 49, wherein the current block comprises a 4x4 sub-block of a prediction unit (PU).

[0210] Clause 51 : The device of clause 50, wherein the 4x4 sub-block comprises a first 4x4 sub-block of the PU, and wherein the processing system is further configured to determine a different value for a bi-prediction syntax element of a second, different 4x4 sub-block of the PU than the substitution value.

[0211] Clause 52: The device of clause 48, wherein the processing system is further configured to determine the substitution value from a value of a corresponding bi-prediction syntax element of a neighboring block of the current block.

[0212] Clause 53: The device of clause 52, wherein the corresponding bi-prediction syntax element comprises an inter-prediction direction (interDir) syntax element.

[0213] Clause 54: The device of clause 52, wherein the corresponding bi-prediction syntax element comprises a syntax element used to determine a boundary strength of a deblocking filter.

[0214] Clause 55: The method of any of clauses 48 to 54, wherein the processing system is further configured to encode the current block prior to decoding the current block.

[0215] Clause 56: The device of any of clauses 48 to 55, further comprising a display configured to display the decoded video data.

[0216] Clause 57: The device of any of clauses 48 to 56, wherein the device comprises one or more of a camera, a computer, a mobile device, a broadcast receiver device, or a set-top box.

[0217] It is to be recognized that depending on the example, certain acts or events of any of the techniques described herein can be performed in a different sequence, can be added, omitted, or can be partially or wholly combined (e.g., some acts or events from two or more techniques can be combined into a single technique), with other techniques disclosed herein, or with other techniques not specifically disclosed.

[0218] In one or more examples, the functions described can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions can be stored on or transmitted over as one or more instructions or code on a computer-readable medium and executed by a hardware-based processing unit. Computer-readable media can include computer-readable storage media, which corresponds to a tangible medium such as data storage media, or communication media including any medium that facilitates transfer of a computer program from one place to another, e.g., according to a communication protocol. In this manner, computer- readable media generally can correspond to (1) tangible computer-readable storage media which is non-transitory or (2) a communication medium such as a signal or carrier wave. Data storage media can be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementation of the techniques described in this disclosure. A computer program product can include a computer-readable medium.

[0219] By way of example, and not limitation, such computer-readable storage media can include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage, or other magnetic storage devices, flash memory, or any other storage medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Also, any

[0220] Instructions can be executed by one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Accordingly, the term "processor" as used herein can refer to any of the foregoing structure or any other structure suitable for implementation of the techniques described herein. In addition, in some aspects, the functionality described herein can be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated in a combined codec. Also, the techniques could be fully implemented in one or more circuits or logic elements.

[0221] The techniques of this disclosure can be implemented in a wide variety of devices or apparatuses, including a wireless handset, an integrated circuit (IC) or a set of ICs (e.g., a chip set). Various components, modules, or units are described herein as being configured to perform a particular function or task. However, such components, modules, or units can be configured to perform a variety of tasks that are not explicitly described or that can be described in the context of other components, modules, or units. For example, a component, module, or unit can be configured to perform a function in the context of one

[0222] Various examples have been described. These and other examples are within the scope of the following claims.

Claims

1. A method for decoding video data, the method comprising: It was determined that the previously decoded blocks of video data were decoded using a one-way prediction mode that did not assign values ​​to bidirectional prediction syntax elements; The current block of the video data is determined to be decoded using a bidirectional prediction mode, and the motion information of the current block is to be predicted based on the previous decoded block, including prediction based on the bidirectional prediction syntax elements of the previous decoded block. as well as In response to the fact that the bidirectional prediction syntax element of the previously decoded block does not have an assignment value, the current block is decoded using the substitution value of the bidirectional prediction syntax element.

2. The method of claim 1, wherein the bidirectional prediction syntax element comprises a bidirectional prediction (bcwIdx) syntax element having a decoding unit (CU) level weight index, wherein the replacement value comprises a preset value (BCW_DEFAULT), and the method further comprises deriving BCW_DEFAULT = (BCW_NUM>>1), wherein BCW_NUM is the number of weight options.

3. The method according to claim 2, wherein BCW_NUM is equal to 1.

4. The method of claim 2, wherein BCW_DEFAULT corresponds to an equal-weighted mixing of BCW modes.

5. The method of claim 1, wherein the current block comprises a 4x4 sub-block of a prediction unit (PU).

6. The method of claim 5, wherein the 4x4 sub-block comprises a first 4x4 sub-block of the PU, and the method further comprises determining a value different from the substitution value for a bidirectional predictive syntax element of a second distinct 4x4 sub-block of the PU.

7. The method of claim 1, further comprising determining the substitution value based on the values ​​of corresponding bidirectional predictive syntax elements of neighboring blocks of the current block.

8. The method of claim 7, wherein the corresponding bidirectional prediction syntax element includes an inter-prediction direction (interDir) syntax element.

9. The method of claim 7, wherein the corresponding bidirectional prediction syntax element includes a syntax element for determining the boundary strength of the unblocking filter.

10. The method of claim 1, further comprising encoding the current block before decoding the current block.

11. An apparatus for decoding video data, the apparatus comprising: A memory configured to store video data; and A processing system comprising one or more processors implemented in a circuit, the processing system being configured to: It was determined that the previously decoded blocks of video data were decoded using a one-way prediction mode that did not assign values ​​to bidirectional prediction syntax elements; The current block of the video data is determined to be decoded using a bidirectional prediction mode, and the motion information of the current block is to be predicted based on the previous decoded block, including prediction based on the bidirectional prediction syntax elements of the previous decoded block. as well as In response to the fact that the bidirectional prediction syntax element of the previously decoded block does not have an assignment value, the current block is decoded using the substitution value of the bidirectional prediction syntax element.

12. The apparatus of claim 11, wherein the bidirectional prediction syntax element comprises a bidirectional prediction (bcwIdx) syntax element having a decoding unit (CU) level weight index, wherein the replacement value comprises a preset value (BCW_DEFAULT), and wherein the processing system is further configured to derive BCW_DEFAULT = (BCW_NUM>>1), where BCW_NUM is the number of weight options.

13. The device of claim 11, wherein the current block comprises a 4x4 sub-block of a prediction unit (PU).

14. The device of claim 13, wherein the 4x4 sub-block comprises a first 4x4 sub-block of the PU, and wherein the processing system is further configured to determine a value different from the substitution value for a bidirectional predictive syntax element of a second distinct 4x4 sub-block of the PU.

15. The apparatus of claim 11, wherein the processing system is further configured to determine the substitution value based on the values ​​of corresponding bidirectional predictive syntax elements of neighboring blocks of the current block.

16. The apparatus of claim 15, wherein the corresponding bidirectional prediction syntax element includes an inter-prediction direction (interDir) syntax element.

17. The apparatus of claim 15, wherein the corresponding bidirectional prediction syntax element includes a syntax element for determining the boundary strength of the unblocking filter.

18. The apparatus of claim 11, wherein the processing system is further configured to encode the current block prior to decoding the current block.

19. The device of claim 11, further comprising a display configured to display decoded video data.

20. The device of claim 11, wherein the device comprises one or more of a camera, a computer, a mobile device, a broadcast receiver device, or a set-top box.

Images