Merge candidates for motion vector prediction for video coding

Abstract

To provide methods for coding (e.g., encoding or decoding) of motion information for a block of video data.SOLUTION: A method of decoding video data includes constructing a motion vector candidate list of merge candidates for a current block of video data based on motion information from a number of neighboring blocks relative to the current block, wherein the number of neighboring blocks considered for the motion vector candidate list is based on the size of the current block, and the number of neighboring blocks is greater than 5. In some examples, the method includes deriving a histogram of motion vector information for the neighboring blocks, and constructing the motion vector candidate list based on the derived histogram.SELECTED DRAWING: Figure 17

Description

[Technical Field]

[0001]

[0001] This application claims the benefit of U.S. Provisional Application No. 62 / 336,449, filed May 13, 2016, the entire contents of which are incorporated herein by reference.

[0002]

[0002] This disclosure relates to video coding. [Background technology]

[0003] Digital video capabilities may be incorporated into a wide range of devices, including digital televisions, digital direct broadcast systems, wireless broadcast systems, personal digital assistants (PDAs), laptop or desktop computers, tablet computers, e-book readers, digital cameras, digital recording devices, digital media players, video game devices, video game consoles, cellular or satellite radiotelephones, so-called "smartphones," video teleconferencing devices, video streaming devices, and the like. Digital video devices implement video coding techniques, such as those described in standards defined by MPEG-2, MPEG-4, ITU-T H.263, ITU-T H.264 / MPEG-4, Part 10, Advanced Video Coding (AVC), ITU-T H.265, also known as High Efficiency Video Coding (HEVC), and extensions to such standards. By implementing such video coding techniques, video devices may transmit, receive, encode, decode, and / or store digital video information more efficiently.

[0004]

[0004] Video coding techniques include spatial (intra-picture) prediction and / or temporal (inter-picture) prediction to reduce or remove redundancy inherent in video sequences. In block-based video coding, video slices (e.g., video frames or portions of video frames) may be partitioned into video blocks, which in some techniques may also be referred to as treeblocks, coding units (CUs), and / or coding nodes. Video blocks in intra-coded (I) slices of a picture are encoded using spatial prediction with respect to reference samples in neighboring blocks in the same picture. Video blocks in inter-coded (P or B) slices of a picture may use spatial prediction with respect to reference samples in neighboring blocks in the same picture or temporal prediction with respect to reference samples in other reference pictures. A picture may be referred to as a frame, and a reference picture may be referred to as a reference frame.

[0005]

[0005] Spatial or temporal prediction produces a predictive block for a block to be coded. Residual data represents pixel differences between the original block to be coded and the predictive block. Inter-coded blocks are coded according to a motion vector that points to a block of reference samples that form the predictive block, and the residual data indicates the differences between the coded block and the predictive block. Intra-coded blocks are coded according to an intra-coding mode and the residual data. For further compression, the residual data may be transformed from the pixel domain to a transform domain, resulting in residual transform coefficients, which may then be quantized. The quantized transform coefficients, initially organized into a two-dimensional array, may be scanned to generate a one-dimensional vector of transform coefficients, and entropy coding may be applied to achieve even further compression. Summary of the Invention

[0006] Generally, this disclosure describes techniques related to coding (e.g., encoding or decoding) motion information for blocks of video data. In various examples of this disclosure, a motion vector candidate list (also referred to as a merge candidate list or simply a candidate list) may be constructed using motion information from multiple neighboring blocks. A histogram of the motion information may then be derived and used to determine the order and / or location of spatial merge candidates for the motion vector candidate list.

[0007]

[0007] In one example, the present disclosure describes a method for decoding video data, the method comprising receiving a current block of video data encoded in merge mode; constructing a motion vector candidate list of merge candidates for the current block based on motion information from a certain number of neighboring blocks relative to the current block of the video data; determining a current motion vector from the motion vector candidate list, wherein the number of neighboring blocks considered for the motion vector candidate list is based on the size of the current block, wherein the number of neighboring blocks is greater than 5; and decoding the current block of the video data using the current motion vector.

[0008]

[0008] In another example, the present disclosure describes an apparatus configured to decode video data, the apparatus comprising: a memory configured to store a current block of the video data; and one or more processors configured to: receive the current block of video data encoded in a merge mode; construct a motion vector candidate list of merge candidates for the current block based on motion information from a certain number of neighboring blocks relative to the current block of the video data; determine a current motion vector from the motion vector candidate list, wherein the number of neighboring blocks considered for the motion vector candidate list is based on a size of the current block, wherein the number of neighboring blocks is greater than 5; and decode the current block of the video data using the current motion vector.

[0009]

[0009] In another example, the present disclosure describes a computer-readable storage medium storing instructions that, when executed, cause one or more processors configured to decode video data to receive a current block of video data encoded in merge mode; construct a motion vector candidate list of merge candidates for the current block based on motion information from a certain number of neighboring blocks relative to the current block of video data; determine a current motion vector from the motion vector candidate list, wherein the number of neighboring blocks considered for the motion vector candidate list is based on the size of the current block, wherein the number of neighboring blocks is greater than 5; and decode the current block of video data using the current motion vector.

[0010]

[0010] In one example, the present disclosure describes an apparatus configured to encode video data, the apparatus comprising: a memory configured to store a current block of the video data; and one or more processors configured to receive the current block of the video data; construct a motion vector candidate list of merging candidates for the current block based on motion information from a certain number of neighboring blocks relative to the current block of the video data; determine a current motion vector from the motion vector candidate list, wherein the number of neighboring blocks considered for the motion vector candidate list is based on the size of the current block, wherein the number of neighboring blocks is greater than 5; and encode the current block of the video data using the current motion vector.

[0011] The details of one or more examples are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will become apparent from the description and drawings, and from the claims. [Brief explanation of the drawings]

[0012] [Figure 1]

[0012] FIG. 1 is a block diagram illustrating an example video encoding and decoding system that may be configured to perform the techniques of this disclosure. [Figure 2]

[0013] 1 is a block diagram illustrating an example of a video encoder that may be configured to perform the techniques of this disclosure. [Figure 3]

[0014] 1 is a block diagram illustrating an example of a video decoder that may be configured to perform the techniques of this disclosure. [Figure 4]

[0015] 1 is a conceptual diagram showing a coding unit (CU) structure in High Efficiency Video Coding (HEVC). [Figure 5]

[0016] 1 is a conceptual diagram illustrating example partition types for inter prediction modes. [Figure 6A]

[0017] 1 is a conceptual diagram illustrating an example of block partitioning using a quad-tree-binary-tree (QTBT) structure. [Figure 6B]

[0018] 6B is a conceptual diagram illustrating an example tree structure corresponding to block partitioning using the QTBT structure of FIG. 6A. [Figure 7]

[0019] 1 is a conceptual diagram showing spatial neighbor candidates in HEVC. [Figure 8]

[0020] 1 is a conceptual diagram showing temporal motion vector prediction (TMVP) in HEVC. [Figure 9]

[0021] 1 is a conceptual diagram illustrating an example prediction structure for 3D-HEVC. [Figure 10]

[0022] A conceptual diagram showing sub-PU-based inter-view motion prediction in 3D-HEVC. [Figure 11]

[0023] 1 is a conceptual diagram showing sub-PU motion prediction from a reference picture. [Figure 12]

[0024] A conceptual diagram showing related pictures in ATMVP. [Figure 13]

[0025] 1 is a flowchart illustrating an exemplary method in accordance with techniques of this disclosure. [Figure 14]

[0026] FIG. 10 is a conceptual diagram showing an example of a PU and adjacent blocks. [Figure 15]

[0027] FIG. 10 is a conceptual diagram showing another example of a PU and neighboring blocks. [Figure 16]

[0028] FIG. 10 is a conceptual diagram showing another example of a PU and neighboring blocks. [Figure 17]

[0029] FIG. 10 is a conceptual diagram showing another example of a PU and neighboring blocks. [Figure 18]

[0030] 1 is a flowchart illustrating an exemplary encoding method of the present disclosure. [Figure 19]

[0031] 1 is a flowchart illustrating an exemplary decoding method of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0013]

[0032] This disclosure describes techniques for improving the efficiency of merge-based motion vector prediction. This disclosure describes techniques for determining merge candidates to use for motion vector prediction. Example techniques of this disclosure may include adaptive ordering and pruning of merge candidates. Example adaptive ordering techniques of this disclosure may include adaptive ordering of spatial, sub-prediction unit (PU), and combined motion vector (combi-mv) candidates. In some examples, the proposed adaptive pruning technique of this disclosure may be applied to all merge candidates, including temporal motion vector prediction (TMVP) candidates, zero motion vector (0mv) candidates, and the candidates mentioned above.

[0014]

[0033] The techniques of this disclosure may provide one or more of the following benefits: (1) higher efficiency, (2) simplicity (e.g., lower decoder implementation complexity), and (3) flexibility. The techniques described herein may provide higher bit-savings by assigning higher priority to merge candidates whose values ​​are closer (or likely to be closer) to the actual motion vector. Furthermore, video encoder and video decoder complexity and memory requirements are relatively small. Finally, the proposed techniques may be applied to various codecs, such as H.266 and quad-tree plus binary tree (QTBT)-based video codecs. Furthermore, the proposed techniques provide flexibility such that the techniques described herein can be combined in any manner, so that they can be used independently or together.

[0015]

[0034] 1 is a block diagram illustrating an exemplary video encoding and decoding system 10 that may be configured to perform the techniques of this disclosure for motion vector prediction. As shown in FIG. 1, system 10 includes a source device 12 that provides encoded video data to be subsequently decoded by a destination device 14. In particular, source device 12 provides video data to destination device 14 via a computer-readable medium 16. Source device 12 and destination device 14 may comprise any of a wide range of devices, including desktop computers, notebook (i.e., laptop) computers, tablet computers, set-top boxes, telephone handsets such as so-called "smart" phones, so-called "smart" pads, televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, etc. In some cases, source device 12 and destination device 14 may be equipped for wireless communication.

[0016]

[0035] The destination device 14 may receive the encoded video data to be decoded via a computer-readable medium 16. The computer-readable medium 16 may comprise any type of medium or device capable of moving encoded video data from the source device 12 to the destination device 14. In one example, the computer-readable medium 16 may comprise a communication medium for enabling the source device 12 to transmit the encoded video data directly to the destination device 14 in real time. The encoded video data may be modulated according to a communication standard, such as a wireless communication protocol, and transmitted to the destination device 14. The communication medium may comprise any wireless or wired communication medium, such as a radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form part of a packet-based network, such as a local area network, a wide area network, or a global network such as the Internet. The communication medium may include routers, switches, base stations, or any other equipment that may be useful for enabling communication from the source device 12 to the destination device 14.

[0017]

[0036] In some examples, the encoded data may be output from output interface 22 to a storage device. Similarly, the encoded data may be accessed from a storage device by an input interface. The storage device may include any of a variety of distributed or locally accessed data storage media, such as a hard drive, a Blu-ray disc, a DVD, a CD-ROM, flash memory, volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded video data. In a further example, the storage device may correspond to a file server or another intermediate storage device that may store the encoded video generated by source device 12. Destination device 14 may access the stored video data from the storage device via streaming or download. The file server may be any type of server capable of storing the encoded video data and transmitting the encoded video data to destination device 14. Exemplary file servers include a web server (e.g., for a website), an FTP server, a network-attached storage (NAS) device, or a local disk drive. Destination device 14 may access the encoded video data through any standard data connection, including an Internet connection. This may include a wireless channel (e.g., a Wi-Fi® connection), a wired connection (e.g., DSL, cable modem, etc.), or a combination of both, that is suitable for accessing the encoded video data stored on the file server. The transmission of the encoded video data from the storage device may be a streaming transmission, a download transmission, or a combination thereof.

[0018]

[0037] The techniques of this disclosure are not necessarily limited to wireless applications or settings. The techniques may be applied to video coding supporting any of a variety of multimedia applications, such as over-the-air television broadcast, cable television transmission, satellite television transmission, Internet streaming video transmission such as Dynamic Adaptive Streaming over HTTP (DASH), digital video encoded on a data storage medium, decoding of digital video stored on a data storage medium, or other applications. In some examples, system 10 may be configured to support one-way or two-way video transmission to support applications such as video streaming, video playback, video broadcasting, and / or video telephony.

[0019]

[0038] In the example of FIG. 1 , source device 12 includes a video source 18, a video encoder 20, and an output interface 22. Destination device 14 includes an input interface 28, a video decoder 30, and a display device 32. According to this disclosure, video encoder 20 of source device 12 may be configured to apply techniques of this disclosure for motion vector prediction. In other examples, the source device and destination device may include other components or configurations. For example, source device 12 may receive video data from an external video source 18, such as an external camera. Similarly, destination device 14 may interface with an external display device rather than including a built-in display device.

[0020]

[0039] The illustrated system 10 of FIG. 1 is merely an example. The techniques of this disclosure for motion vector prediction may be performed by any digital video encoding and / or decoding device. Generally, the techniques of this disclosure are performed by a video encoding device, but the techniques may also be performed by a video encoder / decoder, commonly referred to as a “codec.” Moreover, the techniques of this disclosure may also be performed by a video preprocessor. Source device 12 and destination device 14 are merely examples of coding devices, such that source device 12 generates coded video data for transmission to destination device 14. In some examples, devices 12, 14 may operate substantially symmetrically, such that each of devices 12, 14 includes a video encoding component and a video decoding component. Thus, system 10 may support unidirectional or bidirectional video transmission between video device 12 and video device 14, for example, video streaming, video playback, video broadcasting, or video telephony.

[0021]

[0040] Video source 18 of source device 12 may include a video capture device such as a video camera, a video archive containing previously captured video, and / or a video feed interface for receiving video from a video content provider. As a further alternative, video source 18 may generate computer graphics-based data as the source video, or a combination of live, archived, and computer-generated video. In some cases, if video source 18 is a video camera, source device 12 and destination device 14 may form a so-called camera phone or video phone. However, as mentioned above, the techniques described in this disclosure may be applicable to video coding generally and may be applied to wireless and / or wired applications. In each case, the captured video, previously captured video, or computer-generated video may be encoded by video encoder 20. The encoded video information may then be output onto computer-readable medium 16 by output interface 22.

[0022]

[0041] Computer-readable medium 16 may include a transient medium, such as a wireless broadcast or wired network transmission, or a storage medium (i.e., a non-transitory storage medium), such as a hard disk, flash drive, compact disc, digital video disc, Blu-ray disc, or other computer-readable medium. In some examples, a network server (not shown) may receive encoded video data from source device 12 and provide the encoded video data to destination device 14, e.g., via a network transmission. Similarly, a computing device of a media production facility, such as a disc stamping facility, may receive encoded video data from source device 12 and generate a disc containing the encoded video data. Thus, it will be understood that computer-readable medium 16, in various examples, includes one or more computer-readable media of various forms.

[0023]

[0042] Input interface 28 of destination device 14 receives information from computer-readable medium 16. The information on computer-readable medium 16 may include syntax information, including syntax elements that describe characteristics and / or processing of blocks and other coded units, e.g., GOPs, defined by video encoder 20 and used by video decoder 30. Display device 32 displays the decoded video data to a user and may comprise any of a variety of display devices, such as a cathode ray tube (CRT), a liquid crystal display (LCD), a plasma display, an organic light-emitting diode (OLED) display, or another type of display device.

[0024]

[0043] Video encoder 20 and video decoder 30 may operate in accordance with a video coding standard, such as the High Efficiency Video Coding (HEVC) standard, extensions to the HEVC standard, or a subsequent standard such as ITU-T H.266. Alternatively or additionally, video encoder 20 and video decoder 30 may operate in accordance with other proprietary or industry standards, such as the ITU-T H.264 standard, alternatively referred to as MPEG-4, Part 10, Advanced Video Coding (AVC), or extensions to such standards. However, the techniques of this disclosure are not limited to any particular coding standard. Other examples of video coding standards include MPEG-2 and ITU-T H.263. Although not shown in FIG. 1 , in some aspects, video encoder 20 and video decoder 30 may be integrated with an audio encoder and audio decoder, respectively, and may include appropriate MUX-DEMUX units or other hardware and software to handle the encoding of both audio and video in a common data stream or separate data streams. Where applicable, the MUX-DEMUX unit may conform to the ITU H.223 multiplexer protocol or other protocols such as the User Datagram Protocol (UDP).

[0025]

[0044] Video encoder 20 and video decoder 30 may each be implemented as any of a variety of suitable encoder or decoder circuits, or any combination thereof, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, etc. When the techniques are implemented partially in software, a device may store instructions for the software on a suitable non-transitory computer-readable medium and execute the instructions in hardware using one or more processors to perform the techniques of this disclosure. Video encoder 20 and video decoder 30 may each be included in one or more encoders or decoders, any of which may be integrated as part of a combined encoder / decoder (codec) in the respective device.

[0026]

[0045] As described in more detail below, video encoder 20 and video decoder 30 may be configured to receive a current block of video data, construct a motion vector candidate list of merge candidates for the current block based on motion information from a number of neighboring blocks relative to the current block of video data, determine a current motion vector from the motion vector candidate list, where the number of neighboring blocks considered for the motion vector candidate list is based on the size of the current block, and where the number of neighboring blocks is greater than five, and code (e.g., encode or decode) the current block of video data using the current motion vector.

[0027]

[0046] Video coding standards include ITU-T H.261, ISO / IEC MPEG-1 Visual, ITU-T H.262 or ISO / IEC MPEG-2 Visual, ITU-T H.263, ISO / IEC MPEG-4 Visual, and ITU-T H.264 (also known as ISO / IEC MPEG-4 AVC), which includes its Scalable Video Coding (SVC) and Multiview Video Coding (MVC) extensions. One joint draft of MVC is described in "Advanced video coding for generic audiovisual services," ITU-T Recommendation H.264, March 2010.

[0028]

[0047] Additionally, there is a newly developed video coding standard, namely, HEVC, developed by the Joint Collaboration Team on Video Coding (JCT-VC) between the ITU-T Video Coding Experts Group (VCEG) and the ISO / IEC Motion Picture Experts Group (MPEG). A recent draft of HEVC is available at http: / / phenix.int-evry.fr / jct / doc_end_user / documents / 12_Geneva / wg11 / JCTVC-L1003-v34.zip. The HEVC standard is also presented together in Recommendation ITU-T H.265 and International Standard ISO / IEC 23008-2, both entitled "High efficiency video coding," and both published in October 2014.

[0029]

[0048] The JCT-VC developed the HEVC standard. The HEVC standardization effort was based on an evolutionary model of a video coding device called the HEVC Test Model (HM). The HM assumed some additional capabilities of a video coding device relative to existing devices that conform, for example, to ITU-T H.264 / AVC. For example, H.264 provides nine intra-prediction coding modes, while the HEVC HM may provide as many as 33 intra-prediction coding modes. While this disclosure may use some HEVC terminology for purposes of explanation, the techniques of this disclosure are not limited to HEVC, and in fact, it is expressly contemplated that the techniques of this disclosure may be implemented in successor standards to HEVC.

[0030]

[0049] Generally, the HM working model described that a video frame or picture can be divided into a series of treeblocks or largest coding units (LCUs), each containing both luma and chroma samples. Syntax data in the bitstream may define the size of the LCU, which is the largest coding unit in terms of the number of pixels. A slice includes several consecutive treeblocks in coding order. A video frame or picture can be partitioned into one or more slices. Each treeblock can be split into coding units (CUs) according to a quadtree. Generally, the quadtree data structure includes one node per CU, with the root node corresponding to the treeblock. If a CU is split into four sub-CUs, the node corresponding to the CU includes four leaf nodes, each of which corresponds to one of the sub-CUs.

[0031]

[0050] Each node of the quadtree data structure may provide syntax data for the corresponding CU. For example, a quadtree node may include a split flag indicating whether the CU corresponding to that node is split into sub-CUs. Syntax elements for a CU may be defined recursively and may depend on whether the CU is split into sub-CUs. If a CU is not further split, the CU is called a leaf CU. In this disclosure, the four sub-CUs of a leaf CU are also called leaf CUs, even if there is no explicit splitting of the original leaf CU. For example, if a CU of 16x16 size is not further split, the four 8x8 sub-CUs are also called leaf CUs, even if the 16x16 CU is never split.

[0032]

[0051] A CU has a purpose similar to a macroblock in the H.264 standard, except that CUs have no size variation. For example, a treeblock may be split into four child nodes (also called sub-CUs), and each child node, in turn, becomes a parent node and may split into another four child nodes. The last unsplit child node, called a leaf node of the quadtree, comprises a coding node, also called a leaf CU. Syntax data associated with the coded bitstream may define the maximum number of times a treeblock may be split, called the maximum CU depth, and may also define the minimum size of a coding node. Accordingly, the bitstream may also define the smallest coding unit (SCU). This disclosure uses the term "block" to refer to either a CU, PU, ​​or TU in the context of HEVC, or similar data structures in the context of other standards (e.g., a macroblock and its sub-blocks in H.264 / AVC).

[0033]

[0052] A CU includes a coding node and prediction units (PUs) and transform units (TUs) associated with the coding node. The size of a CU corresponds to the size of the coding node and must be square in shape. The size of a CU may range from 8x8 pixels to the size of a treeblock, with a maximum of 64x64 pixels or more. Each CU may include one or more PUs and one or more TUs. Syntax data associated with a CU may, for example, describe partitioning of the CU into one or more PUs. The partitioning mode may vary between whether the CU is skip-mode coded or direct-mode coded, or intra-prediction mode coded or inter-prediction mode coded. A PU may be partitioned to be non-square in shape. Syntax data associated with a CU may also, for example, describe partitioning of the CU into one or more TUs according to a quadtree. A TU may be square or non-square (e.g., rectangular) in shape.

[0034]

[0053] The HEVC standard allows for transforms according to TUs, which may vary for each CU. TUs are generally sized based on the size of the PUs within a given CU defined for partitioned LCUs, although this is not always the case. TUs are generally the same size as or smaller than the PUs. In some examples, residual samples corresponding to a CU may be subdivided into smaller units using a quadtree structure known as a "residual quad tree" (RQT). Leaf nodes of an RQT may be referred to as transform units (TUs). Pixel difference values ​​associated with a TU may be transformed to generate transform coefficients, which may be quantized.

[0035]

[0054] A leaf CU may include one or more prediction units (PUs). Generally, a PU represents a spatial area corresponding to all or a portion of a corresponding CU and may include data for retrieving reference samples for the PU. Moreover, a PU includes data related to prediction. For example, when a PU is intra-mode coded, the data of the PU may be included in a residual quadtree (RQT), which may include data describing an intra-prediction mode for the TU corresponding to the PU. As another example, when a PU is inter-mode coded, the PU may include data defining one or more motion vectors for the PU. The data defining a motion vector for the PU may describe, for example, the horizontal component of the motion vector, the vertical component of the motion vector, the resolution of the motion vector (e.g., ¼-pixel precision or ⅛-pixel precision), the reference picture to which the motion vector points, and / or the reference picture list (e.g., List 0, List 1, or List C) of the motion vector.

[0036]

[0055] A leaf CU having one or more PUs may also include one or more transform units (TUs). Transform units may be specified using an RQT (also called a TU quadtree structure), as described above. For example, a split flag may indicate whether a leaf CU is split into four transform units. Each transform unit may then be further split into additional sub-TUs. When a TU is not further split, the TU may be referred to as a leaf TU. Generally, in intra-coding, all leaf TUs belonging to a leaf CU share the same intra-prediction mode. That is, generally, the same intra-prediction mode is applied to calculate predicted values ​​of all TUs of a leaf CU. In intra-coding, a video encoder may use the intra-prediction mode to calculate a residual value of each leaf TU as the difference between a portion of the CU corresponding to the TU and the original block. A TU is not necessarily limited to the size of a PU. Thus, a TU may be larger or smaller than a PU. In intra-coding, a PU may be co-located with a corresponding leaf TU for the same CU. In some examples, the maximum size of a leaf TU may correspond to the size of the corresponding leaf CU.

[0037]

[0056] Furthermore, the TUs of a leaf CU may also be associated with a respective quadtree data structure called a residual quadtree (RQT). That is, a leaf CU may include a quadtree that indicates how the leaf CU is partitioned into TUs. The root node of a TU quadtree generally corresponds to a leaf CU, and the root node of a CU quadtree generally corresponds to a treeblock (or LCU). TUs of an RQT that are not split are called leaf TUs. Generally, this disclosure uses the terms CU and TU to refer to leaf CUs and leaf TUs, respectively, unless otherwise specified.

[0038]

[0057] A video sequence generally includes a series of video frames or pictures. A group of pictures (GOP) generally comprises a series of one or more of the video pictures. A GOP may include syntax data describing the pictures included in the GOP in a header of the GOP, in the headers of one or more of the pictures, or elsewhere. Each slice of a picture may include slice syntax data describing the coding mode for the respective slice. Video encoder 20 generally operates on video blocks within individual video slices to encode video data. Video blocks may correspond to coding nodes within a CU. Video blocks may have a fixed size or a variable size and may vary in size depending on a specified coding standard.

[0039]

[0058] As an example, the HM supports prediction at various PU sizes. Assuming a particular CU size is 2N×2N, the HM supports intra prediction at PU sizes of 2N×2N or N×N, and inter prediction at symmetric PU sizes of 2N×2N, 2N×N, N×2N, or N×N. The HM also supports asymmetric partitioning for inter prediction at PU sizes of 2N×nU, 2N×nD, nL×2N, and nR×2N. In asymmetric partitioning, one direction of the CU is not partitioned, while the other direction is partitioned into 25% and 75%. The portion of the CU corresponding to the 25% partition is indicated by an “n” followed by “Up,” “Down,” “Left,” or “Right.” Thus, for example, “2N×nU” refers to a 2N×2N CU partitioned horizontally with a 2N×0.5N PU at the top and a 2N×1.5N PU at the bottom.

[0040]

[0059] In this disclosure, "N x N (NxN)" and "N by N (N by N)" may be used interchangeably to refer to the pixel dimensions of a video block in terms of vertical and horizontal dimensions, e.g., 16 x 16 (16x16) pixels or 16 x 16 (16 by 16) pixels. Generally, a 16x16 block has 16 pixels in the vertical direction (y=16) and 16 pixels in the horizontal direction (x=16). Similarly, an NxN block generally has N pixels in the vertical direction and N pixels in the horizontal direction, where N represents a non-negative integer value. Pixels in a block may be arranged in rows and columns. Moreover, a block does not necessarily have to have the same number of pixels in the horizontal direction as in the vertical direction. For example, a block may comprise N x M pixels, where M is not necessarily equal to N.

[0041]

[0060] After intra-predictive coding or inter-predictive coding using the PU of a CU, video encoder 20 may calculate residual data for the TUs of the CU. The PU may comprise syntax data describing a method or mode of generating predictive pixel data in the spatial domain (also referred to as the pixel domain), and the TUs may comprise coefficients in the transform domain after application of a transform, e.g., a discrete cosine transform (DCT), an integer transform, a wavelet transform, or a conceptually similar transform, to the residual video data. The residual data may correspond to pixel differences between pixels of the uncoded picture and predicted values ​​corresponding to the PU. Video encoder 20 may form TUs including the residual data for the CU and then transform the TUs to generate transform coefficients for the CU.

[0042]

[0061] After any transformation to generate transform coefficients, video encoder 20 may perform quantization of the transform coefficients. Quantization generally refers to a process in which transform coefficients are quantized to possibly reduce the amount of data used to represent the coefficients, providing further compression. The quantization process may reduce the bit depth associated with some or all of the coefficients. For example, during quantization, an n-bit value may be truncated to an m-bit value, where n is greater than m.

[0043]

[0062] After quantization, the video encoder may scan the transform coefficients and generate a one-dimensional vector from a two-dimensional matrix containing the quantized transform coefficients. The scan may be designed to place higher energy (and therefore lower frequency) coefficients at the front of the array and lower energy (and therefore higher frequency) coefficients at the back of the array. In some examples, the video encoder 20 may utilize a predefined scan order to scan the quantized transform coefficients to generate a serialized vector that can be entropy coded. In other examples, the video encoder 20 may perform adaptive scanning. After scanning the quantized transform coefficients to form the one-dimensional vector, video encoder 20 may entropy encode the one-dimensional vector according to, for example, context-adaptive variable length coding (CAVLC), context-adaptive binary arithmetic coding (CABAC), syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or another entropy encoding method. Video encoder 20 may also entropy encode syntax elements associated with the encoded video data for use by video decoder 30 in decoding the video data.

[0044]

[0063] To perform CABAC, video encoder 20 may assign a context within a context model to a symbol to be transmitted. The context may relate, for example, to whether neighboring values ​​of the symbol are non-zero. To perform CAVLC, video encoder 20 may select a variable-length code for the symbol to be transmitted. Codewords in VLC may be configured such that relatively shorter codes correspond to more-probable symbols and longer codes correspond to less-probable symbols. In this manner, use of VLC may achieve bit savings over, for example, using equal-length codewords for each symbol to be transmitted. The probability determination may be based on the context assigned to the symbol.

[0045]

[0064] 2 is a block diagram illustrating an example of a video encoder 20 that may be configured to perform the techniques of this disclosure for motion vector prediction, as described in more detail below. Video encoder 20 may perform intra-coding and inter-coding of video blocks within video slices. Intra-coding relies on spatial prediction to reduce or remove spatial redundancy in video within a given video frame or picture. Inter-coding relies on temporal prediction to reduce or remove temporal redundancy in video within adjacent frames or pictures of a video sequence. Intra-mode (I-mode) may refer to any of several spatial-based coding modes. Inter-mode, such as unidirectional prediction (P-mode) or bi-prediction (B-mode), may refer to any of several temporal-based coding modes.

[0046]

[0065] As shown in FIG. 2, video encoder 20 receives a current video block in a video frame to be encoded. In the example of FIG. 2, video encoder 20 includes a video data memory 41, a mode select unit 40, a reference picture memory 64, an adder 50, a transform processing unit 52, a quantization unit 54, and an entropy coding unit 56. Mode select unit 40, in turn, includes a motion compensation unit 44, a motion estimation unit 42, an intra prediction unit 46, and a partition unit 48. For video block reconstruction, video encoder 20 also includes an inverse quantization unit 58, an inverse transform unit 60, and an adder 62. A deblocking filter (not shown in FIG. 2) may also be included to filter block boundaries to remove blockiness artifacts from the reconstructed video. If desired, the deblocking filter will generally filter the output of adder 62. Additional filters (in or after the loop) may also be used in addition to the deblocking filter. Such a filter is not shown for simplicity, but may filter the output of summer 50 (as an in-loop filter) if desired.

[0047]

[0066] Video data memory 41 may be configured to store video data to be encoded by components of video encoder 20. The video data stored in video data memory 41 may be obtained, for example, from video source 18. Reference picture memory 64 (sometimes referred to as a decoded picture buffer) may be a reference picture memory that stores reference video data for use in encoding video data by video encoder 20, for example, in intra-coding or inter-coding modes. Video data memory 41 and reference picture memory 64 may be formed by any of a variety of memory devices, such as dynamic random access memory (DRAM), including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. Video data memory 41 and reference picture memory 64 may be provided by the same memory device or separate memory devices. In various examples, video data memory 41 may be on-chip with other components of video encoder 20 or off-chip relative to those components.

[0048]

[0067] During the encoding process, video encoder 20 receives a video frame or slice to be coded. The frame or slice may be divided into multiple video blocks. Motion estimation unit 42 and motion compensation unit 44 perform inter-predictive coding of the received video block relative to one or more blocks in one or more reference frames to perform temporal prediction. Intra-prediction unit 46 may alternatively perform intra-predictive coding of the received video block relative to one or more neighboring blocks in the same frame or slice as the block to be coded to perform spatial prediction. Video encoder 20 may perform multiple coding passes, for example, to select an appropriate coding mode for each block of video data.

[0049]

[0068] Moreover, partition unit 48 may partition blocks of video data into sub-blocks based on an evaluation of a previous partitioning scheme in a previous coding pass. For example, partition unit 48 may first partition a frame or slice into LCUs and then partition each of the LCUs into sub-CUs based on a rate-distortion analysis (e.g., rate-distortion optimization). Mode selection unit 40 may further generate a quadtree data structure indicating the partitioning of the LCUs into sub-CUs. A leaf-node CU of the quadtree may include one or more PUs and one or more TUs.

[0050]

[0069] Mode select unit 40 may, for example, select a coding mode, i.e., one of intra or inter, based on the error result, and provide the resulting intra- or inter-coded blocks to summer 50 to generate residual block data, and to summer 62 to reconstruct coded blocks for use as reference frames. Mode select unit 40 also provides syntax elements, such as motion vectors, intra-mode indicators, partition information, and other such syntax information, to entropy coding unit 56.

[0051]

[0070] Motion estimation unit 42 and motion compensation unit 44 may be highly integrated but are shown separately for conceptual purposes. Motion estimation performed by motion estimation unit 42 is the process of generating motion vectors that estimate the motion of video blocks. A motion vector may indicate, for example, the displacement of a PU of a video block in a current video frame or picture relative to a predictive block in a reference frame (or other coding unit) relative to a current block being coded in the current frame (or other coding unit). A predictive block is a block that is found to closely match a block to be coded in terms of pixel differences, which may be determined by sum of absolute difference (SAD), sum of square difference (SSD), or other difference metric. In some examples, video encoder 20 may calculate values ​​for sub-integer pixel positions of reference pictures stored in reference picture memory 64. For example, video encoder 20 may interpolate values ​​for quarter-pixel positions, eighth-pixel positions, or other fractional pixel positions of a reference picture. Thus, motion estimation unit 42 may perform motion searches for full-pixel and fractional-pixel locations and output motion vectors with fractional-pixel accuracy.

[0052]

[0071] Motion estimation unit 42 calculates a motion vector for a PU of a video block in an inter-coded slice by comparing the position of the PU with the position of a predictive block of a reference picture. The reference picture may be selected from a first reference picture list (List 0) or a second reference picture list (List 1), each of which identifies one or more reference pictures stored in reference picture memory 64. Motion estimation unit 42 sends the calculated motion vector to entropy coding unit 56 and motion compensation unit 44.

[0053]

[0072] The motion compensation performed by motion compensation unit 44 may involve fetching or generating a predictive block based on the motion vector determined by motion estimation unit 42. Again, motion estimation unit 42 and motion compensation unit 44 may, in some examples, be functionally integrated. Upon receiving the motion vector for the PU of the current video block, motion compensation unit 44 may locate the predictive block to which the motion vector points in one of the reference picture lists. Adder 50 forms a residual video block by subtracting pixel values ​​of the predictive block from pixel values ​​of the current video block being coded and forming pixel difference values, as described below. Generally, motion estimation unit 42 performs motion estimation on the luma component, and motion compensation unit 44 uses the motion vector calculated based on the luma component for both the chroma and luma components. Mode select unit 40 may also generate syntax elements associated with the video block and the video slice for use by video decoder 30 in decoding the video block of the video slice.

[0054]

[0073] Video encoder 20 including motion estimation unit 42 and motion compensation unit 44 may be configured to perform any of the various techniques of this disclosure, described above with respect to FIG. 1 and in more detail below. For example, motion compensation unit 44 may be configured to code motion information for blocks of video data using AMVP mode or merge mode in accordance with the techniques of this disclosure. Furthermore, video encoder 20 including motion estimation unit 42 and motion compensation unit 44 may be configured to perform any combination of the motion vector candidate list construction techniques of this disclosure, described in more detail below. In the context of this disclosure, the terms motion vector candidate list, merge candidate list, and candidate list may be used interchangeably.

[0055]

[0074] Assuming that motion compensation unit 44 elects to perform merge mode, motion compensation unit 44 may form a candidate list including a set of merge candidates. Motion compensation unit 44 may add candidates to the candidate list based on a particular, predetermined order. In other examples of this disclosure, motion compensation unit 44 may be configured to dynamically add candidates to the candidate list in a different order based on histogram information of motion vectors from neighboring blocks. Motion compensation unit 44 may also add additional candidates and perform candidate list pruning, as described in more detail below. Finally, mode selection unit 40 may determine which of those candidates should be used to encode the motion information of the current block and encode a merge index representing the selected candidate.

[0056]

[0075] Intra prediction unit 46 may intra predict the current block as an alternative to the inter prediction performed by motion estimation unit 42 and motion compensation unit 44, as described above. In particular, intra prediction unit 46 may determine an intra prediction mode to use to encode the current block. In some examples, intra prediction unit 46 may encode the current block using different intra prediction modes, e.g., during separate encoding passes, and intra prediction unit 46 (or, in some examples, mode selection unit 40) may select an appropriate intra prediction mode to use from the tested modes.

[0057]

[0076] For example, intra prediction unit 46 may calculate rate-distortion values ​​for various tested intra prediction modes using a rate-distortion analysis and select the intra prediction mode with the best rate-distortion characteristics among the tested modes. The rate-distortion analysis generally determines the amount of distortion (or error) between a coded block and the original uncoded block that was coded to generate the coded block, as well as the bit rate (i.e., number of bits) used to generate the coded block. Intra prediction unit 46 may calculate a ratio from the distortion and rate for the various coded blocks to determine which intra prediction mode exhibits the best rate-distortion value for the block.

[0058]

[0077] After selecting an intra-prediction mode for the block, intra-prediction unit 46 may provide information indicating the selected intra-prediction mode for the block to entropy coding unit 56. Entropy coding unit 56 may encode the information indicating the selected intra-prediction mode. Video encoder 20 may include configuration data in the transmitted bitstream, which may include multiple intra-prediction mode index tables and multiple modified intra-prediction mode index tables (also referred to as codeword mapping tables), definitions of coding contexts for various blocks, and an indication of the most probable intra-prediction mode, intra-prediction mode index table, and modified intra-prediction mode index table to use for each of the contexts.

[0059]

[0078] Video encoder 20 forms a residual video block by subtracting prediction data from mode select unit 40 from the original video block being coded. Summer 50 represents one or more components that perform this subtraction operation. Transform processing unit 52 applies a transform, such as a discrete cosine transform (DCT) or a conceptually similar transform, to the residual block, generating a video block comprising residual transform coefficient values. Transform processing unit 52 may perform other transforms that are conceptually similar to the DCT. Wavelet transforms, integer transforms, subband transforms, or other types of transforms may also be used.

[0060]

[0079] In either case, transform processing unit 52 applies a transform to the residual block, producing a block of residual transform coefficients. The transform may convert the residual information from the pixel value domain to a transform domain, such as the frequency domain. Transform processing unit 52 may send the resulting transform coefficients to quantization unit 54. Quantization unit 54 quantizes the transform coefficients to further reduce the bit rate. The quantization process may reduce the bit depth associated with some or all of the coefficients. The degree of quantization may be changed by adjusting a quantization parameter. In some examples, quantization unit 54 may then perform a scan of a matrix including the quantized transform coefficients. Alternatively, entropy coding unit 56 may perform the scan.

[0061]

[0080] After quantization, entropy coding unit 56 entropy codes the quantized transform coefficients. For example, entropy coding unit 56 may perform context-adaptive variable length coding (CAVLC), context-adaptive binary arithmetic coding (CABAC), syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioned entropy (PIPE) coding, or another entropy coding technique. In the case of context-based entropy coding, the context may be based on neighboring blocks. After entropy coding by entropy coding unit 56, the coded bitstream may be transmitted to another device (e.g., video decoder 30) or archived for later transmission or retrieval.

[0062]

[0081] Inverse quantization unit 58 and inverse transform unit 60 apply inverse quantization and inverse transform, respectively, to reconstruct the residual block in the pixel domain, e.g., for later use as a reference block. Motion compensation unit 44 may calculate a reference block by adding the residual block to a predictive block of one of the frames in reference picture memory 64. Motion compensation unit 44 may also apply one or more interpolation filters to the reconstructed residual block to calculate sub-integer pixel values ​​for use in motion estimation. Adder 62 adds the reconstructed residual block to the motion-compensated predictive block generated by motion compensation unit 44 to generate a reconstructed video block for storage in reference picture memory 64. The reconstructed video block may be used as a reference block by motion estimation unit 42 and motion compensation unit 44 for inter-coding blocks in subsequent video frames.

[0063]

[0082] In this manner, video encoder 20 of FIG. 2 represents an example of a video coder configured to derive a histogram of motion vector information from neighboring blocks relative to a current block, determine merge candidates for a motion vector candidate list for motion vector prediction for the current block based on the derived histogram, order the motion vector candidate list based on the derived histogram, and perform merge vector prediction using the motion vector candidate list.

[0064]

[0083] 3 is a block diagram illustrating an example of a video decoder 30 that may be configured to perform the motion vector prediction techniques of this disclosure. In the example of FIG. 3, video decoder 30 includes a video data memory 71, an entropy decoding unit 70, a motion compensation unit 72, an intra prediction unit 74, an inverse quantization unit 76, an inverse transform unit 78, a reference picture memory 82, and an adder 80. Video decoder 30 may, in some examples, perform a decoding path that is generally inverse to the encoding path described with respect to video encoder 20 (FIG. 2). Motion compensation unit 72 may generate prediction data based on a motion vector received from entropy decoding unit 70, and intra prediction unit 74 may generate prediction data based on an intra prediction mode indicator received from entropy decoding unit 70.

[0065]

[0084] Video data memory 71 may store video data, such as an encoded video bitstream, to be decoded by components of video decoder 30. The video data stored in video data memory 71 may be obtained, for example, from computer-readable medium 16, from a local video source such as a camera, via wired or wireless network communication of video data, or by accessing a physical data storage medium. Video data memory 71 may form a coded picture buffer (CPB) that stores coded video data from the coded video bitstream. Reference picture memory 82 (also referred to as a decoded picture buffer (DPB)) may be a reference picture memory that stores reference video data for use in decoding video data by video decoder 30 in, for example, an intra-coding mode or an inter-coding mode, or for output. Video data memory 71 and reference picture memory 82 may be formed by any of a variety of memory devices, such as DRAM, SDRAM, MRAM, RRAM, or other types of memory devices. Video data memory 71 and reference picture memory 82 may be provided by the same memory device or separate memory devices. In various examples, video data memory 71 may be on-chip with other components of video decoder 30, or off-chip relative to those components.

[0066]

[0085] During the decoding process, video decoder 30 receives from video encoder 20 an encoded video bitstream representing video blocks of encoded video slices and associated syntax elements. Entropy decoding unit 70 of video decoder 30 entropy decodes the bitstream to generate quantized coefficients, motion vectors or intra-prediction mode indicators, and other syntax elements. Entropy decoding unit 70 forwards the motion vectors and other syntax elements to motion compensation unit 72. Video decoder 30 may receive syntax elements at the video slice level and / or the video block level.

[0067]

[0086] When a video slice is coded as an intra-coded (I) slice, intra prediction unit 74 may generate predictive data for video blocks of the current video slice based on the signaled intra prediction mode and data from previously decoded blocks of the current frame or picture. When a video frame is coded as an inter-coded (e.g., BB or P) slice, motion compensation unit 72 generates predictive blocks for video blocks of the current video slice based on motion vectors and other syntax elements received from entropy decoding unit 70. The predictive blocks may be generated from one of the reference pictures in one of the reference picture lists. Video decoder 30 may construct the reference frame lists, i.e., List 0 and List 1, using a default construction technique based on the reference pictures stored in reference picture memory 82.

[0068]

[0087] Motion compensation unit 72 determines prediction information for video blocks of the current video slice by parsing the motion vectors and other syntax elements and uses the prediction information to generate predictive blocks for the current video block being decoded. For example, motion compensation unit 72 uses some of the received syntax elements to determine the prediction mode (e.g., intra- or inter-prediction) used to code the video blocks of the video slice, the inter-prediction slice type (e.g., B slice or P slice), construction information for one or more of the reference picture lists for the slice, motion vectors for each inter-coded video block of the slice, inter-prediction status for each inter-coded video block of the slice, and other information for decoding video blocks in the current video slice.

[0069]

[0088] Motion compensation unit 72 may also perform interpolation based on an interpolation filter. Motion compensation unit 72 may use the interpolation filter used by video encoder 20 during encoding of the video block to calculate sub-integer pixel interpolated values ​​of the reference block. In this case, motion compensation unit 72 may determine the interpolation filter used by video encoder 20 from the received syntax element and use that interpolation filter to generate the predictive block.

[0070]

[0089] Video decoder 30 including motion compensation unit 72 may be configured to perform any of the various techniques of this disclosure, described above with respect to FIG. 1 and in more detail below. For example, motion compensation unit 72 may be configured to perform motion vector prediction using AMVP mode or merge mode in accordance with the techniques of this disclosure. Furthermore, video decoder 30 including motion compensation unit 72 may be configured to perform any combination of the motion vector candidate list construction techniques of this disclosure, described in more detail below. Entropy decoding unit 70 may decode one or more syntax elements that represent how motion information is coded for the current block.

[0071]

[0090] Assuming the syntax element indicates that merge mode is performed, motion compensation unit 72 may form a candidate list including a set of merge candidates. Motion compensation unit 72 may add candidates to the candidate list based on a particular, predetermined order. In other examples of this disclosure, motion compensation unit 72 may be configured to dynamically add candidates to the candidate list in a different order based on histogram information of motion vectors from neighboring blocks. Motion compensation unit 72 may also add additional candidates and perform candidate list pruning, as described in more detail below. Finally, motion compensation unit 72 may decode a merge index that represents which of the candidates will be used to code the motion information for the current block.

[0072]

[0091] Inverse quantization unit 76 inverse quantizes, or dequantizes, the quantized transform coefficients provided in the bitstream and entropy decoded by entropy decoding unit 70. The inverse quantization process determines the degree of quantization, and similarly, depends on a quantization parameter QP calculated by video encoder 30 for each video block in a video slice to determine the degree of inverse quantization to be applied. Y This may include the use of

[0073]

[0092] Inverse transform unit 78 applies an inverse transform, eg, an inverse DCT, an inverse integer transform, or a conceptually similar inverse transform process, to the transform coefficients in order to produce residual blocks in the pixel domain.

[0074]

[0093] After motion compensation unit 72 generates a predictive block for a current video block based on the motion vector and other syntax elements, video decoder 30 forms a decoded video block by adding a residual block from inverse transform unit 78 with the corresponding predictive block generated by motion compensation unit 72. Adder 80 represents one or more components that perform this addition operation. If desired, a deblocking filter may also be applied to filter the decoded block to remove blockiness artifacts. Other loop filters may also be used (either during or after the coding loop) to smooth pixel transitions or possibly improve video quality. The decoded video blocks of a given frame or picture are then stored in reference picture memory 82, which stores reference pictures used for subsequent motion compensation. Reference picture memory 82 also stores decoded video for later presentation on a display device, such as display device 32 of FIG. 1.

[0075]

[0094] In this manner, video decoder 30 represents an example of a video coder configured to derive a histogram of motion vector information from neighboring blocks relative to a current block, determine merge candidates for a motion vector candidate list for motion vector prediction for the current block based on the derived histogram, order the motion vector candidate list based on the derived histogram, and perform merge vector prediction using the motion vector candidate list.

[0076]

[0095] The following sections describe some aspects of video coding techniques and standards, particularly with regard to motion vector prediction and related techniques. First, motion information is described. For each block of video data coded using an inter-prediction mode, a set of motion information may be available. The set of motion information includes motion information for a forward prediction direction and a backward prediction direction. Here, the forward prediction direction and the backward prediction direction are two prediction directions corresponding to reference picture list 0 (RefPicList0) and reference picture list 1 (RefPicList1) of the current picture or slice. The terms "forward" and "backward" do not necessarily have a geometric meaning. Instead, they are used to distinguish which reference picture list a motion vector is based on. Forward prediction refers to a prediction formed based on reference list 0, and backward prediction refers to a prediction formed based on reference list 1. When both reference list 0 and reference list 1 are used to form a prediction for a given block, it is called bidirectional prediction.

[0077]

[0096] If only one reference picture list is used for a given picture or slice, every block in the picture or slice is forward predicted. If both reference picture lists are used for a given picture or slice, the blocks in the picture or slice can be forward predicted, backward predicted, or bidirectionally predicted.

[0078]

[0097] For each prediction direction, the motion information also includes a reference index and a motion vector. The reference index is used to identify a reference picture in a corresponding reference picture list (e.g., RefPicList0 or RefPicList1). The motion vector has both horizontal and vertical components, each component indicating an offset value along the horizontal and vertical directions, respectively. The motion vector indicates the position of a predictor block relative to the position of the current block being coded. The reference index indicates the picture containing the predictor block. In some descriptions, for simplicity, the term "motion vector" may be used interchangeably with motion information to refer to both a motion vector and its associated reference index.

[0079]

[0098] In video coding standards, picture order count (POC) is widely used to identify the display order of pictures. Although there are cases where two pictures in one coded video sequence can have the same POC value, this does not generally occur within a coded video sequence. When multiple coded video sequences are present in a bitstream, pictures with the same value of POC may be close to each other in terms of decoding order. The POC value of a picture is generally used for reference picture list construction, derivation of reference picture sets as in HEVC, and motion vector scaling.

[0080]

[0099] The following section describes the macroblock (MB) structure in Advanced Video Coding (AVC) (H.264). In H.264 / AVC, each inter macroblock (MB) (e.g., an MB coded using inter prediction) can be partitioned in four different ways: One 16x16MB partition Two 16x8MB partitions Two 8x16MB partitions Four 8x8MB partitions

[0100] Different MB partitions within one MB may have different reference index values ​​(RefPicList0 or RefPicList1) for each prediction direction. When an MB is not partitioned into four 8x8 MB partitions, the MB has only one motion vector in each prediction direction for each MB partition.

[0081]

[0101] When an MB is partitioned into four 8x8MB partitions, each 8x8MB partition can be further partitioned into sub-blocks, each of which can have a different motion vector for each prediction direction. There are four different ways to divide an 8x8MB partition into sub-blocks: One 8x8 sub-block Two 8x4 sub-blocks Two 4x8 sub-blocks Four 4x4 sub-blocks

[0102] Each sub-block may have a different motion vector for each prediction direction, and therefore the motion vectors are at levels equal to or higher than the sub-block.

[0082]

[0103] Temporal direct mode in AVC is not described. In AVC, temporal direct mode can be enabled at either the MB level or the MB partition level for skip or direct mode in B slices. For each MB partition, the motion vectors of the blocks co-located with the current MB partition in the current block's RefPicList1[0] are used to derive a motion vector. Each motion vector in the co-located block is scaled based on the POC distance. In AVC, direct mode can also predict motion information from spatial neighbors.

[0083]

[0104] Next, the coding unit (CU) structure in HEVC will be described. In HEVC, the largest coding unit in a slice is called a coding tree block (CTB) or coding tree unit (CTU). The CTB contains a quadtree whose nodes are coding units. The CTB can be recursively split into CUs in a quadtree manner, as described in W. J. Han et al., "Improved Video Compression Efficiency Through Flexible Unit Representation and Corresponding Extension of Coding Tools," IEEE Transaction on Circuits and Systems for Video Technology, vol. 20, no. 12, pp. 1709-1720, December 2010, and shown in FIG. 4. As shown in FIG. 4, each level of partitioning is a quadtree split into four sub-blocks. The black blocks are examples of leaf nodes (i.e., blocks that are not further split).

[0084]

[0105] The size of the CTB can range from 16x16 to 64x64 in the HEVC Main Profile (although technically, an 8x8 CTB size can be supported). CUs can be the same size of CTB, but as small as 8x8. Each CU is coded using one mode (e.g., intra-prediction mode or inter-prediction mode). When a CU is inter-coded, it can be further partitioned into two or four prediction units (PUs), or can be just one PU when no further partitioning applies. When there are two PUs in a CU, the PUs can be a half-size rectangle, or two rectangles that are 1 / 4 or 3 / 4 the size of the CU.

[0085]

[0106] When a CU is inter-coded, there is one set of motion information (e.g., motion vector, prediction direction, and reference picture) for each PU. Furthermore, each PU is coded using a unique inter-prediction mode to derive the set of motion information. However, it should be understood that even if two PUs are uniquely coded, they may still have the same motion information in some situations.

[0086]

[0107] In HEVC, as shown in FIG. 5, there are eight partition modes for CUs coded using inter prediction modes: PART_2N×2N, PART_2N×N, PART_N×2N, PART_N×N, PART_2N×nU, PART_2N×nD, PART_nL×2N, and PART_nR×2N. A CU coded using the partition mode PART_2N×2N is not further split. That is, the entire CU is treated as a single PU (PU0). A CU coded using the partition mode PART_2N×N is symmetrically split horizontally into two PUs (PU0 and PU1). A CU coded using the partition mode PART_N×2N is symmetrically split vertically into two PUs. A CU coded using the partition mode PART_N×N is symmetrically split into four equally sized PUs (PU0, PU1, PU2, PU3).

[0087]

[0108] A CU coded using the partitioning mode PART_2N×nU is asymmetrically split horizontally into one PU0 (top PU) having 1 / 4 the size of the CU and one PU1 (bottom PU) having 3 / 4 the size of the CU. A CU coded using the partitioning mode PART_2N×nD is asymmetrically split horizontally into one PU0 (top PU) having 3 / 4 the size of the CU and one PU1 (bottom PU) having 1 / 4 the size of the CU. A CU coded using the partitioning mode PART_nL×2N is asymmetrically split vertically into one PU0 (left PU) having 1 / 4 the size of the CU and one PU1 (right PU) having 3 / 4 the size of the CU. A CU coded using the partitioning mode PART_nR×2N is asymmetrically split vertically into one PU0 (left PU) having 3 / 4 the size of the CU and one PU1 (right PU) having 1 / 4 the size of the CU.

[0088]

[0109] Although HEVC uses a quadtree partitioning structure, other partitioning structures are being researched for future video coding standards. For example, in J. An et al., "Block partitioning structure for next generation video coding," International Telecommunication Union, COM16-C966, September 2015 (hereinafter "VCEG Proposal COM16-C966"), a quadtree-binary-tree (QTBT) partitioning technique was proposed for future video coding standards beyond HEVC (e.g., H.266). Simulations showed that the proposed QTBT structure is more efficient than the quadtree structure used in HEVC for some video sequences.

[0089]

[0110] In the proposed QTBT structure of VCEG proposal COM16-C966, the CTB is first partitioned using a quadtree partitioning technique, where quadtree splitting of one node may be repeated until the node reaches the minimum allowed quadtree leaf node size. The minimum allowed quadtree leaf node size may be indicated to a video decoder by the value of the syntax element MinQTSize. If the quadtree leaf node size is not larger than the maximum allowed binary tree root node size (e.g., indicated by the syntax element MaxBTSize), the quadtree leaf node may be further partitioned using binary tree partitioning. The binary tree partitioning of one node may be repeated until the node reaches the minimum allowed binary tree leaf node size (e.g., as indicated by the syntax element MinBTSize) or the maximum allowed binary tree depth (e.g., as indicated by the syntax element MaxBTDepth). VCEG proposal COM16-C966 uses the term "CU" to refer to a binary tree leaf node. In VCEG proposal COM16-C966, CUs are used for prediction (e.g., intra-prediction, inter-prediction, etc.) and transformation without any further partitioning. Generally, according to the QTBT technique, there are two split types for binary tree splitting: symmetric horizontal splitting and symmetric vertical splitting. In each case, a block is split by dividing the block in half (down the middle) either horizontally or vertically.

[0090]

[0111] In one example of a QTBT partitioning structure, the CTU size is set as 128x128 (e.g., a 128x128 luma block and two corresponding 64x64 chroma blocks), MinQTSize is set as 16x16, MaxBTSize is set as 64x64, MinBTSize (for both width and height) is set as 4, and MaxBTDepth is set as 4. Quadtree partitioning is first applied to the CTU to generate quadtree leaf nodes. The quadtree leaf nodes may have sizes from 16x16 (i.e., MinQTSize is 16x16) to 128x128 (i.e., the CTU size). According to one example of QTBT partitioning, if the leaf quadtree node is 128x128, the leaf quadtree node cannot be further split by a binary tree because the size of the leaf quadtree node exceeds MaxBTSize (i.e., 64x64). Otherwise, the leaf quadtree node is further partitioned by the binary tree. Therefore, the quadtree leaf node is also the root node for the binary tree and has the binary tree depth as 0. The binary tree depth reaching MaxBTDepth (e.g., 4) implies no further splitting. The binary tree node having a width equal to MinBTSize (e.g., 4) implies no further horizontal splitting. Similarly, the binary tree node having a height equal to MinBTSize implies no further vertical splitting. The leaf node (CU) of the binary tree is further processed (e.g., by performing a prediction process and a transformation process) without any further partitioning.

[0091]

[0112] FIG. 6A shows an example of a block 150 (e.g., CTB) partitioned using the QTBT partitioning technique. As shown in FIG. 6A, using the QTBT partitioning technique, each of the resultant blocks is split symmetrically through the center of each block. FIG. 6B shows a tree structure corresponding to the block partitioning of FIG. 6A. The solid lines in FIG. 6B indicate quadtree splitting, and the dotted lines indicate binary tree splitting. In one example, at each splitting (i.e., non-leaf) node of the binary tree, a syntax element (e.g., a flag) is signaled to indicate the type of splitting (e.g., horizontal or vertical) to be performed, where 0 indicates horizontal splitting and 1 indicates vertical splitting. In the case of quadtree splitting, there is no need to indicate the splitting type, as quadtree splitting always splits a block horizontally and vertically into four sub-blocks of equal size.

[0092]

[0113] As shown in FIG. 6B, at node 170, block 150 is split using QT partitioning into four blocks 151, 152, 153, and 154, shown in FIG. 6A. Block 154 is not further split and is therefore a leaf node. At node 172, block 151 is further split into two blocks using BT partitioning. As shown in FIG. 6B, node 172 is marked with a 1, indicating a vertical split. Thus, the split at node 172 results in block 157 and a block containing both blocks 155 and 156. Blocks 155 and 156 are created by a further vertical split at node 174. At node 176, block 152 is further split into two blocks 158 and 159 using BT partitioning. As shown in FIG. 6B, node 176 is marked with a 1, indicating a horizontal split.

[0093]

[0114] At node 178, block 153 is split into four equally sized blocks using QT partitioning. Blocks 163 and 166 are created from this QT partitioning and are not further split. At node 180, the top-left block is first split using vertical binary tree splitting, resulting in block 160 and the right vertical block. The right vertical block is then split into blocks 161 and 162 using horizontal binary tree splitting. The bottom-right block created from quad-tree splitting at node 178 is split at node 184 into blocks 164 and 165 using horizontal binary tree splitting.

[0094]

[0115] The motion vector candidate list construction techniques described in more detail below may be used in conjunction with any video block partitioning technique, including the MB partitioning structure of H.264 / AVC, the quadtree partitioning structure of HEVC, or a QTBT partitioning structure such as the QTBT structure proposed for H.266.

[0095]

[0116] Next, motion prediction in HEVC will be described. The HEVC standard has two inter-prediction modes for a PU, referred to as merge mode (skip is considered a special case of merge) and advanced motion vector prediction (AMVP) mode, respectively. In either AMVP mode or merge mode, video encoder 20 and video decoder 30 are configured to build a motion vector (MV) candidate list for multiple motion vector predictors. The motion vector predictors may be motion vectors from neighboring blocks or artificially generated motion vectors, which may be used to predict a motion vector for a currently coded block of video data. For merge mode, instead of encoding the motion vector of the current block itself, video encoder 20 selects one motion vector from the motion vector candidate list, as well as a reference index associated with that candidate, and uses that candidate motion vector for inter prediction. Video encoder 20 may signal an index (e.g., a merge index) of the selected motion vector candidate to video decoder 30. Video decoder 30 may build a motion vector candidate list for merge mode in the same manner as video encoder 20. Video decoder 30 may use the signaled index into the motion vector candidate list to identify the selected candidate, and then retrieve the motion vector and a reference index associated with that candidate to use as the motion vector for the current block.

[0096]

[0117] The MV candidate list includes up to five candidates for merge mode and two candidates for AMVP mode. A merge candidate may include a set of motion information, e.g., a motion vector corresponding to both a reference picture list (List 0 and List 1) and a reference index. When a merge candidate is identified by a merge index, the reference picture is used for prediction of the current block, and an associated motion vector is determined. However, under AMVP mode, for each potential prediction direction from either List 0 or List 1, the reference index is explicitly signaled along with the motion vector predictor (MVP) index to the MV candidate list, since the AMVP candidate includes only a motion vector. In AMVP mode, the predicted motion vector can be further refined. In some examples of AMVP, video encoder 20 may also signal a motion vector difference (MVD). The MVD is the difference between the selected MVP and the actual determined motion vector for the current block. As seen above, a merge candidate corresponds to a full set of motion information, while an AMVP candidate includes only one motion vector for a particular prediction direction and reference index.

[0097]

[0118] Figure 7 is a conceptual diagram showing spatial neighboring candidates in HEVC. Spatial MV candidates are derived from the neighboring blocks shown on Figure 7 for a particular PU (PU0), but the method of generating candidates from blocks differs for merge mode and AMVP mode.

[0098]

[0119] In merge mode, up to four spatial MV candidates may be derived in the order shown in Figure 7(a). The order is left (0,A1), top (1,B1), top right (2,B0), bottom left (3,A0), and top left (4,B2) as shown in Figure 7(a). That is, in Figure 7(a), block 200 includes PU0 204A and PU1 204B. When a video coder (e.g., video encoder 20 and / or video decoder 30) is to code motion information for PU0 204A using merge mode, the video coder adds motion information from spatial neighboring blocks 208A, 208B, 208C, 208D, and 208E to a candidate list in that order. Blocks 208A, 208B, 208C, 208D, and 208E are sometimes referred to as blocks A1, B1, B0, A0, and B2, respectively, in HEVC.

[0099]

[0120] In AVMP mode, neighboring blocks are divided into two groups: a left group including blocks 0 and 1, and an upper group including blocks 2, 3, and 4, as shown in FIG. 7(b). These blocks are labeled as blocks 210A, 210B, 210C, 210D, and 210E in FIG. 7(b), respectively. Block 202 includes PU0 206A and PU1 206B, and blocks 210A, 210B, 210C, 210D, and 210E represent spatial neighbors to PU0 206A. For each group, potential candidates among neighboring blocks that reference the same reference picture as the reference picture indicated by the signaled reference index have the highest priority to be selected to form the final candidates of the group. It is possible that not all neighboring blocks contain motion vectors pointing to the same reference picture. Therefore, if no such candidate can be found, the first available candidate will be scaled to form the final candidate, and thus the time distance difference can be compensated.

[0100]

[0121] Figure 8 is a conceptual diagram illustrating temporal motion vector prediction (TMVP) in HEVC. In particular, Figure 8(a) shows an example CU 220 including PU0 222A and PU1 222B. PU0 222A includes a center block 226 for PU 222A and a bottom-right block 224 for PU0 222A. Figure 8(a) also shows an outer block 228 for which motion information can be predicted from the motion information of PU0 222A, as described below. Figure 8(b) shows a current picture 230 including a current block 238 for which motion information should be predicted. In particular, Figure 8(b) shows a co-located picture 234 for the current picture 230 (including a co-located block 240 for the current block 238), a current reference picture 232, and a co-located reference picture 236. The co-located block 240 is predicted using a motion vector 244 , which is used as a temporal motion vector predictor (TMVP) 242 for the motion information of block 238 .

[0101]

[0122] A video coder (e.g., video encoder 20 and / or video decoder 30) may add a TMVP candidate (e.g., TMVP candidate 242) to the MV candidate list after any spatial motion vector candidate if TMVP is enabled and a TMVP candidate is available. The process of motion vector derivation for a TMVP candidate is the same for both merge mode and AMVP mode. However, the target reference index for a TMVP candidate in merge mode is set to 0, in accordance with HEVC.

[0102]

[0123] The primary block location for TMVP candidate derivation is the outer, bottom-right block of the co-located PU, as shown in Figure 8(a) as block 224 for PU0 222A, to compensate for the bias toward the above and left blocks used to generate spatial neighbor candidates. However, if block 224 is located outside the current CTB row or motion information is not available for block 224, the block is replaced with the center block 226 of the PU, as shown in Figure 8(a). The motion vector for TMVP candidate 242 is derived from co-located block 240 of co-located picture 234, as indicated in the slice-level information.

[0103]

[0124] Similar to the temporal direct mode in AVC, the motion vectors of the TMVP candidates may undergo motion vector scaling, which is performed to compensate for the POC distance differences between the current picture 230 and the current reference picture 232, and between the co-located picture 234 and the co-located reference picture 236. That is, the motion vectors 244 may be scaled to generate the TMVP candidates 242 based on these POC differences. Other aspects of motion prediction in HEVC are described below.

[0104]

[0125] Motion vector scaling can be performed in HEVC. The value of a motion vector is assumed to be proportional to the distance of pictures in presentation time. A motion vector relates two pictures: a reference picture and a picture that contains the motion vector (i.e., a containing picture). When a motion vector is used to predict another motion vector, the distance between the containing picture and the reference picture is calculated based on the Picture Order Count (POC) value.

[0105]

[0126] For a motion vector to be predicted, both its associated containing picture and reference picture may be different. Therefore, a new distance (based on POC) is calculated. And the motion vector is scaled based on these two POC distances. For spatially adjacent candidates, the containing picture for two motion vectors is the same, but the reference picture is different. In HEVC, motion vector scaling is applied to both TMVP and AMVP for spatial and temporally adjacent candidates.

[0106]

[0127] Also, artificial motion vector candidate generation may be performed in HEVC. If the motion vector candidate list is not complete (e.g., contains fewer than a prescribed number of candidates), video encoder 20 and / or video decoder 30 may generate artificial motion vector candidates. Video encoder 20 and / or video decoder 30 generate and insert artificial motion vector candidates at the end of the motion vector candidate list until the motion vector candidate list has the prescribed number of candidates.

[0107]

[0128] In merge mode, there can be two types of pseudo MV candidates: bidirectional combined pseudo motion vector candidates derived for B slices, and 0 motion vector candidates that are only used for AMVP when the first type (i.e., combined pseudo motion vector candidates) does not provide enough pseudo candidates to fill the candidate list.

[0108]

[0129] For each pair of candidates already in the candidate list and with the necessary motion information, a bidirectional combined motion vector candidate is derived by combining the motion vector of the first candidate that references a picture in list 0 and the motion vector of the second candidate that references a picture in list 1. The 0 motion vector candidate simply points to a co-located block in another picture (e.g., the 0 motion vector candidate is (0,0)).

[0109]

[0130] A video coder configured according to HEVC may also perform a pruning process for candidate insertion. Motion vector candidates from different blocks may happen to be the same, which reduces the efficiency of the merge / AMVP candidate list. That is, having multiple motion vector candidates with the same value reduces the actual number of motion vectors to be tested and selected. To solve this problem, a pruning process may be applied. The pruning process compares a motion vector candidate with other motion vector candidates currently in the candidate list to avoid inserting an equivalent candidate in some situations. To reduce complexity, instead of comparing each potential motion vector candidate with all other existing motion vector candidates, only a limited number of pruning processes are applied.

[0110]

[0131] Figure 9 shows an example prediction structure for 3D-HEVC. 3D-HEVC is a 3D video extension of HEVC under development by JCT-3V. Some techniques related to the techniques of this disclosure are described with reference to Figures 9 and 10 below. Figure 9 shows a multiview prediction structure for three views. V3 indicates the base view, and pictures in non-base views (V1 or V5) can be predicted from pictures in dependent (base) views of the same time instance. Inter-view sample prediction (from reconstructed samples) is supported in multiview HEVC (MV-HEVC), and its general prediction structure is shown in Figure 10.

[0111]

[0132] Both MV-HEVC and 3D-HEVC conform to HEVC such that the base (texture) views are decodable by an HEVC (Version 1) decoder. Test models for MV-HEVC and 3D-HEVC are described in Zhang et al., "Test Model 6 of 3D-HEVC and MV-HEVC," JCT-3V document ISO / IEC JTC1 / SC29 / WG11 N13940, available as of January 26, 2015, at the website mpeg.chiariglione.org / standards / mpeg-h / high-efficiency-video-coding / test-model-6-3d-hevc-and-mv-hevc.

[0112]

[0133] In MV-HEVC, a current picture in a non-base view can be predicted by pictures in the same view and in reference views of the same time instance by including all of these pictures in the reference picture list of that picture. Thus, the reference picture list of the current picture includes both temporal reference pictures and inter-view reference pictures. A motion vector associated with a reference index corresponding to a temporal reference picture is denoted as a temporal motion vector. A motion vector associated with a reference index corresponding to an inter-view reference picture is denoted as a disparity motion vector. 3D-HEVC supports all features in MV-HEVC. Therefore, inter-view sample prediction as described above is enabled.

[0113]

[0134] Additionally, 3D-HEVC supports more advanced texture-only coding tools and depth-related / dependent coding tools. Texture-only coding tools often use identification of corresponding blocks (between views) that may belong to the same object. Disparity vector derivation is therefore one technique used in 3D-HEVC.

[0114]

[0135] Figure 10 is a conceptual diagram illustrating sub-PU-based inter-view motion prediction in 3D-HEVC. Figure 10 shows a current picture 360 ​​of a current view (V1) and a co-located picture 362 in a reference view (V0). The current picture 360 ​​includes a current PU 364, which includes four sub-PUs 366A-366D (sub-PUs 366). Each disparity vector 374A-374D (disparity vector 374) identifies a corresponding sub-PU 368A-368D to the sub-PU 366 in the co-located picture 362. In 3D-HEVC, the sub-PU level inter-view motion prediction method is used for inter-view merge candidates, i.e., candidates derived from reference blocks in the reference view.

[0115]

[0136] When such a mode is enabled, the current PU 364 may correspond to a reference area (having the same size as the current PU identified by the disparity vector) in the reference view, which may have richer motion information (e.g., many different associated motion vectors) than required for generating one set of motion information for the PU. Therefore, as shown in FIG. 10, a sub-PU level inter-view motion prediction (SPIVMP) method may be used. This mode may also be signaled as a special merge candidate. Each of the sub-PUs contains a full set of motion information. Therefore, a PU may contain multiple sets of motion information.

[0116]

[0137] Also, in 3D-HEVC, sub-PU-based motion parameter inheritance (MPI) may be used. In 3D-HEVC depth coding, MPI candidates derived from texture views may also be extended in a manner similar to sub-PU level inter-view motion prediction. For example, if a current depth PU has a collocated region containing multiple PUs, the current depth PU may be separated into sub-PUs, and each sub-PU may have a different set of motion information. This method is called sub-PU MPI.

[0117]

[0138] Exemplary sub-PU related techniques for 2D video coding are described in U.S. Application No. 14 / 497,128, filed September 25, 2014, which is incorporated herein by reference in its entirety. In U.S. Application No. 14 / 497,128, a sub-PU-based advanced TMVP (ATMVP) design was proposed.

[0118]

[0139] In single-layer coding, a two-stage advanced temporal motion vector prediction design may be used. The first stage is utilized to derive a vector identifying the corresponding block of the current prediction unit (PU) in the reference picture, and the second stage is to extract multiple sets of motion information from the corresponding block and assign them to the sub-PUs of the PU. Each sub-PU of the PU is therefore motion compensated separately. The ATMVP concept is summarized as follows: (1) The vector in the first stage may be derived from the spatial and temporal neighboring blocks of the current PU. (2) This process may be achieved by activating a merge candidate among all other merge candidates. While applicable to single-layer coding and sub-PU temporal motion vector prediction, the PU or CU may have motion refinement data to be propagated on the predictor.

[0119]

[0140] Some aspects of U.S. Application No. 14 / 497,128 are highlighted as follows: 1. The first stage of vector derivation can also be simplified by just a 0 vector. 2. The first stage of vector derivation can include jointly identifying a motion vector and its associated picture. Various methods have been proposed for selecting the associated picture and further determining which motion vector is the first-stage vector. 3. If motion information during the above process is unavailable, the "first-stage vector" is used for substitution. 4. The motion vector identified from the temporal neighbor can be scaled to be used for the current sub-PU in a manner similar to motion vector scaling in TMVP. However, which reference picture such a motion vector can be scaled to can be designed using one of the following methods:

[0120] The picture is identified by a fixed reference index of the current picture.

[0121] b. If the picture is also available in the reference picture list of the current picture, it is identified as a reference picture of the corresponding temporal neighbor.

[0122] c. The picture is set to be the co-located picture identified in the first stage and from where the motion vector was captured.

[0123]

[0141] To address some of the design issues in U.S. Application No. 14 / 497,128, the following technique was proposed in U.S. Application No. 15 / 005,564, filed January 25, 2016, the entire contents of which are incorporated herein by reference.

[0124] 1. Position of ATMVP candidates when inserted, for example, as a merge candidate list: a. Assume that spatial and TMVP candidates are inserted into the merge candidate list in a certain order. ATMVP candidates can be inserted into any relatively fixed position of those candidates.

[0125] i. In one alternative, for example, the ATMVP candidate may be inserted into the merge candidate list after the first two spatial candidates, eg, A1 and B1.

[0126] ii. In one alternative, for example, the ATMVP candidate may be inserted after the first three spatial candidates, eg, A1, B1 and B0.

[0127] iii. In one alternative, for example, the ATMVP candidate may be inserted after the first four candidates, eg, A1, B1, B0, and A0.

[0128] iv. In one alternative, for example, the ATMVP candidate may be inserted immediately before the TMVP candidate.

[0129] v. In one alternative, for example, the ATMVP candidate may be inserted immediately after the TMVP candidate.

[0130] Alternatively, the position of the ATMVP candidate in the candidate list may be signaled in the bitstream. Additionally, the positions of other candidates, including TMVP candidates, may be signaled.

[0131] 2. The availability check of ATMVP candidates can be applied by accessing only one set of motion information. If such a set of information is unavailable, for example, when one block is intra-coded, all ATMVP candidates are considered unavailable. In that case, the ATMVP is not inserted into the merge list.

[0132] a. Purely to check the availability of ATMVP candidates, a central position or central sub-PU is used. When a central sub-PU is used, it is selected to cover the central position (e.g., a central 3-position with relative coordinates of (W / 2, H / 2) to the top-left sample of the PU, where W×H is the size of the PU). Such a position or central sub-PU can be used together with a temporal vector to identify a corresponding block in the motion source picture. A set of motion information from blocks covering the central position of the corresponding block is identified.

[0133] 3. A representative set of motion information for ATMVP coded PUs from sub-PUs.

[0134] a. To form ATMVP candidates, a representative set of motion information is first formed.

[0135] b. Such a representative set of motion information may be derived from fixed positions or fixed sub-PUs, and may be selected in a manner similar to that of the set of motion information used to determine the availability of ATMVP candidates, as described in Clause 2.

[0136] c. A sub-PU has identified its own set of motion information, and when unavailable, it is set equal to the representative set of motion information.

[0137] d. If the representative set of motion information is set to be that of the sub-PU, no additional motion storage is required at the decoder side for the current CTU or slice in the worst-case scenario.

[0138] e. Such a representative set of motion information is used in all scenarios, including pruning, when the decoding process requires that all PUs be represented by one set of motion information, so that the process is used to generate combined bi-predictive merging candidates.

[0139] 4. ATMVP candidates are pruned using TMVP candidates, and the interaction between TMVP and ATMVP can be considered, and the detailed technique is described below.

[0140] a. Pruning of sub-PU-based candidates, e.g., ATMVP candidates, using normal candidates may be done by using a representative set of motion information for such sub-PU-based candidates (as in Clause 3). If such set of motion information is the same as the normal merge candidate, the two candidates are considered to be the same.

[0141] b. Alternatively, or additionally, a check is performed to determine whether the ATMVP includes multiple different sets of motion information for multiple sub-PUs, and if at least two different sets are identified, the sub-PU-based candidate is not used for pruning, i.e., is considered different from other candidates. Otherwise, it may be used for pruning (e.g., may be pruned during the pruning process).

[0142] c. Alternatively, additionally, ATMVP candidates may be pruned using only spatial candidates, eg, left and top spatial candidates, with positions indicated as A1 and B1.

[0143] d. Alternatively, only one candidate, either an ATMVP candidate or a TMVP candidate, is formed from the temporal reference. When ATMVP is available, the candidate is ATMVP; otherwise, the candidate is TMVP. Such a candidate is inserted into the merge candidate list at a position similar to that of TMVP. In this case, the maximum number of candidates can be kept unchanged.

[0144] i. Alternatively, TMVP is always disabled even when ATMVP is unavailable.

[0145] ii. Alternatively, TMVP is used only when ATMVP is unavailable.

[0146] Alternatively, when ATMVP is available and TMVP is unavailable, one set of motion information of one sub-PU is used as a TMVP candidate. In this case, the pruning process between ATMVP and TMVP is not applied.

[0147] f. Alternatively or additionally, the time vector used for ATMVP may also be used for TMVP, such that the bottom right or center 3 positions as currently used for TMVP in HEVC do not need to be used.

[0148] i. Alternatively, the location identified by the time vector and the bottom right and center three locations are considered together to give usable TMVP candidates.

[0149] 5. Multiple availability checks for ATMVP are supported to provide a higher likelihood that the ATMVP candidate will be more accurate and efficient. When the current ATMVP candidate from the motion source picture as identified by the first time vector (e.g., as shown in FIG. 9) is unavailable, another picture may be considered as the motion source picture. When another picture is considered, it may be associated with a different second time vector, or simply associated with a second time vector that is scaled from the first time vector that points to the unavailable ATMVP candidate.

[0150] a. The second temporal vector can identify an ATMVP candidate in the second motion source picture, and the same availability check can be applied. If an ATMVP candidate is available as derived from the second motion source picture, the ATMVP candidate is derived and no other pictures need to be checked; otherwise, other pictures as motion source pictures need to be checked.

[0151] b. The pictures to be examined may be pictures in the reference picture lists of the current picture, with a given order. For each list, the pictures are examined in ascending order of reference index. List X is examined first, followed by pictures in list Y (which is 1-X).

[0152] i. List X is chosen such that list X is the list containing the co-located pictures used for TMVP.

[0153] ii. Alternatively, X is simply set to be 1 or 0.

[0154] c. The pictures to be examined are the pictures identified by the motion vectors of their spatial neighbors with a given order.

[0155] 6. The partition of the PU to which ATMVP is currently applied can be an asymmetric motion partition (AMP) partition such as 2Nx2N, NxN, 2NxN, Nx2N, or 2NxN / 2.

[0156] a. Alternatively, and in addition, ATMVP may also be supported where other partition sizes may be allowed, such sizes may include, for example, 64x8.

[0157] b. Alternatively, the mode may only be applied to some partitions, for example 2N×2N.

[0158] 7. ATMVP candidates are marked as different types of merge candidates.

[0159] 8. When identifying a vector (a temporal vector as in the first stage) from a neighbor, multiple neighboring positions, for example, neighboring positions used in merging candidate list construction, may be examined in sequence. For each neighbor, the motion vector corresponding to reference picture list 0 (List 0) or reference picture list 1 (List 1) may be examined in sequence. When two motion vectors are available, the motion vector in list X may be examined first, followed by list Y (where Y is equal to 1-X), so that list X becomes the list containing the co-located pictures used for TMVP. In ATMVP, a temporal vector is additionally used as a shift of the center position of the sub-PU, where the components of the temporal vector may need to be shifted by an integer. Such a shifted center position is used to identify the smallest unit to which a motion vector can be allocated, for example, with a size of 4x4 covering the current center position.

[0160] Alternatively, the motion vectors corresponding to list 0 may be examined before the motion vectors corresponding to list 1.

[0161] b. Alternatively, the motion vectors corresponding to list 1 may be examined before the motion vectors corresponding to list 0.

[0162] c. Alternatively, all motion vectors corresponding to list X in all spatial neighbors are examined in order, followed by the motion vector corresponding to list Y (where Y equals 1-X), where list "X" is a list indicating where the co-located picture belongs, or can simply be set to be 0 or 1.

[0163] d. The order of spatial neighbors may be the same as the order used in HEVC merge mode.

[0164] 9. In the first stage of identification, when the time vector does not contain information identifying a reference picture, the motion source picture as shown in FIG. 9 may simply be set to be a fixed picture, for example, a co-located picture used for TMVP.

[0165] In such a case, vectors can be identified only from motion vectors that point to such fixed pictures.

[0166] b. In such a case, the vectors are only distinguished from motion vectors that point to any picture, but can also be scaled towards a fixed picture.

[0167] 10. When in the first stage of identifying a vector, rather than identifying a reference picture, a motion source picture as shown in Figure 9, one or more of the following additional tests may be applied to the candidate motion vector:

[0168] a. If a motion vector is associated with a picture or slice that is intra-coded, such motion vector may be considered unavailable and not used to be converted to that vector.

[0169] b. If a motion vector identifies an intra-block in the associated picture (e.g., by adding the current center coordinate with the motion vector), such motion vector may be considered unavailable and may not be used to convert to that vector.

[0170] 11. When in the first stage of identifying a vector, the components of the vector can be set to be (1 / 2 width of current PU, 1 / 2 height of current PU) so that it identifies the bottom right pixel position in the motion source picture, where (x, y) indicate the horizontal and vertical components of one motion vector.

[0171] a. Alternatively, the components of the vector may be set to be (sum(half width of current PU, M), sum(half height of current PU, N)), where the function sum(a, b) returns the sum of a and b. In one example, when motion information is stored in 4x4 units, M and N are both set to be equal to 2. In another example, when motion information is stored in 8x8 units, M and N are both set to be equal to 4.

[0172] 12. The sub-block / sub-PU size when ATMVP is applied is signaled in a parameter set, e.g., a sequence parameter set of a picture parameter set. The size ranges from the smallest PU size to the CTU size. The size may also be predefined or signaled. The size may be as small as 4x4, for example. Alternatively, the sub-block / sub-PU size may be derived based on the size of the PU or CU. For example, the sub-block / sub-PU may be set equal to max(4x4, (CU width) >> M). The value of M may be predefined or signaled in the bitstream.

[0173] 13. The maximum number of merge candidates may be increased by 1, with ATMVP being considered as a new merge candidate. For example, the maximum number of merge candidates may be increased to 6, compared to HEVC, which requires a maximum of 5 candidates in the merge candidate list after pruning.

[0174] a. Alternatively, pruning with or unification with conventional TMVP candidates can be performed for ATMVP so that the maximum number of merge candidates can be kept constant.

[0175] b. Alternatively, when ATMVP is identified as available, spatially adjacent candidates are removed from the merge candidate list, for example, the last spatially adjacent candidate in fetching order is removed.

[0176] 14. When multiple spatially adjacent motion vectors are considered to derive a temporal vector, the similarity of the motion vectors can be calculated based on the adjacent motion vectors of the current PU and the adjacent motion vectors identified by the particular temporal vector that is set equal to the motion vector. The one that results in the highest motion similarity can be selected as the final temporal vector.

[0177] a. In one alternative, for each motion vector from neighboring location N, the motion vector identifies a block (the same size as the current PU) in the motion source picture, whose neighboring location N contains a set of motion information. This set of motion vectors is compared to the set of motion information as for neighboring location N of the current block.

[0178] b. In another alternative, for each motion vector from neighboring location N, the motion vector identifies a block in the motion source picture, whose neighboring locations contain multiple sets of motion information. These multiple sets of motion vectors are compared with multiple sets of motion information from neighboring locations of the current PU at the same relative locations. The similarity of the motion information is calculated. For example, the current PU has the following sets of motion information from A1, B1, A0, and B0, denoted as MIA1, MIB1, MIA0, and MIB0. For the temporal vector TV, it identifies a block corresponding to the PU in the motion source picture. Such a block has motion information from the same relative A1, B1, A0, and B0 locations, denoted as TMIA1, TMIB1, TMIA0, and TMIB0. The motion similarity determined by TV is

[0179]

number

[0180] where MVSim defines the similarity between two sets of motion information.

[0181] c. In both of the above cases, a motion similarity measure MVSim may be used, where the two input parameters are two sets of motion information, each containing up to two motion vectors and two reference indices. Each pair of motion vectors in list X actually relates to different pictures, i.e., reference pictures in different lists X of the current picture and the motion source picture. For each of two motion vectors MVXN and TMVXN (where X is equal to 0 or 1), a motion vector difference MVDXN may be calculated as MVXN-TMVXN. Then, the difference MVSimX may be calculated as, for example,

[0182]

number

[0183] or

[0184]

number

[0185] i. In order to have a unified calculation of the motion differential, both of the motion vectors need to be scaled towards the same fixed picture, which may be, for example, the first reference picture RefPicListX[0] of the list X of the current picture.

[0186] ii. If the availability of motion vectors in list X from the first set and the availability of motion vectors in list X from the second set are different, i.e., one reference index is −1 and the other reference index is not −1, then such two sets of motion information are considered to be dissimilar in direction X. If the two sets are dissimilar in both sets, the final MVSim function may return a large value T, which may be considered, for example, to be infinity.

[0187] iii. Alternatively, for a pair of sets of motion information, if one is predicted from list X (X equals 0 or 1) rather than list Y (Y equals 1-X) and the other has the same status, a weighting between 1 and 2 (e.g., MVSim equals MVSimX*1.5) may be used. When one set is predicted only from list X and the other set is predicted only from list Y, MVSim is set to a large value T.

[0188] iv. Alternatively, for any set of motion information, both motion vectors are generated as long as one motion vector is available. If only one motion vector is available (corresponding to list X), it is scaled to form the motion vector corresponding to the other list Y.

[0189] d. Alternatively, the motion vector may be measured based on the difference between the neighboring pixels of the current PU and the neighboring pixels of the block (same size as the current PU) identified by the motion vector. The motion vector that results in the smallest difference may be selected as the final temporal vector.

[0190] 15. When deriving a temporal vector for the current block, motion vectors and / or temporal vectors from neighboring blocks coded using ATMVP may have higher priority than motion vectors from other neighboring blocks.

[0191] a. In one example, only the time vectors of neighboring blocks are checked first, and the first available one can be set as the time vector of the current block. Only if no such time vector exists, are further normal motion vectors checked. In this case, the time vector for the ATMVP coded block needs to be stored.

[0192] b. In another example, only motion vectors from ATMVP-coded neighboring blocks may be checked first, and the first available one may be set as the time vector of the current block. Only if no such time vector exists are further normal motion vectors checked.

[0193] c. In another example, only motion vectors from ATMVP-coded neighboring blocks may be examined first, and the first available one may be set as the temporal vector of the current block. If no such motion vector is available, the examination of temporal vectors continues as in clause 15a.

[0194] d. In another example, the time vectors from neighboring blocks may be examined first, and the first available one may be set as the time vector of the current block. If no such motion vector is available, the examination of the time vectors continues as in clause 15b.

[0195] e. In another example, the time vectors and motion vectors of ATMVP coded neighboring blocks may be first checked, and the first available one may be set as the time vector of the current block. Only if no such time vector and motion vector exist, further regular motion vectors may be checked.

[0196] 16. When multiple spatially adjacent motion vectors are considered to derive a temporal vector, the motion vector may be selected so that it minimizes distortion calculated from the pixel domain; for example, template matching may be used to derive the temporal vector so that the one resulting in the smallest matching cost is selected as the final temporal vector.

[0197] 17. The derivation of a set of motion information from corresponding blocks (in the motion source picture) is performed such that, when a motion vector is available in the corresponding block for any list X (denoting the motion vector as MVX), for the current sub-PU of the ATMVP candidate, the motion vector is considered to be available for list X (by scaling MVX). If a motion vector is not available in the corresponding block for any list X, the motion vector is considered to be unavailable for list X.

[0198] a. Alternatively, when a motion vector in a corresponding block is unavailable for list X but available for list 1-X (where 1-X is denoted by Y and the motion vector is shown as being MVY), the motion vector is still considered to be available for list X (by scaling MVY towards the target reference picture in list X).

[0199] b. Alternatively, or additionally, when both motion vectors in the corresponding block for list X and list (equal to 1-X) Y are available, the motion vectors from list X and list Y are not used as they are needed to directly scale and generate two motion vectors for the current sub-PU by scaling.

[0200] i. In one example, when constructing ATMVP candidates, a low-latency check as done in TMVP is applied to each sub-PU. For every picture in every reference picture list (indicated by refPic) of the current slice, if the picture order count (POC) value of refPic is smaller than the POC of the current slice, the current slice is considered in low-latency mode. In this low-latency mode, motion vectors from list X and list Y are scaled to generate the current sub-PU's motion vectors for list X and list Y, respectively. When not in low-latency mode, only one motion vector MVZ from MVX or MVY is selected and scaled to generate two motion vectors for the current sub-PU. As in TMVP, in such a case, Z is set equal to collocated_from_l0_flag, which means that it depends on whether the collocated picture is in the current picture's list X or list Y as in TMVP. Alternatively, Z is set as follows: if a motion source picture is identified from list X, Z is set to X. Alternatively, additionally, when a motion source picture belongs to both reference picture lists, and RefPicList0[idx0] is the first motion source picture in list 0, and RefPicList(1)[idx1] is the first motion source picture in list 1, Z is set to 0 if idx0 is less than or equal to idx1, and set to 1 otherwise.

[0201] 18. The motion source picture may be signaled, e.g., generated, by video encoder 20 in the coded bitstream. In particular, a flag indicating whether the motion source picture is from list 0 or list 1 is signaled for a B slice. Alternatively, or additionally, a reference index into list 0 or list 1 of the current picture may be signaled to identify the motion source picture.

[0202]

[0142] When identifying a temporal vector, if it points to an intra-coded block in the associated motion source picture, the vector is considered unavailable (so that other vectors can be considered).

[0203]

[0143] Figure 11 is a conceptual diagram illustrating sub-PU motion prediction from a reference picture. In this example, current picture 380 includes current PU 384 (e.g., PU). In this example, motion vector 392 identifies PU 386 of reference picture 382 for PU 384. PU 386 is partitioned into sub-PUs 388A-388D, each having a respective motion vector 390A-390D. Thus, although current PU 384 is not actually partitioned into separate sub-PUs, in this example, current PU 384 may be predicted using motion information from sub-PUs 388A-388D. In particular, a video coder may code the sub-PUs of current PU 384 using the respective motion vectors 390A-390D. However, a video coder need not code a syntax element indicating that current PU 384 is split into sub-PUs. In this way, the current PU 384 can be effectively predicted using multiple motion vectors 390A-390D inherited from each sub-PU 388A-388D without the signaling overhead of syntax elements used to split the current PU 384 into multiple sub-PUs.

[0204]

[0144] Figure 12 is a conceptual diagram illustrating related pictures in ATMVP (similar to TMVP). In particular, Figure 12 shows a current picture 404, a motion source picture 406, and reference pictures 400 and 402. More specifically, the current picture 404 includes a current block 408. A temporal motion vector 412 identifies a corresponding block 410 in the motion source picture 406 for the current block 408. The corresponding block 410, in turn, includes a motion vector 414, which references the reference picture 402 and serves as an advanced temporal motion vector predictor for at least a portion of the current block 408, e.g., a sub-PU of the current block 408. That is, the motion vector 414 may be added as a candidate motion vector predictor for the current block 408. If selected, at least a portion of the current block 408 may be predicted using the corresponding motion vector, i.e., motion vector 416, which references the reference picture 400.

[0205]

[0145] Sub-PU-related techniques for HEVC are also described in U.S. Application No. 15 / 176,790, filed July 9, 2016, the entire contents of both of which are incorporated herein by reference. To improve performance using sub-PU motion prediction, spatio-temporal motion information (ATMVP_EXT) of neighboring sub-PUs is exploited. In this example, the motion vector for each sub-PU is derived from information of neighboring blocks in the three-dimensional domain. This means that the neighboring blocks can be spatial neighbors in the current picture or temporal neighbors in the previous coded picture. Figure 13 shows a flowchart of the spatial-temporal motion vector predictor (STMVP) derivation process. In addition to what is described below, the methods described above for ATMVP (e.g., clauses 1, 2, 3, 4, 6, 7, 12, and 13) can be directly extended to STMVP.

[0206] As shown in FIG. 13, video encoder 20 and / or video decoder 30 may be configured to obtain available motion fields from spatial or temporal neighboring blocks for the current sub-PU (430). In this context, a motion field is a collection of motion vectors selected as the best for spatially / temporally neighboring blocks. For example, a block located to the left or above the current block has already been coded, and the best motion vector is available before coding the current sub-PU. The available motion information from neighboring blocks is equivalent in both video encoder 20 and video decoder 30. The motion information includes one or two three-dimensional vectors (MVx, Mvy, temporal direction), i.e., one vector for uni-prediction and two vectors for bi-prediction. Video encoder 20 and / or video decoder 30 then derives the motion information from the obtained neighboring motion fields (432). Video encoder 20 and / or video decoder 30 then determines whether all of the sub-PUs have been processed (434). If no, video encoder 20 and / or video decoder 30 may move to the next sub-PU. If yes, video encoder 20 and / or video decoder 30 may determine the availability of a spatiotemporal sub-PU motion predictor (436). If available, video encoder 20 and / or video decoder 30 inserts the spatiotemporal sub-PU motion predictor into the merge list.

[0207]

[0147] In the following description, the term "block" is used to refer to a block unit for storage of prediction-related information, e.g., inter- or intra-prediction, intra-prediction mode, motion information, etc. Such prediction information may be stored and used to code future blocks, e.g., to predict prediction mode information for future blocks. In AVC and HEVC, the size of such a block is 4x4. Note that in the following description, "PU" is used to indicate an inter-coded block unit and a sub-PU to indicate a unit that derives motion information from neighboring blocks. Any combination of the following techniques may be applied.

[0208]

[0148] In one example, video encoder 20 and / or video decoder 30 may be configured to obtain motion information from neighboring blocks. Sub-PUs and neighboring blocks may have different sizes. Consider a PU with multiple sub-PUs. The size of a sub-PU is typically equal to or larger than its neighboring block size. In one example, as shown in Figure 14, hashed squares represent neighboring blocks (a, b, ... i) outside the current PU, and the remaining unhashed squares (A, B, ... P) represent sub-PUs within the current PU. As shown in Figure 14, the size of a sub-PU and the size of its neighboring blocks are the same. In one example, the size of a sub-PU is equal to 4x4, although sub-PUs of different sizes may be used. Figure 15 shows another example in which a sub-PU is larger than a neighboring block. In other examples, a sub-PU may have a non-square shape, such as a rectangle or a triangle. In one example, the size of a sub-PU may be signaled in a slice header.

[0209]

[0149] In another example, the process in clause 12 of the above description related to ATMPV may be extended to STMVP. For example, the sub-block / sub-PU size when STMVP is applied is signaled in a parameter set, e.g., a sequence parameter set of a picture parameter set. The size ranges from the smallest PU size to the CTU size. The size may also be predefined or signaled. The size may be as small as 4x4, for example. Alternatively, the sub-block / sub-PU size may be derived based on the size of the PU or CU. For example, the sub-block / sub-PU may be set equal to max(4x4, (CU width) >> M). The value of M may be predefined or signaled in the bitstream.

[0210]

[0150] In STMVP, different test orders of sub-PUs may be used. In the example of Figure 14, in the following description, it is assumed that a raster scan order (A, B, C, D, E...) is applied to the sub-PUs for their motion prediction derivation. However, it should be noted that other scan orders may also be applied, and the techniques of this disclosure are not limited to only the raster scan order.

[0211]

[0151] In STMVP, neighboring blocks can be classified into two different types: spatial and temporal. A spatial neighboring block is an already coded block or an already scanned sub-PU in the current picture or slice that is adjacent to the current sub-PU. A temporal neighboring block is a block in a previous coded picture that is adjacent to the co-located block of the current sub-PU. In one example, all reference pictures associated with the current PU are used to obtain the temporal neighboring block. In another example, a subset of reference pictures is used for STMVP derivation. For example, only the first entry of each reference picture list is used.

[0212]

[0152] Following this definition, and referring to Figure 14, for sub-PU(A), all neighboring blocks (a, b, ... i) and their co-located blocks in the previous coded picture are spatial and temporal neighboring blocks that are treated as available. According to the raster scan order, blocks B, C, D, E...P are not spatially available. Nevertheless, all sub-PUs (A to P) are temporally available neighboring blocks for sub-PU(A) because their motion information can be found in their co-located blocks in the previous coded picture. Taking sub-PU(G) as another example, its spatial neighboring blocks that are available include a, b... to i, and also include A to F. In some examples, some restrictions may be applied to spatial neighboring blocks, such as that the spatial neighboring blocks (i.e., a, b... to i) are in the same LCU / slice / tile.

[0213]

[0153] Video encoder 20 and / or video decoder 30 select a subset of all available neighboring blocks to derive motion information or a motion field for each sub-PU. The subset used for the derivation of each PU may be predefined. In another example, the subset used for the derivation may be signaled as high-level syntax in a slice header, a picture parameter set (PPS), and / or a sequence parameter set (SPS). To optimize coding performance, the subset may be different for each sub-PU. In practice, a fixed pattern of locations for the subsets is preferred for simplicity. For example, each sub-PU may use its immediately above spatial neighbor, its immediately left spatial neighbor, and its immediately below-right temporal neighbor as its subset. As shown in Figure 14, when considering sub-PU(J), the above block (F) and the left block (I) are spatially available neighboring blocks, and the below-right block (O) is a temporally available neighboring block. For such a subset, the sub-PUs in the current PU are processed consecutively due to processing dependencies.

[0214]

[0154] To enable parallel processing of each sub-PU in the current PU, different subsets of neighboring blocks may be defined and used. In one example, the subset includes only spatial neighboring blocks that do not belong to the current PU, e.g., blocks a, b, ... i. In this case, parallel processing may be possible. In another example, for a given sub-PU, if its spatial neighboring blocks are within the current PU, the collocated blocks of the spatial neighboring blocks may be included in the subset and used to derive motion information for the current sub-PU. For example, when considering sub-PU(J), the temporally collocated blocks of the top block (F), left block (I), and bottom right block (O) are selected as the subset for deriving motion for sub-PU(J). In this case, the subset for sub-PU(J) includes three temporally neighboring blocks. In another example, partially parallel processing may be enabled, in which one PU is split into several regions, and each region (covering several sub-PUs) may be processed independently.

[0215]

[0155] Sometimes, neighboring blocks are intra-coded, and here, for better motion prediction and coding efficiency, it is desirable to have a rule for determining alternative motion information for those blocks. For example, considering sub-PU(A), it may be the case that blocks b, c, and f are intra-coded, and blocks a, d, e, g, h, and i are inter-coded.

[0216]

[0156] For spatial neighbors, a predefined order may be used to populate the motion information of an intra-coded block with the motion information of the first found inter-coded block. For example, the search order for the top neighbor may be set to start from the immediately above neighbor rightwards to the rightmost neighbor, which means the order b, c, d, and e. The search order for the left neighbor may be set to start from the immediately left neighbor downwards to the bottommost neighbor. In this example, the order is f, g, h, then i. If an inter-coded block is not found through the search process, the top or left spatial neighbor is considered unavailable.

[0217]

[0157] For temporal neighbors, the same rules as specified in TMVP derivation can be used. However, note that other rules can also be used, such as rules based on motion direction, temporal distance (searching in different reference pictures), and spatial location.

[0218]

[0158] Video encoder 20 and / or video decoder 30 may then derive motion information for a given sub-PU. As part of this process, video encoder 20 and / or video decoder 30 may perform target reference picture determination and motion vector scaling. For neighboring blocks, motion vector scaling may be applied to the motion vectors associated with the neighboring blocks based on each reference picture list to map the motion vectors of all neighboring blocks to the same reference picture in each list. In this example, there may be two steps: first, determine a source motion vector to use for scaling, and second, determine a target reference picture onto which the source motion vector is projected.

[0219]

[0159] In the first step, several methods can be used.

[0220] (a) For each reference list, motion vector scaling is independent of motion vectors in other reference lists. For the motion information of a given block, if there is no motion vector in a reference list (e.g., instead of bi-prediction mode), motion vector scaling is not performed for that list.

[0221] (b) Motion vector scaling is not independent of motion vectors in other reference lists: For the motion information of a given block, if a motion vector is not available in the reference list, the motion vector may be scaled from a motion vector in another reference list.

[0222] (c) Both motion vectors are scaled from one predefined reference list (as in the TMVP case above).

[0223]

[0160] As an example, method (a) is used to scale the motion vectors of spatially adjacent blocks, and method (c) is used to scale the motion vectors of temporally adjacent blocks.

[0224]

[0161] With regard to the second step, video encoder 20 and / or video decoder 30 may select a target reference picture according to a certain rule based on the motion information (e.g., reference pictures) of available spatially neighboring blocks. An example of such a rule is the majority rule, i.e., selecting a reference picture shared by a majority of blocks. In this case, no signaling is needed from the encoder to the decoder for the target reference picture, since the same information can also be inferred at the decoder side using the same rule. Alternatively, such a reference picture may also be explicitly specified in the slice header or signaled to the decoder in some other way. The target reference picture is determined as the first reference picture (refidx=0) in each reference list.

[0225]

[0162] Video encoder 20 and / or video decoder 30 may be configured to derive motion information for a given sub-PU. As shown in the previous section, the motion information of the current sub-PU is derived after extracting motion information from neighboring blocks and performing a motion scaling process (if necessary). Assume that there are N available neighboring blocks with motion information for one given sub-PU. First, a prediction direction (InterDir) is determined. An exemplary method is as follows:

[0226] a.InterDir is initialized as 0, then loop through the motion information of N available neighboring blocks; b. If there is at least one motion vector in list 0, then InterDir = (InterDir bitwise OR 1); c. If there is at least one motion vector in list 1, then InterDir=(InterDir bitwise OR 2).

[0227] Here, "bitwiseOR" represents a bitwise OR operation. The values ​​of InterDir are defined as 0 (no inter prediction), 1 (inter prediction based on list 0), 2 (inter prediction based on list 1), and 3 (inter prediction based on both list 0 and list 1).

[0228]

[0163] In another example, similar to the decision regarding the target reference picture for motion vector scaling described above, a majority rule may be used to determine the value of InterDir for a given sub-PU based on the motion information of all available neighboring blocks.

[0229]

[0164] After InterDir is determined, motion vectors may be derived. For each reference list based on the derived InterDir, there may be M motion vectors (M≦N) available through motion vector scaling relative to the target reference picture, as described above. The motion vectors for a reference list may be derived as follows:

[0230]

number

[0231] where w i and w j are the weighting factors for the horizontal and vertical motion components, respectively, and O i and O j is an offset value that depends on the weighting factor.

[0232]

[0165] The weighting factor may be determined based on various factors. In one example, the same rule may be applied to all sub-PUs in one PU. The rule may be defined as follows: For example, the weighting factor may be determined based on the location distance between the current sub-PU and the corresponding neighboring block. In another example, the weighting factor may also be determined based on the POC distance between the target reference picture and the reference picture associated with the motion vector of the corresponding neighboring block before scaling. In yet another example, the weighting factor may be determined based on the motion vector difference or consistency. Also, for simplicity, all weighting factors may be set to 1.

[0233]

[0166] In another example, different rules may be applied to sub-PUs within one PU. For example, the above rules may be applied, and further, for the sub-PU located in the first row / first column, the weighting factor for the motion vector derived from the temporal neighboring block is set to 0, and for the remaining blocks, the weighting factor for the motion vector derived from the spatial neighboring block is set to 0.

[0234]

[0167] Note that in practice, the above equations can be implemented as is or simplified for easy implementation. For example, to avoid division or floating-point operations, fixed-point arithmetic can be used to approximate the above equations. One example is to avoid dividing by 3, and instead choose to multiply by 43 / 128 to replace the division operation with a multiplication and bit shift. These variations in implementation should be considered to be covered under the same spirit of the techniques of this disclosure. Alternatively, non-linear operations, such as a median filter, can also be applied to derive motion vectors.

[0235]

[0168] Video encoder 20 and / or video decoder 30 may also be configured to perform an availability check during the candidate list construction process for STMVP. It is proposed that even if the motion vector predictor of each sub-PU is available, STMVP mode may be reset to be unavailable for one PU. For example, once the motion vector predictor of each sub-PU is derived for a given PU, several availability checks are performed to determine whether STMVP mode should be made available for the given PU. Such an operation is used to eliminate cases where it is extremely unlikely that STMVP mode will ultimately be selected for a given PU. When STMVP mode is not available, the mode signaling does not include STMVP. If STMVP mode is implemented by inserting SMTVP into the merge list, the merge list does not include this STMVP candidate when it is determined that STMVP mode is unavailable. As a result, signaling overhead may be reduced.

[0236]

[0169] Consider a PU partitioned into M sub-PUs. In one example, if N1 (N1≦M) sub-PUs among the M sub-PUs have the same motion vector predictor (i.e., the same motion vector and the same reference picture index), STMVP is enabled only when N1 is smaller than a threshold or the predictor is different from other motion vector predictors (with smaller merge indexes) in the merge list. In another example, if N2 (N2≦M) sub-PUs under STMVP mode share the same motion vector predictor with corresponding sub-PUs under ATMVP, STMVP is enabled only when N2 is smaller than another threshold. In one example of the present disclosure, both the threshold for N1 and the threshold for N2 are set equal to M.

[0237]

[0170] If an STMVP is available, video encoder 20 and / or video decoder 30 inserts the STMVP candidate into the merge list. The process in Clause 1 above for ATMVP can be extended so that the STMVP candidate can be inserted either before or after the ATMVP candidate. In one example, the STMVP candidate is inserted right after the ATMVP candidate in the merge list.

[0238]

[0171] A POC-based MV pruning technique is described in U.S. Application No. 15 / 431,321, filed February 13, 2017, the entire contents of which are incorporated herein by reference. To maximize the efficiency of MV prediction, the uniqueness of available MVs may be examined. Otherwise, redundant MVs may lead to inefficient resource utilization, such as wasting the bit budget or resources of the target device. Therefore, eliminating redundancy in MV candidates, or so-called pruning, may be an important step to keep MVs as unique and diverse as possible, to provide more meaningful MV candidates in MV prediction.

[0239] This disclosure describes POC-based pruning with three main strengths: (1) higher accuracy, (2) simplicity, and (3) generality. The proposed technique has higher pruning accuracy because it can detect redundant MVs not captured by existing pruning methods. Furthermore, it is simple because no additional complexity is required. Finally, POC-based pruning is general in that it can be applied to versatile situations, such as spatial MVs for ATMVP / merge candidates, sub-PU (ATMVP and STMVP) MVs, TMVP, combinatorial MVs, and even 0 MVs.

[0240] [Table 1]

[0241] Table 1 summarizes what kinds of MV pairs can be pruned using the POC-based method. In category C1, MVs from normal PUs (not sub-PUs) are compared. The comparison can be between two uni-MVs (e.g., motion vectors for uni-prediction) or two bi-MVs (e.g., motion vectors for bi-prediction). From C2 to C4, one or more sub-PU MVs are included in the comparison. In C2, POC-based pruning is used to determine whether the sub-PU MVs within a PU are all equivalent. This can be handled by applying the same technique as in C1 to the MVs from the sub-PUs. The situation where all sub-PU MVs are equivalent falls into C3, where the MVs represent all MVs from the sub-PUs, and therefore the same comparison as in C1 is applied. However, if all sub-PU MVs of a candidate are not equivalent and another candidate with sub-PU MVs exists, i.e., C4, POC-based pruning is applied to each pair of MVs from sub-PUs located at the same position within the PU. Both C5 and C6 involve bi-MV construction by combining two mono-MVs, i.e., a mono-MV from L0 and another mono-MV from L1. If two mono-MVs are equivalent (e.g., the same MV from the same reference picture), bi-MV construction is not required because the resulting bi-MV will be the same as the mono-MV. Therefore, POC-based pruning can help save resources by more accurately detecting equivalent MVs, especially when L0 and L1 have the same reference picture.

[0242]

[0174] For a given list of candidates, two factors can determine the efficiency of the merge candidate list: (1) the ordering of the candidate list (e.g., how to assign an order to the candidates in the list), and (2) pruning (e.g., removing redundancy among those candidates). Generally, it is preferable to have the selected candidate that is most likely to be first in the ordering in the candidate list, since fewer indices to the first candidate can be signaled. Also, having more variety of candidates in the list (e.g., less redundancy) increases the chance that a more accurate motion vector will be present among the candidates in the list.

[0243]

[0175] Techniques of this disclosure include techniques for determining a set of merge candidates from a larger group of possible candidates. Furthermore, this disclosure describes techniques for adaptive selection, ordering, and pruning of merge candidates to achieve higher efficiency of the motion vector candidate list. In the case of adaptive ordering, the proposed techniques leverage additional MV information to assign higher priority (e.g., resulting in a smaller index in the list) to more likely candidates with more accurate motion information. In the case of adaptive pruning, a motion vector difference (MVD) may be used to determine whether two MVs are equivalent (or very close) by comparing the MVD to an adaptive threshold.

[0244]

[0176] Due to the flexibility of the proposed techniques, the techniques of this disclosure can be applied to most of the existing state-of-the-art codecs, such as H.264, HEVC, or H.266, and can be easily extended to different partitioning frameworks, such as the QTBT structure described above. Furthermore, different combinations of the proposed techniques can be combined to a desired solution for a specific application. That is, the following techniques can be applied independently or in any non-mutually exclusive combination.

[0245] Furthermore, the following proposed techniques may be performed without any additional signaling other than a merge index, as is the case in HEVC or H.266 reference software. That is, in some examples, video encoder 20 and video decoder 30 may be configured to perform the following techniques based on a set of predetermined rules and without using explicit signaling. Video encoder 20 may be configured to signal a merge index for the current block, and video decoder 30 may be configured to perform the same procedure as video encoder 20 does to derive merge candidates. Thus, using the received merge index, video decoder 30 may be configured to determine equivalent MV information without any mismatch.

[0246]

[0178] Figure 16 shows an example set of neighboring blocks for current block 450. As shown in Figure 16, the shaded neighboring blocks a, e, f, j, and k are the same as those used as spatial merge candidates in HEVC. This disclosure proposes using motion information from additional neighboring blocks coded before current block 450. Such additional neighboring blocks may include neighboring blocks b, c, d, g, h, and i. By deriving the final motion vector candidate list from more neighboring blocks, the likelihood that a more accurate motion vector will be among the motion vector candidate list is increased.

[0247]

[0179] In the example of Figure 16, current block 450 is 16x16 and each of the neighboring blocks is a 4x4 block. Note, however, that the neighboring blocks may be of different sizes based on the size of the current block. In general, video encoder 20 and video decoder 30 may be configured to build a motion vector candidate list of candidates for current block 450, where the motion vector candidate list includes motion vector information from a number of neighboring blocks to the current block, where the number of neighboring blocks is greater than five.

[0248]

[0180] In another example of this disclosure, video encoder 20 and video decoder 30 may be configured to construct a motion vector candidate list using a motion vector histogram-based ordering of motion information from neighboring blocks. Based on the assumption that motion is spatially homogeneous (e.g., likely to be the same or nearly the same at a given spatial location in a picture), dominant motion information of a neighboring block is likely to be the motion information selected for the current block. Thus, video encoder 20 and video decoder 30 may be configured to derive a motion vector histogram from the motion vector distribution of neighboring blocks. As described above, motion vector information includes a three-dimensional vector (MVx, MVy, direction), where MVx is the horizontal component of the motion vector and MVy is the vertical component of the motion vector, where direction refers to either the past (reference list L0) prediction direction or the future (reference list L1) prediction direction. Referring to Figure 13, video encoder 20 and video decoder 30 may determine how often a particular motion vector is the same for each of neighboring blocks a-k.

[0249]

[0181] Video encoder 20 and video decoder 30 may use the histogram information in a number of different ways. In one example, video encoder 20 and video decoder 30 may use the histogram information to determine which motion vectors, and therefore which neighboring blocks, can be used as spatial merge candidates in a candidate list. In another example, video encoder 20 and video decoder 30 may use the histogram to determine in what order certain spatial merge candidates should be added to the list.

[0250]

[0182] In general, video encoder 20 and video decoder 30 may be configured to derive motion vector histograms from neighboring pixels or blocks. As explained above, Figure 16 shows an example of 4x4 neighboring blocks (a-k) to be used for a motion vector histogram of a 16x16 current block 450. The highlighted blocks (a, e, f, j, and k) are locations of spatial merge candidates in HEVC.

[0251]

[0183] In some examples, video encoder 20 and video decoder 30 derive a motion vector histogram from the motion vector distribution of neighboring blocks having a certain size. Figure 16 shows what neighboring blocks (a to k) are used to construct the MV histogram. The unit size of the neighboring blocks may be a specific size, for example, 4x4, or some predefined minimum size for motion compensation. If blocks do not have associated motion information (e.g., intra-predicted blocks), they may be ignored or filled with motion information from other neighboring blocks. For example, if neighboring block h is an intra-predicted block, video encoder 20 and video decoder 30 may simply not use that neighboring block. In another example, if neighboring block h is an intra-predicted block, video encoder 20 and video decoder 30 may use motion information from the block to the left of neighboring block h.

[0252]

[0184] As shown in the example in Figure 16, to construct an MV histogram for a 16x16 current block, video encoder 20 and video decoder 30 may look at 11 neighboring blocks (block a through block k) that have a size of 4x4. Note that the neighboring blocks may be pre-defined, as in the case of Figure 16 (including the top row / left column), or may depend on the size and / or shape of the current block.

[0253]

[0185] In another example, the histogram may be constructed with weights proportional to the size of the neighboring blocks. For example, the number of pixels (or unit blocks, i.e., 4x4 blocks) belonging to the neighboring blocks may be used as the weight for the histogram. That is, motion vectors from larger blocks (specifically, blocks containing more pixels) have higher weights proportional to the number of pixels in their blocks. In another example, the weight for the histogram may be determined by a combination of the two factors mentioned above, i.e., the number of pixels (or unit blocks) in the neighboring blocks and the number of pixels (or unit blocks) neighboring the current block.

[0254]

[0186] Note that both video encoder 20 and video decoder 30 should follow equivalent rules for constructing histograms to avoid inconsistencies. Assuming equivalent histograms in both video encoder 20 and video decoder 30, all of the following adaptation schemes for merge candidates will lead to equivalent merge lists.

[0255] After determining the motion vector histogram, video encoder 20 and video decoder 30 may then use the histogram to determine the order of spatial merge candidates in the motion vector candidate list. In some examples, the constructed histogram may be f may be used to determine the order of the N spatial merge candidates, where fis the number of fixed space candidates. For example, f The spatial candidates may be neighboring blocks a, e, f, j, and k, as used in HEVC. However, any subset of the total number of candidates may be used. For example, referring to Figure 16, any fixed subset of neighboring blocks a through k may be used as spatial merging candidates.

[0256]

[0188] According to the frequency of each motion vector of the available neighboring blocks, the most frequent motion vector from the histogram is inserted into the merge list first, and the least frequent motion vector from the histogram is the last one among the spatial merge candidates to be inserted into the list. For example, FIG. 16 shows five spatial merge candidates (a, e, f, j, k) used in HEVC. Instead of following a fixed order of the candidates (Jefka order in HEVC), both video encoder 20 and video decoder 30 may be configured to adaptively determine the order from the MV histogram. In another example, instead of examining each neighboring block (e.g., 4x4 spatial merge candidates), the reordering of spatial merging candidates is based on the size of the prediction block (e.g., PU in HEVC) containing the block used to derive the spatial merging candidate.

[0257]

[0189] Consider the example of Figure 17. As shown in Figure 17, three neighboring blocks (e, g, and h) have motion vector 0 (MV0), four different neighboring blocks (a, b, c, and d) have motion vector 1 (MV1), one neighboring block (f) has motion vector 2 (MV2), two different neighboring blocks (i and j) have motion vector 3 (MV3), and one neighboring block (k) has motion vector 4 (MV4). Therefore, video encoder 20 and video decoder 30 would be configured to use fixed candidates a, e, f, j, and k to order the motion vector candidate list as follows: MV1-candidate a (index 0), MV0-candidate e (index 0), MV3-candidate j (index 0), MV2-candidate f (index 0), MV4-candidate k (index 0). The example of Figure 17 assumes that all neighboring blocks have the same weight. In some examples, if two or more of the fixed candidates have the same associated motion vector with the same number of occurrences in the histogram (e.g., MV2 and MV4 in FIG. 17), a predetermined order may be used for examining the candidates. In the example of FIG. 17, candidate f is placed in the list before candidate k. However, any predetermined order may be used.

[0258] In another example of this disclosure, after determining the motion vector histogram, video encoder 20 and video decoder 30 may then use the histogram to determine which of the neighboring blocks to use as spatial merge candidates in the motion vector candidate list, regardless of order. f Rather than using a number of spatial merging candidates, video encoder 20 and video decoder 30 may determine which of all possible neighboring blocks to use as spatial merge candidates in the list. In this example, with respect to Figure 16, all neighboring blocks a-k may be considered for inclusion as spatial merge candidates in the motion vector candidate list.

[0259]

[0191] The video encoder 20 and the video decoder 30 determine which neighboring blocks out of the total number of available neighboring blocks are in the motion vector candidate list by a predetermined number (N h ) candidates to make up. As explained above, instead of changing only the order of a given list of candidates, both the location (e.g., which actual neighboring blocks) and the order of the determined spatial merge candidates can be adaptively derived from the distribution of neighboring motion vectors in the determined histogram. For example, h If MV0=2, the two most frequent motion vectors from neighboring blocks are arranged in the merge list in order of frequency. If more than one neighboring block is associated with the most frequent motion vector in the histogram, video encoder 20 and video decoder 30 may use a predetermined rule to determine which of the neighboring blocks should be placed in the candidate list. However, it should be noted that no matter what rule is used, the neighboring block associated with the motion vector that appears most frequently in the histogram will be added to the motion vector candidate list. According to this example, referring to Figure 17, the neighboring block associated with MV0 and the neighboring block associated with MV1 will be added to the motion vector candidate list.

[0260] In some examples, video encoder 20 and video decoder 30 may be configured to use both of the above techniques for merge list construction using a determined histogram. That is, video encoder 20 and video decoder 30 may both order a fixed set of candidates using a histogram, and select a number (N h ) can be added as non-fixed candidates. f The locations of the spatial merge candidates are fixed across all blocks, e.g., blocks a, e, f, j, and k in Figure 16. Furthermore, the N hThe most frequently occurring motion information of is added to the list as a spatial merging candidate, and then (N f +N h ) candidates is ordered based on the frequency of occurrence of the associated motion vectors in the determined histogram.

[0261] In another example, N f The locations of the spatial merge candidates are fixed across all blocks, e.g., blocks a, e, f, j, and k in FIG. 16, and video encoder 20 and video decoder 30 use the determined histograms to determine the order of the fixed candidates. h The most frequently occurring motion information of is added to the list, but additional N h This candidate is inserted at a certain predetermined position (for example, before or after the motion vector from block e in FIG. 16).

[0262] In another example, this disclosure describes adaptive ordering of sub-PU merge candidates, such as the ATMVP and ATMVP_EXT candidates described above. In one example of JEM2.0 software, ATMVP and ATMVP_EXT are always placed between candidate k and candidate a (e.g., as shown in FIG. 16). Instead of placing ATMVP / ATMVP_EXT at a fixed location in the merge list, video encoder 20 and video decoder 30 may be configured to adaptively place ATMVP / ATMVP_EXT candidates depending on the state related to other available merge candidates, ATMVP / ATMVP_EXT, or a combination thereof.

[0263] In some examples, the motion vector difference (MVD) between two spatial merge candidates may be utilized to determine the location of the ATMVP / ATMVP_EXT candidate. Video encoder 20 and video decoder 30 may be configured to calculate the MVD as a function of the motion vector. In one example, the sum of absolute differences between two MVDs is: MVD=abs(MV x [1]-MV x [0])+abs(MV y [1]-MV y [0]). In another example, the function is MVD=(MV x [1]-MV x [0])*(MV x [1]-MV x [0])+(MV y [1]-MV y [0])*(MV y [1]-MV y [0]). Note that the function for calculating MVD may differ based on the motion vector precision, such as integer, ½, ¼, ⅛, or 1 / 16 pixel precision.

[0264] For example, unlike the latest JEM software, which always places ATMVP / ATMVP_EXT between candidate k and candidate a in FIG. 16, the video encoder 20 and video decoder 30 place the MVD (MVD jk ) can be configured to position the ATMVP / ATMVP_EXT candidate before candidate k. jk is less than a threshold TH1 or greater than another threshold TH2, i.e., MVD jk <TH1またはMVD jk>TH2, then ATMVP / ATMVP_EXT precedes candidate k. For example, an adaptive threshold may be used by calculating TH1 by taking the minimum MVD between all or (some) spatially adjacent motion vectors and calculating TH2 by taking the maximum MVD, where the calculation excludes identical MV pairs as well as pairs (MVj and MVk). Since both video encoder 20 and video decoder 30 have access identical neighboring motion vectors, the calculation will result in the same TH1 and TH2. Alternatively, both TH1 and TH2 may be determined experimentally, e.g., TH1=2 and TH2=8 for 1 / 16 pixel MV precision. Otherwise, candidate k precedes ATMVP / ATMVP_EXT in the list. Similarly, video encoder 20 and video decoder 30 may calculate the MVD (MVD ef ), the order of ATMVP / ATMVP_EXT candidates and candidate f may be determined by examining the merge list. If one or neither of the candidates used for MVD computation, candidate k or candidate a in the above example, is available in the merge list, the ATMVP / ATMVP_EXT candidates may be arranged in the motion vector candidate list in a default order.

[0265]

[0197] In another example, video encoder 20 and video decoder 30 may be configured to analyze characteristics of ATMVP / ATMVP_EXT candidates, such as the variance of sub-block MVs or the spatial distribution of sub-block MVs, to determine where those candidates will be located in the motion vector candidate list. If the variance is within the range [TH1, TH2], a higher priority, i.e., a smaller index in the list, is assigned. The range [TH1, TH2] may be determined by the average variance of the sub-block MVs of the previously coded block that selected ATMVP or ATMVP_EXT as the best merge candidate, as follows: TH1 = C1 * Var1 and TH2 = C2 * Var2, where Var1 and Var2 are calculated and stored from the previously coded block. The coefficients C1 and C2 may be fixed as constants or may depend on the size and / or shape of the current block. The range may depend on the size and / or shape of the current block. For larger blocks, both TH1 and TH2 increase, resulting in a wider range. The range may depend on the motion vector accuracy.

[0266]

[0198] In another example, video encoder 20 and video decoder 30 may be configured to analyze the status of both spatial merge candidates and ATMVP / ATMVP_EXT candidates to determine an order among those candidates. For example, the most frequent subblock motion vector, or the average motion vector, from the ATMVP or ATMVP_EXT candidate may be considered as the delegate MV for the subblock. The delegate motion vector may be used to calculate the MVD for a spatial candidate, for example, block f in FIG. 16. If the MVD is greater than TH1 but less than TH2, video encoder 20 and video decoder 30 may be configured to place the ATMVP / ATMVP_EXT candidate before the spatial candidate.

[0267] In HEVC, a combination motion vector (combi-mv) candidate is derived using two available bidirectional merge candidates, i.e., C1 and C2, which contain two motion vectors for both prediction directions, i.e., reference list L0 and reference list L1. Both C1 and C2 have bidirectional MVs, i.e., MV for candidate C1 L0C1 and MV L1C1 , as well as the MV for candidate C2 L0C2 and MV L1C2 Assume that video encoder 20 and video decoder 30 may be configured to derive a new combi-mv, i.e., (MV0,MV1), by taking L0 MV from C1 and L1 MV from C2, where (MV0,MV1)=(MV L0C1 ,MV L1C2 ). Similarly, video encoder 20 and video decoder 30 may be configured to derive another combi-mv by taking the remaining MVs, such that (MV0′, MV1′)=(MV L0C2 ,MV L1C1 ) can be.

[0268] In some examples, the maximum number of combi-mv candidates is fixed. In HEVC, if the number of available merge candidates is less than the maximum number of merge candidates, e.g., 5 in HEVC, and two or more bidirectional merge candidates are available in the list, then at most 12 combi-mv candidates can be considered as merge candidates. In extensions of HEVC, more merge candidates such as ATMVP and ATMVP_EXT candidates are added, and therefore, increasing the maximum number of combi-mv candidates from 12 to a certain large number, e.g., 30, is a possible extension.

[0269] In another example of this disclosure, the video encoder 20 and the video decoder 30 may be configured to consider more combi-mvs if necessary (e.g., if the maximum number of merging candidates has not yet been reached). For example, if the similarity between available merging candidates is higher than a certain threshold, the combi-mv candidate will also be similar to the existing candidate, and thus the maximum number of combi-mvs is suppressed. The similarity may be measured by the sum of absolute differences (SAD), SATD, average luminance or chrominance value, pixel variance, and / or MV trajectory.

[0270]

[0202] When more combi-mvs are considered, adaptive ordering can be considered to maximize the benefit from spare combi-mv candidates. Given an ordering of combi-mv candidates, the following technique reorders the candidates with respect to some criterion. Candidates that do not meet the criterion follow a default ordering.

[0271] In some examples, the video encoder 20 and the video decoder 30 may be configured to reorder the combi-mv candidates with respect to the similarity between the derived combi-mv and the existing mvs from the available candidates. Both C1 and C2 are bidirectional MVs, i.e., MVs C1 =(MV L0C1 ,MV L1C1 ) and MV C2 =(MV L0C2 ,MV L1C2 ) and two combi-MVs are MVs combi-1 =(MV L0C1 ,MV L1C2 ) and MV combi-2 =(MV L0C2 ,MV L1C1 ) can be derived as follows: MV L0C1 and MV L0C2 (and / or MV L1C2 and MV L1C1 ) point to the same picture, MV L0C1 and MV L0C2and (and / or MV L1C2 and MV L1C1 The MVD between and) is calculated. Then, the video encoder 20 and the video decoder 30 are configured to add the derived combi-mv to the motion vector candidate list after pruning if one of the following conditions is met, i.e., (1) the MVD is between two thresholds, i.e., TH1 < MVD < TH2, or (2) MV L0C1 and MV L0C2 and point to different pictures. In other cases, the combi-mv is left behind. In one example for 1 / 16 pixel motion vector accuracy, TH1 = 2 and TH2 = 8 when both the width and height of the current block are less than 8. When the width and height of the current block are greater than 8 and less than 32, TH1 = 8 and TH2 = 32. When the width and height are greater than 32, TH1 = 16 and TH2 = 64. Once all combi-mv candidates that meet those conditions are first added to the merge list, the video encoder 20 and the video decoder 30 can be configured to add the remaining combi-mv to the list after pruning. The thresholds, TH1 and TH2, can be adaptively selected according to the size or shape of the current block, e.g., max(width, height).

[0272]

[0204] In another example, the video encoder 20 and the video decoder 30 can be configured to sort the combi-mv candidates with respect to the MVD described above. For simplicity, the MVD of a combi-mv candidate with MV L0C1 and MV L0C2 (or MV L1C1 and MV L1C2 ) pointing to different pictures is set to 0. If the combi-mv candidates have equal MVD values, they follow the default order. Once sorted, the video encoder 20 and the video decoder 30 can be configured to add the candidates to the motion vector candidate list after pruning.

[0273]

[0205] In addition to POC-based pruning, as described above, video encoder 20 and video decoder 30 may be configured to further prune merge candidates using adaptive criteria to be determined by additional information other than the motion vector itself (e.g., the size and / or shape of the current block, the type of target merge candidate, and / or the location of the merge candidate if spatial).

[0274]

[0206] In some examples, video encoder 20 and video decoder 30 may be configured to consider a pair of MVs with smaller MVD than an adaptive threshold as identical, and thus pruned to diversify the motion vector candidate list further. The threshold may be chosen adaptively depending on the size and / or shape of the current block.

[0275]

[0207] In some examples, a pruning method such as the above-described MVD-based pruning with an adaptive threshold may be applied to all types of merge candidates, i.e., spatial candidates, temporal candidates, sub-block candidates, or combi-mv candidates. In yet another example, different criteria may be considered for different types of candidates. As an example for spatial candidates, the distance between the locations from which the candidates are derived may be used as a metric to determine the adaptive threshold. For example, if two motion vectors are derived from adjacent blocks, e.g., blocks e and f in Figure 16, the motion vectors are pruned by a threshold that is smaller than the threshold derived from the block whose motion vector is farther away, e.g., blocks f and k in Figure 16.

[0276]

[0208] In some examples, for bidirectional merge candidates, the similarity between two uni-predicted blocks (a uni-predicted block from L0 and another uni-predicted block from the L1 direction) could indicate how reliable the merge candidate is. Based on this observation, the video encoder 20 and the video decoder 30 may be configured to use a metric to differentiate bidirectional merge candidates by using the similarity between the two uni-predicted blocks and sort the bidirectional merge candidates accordingly. For example, the video encoder 20 and the video decoder 30 may be configured to use sum of absolute differences (SAD), SSE, SATD, average luminance or chrominance value, pixel variance, and / or MV trajectory to determine the similarity. More complex metrics can provide greater accuracy for measuring prediction performance. The determination of the metric may depend on the requirements of the target application.

[0277] If SAD is used, then for two given bidirectional merge candidates, C1 and C2, two SADs are calculated between the L0 and L1 directions for each bidirectional candidate, i.e., SAD C1 and S.A.D. C2 Video encoder 20 and video decoder 30 may be configured to place the candidate with the smaller final SAD, i.e., SADC1 or SADC2, ahead of the other in the merge list.

[0278]

[0210] In some examples, all of the proposed techniques described above may be combined to construct a motion vector candidate list, while in other examples, a set or subset of the proposed techniques may be incorporated.

[0279]

[0211] Figure 18 is a flowchart illustrating an example encoding method of this disclosure. The technique of Figure 18 may be performed by one or more hardware units of video encoder 20, including motion estimation unit 42 and motion compensation unit 44.

[0280] In one example of this disclosure, video encoder 20 may be configured to receive a current block of video data (500). Video encoder 20 may derive a histogram of motion vector information for neighboring blocks relative to the current block (502). In one example of this disclosure, the number of neighboring blocks considered for the motion vector candidate list is based on the size of the current block, and the number of neighboring blocks is greater than five. In this context, the term “considered” may include video encoder 20 analyzing the neighboring blocks, determining whether the neighboring blocks have associated motion information, and, if the neighboring blocks have associated motion information, using the motion information to construct the motion vector candidate list. As described above, the motion information may be added directly to the motion vector candidate list or may be used to construct a histogram that may be used to determine the order and / or location of neighboring blocks to use as spatial merge candidates in the motion vector candidate list. Video encoder 20 may further be configured to construct a motion vector candidate list of merge candidates for the current block based on the motion information from the number of neighboring blocks relative to the current block of video data. In some examples, the motion information considered is the derived histogram (504). Video encoder 20 then determines a current motion vector from the motion vector candidate list (506) and encodes the current block of video data using the current motion vector (508).

[0281]

[0213] In another example of the present disclosure, video encoder 20 may be configured to order a predetermined fixed subset of spatial merge candidates in the motion vector candidate list based on the derived histogram.

[0282]

[0214] In another example of the present disclosure, video encoder 20 may be configured to determine a fixed number of spatial merge candidates to add to the motion vector candidate list from the total number of neighboring blocks based on the derived histogram.

[0283]

[0215] In another example of the present disclosure, the video encoder 20 may be configured to determine, based on the derived histogram, a fixed number of spatial merge candidates to add to the motion vector candidate list from the total number of adjacent blocks, and to order, based on the derived histogram, a predetermined fixed subset of spatial merge candidates and the determined fixed number of spatial merge candidates in the motion vector candidate list.

[0284]

[0216] In another example of the present disclosure, the video encoder 20 may be configured to order a predetermined fixed subset of spatial merge candidates in the motion vector candidate list based on the derived histogram, determine a fixed number of spatial merge candidates to add to the motion vector candidate list from the total number of adjacent blocks based on the derived histogram, and insert the determined fixed number of spatial merge candidates at a predetermined location in the motion vector candidate list.

[0285] In another example of this disclosure, video encoder 20 may be configured to add an advanced temporal motion vector prediction (ATMVP) candidate to a motion vector candidate list based on a function of the motion vectors for one or more ATMVP candidates. In another example of this disclosure, video encoder 20 may be configured to determine a location in the motion vector candidate list to add the ATMVP candidate based on a function of the motion vectors for the one or more ATMVP candidates.

[0286]

[0218] In another example of the present disclosure, the video encoder 20 may be configured to determine a combined motion vector candidate by combining motion vector information from two bidirectional motion vector candidates and to add the combined motion vector candidate to a motion vector candidate list.

[0287]

[0219] In another example of the present disclosure, the video encoder 20 may be configured to determine a location in the motion vector candidate list for adding a combined motion vector candidate based on a function of the motion vectors for one or more combined motion vector candidates.

[0288]

[0220] In another example of this disclosure, video encoder 20 may be configured to prune a motion vector candidate list based on motion vector differential information of the motion vector candidates in the motion vector candidate list.

[0289]

[0221] In another example of this disclosure, video encoder 20 may be configured to order bidirectional candidates based on their motion vector differential information in the motion vector candidate list.

[0290]

[0222] Figure 19 is a flowchart illustrating an example decoding method of this disclosure. The technique of Figure 19 may be performed by one or more hardware units of video decoder 30, including motion compensation unit 72.

[0291] In one example of this disclosure, video decoder 30 may be configured to receive a current block of video data encoded using a merge mode (550). Video decoder 30 derives a histogram of motion vector information for neighboring blocks relative to the current block (552). In one example of this disclosure, the number of neighboring blocks considered for the motion vector candidate list is based on the size of the current block, and the number of neighboring blocks is greater than five. In this context, the term “considered” may include video decoder 30 analyzing the neighboring blocks, determining whether the neighboring blocks have associated motion information, and, if the neighboring blocks have associated motion information, using the motion information to construct the motion vector candidate list. As described above, the motion information may be added directly to the motion vector candidate list or may be used to construct a histogram that may be used to determine the order and / or location of neighboring blocks to use as spatial merge candidates in the motion vector candidate list. Video decoder 30 may further be configured to construct a motion vector candidate list of merge candidates for the current block based on the motion information from the above number of neighboring blocks relative to the current block of video data. In some examples, the motion information considered is the derived histogram (554). Video decoder 30 then determines a current motion vector from the motion vector candidate list (556) and decodes the current block of video data using the current motion vector (558).

[0292]

[0224] In another example of this disclosure, video decoder 30 may be configured to order a predetermined fixed subset of spatial merge candidates in a motion vector candidate list based on a derived histogram.

[0293]

[0225] In another example of the present disclosure, video decoder 30 may be configured to determine a fixed number of spatial merge candidates to add to the motion vector candidate list from the total number of neighboring blocks based on the derived histogram.

[0294]

[0226] In another example of the present disclosure, the video decoder 30 may be configured to determine, based on the derived histogram, a fixed number of spatial merge candidates to add to the motion vector candidate list from the total number of adjacent blocks, and to order, based on the derived histogram, the spatial merge candidates of a predetermined fixed subset and the determined fixed number of spatial merge candidates in the motion vector candidate list.

[0295]

[0227] In another example of the present disclosure, the video decoder 30 may be configured to order a predetermined fixed subset of spatial merge candidates in the motion vector candidate list based on the derived histogram, determine a fixed number of spatial merge candidates to add to the motion vector candidate list from the total number of adjacent blocks based on the derived histogram, and insert the determined fixed number of spatial merge candidates at a predetermined location in the motion vector candidate list.

[0296] In another example of this disclosure, video decoder 30 may be configured to add advanced temporal motion vector prediction (ATMVP) candidates to a motion vector candidate list based on a function of the motion vectors for one or more ATMVP candidates. In another example of this disclosure, video encoder 20 may be configured to determine a location in the motion vector candidate list to add the ATMVP candidate based on a function of the motion vectors for one or more ATMVP candidates.

[0297]

[0229] In another example of the present disclosure, the video decoder 30 may be configured to determine a combined motion vector candidate by combining motion vector information from two bidirectional motion vector candidates and to add the combined motion vector candidate to a motion vector candidate list.

[0298]

[0230] In another example of the present disclosure, the video decoder 30 may be configured to determine a location in the motion vector candidate list for adding a combined motion vector candidate based on a function of the motion vectors for one or more combined motion vector candidates.

[0299]

[0231] In another example of this disclosure, video decoder 30 may be configured to prune the motion vector candidate list based on motion vector differential information of the motion vector candidates in the motion vector candidate list.

[0300]

[0232] In another example of this disclosure, video decoder 30 may be configured to order bidirectional candidates based on the motion vector differential information of the bidirectional candidates in the motion vector candidate list.

[0301] As an example, the combination of the proposed techniques shows a 0.4% BD rate improvement in random access configuration on JEM2.0 software, as in the table below. The gain in the example below comes from the combination of (1) histogram-based spatial merge candidate ordering, (2) MVD-based combined merge candidate ordering and pruning, (3) ATMVP, combined, pruning on 0mv candidates, and (4) an increased number of merge candidates and combi-merge candidates tools.

[0302] [Table 2]

[0303]

[0234] In accordance with the above examples, it should be recognized that some acts or events of any of the techniques described herein may be performed in a different sequence, added, merged, or omitted entirely (e.g., not all described acts or events may be required for the practice of the techniques). Moreover, in some examples, acts or events may be performed simultaneously rather than sequentially, for example, through multithreaded processing, interrupt processing, or multiple processors.

[0304] In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted via a computer-readable medium as one or more instructions or code and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media, which correspond to tangible media such as data storage media, or may include communication media, including any medium that enables transfer of a computer program from one place to another, for example, according to a communications protocol. In this manner, computer-readable media may generally correspond to (1) tangible computer-readable storage media that is non-transitory, or (2) a communication medium such as a signal or carrier wave. Data storage media may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementing the techniques described in this disclosure. A computer program product may include a computer-readable medium.

[0305]

[0236] By way of example, and not limitation, such computer-readable storage media may comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Also, any connection is properly termed a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included within the definition of medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transitory media, but instead cover non-transitory tangible storage media. As used herein, disk and disc include compact discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), floppy disks, and Blu-ray discs, where disks typically reproduce data magnetically and discs reproduce data optically with a laser. Combinations of the above should also be included within the scope of computer-readable media.

[0306]

[0237] The instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other equivalent integrated circuits or discrete logic circuitry. Accordingly, the term "processor," as used herein, may refer to any of the above structures or other structures suitable for implementing the techniques described herein. Furthermore, in some aspects, the functionality described herein may be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated into a combined codec. Also, the techniques may be fully implemented in one or more circuits or logic elements.

[0307] The techniques of this disclosure may be implemented in a wide variety of devices or apparatuses, including wireless handsets, integrated circuits (ICs), or sets of ICs (e.g., chipsets). Although various components, modules, or units have been described in this disclosure to highlight functional aspects of devices configured to perform the disclosed techniques, those components, modules, or units do not necessarily require realization by different hardware units. Rather, as described above, the various units may be combined in a codec hardware unit or provided by a collection of interoperable hardware units, including one or more processors described above, along with suitable software and / or firmware.

[0308]

[0239] Various examples have been described. These and other examples are within the scope of the following claims.

Claims

1. 1. A method for decoding video data, the method comprising: receiving a current block of video data encoded in merge mode; constructing a motion vector candidate list of merging candidates for a current block of video data based on motion information from a subset of a plurality of neighboring blocks relative to the current block, wherein the plurality of neighboring blocks include all neighboring blocks adjacent to a top row of the current block or adjacent to a left column of the current block, an upper right neighboring block adjacent to the current block, an upper left neighboring block adjacent to the current block, and a lower left neighboring block adjacent to the current block, wherein each neighboring block has a respective number of neighboring unit blocks adjacent to the current block, and wherein the number of neighboring blocks in the subset of neighboring blocks is greater than 5; deriving a histogram of motion vector information from the motion vector distribution of the subset of neighboring blocks, wherein the motion vector distribution indicates a number of occurrences of a particular motion vector for the subset of neighboring blocks, each occurrence being weighted by the number of the respective neighboring unit blocks in the respective neighboring block; ordering the spatial merge candidates in the motion vector candidate list according to the frequency of the associated motion vectors in the histogram, wherein a predetermined order is used if two or more spatial merge candidates have associated motion vectors with the same frequency in the histogram. determining a current motion vector from the motion vector candidate list; and decoding the current block of video data using the current motion vector.

2. The method described in claim 1, wherein the unit block has a predefined minimum size for motion compensation.

3. The method described in claim 1, wherein the unit block has a size of 4x4.

4. A method according to any one of claims 1 to 3, wherein the subset of adjacent blocks is predefined.

5. 1. An apparatus configured to decode video data, said apparatus comprising: a memory configured to store a current block of video data; one or more processors, wherein the one or more processors: receiving the current block of video data encoded in merge mode; constructing a motion vector candidate list of merging candidates for a current block of video data based on motion information from a subset of a plurality of neighboring blocks relative to the current block, wherein the plurality of neighboring blocks include all neighboring blocks adjacent to a top row of the current block or adjacent to a left column of the current block, an upper right neighboring block adjacent to the current block, an upper left neighboring block adjacent to the current block, and a lower left neighboring block adjacent to the current block, wherein each neighboring block has a respective number of neighboring unit blocks adjacent to the current block, and wherein the number of neighboring blocks in the subset of neighboring blocks is greater than 5; deriving a histogram of motion vector information from the motion vector distribution of the subset of neighboring blocks, wherein the motion vector distribution indicates a number of occurrences of a particular motion vector for the subset of neighboring blocks, each occurrence being weighted by the number of the respective neighboring unit blocks in the respective neighboring block; ordering the spatial merge candidates in the motion vector candidate list according to the frequency of the associated motion vectors in the histogram, wherein a predetermined order is used if two or more spatial merge candidates have associated motion vectors with the same frequency in the histogram. determining a current motion vector from the motion vector candidate list; and decoding the current block of video data using the current motion vector.

6. The device described in claim 5, wherein the unit blocks have a predefined minimum size for motion compensation.

7. The device described in claim 5, wherein the unit block has a size of 4x4.

8. The device of claim 5, wherein the subset of adjacent blocks is predefined.

9. 5. A computer-readable storage medium storing instructions that, when executed, cause one or more processors configured to decode video data to perform the method of any one of claims 1 to 4.

10. 1. An apparatus configured to encode video data, said apparatus comprising: a memory configured to store a current block of video data; one or more processors, wherein the one or more processors: receiving the current block of video data; constructing a motion vector candidate list of merging candidates for a current block of video data based on motion information from a subset of a plurality of neighboring blocks relative to the current block, wherein the plurality of neighboring blocks include all neighboring blocks adjacent to a top row of the current block or adjacent to a left column of the current block, an upper right neighboring block adjacent to the current block, an upper left neighboring block adjacent to the current block, and a lower left neighboring block adjacent to the current block, wherein each neighboring block has a respective number of neighboring unit blocks adjacent to the current block, and wherein the number of neighboring blocks in the subset of neighboring blocks is greater than 5; deriving a histogram of motion vector information from the motion vector distribution of the subset of neighboring blocks, wherein the motion vector distribution indicates a number of occurrences of a particular motion vector for the subset of neighboring blocks, each occurrence being weighted by the number of the respective neighboring unit blocks in the respective neighboring block; ordering the spatial merge candidates in the motion vector candidate list according to the frequency of the associated motion vectors in the histogram, wherein a predetermined order is used if two or more spatial merge candidates have associated motion vectors with the same frequency in the histogram. determining a current motion vector from the motion vector candidate list; encoding the current block of video data using the current motion vector.

11. The device of claim 10, wherein the unit blocks have a predefined minimum size for motion compensation.

12. The device described in claim 10, wherein the unit block has a size of 4x4.

13. The device of claim 10, wherein the subset of adjacent blocks is predefined.

14. A method for encoding video data, the method comprising: receiving a current block of video data; constructing a motion vector candidate list of merging candidates for a current block of video data based on motion information from a subset of a plurality of neighboring blocks relative to the current block, wherein the plurality of neighboring blocks include all neighboring blocks adjacent to a top row of the current block or adjacent to a left column of the current block, an upper right neighboring block adjacent to the current block, an upper left neighboring block adjacent to the current block, and a lower left neighboring block adjacent to the current block, wherein each neighboring block has a respective number of neighboring unit blocks adjacent to the current block, and wherein the number of neighboring blocks in the subset of neighboring blocks is greater than 5; deriving a histogram of motion vector information from the motion vector distribution of the subset of neighboring blocks, wherein the motion vector distribution indicates a number of occurrences of a particular motion vector for the subset of neighboring blocks, each occurrence being weighted by the number of the respective neighboring unit blocks in the respective neighboring block; ordering the spatial merge candidates in the motion vector candidate list according to the frequency of the associated motion vectors in the histogram, wherein a predetermined order is used if two or more spatial merge candidates have associated motion vectors with the same frequency in the histogram. determining a current motion vector from the motion vector candidate list; encoding the current block of video data using the current motion vector.

15. The method described in claim 14, wherein the unit blocks have a predefined minimum size for motion compensation.

Images