Merge candidates for motion vector prediction for video coding

Abstract

To provide methods for coding (e.g., encoding or decoding) of motion information for a block of video data.SOLUTION: A method of decoding video data includes constructing a motion vector candidate list of merge candidates for a current block of video data based on motion information from a number of neighboring blocks relative to the current block, wherein the number of neighboring blocks considered for the motion vector candidate list is based on the size of the current block, and the number of neighboring blocks is greater than 5. In some examples, the method includes deriving a histogram of motion vector information for the neighboring blocks, and constructing the motion vector candidate list based on the derived histogram.SELECTED DRAWING: Figure 17

Description

[Technical Field]

[0001]

[0001] This application claims the benefit of U.S. Provisional Application No. 62 / 336,449, filed on 13 May 2016, the entirety of which is incorporated herein by reference.

[0002]

[0002] This disclosure relates to video coding. [Background technology]

[0003]

[0003] Digital video functionality can be incorporated into a wide range of devices, including digital televisions, digital direct broadcast systems, wireless broadcast systems, personal digital assistants (PDAs), laptop or desktop computers, tablet computers, e-book readers, digital cameras, digital recording devices, digital media players, video game devices, video game consoles, cellular or satellite radiophones, so-called "smartphones," video teleconferencing devices, and video streaming devices. Digital video devices implement video coding techniques, such as those described in the standards defined by ITU-T H.265, also known as MPEG-2, MPEG-4, ITU-T H.263, ITU-T H.264 / MPEG-4, Part 10, Advanced Video Coding (AVC), and High Efficiency Video Coding (HEVC), and extensions of such standards. By implementing such video coding techniques, video devices can transmit, receive, encode, decode, and / or store digital video information more efficiently.

[0004]

[0004] Video coding techniques include spatial (intra-picture) prediction and / or temporal (inter-picture) prediction to reduce or remove redundancy inherent in a video sequence. In block-based video coding, a video slice (e.g., a video frame or a portion of a video frame) can be partitioned into video blocks, which in some techniques may also be referred to as tree blocks, coding units (CUs) and / or coding nodes. Video blocks in an intra-coded (I) slice of a picture are encoded using spatial prediction with respect to reference samples in neighboring blocks within the same picture. Video blocks in an inter-coded (P or B) slice of a picture can use spatial prediction with respect to reference samples in adjacent blocks within the same picture or temporal prediction with respect to reference samples in other reference pictures. A picture may sometimes be referred to as a frame, and a reference picture may sometimes be referred to as a reference frame.

[0005]

[0005] Spatial or temporal prediction results in a prediction block for the block to be coded. Residual data represents the pixel difference between the original block to be coded and the prediction block. An inter-coded block is encoded according to a motion vector pointing to a block of reference samples forming the prediction block, and the residual data indicates the difference between the coded block and the prediction block. An intra-coded block is encoded according to an intra-coding mode and residual data. For further compression, the residual data is transformed from the pixel domain to a transform domain, resulting in residual transform coefficients, which can then be quantized. The quantized transform coefficients, initially constituted in a two-dimensional array, can be scanned to generate a one-dimensional vector of transform coefficients, and entropy coding can be applied to achieve further compression.

Summary of the Invention

[0006]

[0006] Generally, the present disclosure describes techniques related to coding (e.g., encoding or decoding) motion information for blocks of video data. In various examples of the present disclosure, a motion vector candidate list (also referred to as a merge candidate list or simply a candidate list) can be constructed using motion information from a plurality of adjacent blocks. A histogram of the motion information can be derived and then used to determine the order and / or location of spatial merge candidates for the motion vector candidate list.

[0007]

[0007] In one example, the present disclosure describes a method of decoding video data, the method comprising receiving a current block of video data encoded in merge mode, constructing a motion vector candidate list of merge candidates for the current block based on motion information from a number of adjacent blocks for the current block of video data, wherein the number of adjacent blocks considered for the motion vector candidate list is based on the size of the current block, wherein the number of adjacent blocks is greater than 5, determining a current motion vector from the motion vector candidate list, and decoding the current block of video data using the current motion vector.

[0008]

[0008] In another example, the present disclosure describes an apparatus configured to decode video data, the apparatus comprising a memory configured to store a current block of video data, and one or more processors, the one or more processors configured to receive a current block of video data encoded in merge mode, construct a motion vector candidate list of merge candidates for the current block based on motion information from a number of adjacent blocks for the current block of video data, wherein the number of adjacent blocks considered for the motion vector candidate list is based on the size of the current block, wherein the number of adjacent blocks is greater than 5, determine a current motion vector from the motion vector candidate list, and decode the current block of video data using the current motion vector.

[0009]

[0009] In another example, the present disclosure describes a computer-readable storage medium for storing instructions, wherein when the instructions are executed, one or more processors configured to decode video data receive the current block of video data encoded in merge mode; construct a list of merge candidate motion vectors for the current block based on motion information from a certain number of adjacent blocks to the current block of video data; determine the current motion vector from the list of motion vector candidates where the number of adjacent blocks considered for the motion vector candidate list is greater than 5, based on the size of the current block; and decode the current block of video data using the current motion vector.

[0010]

[0010] In one example, the present disclosure describes a device configured to encode video data, the device comprising a memory configured to store a current block of video data, and one or more processors, the one or more processors configured to receive a current block of video data, construct a list of merge candidate motion vectors for the current block based on motion information from a certain number of adjacent blocks to the current block of video data, wherein the number of adjacent blocks considered for the motion vector candidate list is based on the size of the current block, wherein the number of adjacent blocks is greater than 5, determine a current motion vector from the motion vector candidate list, and encode the current block of video data using the current motion vector.

[0011]

[0011] Details of one or more examples are described in the accompanying drawings and the following description. Other features, purposes, and advantages will become apparent from the description and drawings, as well as from the claims. [Brief explanation of the drawing]

[0012] [Figure 1]

[0012] A block diagram showing an exemplary video coding and decoding system that may be configured to perform the techniques of the present disclosure. [Figure 2]

[0013] A block diagram showing an example of a video encoder that may be configured to perform the techniques of this disclosure. [Figure 3]

[0014] A block diagram showing an example of a video decoder that may be configured to perform the techniques of this disclosure. [Figure 4]

[0015] A conceptual diagram showing the coding unit (CU) structure in High Efficiency Video Coding (HEVC). [Figure 5]

[0016] A conceptual diagram illustrating exemplary partition types for interpretation modes. [Figure 6A]

[0017] A conceptual diagram showing an example of a block partition using a quad-tree-binary-tree (QTBT) structure. [Figure 6B]

[0018] A conceptual diagram showing an exemplary tree structure corresponding to a block division using the QTBT structure in Figure 6A. [Figure 7]

[0019] Conceptual diagram showing spatially adjacent candidates in HEVC. [Figure 8]

[0020] A conceptual diagram illustrating temporal motion vector prediction (TMVP) in HEVC. [Figure 9]

[0021] A conceptual diagram showing an exemplary prediction structure for 3D-HEVC. [Figure 10]

[0022] A conceptual diagram illustrating sub-PU-based interview motion prediction in 3D-HEVC. [Figure 11]

[0023] A conceptual diagram showing the prediction of sub-PU movement from a reference picture. [Figure 12]

[0024] A conceptual diagram showing relevant pictures in ATMVP. [Figure 13]

[0025] A flowchart illustrating an exemplary method using the techniques of this disclosure. [Figure 14]

[0026] A conceptual diagram showing an example of a PU and adjacent blocks. [Figure 15]

[0027] A conceptual diagram showing another example of a PU and adjacent blocks. [Figure 16]

[0028] A conceptual diagram showing another example of a PU and adjacent blocks. [Figure 17]

[0029] A conceptual diagram showing another example of a PU and adjacent blocks. [Figure 18]

[0030] A flowchart illustrating an exemplary encoding method in this disclosure. [Figure 19]

[0031] A flowchart illustrating an exemplary decoding method in this disclosure. [Modes for carrying out the invention]

[0013]

[0032] This disclosure describes techniques for improving the efficiency of merge-based motion vector prediction. This disclosure describes techniques for determining merge candidates to be used for motion vector prediction. Illustrative techniques in this disclosure may include adaptive ordering and pruning of merge candidates. Illustrative adaptive ordering techniques in this disclosure may include adaptive ordering of spatial, sub-prediction unit (PU), and combined motion vector (combi-mv) candidates. In some examples, the adaptive pruning techniques proposed in this disclosure may be applied to all merge candidates, including time motion vector prediction (TMVP) candidates, zero motion vector (0mv) candidates, and the candidates described above.

[0014]

[0033] The techniques of this disclosure may provide one or more of the following benefits: (1) higher efficiency, (2) simplicity (e.g., lower decoder implementation complexity), and (3) flexibility. The techniques described herein may provide higher bit-savings by assigning higher priority to merge candidates whose values ​​are closer (or potentially closer) to the actual motion vector. Furthermore, video encoder and video decoder complexity, as well as memory requirements, are relatively small. Finally, the proposed techniques may be applied to a variety of codecs, including H.266 and quad-tree plus binary tree (QTBT) based video codecs. In addition, the proposed techniques provide flexibility such that the techniques may be combined in any way, so that the techniques described herein may be used independently or together.

[0015]

[0034] Figure 1 is a block diagram showing an exemplary video coding and decoding system 10 that may be configured to perform the techniques of the present disclosure for motion vector prediction. As shown in Figure 1, the system 10 includes a source device 12 that provides coded video data to be later decoded by a destination device 14. In particular, the source device 12 provides the video data to the destination device 14 via a computer-readable medium 16. The source device 12 and destination device 14 may comprise any of a wide range of devices, including desktop computers, notebook (i.e., laptop) computers, tablet computers, set-top boxes, telephone handsets such as so-called "smart" phones, so-called "smart" pads, televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, and the like. In some cases, the source device 12 and destination device 14 may be equipped for wireless communication.

[0016]

[0035] The destination device 14 may receive encoded video data to be decoded via a computer-readable medium 16. The computer-readable medium 16 may comprise any type of medium or device capable of moving the encoded video data from the source device 12 to the destination device 14. For example, the computer-readable medium 16 may comprise a communication medium that enables the source device 12 to directly transmit the encoded video data to the destination device 14 in real time. The encoded video data may be modulated according to a communication standard, such as a wireless communication protocol, and transmitted to the destination device 14. The communication medium may comprise any wireless or wired communication medium, such as a radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form part of a packet-based network, such as a local area network, a wide area network, or a global network such as the Internet. The communication medium may include routers, switches, base stations, or any other equipment that may be useful in enabling communication from the source device 12 to the destination device 14.

[0017]

[0036] In some examples, encoded data may be output to a storage device via the output interface 22. Similarly, encoded data may be accessed from the storage device via the input interface. The storage device may include any of a variety of distributed or locally accessed data storage media, such as a hard drive, Blu-ray® disc, DVD, CD-ROM, flash memory, volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded video data. In a further example, the storage device may correspond to a file server or another intermediate storage device capable of storing encoded video generated by the source device 12. The destination device 14 may access the stored video data from the storage device via streaming or download. The file server may be any type of server capable of storing encoded video data and transmitting that encoded video data to the destination device 14. Exemplary file servers include a web server (for a website, for example), an FTP server, a network-attached storage (NAS) device, or a local disk drive. The destination device 14 may access the encoded video data through any standard data connection, including an internet connection. This may include wireless channels (e.g., Wi-Fi® connections), wired connections (e.g., DSL, cable modems, etc.), or a combination of both, which are suitable for accessing encoded video data stored on a file server. Transmission of encoded video data from the storage device may be streaming transmission, download transmission, or a combination thereof.

[0018]

[0037] The techniques of this disclosure are not necessarily limited to wireless applications or configurations. The techniques may be applied to video coding supporting any of a variety of multimedia applications, such as over-the-air television broadcasting, cable television transmission, satellite television transmission, internet streaming video transmission such as Dynamic Adaptive Streaming over HTTP (DASH), digital video encoded on a data storage medium, decoding of digital video stored on a data storage medium, or other applications. In some examples, system 10 may be configured to support one-way or two-way video transmission to support applications such as video streaming, video playback, video broadcasting, and / or video telephony.

[0019]

[0038] In the example shown in Figure 1, the source device 12 includes a video source 18, a video encoder 20, and an output interface 22. The destination device 14 includes an input interface 28, a video decoder 30, and a display device 32. According to this disclosure, the video encoder 20 of the source device 12 may be configured to apply the techniques of this disclosure for motion vector prediction. In other examples, the source and destination devices may include other components or configurations. For example, the source device 12 may receive video data from an external video source 18, such as an external camera. Similarly, the destination device 14 may interface with an external display device instead of including an internal display device.

[0020]

[0039] The illustrated system 10 in Figure 1 is merely an example. The technique of the present disclosure for motion vector prediction can be performed by any digital video coding and / or decoding device. Generally, the technique of the present disclosure is performed by a video coding device, but it can also be performed by a video encoder / decoder, commonly referred to as a “codec”. Furthermore, the technique of the present disclosure can also be performed by a video preprocessor. Source device 12 and destination device 14 are merely examples of coding devices, such that source device 12 generates coded video data to send to destination device 14. In some examples, devices 12 and 14 may operate substantially symmetrically, such that each of devices 12 and 14 includes a video coding component and a video decoding component. Thus, system 10 can support one-way or two-way video transmission between video device 12 and video device 14 for, for example, video streaming, video playback, video broadcasting, or video telephony.

[0021]

[0040] The video source 18 of source device 12 may include a video capture device such as a video camera, a video archive containing previously captured video, and / or a video feed interface for receiving video from a video content provider. Alternatively, video source 18 may generate computer graphics-based data as source video, or a combination of live video, archived video, and computer-generated video. In some cases, if video source 18 is a video camera, source device 12 and destination device 14 may form a so-called cameraphone or videophone. However, as stated above, the techniques described herein may generally be applicable to video coding and may be applicable to wireless and / or wired applications. In each case, captured video, previously captured video, or computer-generated video may be encoded by video encoder 20. The encoded video information may then be output onto computer-readable medium 16 via output interface 22.

[0022]

[0041] The computer-readable medium 16 may include temporary media such as wireless broadcast or wired network transmission, or storage media (i.e., non-temporary storage media) such as hard disks, flash drives, compact discs, digital video discs, Blu-ray discs, or other computer-readable media. In some examples, a network server (not shown) may receive encoded video data from a source device 12 and provide that encoded video data to a destination device 14, for example, via network transmission. Similarly, a computing device in a media manufacturing facility, such as a disc stamping machine, may receive encoded video data from a source device 12 and produce a disc containing that encoded video data. Thus, it will be understood that the computer-readable medium 16 may include one or more computer-readable media in various forms in various examples.

[0023]

[0042] The input interface 28 of the destination device 14 receives information from the computer-readable medium 16. The information on the computer-readable medium 16 may include syntax information, which includes syntax elements that describe the characteristics and / or processing of blocks and other coded units, such as GOPs, defined by the video encoder 20 and used by the video decoder 30. The display device 32 displays the decoded video data to the user and may comprise any of various display devices, such as a cathode ray tube (CRT), liquid crystal display (LCD), plasma display, organic light-emitting diode (OLED) display, or another type of display device.

[0024]

[0043] The video encoder 20 and video decoder 30 may operate in accordance with a video coding standard, such as the High Efficiency Video Coding (HEVC) standard, extensions to the HEVC standard, or subsequent standards such as ITU-T H.266. Alternatively or additionally, the video encoder 20 and video decoder 30 may operate in accordance with other proprietary or industry standards, or extensions to such standards, such as the ITU-T H.264 standard, also known as MPEG-4, Part 10, Advanced Video Coding (AVC). However, the techniques of this disclosure are not limited to any particular coding standard. Other examples of video coding standards include MPEG-2 and ITU-T H.263. Although not shown in Figure 1, in some embodiments, the video encoder 20 and video decoder 30 may be integrated with an audio encoder and audio decoder, respectively, and may include a suitable MUX-DEMUX unit, or other hardware and software, to handle the coding of both audio and video in a common data stream or separate data streams. Where applicable, the MUX-DEMUX unit may comply with the ITU H.223 Multiplexer Protocol or other protocols such as the User Datagram Protocol (UDP).

[0025]

[0044] The video encoder 20 and video decoder 30 can each be implemented as one or more suitable encoder or decoder circuits, or any combination thereof, including one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, etc. When the technique is partially implemented in software, the device may store instructions for the software in a suitable non-temporary computer-readable medium and execute those instructions in hardware using one or more processors to perform the technique of the Disclosure. Each of the video encoder 20 and video decoder 30 may be included in one or more encoders or decoders, and each may be integrated as part of a composite encoder / decoder (codec) in its respective device.

[0026]

[0045] As will be described in more detail below, the video encoder 20 and video decoder 30 may be configured to receive the current block of video data, construct a list of merge candidate motion vectors for the current block based on motion information from a number of adjacent blocks to the current block of video data, determine the current motion vector from the list of motion vector candidates where the number of adjacent blocks considered for the motion vector candidate list is based on the size of the current block, and where the number of adjacent blocks is greater than 5, and code (e.g., encode or decode) the current block of video data using the current motion vector.

[0027]

[0046] Video coding standards include ITU-T H.261, ISO / IEC MPEG-1 Visual, ITU-T H.262 or ISO / IEC MPEG-2 Visual, ITU-T H.263, ISO / IEC MPEG-4 Visual, and ITU-T H.264 (also known as ISO / IEC MPEG-4 AVC), which includes its Scalable Video Coding (SVC) and Multiview Video Coding (MVC) extensions. One joint draft of MVC is described in "Advanced video coding for generic audiovisual services," ITU-T Recommendation H.264, March 2010.

[0028]

[0047] Furthermore, there is a newly developed video coding standard, HEVC, developed by the Joint Collaboration Team on Video Coding (JCT-VC), a joint collaboration between the ITU-T Video Coding Experts Group (VCEG) and the ISO / IEC Motion Picture Experts Group (MPEG). The latest draft of HEVC is available at http: / / phenix.int-evry.fr / jct / doc_end_user / documents / 12_Geneva / wg11 / JCTVC-L1003-v34.zip. The HEVC standard is also presented together in Recommendation ITU-T H.265 and the international standard ISO / IEC 23008-2, both titled "High efficiency video coding" and published in October 2014.

[0029]

[0048] JCT-VC developed the HEVC standard. The HEVC standardization efforts were based on an advanced model of a video coding device called the HEVC Test Model (HM). The HM assumed several additional capabilities of a video coding device compared to existing devices that conform to, for example, ITU-T H.264 / AVC. For example, while H.264 gives nine intra-predictive coding modes, the HEVC HM may give as many as 33 intra-predictive coding modes. This disclosure may use some HEVC terminology for illustrative purposes, but the techniques of this disclosure are not limited to HEVC, and in fact, it is explicitly intended that the techniques of this disclosure may be implemented in successor standards to HEVC.

[0030]

[0049] Generally, HM's working model states that a video frame or picture can be divided into a series of tree blocks or largest coding units (LCUs) containing both lumens and chromens. The syntax data in the bitstream can define the size of the LCU, which is the largest coding unit in terms of the number of pixels. A slice contains several consecutive tree blocks in coding order. A video frame or picture can be divided into one or more slices. Each tree block can be split into coding units (CUs) according to a quadtree. Generally, a quadtree data structure contains one node per CU, with the root node corresponding to the tree block. If a CU is split into four sub-CUs, the node corresponding to the CU contains four leaf nodes, each of which corresponds to one of the sub-CUs.

[0031]

[0050] Each node in a quadtree data structure may provide syntax data for its corresponding CU. For example, a node in a quadtree may include a split flag indicating whether the CU corresponding to that node is split into subCUs. Syntax elements for CUs may be defined recursively and depend on whether the CU is split into subCUs. If a CU is not further split, it is called a leaf CU. In this disclosure, the four subCUs of a leaf CU are also called leaf CUs even if there is no explicit splitting of the original leaf CU. For example, if a 16x16 CU is not further split, the four 8x8 subCUs are also called leaf CUs, even if that 16x16 CU is never split.

[0032]

[0051] A CU serves a similar purpose to a macroblock in the H.264 standard, except that CUs do not have size differences. For example, a tree block can be split into four child nodes (also called subCUs), each of which can then become a parent node and split into another four child nodes. The last unsplit child node, called a leaf node in a quadtree, has a coding node, also called a leaf CU. Syntax data associated with the coded bitstream can define the maximum number of times a tree block can be split, called the maximum CU depth, and can also define the minimum size of a coding node. Accordingly, the bitstream can also define the smallest coding unit (SCU). This disclosure uses the term “block” to refer to any of the CU, PU, ​​or TU in the context of HEVC, or similar data structures in the context of other standards (e.g., macroblocks and their subblocks in H.264 / AVC).

[0033]

[0052] A CU includes a coding node and its associated prediction units (PUs) and transformation units (TUs). The size of a CU corresponds to the size of the coding node and must be square in shape. The size of a CU can range from 8x8 pixels to the size of a tree block with up to 64x64 pixels or more. Each CU may contain one or more PUs and one or more TUs. Syntax data associated with a CU may, for example, describe partitioning the CU into one or more PUs. The partitioning mode may differ between skip-mode coding, direct-mode coding, intra-predictive-mode coding, or inter-predictive-mode coding. PUs may be partitioned so that their shape is non-square. Syntax data associated with a CU may also describe partitioning the CU into one or more TUs according to a quadtree. TUs may be square or non-square (e.g., rectangular).

[0034]

[0053] The HEVC standard allows for transformations that follow a TU, which can vary from CU to CU. A TU is generally sized based on the size of the PU within a given CU, as defined for a partitioned LCU, but this is not always the case. A TU is generally the same size as or smaller than a PU. In some examples, residual samples corresponding to a CU can be subdivided into smaller units using a quad tree structure known as a "residual quad tree" (RQT). The leaf nodes of an RQT are sometimes called transformation units (TUs). Pixel difference values ​​associated with a TU can be transformed to generate transformation coefficients, which can then be quantized.

[0035]

[0054] A leaf CU may contain one or more prediction units (PUs). Generally, a PU represents a spatial area corresponding to all or part of the corresponding CU and may contain data for extracting a reference sample for that PU. In addition, a PU contains data related to prediction. For example, when a PU is intra-mode coded, the data for the PU may be contained in a residual quadtree (RQT) which may contain data describing the intra-prediction mode for the TU corresponding to the PU. As another example, when a PU is inter-mode coded, a PU may contain data defining one or more motion vectors for the PU. The data defining a motion vector for a PU may describe, for example, the horizontal component of the motion vector, the vertical component of the motion vector, the resolution of the motion vector (e.g., 1 / 4 pixel precision or 1 / 8 pixel precision), the reference picture pointed to by the motion vector, and / or a list of reference pictures for the motion vector (e.g., list 0, list 1, or list C).

[0036]

[0055] A leaf CU with one or more PUs may also contain one or more translation units (TUs). Translation units can be specified using RQTs (also called TU quadtree structures) as described above. For example, a split flag may indicate whether a leaf CU is split into four translation units. Each translation unit may then be further split into further sub-TUs. When a TU is not further split, it may be called a leaf TU. Generally, in intracoding, all leaf TUs belonging to a leaf CU share the same intra-prediction mode. That is, generally, the same intra-prediction mode is applied to calculate the predicted values ​​for all TUs of a leaf CU. In intracoding, a video encoder may use the intra-prediction mode to calculate the residual value for each leaf TU as the difference between the portion of the CU corresponding to the TU and the original block. A TU is not necessarily limited by the size of a PU. Therefore, a TU can be larger or smaller than a PU. In intracoding, a PU can collate with its corresponding leaf TU for the same CU. In some examples, the maximum size of a leaf TU may correspond to the size of its corresponding leaf CU.

[0037]

[0056] Furthermore, the TU of a leaf CU may also be associated with a quadtree data structure called a residual quadtree (RQT). That is, a leaf CU may contain a quadtree that shows how the leaf CU is divided into TUs. The root node of a TU quadtree generally corresponds to a leaf CU, and the root node of a CU quadtree generally corresponds to a tree block (or LCU). The TU of an unsplit RQT is called a leaf TU. In general, this disclosure uses the terms CU and TU to refer to leaf CUs and leaf TUs, respectively, unless otherwise specified.

[0038]

[0057] A video sequence generally contains a series of video frames or pictures. A group of pictures (GOP) generally comprises one or more video pictures. A GOP may include syntax data describing some of the pictures contained within the GOP in the GOP header, in the headers of one or more of the pictures, or elsewhere. Each slice of a picture may include slice syntax data describing the encoding mode for that slice. The video encoder 20 generally operates on video blocks within individual video slices to encode video data. A video block may correspond to a coding node in a CU. Video blocks may have a fixed or variable size and their size may vary depending on the specified coding standard.

[0039]

[0058] As an example, HM supports predictions for various PU sizes. Assuming a particular CU size is 2N×2N, HM supports intra-predictions for 2N×2N or N×N PU sizes, and inter-predictions for symmetric PU sizes of 2N×2N, 2N×N, N×2N, or N×N. HM also supports asymmetric division for inter-predictions for 2N×nU, 2N×nD, nL×2N, and nR×2N PU sizes. In asymmetric division, one direction of the CU is not divided, while the other direction is divided into 25% and 75%. The portion of the CU corresponding to the 25% division is indicated by "n" followed by "Up," "Down," "Left," or "Right." Therefore, for example, "2N×nU" refers to a 2N×2N CU divided horizontally into an upper 2N×0.5N PU and a lower 2N×1.5N PU.

[0040]

[0059] In this disclosure, “N×N (NxN)” and “N×N (N by N)” may be used interchangeably to refer to the pixel dimensions of a video block with respect to its vertical and horizontal dimensions, for example, 16×16 (16x16) pixels or 16×16 (16 by 16) pixels. Generally, a 16×16 block has 16 pixels vertically (y=16) and 16 pixels horizontally (x=16). Similarly, an N×N block generally has N pixels vertically and N pixels horizontally, where N represents a non-negative integer. Pixels in a block may be arranged in rows and columns. Furthermore, a block does not necessarily have to have the same number of pixels horizontally as vertically. For example, a block may have N×M pixels, where M is not necessarily equal to N.

[0041]

[0060] After intra-predictive coding or inter-predictive coding of the CU using the PU, the video encoder 20 may compute residual data for the TU of the CU. The PU may comprise syntax data describing a method or mode for generating predicted pixel data in the spatial domain (also called the pixel domain), and the TU may comprise coefficients in the transformation domain after applying a transformation, such as a discrete cosine transform (DCT), integer transform, wavelet transform, or conceptually similar transformation to the residual video data. The residual data may correspond to the pixel difference between the pixels of the unencoded picture and the predicted values ​​corresponding to the PU. The video encoder 20 may form a TU containing the residual data for the CU and then transform the TU to generate transformation coefficients for the CU.

[0042]

[0061] After any transformations to generate the transformation coefficients, the video encoder 20 may perform quantization of the transformation coefficients. Quantization generally refers to the process of further compression in which the transformation coefficients are quantized to reduce the amount of data used to represent the coefficients as much as possible. The quantization process may reduce the bit depth associated with some or all of the coefficients. For example, during quantization, an n-bit value may be truncated to an m-bit value, where n is greater than m.

[0043]

[0062] After quantization, the video encoder may scan the transformation coefficients and generate a one-dimensional vector from a two-dimensional matrix containing the quantization transformation coefficients. The scan may be designed to place higher energy (and therefore lower frequency) coefficients at the front of the array and lower energy (and therefore higher frequency) coefficients at the back of the array. In some examples, the video encoder 20 may utilize a predefined scan order to scan the quantization transformation coefficients to generate a serialized vector that can be entropy-encoded. In other examples, the video encoder 20 may perform adaptive scanning. After scanning quantization transformation coefficients to form a one-dimensional vector, the video encoder 20 may entropically encode the one-dimensional vector according to, for example, context-adaptive variable length coding (CAVLC), context-adaptive binary arithmetic coding (CABAC), syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or another entropy encoding method. The video encoder 20 may also entropically encode syntax elements related to the encoded video data for use by the video decoder 30 when decoding the video data.

[0044]

[0063] To perform CABAC, the video encoder 20 may assign a context within a context model to the symbols to be transmitted. The context may relate, for example, to whether the symbol's adjacency value is non-zero. To perform CAVLC, the video encoder 20 may select a variable-length code for the symbols to be transmitted. The codewords in VLC may be configured so that shorter codes correspond to dominant symbols and longer codes correspond to inferior symbols. In this way, the use of VLC may achieve bit savings that would otherwise be achieved by using equal-length codewords for each symbol to be transmitted. Probability decisions may be based on the context assigned to the symbols.

[0045]

[0064] Figure 2 is a block diagram showing an example of a video encoder 20 that may be configured to perform the techniques of the present disclosure for motion vector prediction, as will be described in more detail below. The video encoder 20 may perform intracoding and intercoding of video blocks within a video slice. Intracoding relies on spatial prediction to reduce or eliminate spatial redundancy of video in a given video frame or picture. Intercoding relies on temporal prediction to reduce or eliminate temporal redundancy of video in adjacent frames or pictures of a video sequence. Intra-mode (I-mode) may refer to any of several spatial-based coding modes. Inter-mode, such as unidirectional prediction (P-mode) or bidirectional prediction (B-mode), may refer to any of several temporal-based coding modes.

[0046]

[0065] As shown in Figure 2, the video encoder 20 receives the current video block in the video frame to be encoded. In the example in Figure 2, the video encoder 20 includes a video data memory 41, a mode selection unit 40, a reference picture memory 64, an adder 50, a transformation unit 52, a quantization unit 54, and an entropy coding unit 56. The mode selection unit 40 now includes a motion compensation unit 44, a motion estimation unit 42, an intra-prediction unit 46, and a segmentation unit 48. For video block reconstruction, the video encoder 20 also includes an inverse quantization unit 58, an inverse transformation unit 60, and an adder 62. A deblocking filter (not shown in Figure 2) may also be included for filtering block boundaries to remove blockiness artifacts from the reconstructed video. If desired, the deblocking filter would generally filter the output of the adder 62. Additional filters (in or after the loop) may also be used in addition to the deblocking filter. Such filters are not shown for brevity, but if desired, the output of adder 50 can be filtered (as an in-loop filter).

[0047]

[0066] The video data memory 41 may be configured to store video data to be encoded by the components of the video encoder 20. Video data stored in the video data memory 41 may be retrieved, for example, from a video source 18. The reference picture memory 64 (sometimes called a decoded picture buffer) may be a reference picture memory that stores reference video data for use when the video encoder 20 encodes video data in intra-coding mode or inter-coding mode. The video data memory 41 and the reference picture memory 64 may be formed by any of various memory devices, such as dynamic random access memory (DRAM) including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM®), or other types of memory devices. The video data memory 41 and the reference picture memory 64 may be provided by the same memory device or separate memory devices. In various examples, the video data memory 41 may be on-chip with the other components of the video encoder 20, or off-chip relative to those components.

[0048]

[0067] During the encoding process, the video encoder 20 receives video frames or slices to be coded. A frame or slice may be divided into multiple video blocks. The motion estimation unit 42 and the motion compensation unit 44 perform inter-predictive coding of the received video blocks for one or more blocks in one or more reference frames to perform temporal prediction. The intra-predictive unit 46 may, alternatively, perform intra-predictive coding of the received video blocks for one or more adjacent blocks in the same frame or slice as the block to be coded to perform spatial prediction. The video encoder 20 may perform multiple coding passes, for example, to select an appropriate coding mode for each block of video data.

[0049]

[0068] Furthermore, the partitioning unit 48 may partition blocks of video data into subblocks based on an evaluation of a previous partitioning scheme in a previous coding pass. For example, the partitioning unit 48 may first partition a frame or slice into LCUs and then partition each LCU into subCUs based on rate-distortion analysis (e.g., rate-distortion optimization). The mode selection unit 40 may further generate a quadtree data structure that shows the partitioning of LCUs into subCUs. A leaf node CU in the quadtree may include one or more PUs and one or more TUs.

[0050]

[0069] The mode selection unit 40 may, for example, select a coding mode based on the error result, i.e., one of intra or intercoded, and provide the obtained intracoded or intercoded block to the adder 50 to generate residual block data, and provide the obtained intracoded or intercoded block to the adder 62 to reconstruct the coded block for use as a reference frame. The mode selection unit 40 also provides syntax elements such as motion vectors, intra-mode indicators, segmentation information, and other such syntax information to the entropy coding unit 56.

[0051]

[0070] The motion estimation unit 42 and the motion compensation unit 44 can be highly integrated, but are shown separately for conceptual purposes. The motion estimation performed by the motion estimation unit 42 is the process of generating motion vectors that estimate the motion of a video block. The motion vectors may, for example, represent the displacement of the PU of a video block in the current video frame or picture relative to a predicted block in a reference frame (or other coding unit) relative to the current block being coded in the current frame (or other coding unit). The predicted block is a block that is found to be a perfect match to the block to be coded with respect to a pixel difference that can be determined by an absolute difference sum (SAD), a square difference sum (SSD), or other difference metric. In some examples, the video encoder 20 may calculate values ​​for sub-integer pixel positions of a reference picture stored in the reference picture memory 64. For example, the video encoder 20 may interpolate values ​​for 1 / 4 pixel positions, 1 / 8 pixel positions, or other fractional pixel positions of the reference picture. Therefore, the motion estimation unit 42 can perform motion search on full pixel positions and fractional pixel positions and output motion vectors with fractional pixel accuracy.

[0052]

[0071] The motion estimation unit 42 calculates a motion vector for the PU of a video block in an intercoded slice by comparing the PU's position with the predicted block's position in a reference picture. The reference picture may be selected from a first reference picture list (list 0) or a second reference picture list (list 1), each of which identifies one or more reference pictures stored in the reference picture memory 64. The motion estimation unit 42 sends the calculated motion vector to the entropy coding unit 56 and the motion compensation unit 44.

[0053]

[0072] Motion compensation performed by the motion compensation unit 44 may involve fetching or generating a predicted block based on the motion vector determined by the motion estimation unit 42. In this case as well, the motion estimation unit 42 and the motion compensation unit 44 may be functionally integrated in some examples. Upon receiving the motion vector for the PU of the current video block, the motion compensation unit 44 may locate the position of the predicted block in one of the reference picture lists that the motion vector points to. The adder 50 forms the residual video block by subtracting the pixel values ​​of the predicted block from the pixel values ​​of the coded current video block, as described below, to form a pixel difference value. Generally, the motion estimation unit 42 performs motion estimation for the lumen component, and the motion compensation unit 44 uses the motion vector calculated based on the lumen component for both the chromen and lumen components. The mode selection unit 40 may also generate syntax elements related to the video block and video slice for use by the video decoder 30 when decoding the video block of the video slice.

[0054]

[0073] A video encoder 20, including a motion estimation unit 42 and a motion compensation unit 44, may be configured to perform any of the various techniques of the present disclosure described above with respect to Figure 1 and described in more detail below. For example, the motion compensation unit 44 may be configured to code motion information for blocks of video data using AMVP mode or merge mode in accordance with the techniques of the present disclosure. Furthermore, the video encoder 20, including the motion estimation unit 42 and the motion compensation unit 44, may be configured to perform any combination of the motion vector candidate list construction techniques of the present disclosure, described in more detail below. In the context of the present disclosure, the terms motion vector candidate list, merge candidate list, and candidate list may be used interchangeably.

[0055]

[0074] Assuming that motion compensation unit 44 chooses to perform a merge mode, motion compensation unit 44 may form a candidate list containing a set of merge candidates. Motion compensation unit 44 may add candidates to the candidate list based on a specific, predetermined order. In other examples of this disclosure, motion compensation unit 44 may be configured to dynamically add candidates to the candidate list in a different order based on histogram information of motion vectors from adjacent blocks. Motion compensation unit 44 may also add additional candidates and perform pruning of the candidate list, as described in more detail below. Finally, mode selection unit 40 may determine which of those candidates should be used to encode the motion information of the current block and encode a merge index representing the selected candidate.

[0056]

[0075] The intra-prediction unit 46 may intra-predict the current block as an alternative to the inter-prediction performed by the motion estimation unit 42 and the motion compensation unit 44, as described above. In particular, the intra-prediction unit 46 may determine the intra-prediction mode to be used to encode the current block. In some examples, the intra-prediction unit 46 may encode the current block using various intra-prediction modes, for example, during separate encoding passes, and the intra-prediction unit 46 (or, in some examples, the mode selection unit 40) may select an appropriate intra-prediction mode to use from the tested modes.

[0057]

[0076] For example, the intra-prediction unit 46 may use rate-distortion analysis to calculate rate-distortion values ​​for various intra-prediction modes under test and select the intra-prediction mode with the best rate-distortion characteristics among the tested modes. Rate-distortion analysis generally determines the amount of distortion (or error) between a coded block and the original uncoded block coded to generate the coded block, as well as the bit rate (i.e., number of bits) used to generate the coded block. The intra-prediction unit 46 may calculate a ratio from the distortion and rate for various coded blocks to determine which intra-prediction mode exhibits the best rate-distortion value for the block.

[0058]

[0077] After selecting an intra-prediction mode for a block, the intra-prediction unit 46 may provide the entropy coding unit 56 with information indicating the selected intra-prediction mode for the block. The entropy coding unit 56 may encode information indicating the selected intra-prediction mode. The video encoder 20 may include configuration data in the transmitted bitstream that may include multiple intra-prediction mode index tables and multiple modified intra-prediction mode index tables (also called codeword mapping tables), definitions of coding contexts for various blocks, and instructions for the most probable intra-prediction mode, intra-prediction mode index table, and modified intra-prediction mode index table to be used for each context.

[0059]

[0078] The video encoder 20 forms a residual video block by subtracting predicted data from the mode selection unit 40 from the original video block being coded. The adder 50 represents one or more components that perform this subtraction operation. The transformation processing unit 52 applies a transformation, such as a discrete cosine transform (DCT) or a conceptually similar transformation, to the residual block to generate a video block with residual transformation coefficient values. The transformation processing unit 52 may perform other transformations that are conceptually similar to the DCT. Wavelet transforms, integer transforms, subband transforms, or other types of transformations may also be used.

[0060]

[0079] In either case, the transformation processing unit 52 applies the transformation to the residual block to generate a block of residual transformation coefficients. The transformation may convert the residual information from the pixel value domain to a transformation domain such as the frequency domain. The transformation processing unit 52 may send the obtained transformation coefficients to the quantization unit 54. The quantization unit 54 quantizes the transformation coefficients to further reduce the bit rate. The quantization process may reduce the bit depth associated with some or all of the coefficients. The degree of quantization can be changed by adjusting the quantization parameters. In some examples, the quantization unit 54 may then perform a scan of the matrix containing the quantized transformation coefficients. Alternatively, the entropy coding unit 56 may perform the scan.

[0061]

[0080] After quantization, the entropy coding unit 56 entropy codes the quantization transformation coefficients. For example, the entropy coding unit 56 may perform context-adaptive variable-length coding (CAVLC), context-adaptive binary arithmetic coding (CABAC), syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval piece entropy (PIPE) coding, or another entropy coding technique. In the case of context-based entropy coding, the context may be based on adjacent blocks. After entropy coding by the entropy coding unit 56, the coded bitstream can be transmitted to another device (e.g., video decoder 30) or archived for later transmission or retrieval.

[0062]

[0081] The inverse quantization unit 58 and the inverse transform unit 60 apply inverse quantization and inverse transform, respectively, to reconstruct the residual block in the pixel region for later use as a reference block. The motion compensation unit 44 may compute the reference block by adding the residual block to one of the predicted blocks in the frame of the reference picture memory 64. The motion compensation unit 44 may also apply one or more interpolation filters to the reconstructed residual block to compute sub-integer pixel values ​​to be used in motion estimation. The adder 62 adds the reconstructed residual block to the motion-compensated predicted block generated by the motion compensation unit 44 to generate a reconstructed video block for storage in the reference picture memory 64. The reconstructed video block may be used as a reference block by the motion estimation unit 42 and the motion compensation unit 44 to intercode the block in subsequent video frames.

[0063]

[0082] Thus, the video encoder 20 in Figure 2 represents an example of a video coder configured to derive a histogram of motion vector information from adjacent blocks relative to the current block, determine merge candidates for a motion vector candidate list for motion vector prediction for the current block based on the derived histogram, order the motion vector candidate list based on the derived histogram, and perform merge vector prediction using the motion vector candidate list.

[0064]

[0083] Figure 3 is a block diagram showing an example of a video decoder 30 that may be configured to perform the motion vector prediction technique of the present disclosure. In the example of Figure 3, the video decoder 30 includes a video data memory 71, an entropy decoding unit 70, a motion compensation unit 72, an intra-prediction unit 74, an inverse quantization unit 76, an inverse transform unit 78, a reference picture memory 82, and an adder 80. In some examples, the video decoder 30 may perform a decoding path that is generally the reverse of the encoding path described with respect to the video encoder 20 (Figure 2). The motion compensation unit 72 may generate prediction data based on motion vectors received from the entropy decoding unit 70, and the intra-prediction unit 74 may generate prediction data based on an intra-prediction mode indicator received from the entropy decoding unit 70.

[0065]

[0084] The video data memory 71 may store video data, such as an encoded video bitstream, to be decoded by the components of the video decoder 30. Video data stored in the video data memory 71 may be obtained, for example, from a computer-readable medium 16, from a local video source such as a camera, via wired or wireless network communication of video data, or by accessing a physical data storage medium. The video data memory 71 may form an encoded picture buffer (CPB) that stores encoded video data from the encoded video bitstream. The reference picture memory 82 (also called a decoded picture buffer (DPB)) may store reference video data for use when the video decoder 30 decodes the video data in intra-coding mode or inter-coding mode, or for output. The video data memory 71 and the reference picture memory 82 may be formed by any of various memory devices, such as DRAM, SDRAM, MRAM, RRAM, or other types of memory devices. The video data memory 71 and the reference picture memory 82 may be provided by the same memory device or separate memory devices. In various examples, the video data memory 71 may be on-chip along with the other components of the video decoder 30, or it may be off-chip relative to those components.

[0066]

[0085] During the decoding process, the video decoder 30 receives an encoded video bitstream from the video encoder 20, representing the video blocks of the encoded video slice and the associated syntax elements. The entropy decoding unit 70 of the video decoder 30 entropy-decodes the bitstream to generate quantization coefficients, motion vectors or intra-predictive mode indicators, and other syntax elements. The entropy decoding unit 70 transfers the motion vectors and other syntax elements to the motion compensation unit 72. The video decoder 30 may receive syntax elements at the video slice level and / or video block level.

[0067]

[0086] When a video slice is coded as an intra-coded (I) slice, the intra-prediction unit 74 may generate prediction data for the video block of the current video slice based on the signaled intra-prediction mode and data from a previously decoded block of the current frame or picture. When a video frame is coded as an inter-coded (e.g., BB or P) slice, the motion compensation unit 72 generates prediction blocks for the video block of the current video slice based on the motion vector and other syntax elements received from the entropy decoding unit 70. The prediction blocks may be generated from one of the reference pictures in one of the reference picture lists. The video decoder 30 may construct reference frame lists, i.e., list 0 and list 1, using default construction techniques based on the reference pictures stored in the reference picture memory 82.

[0068]

[0087] The motion compensation unit 72 determines prediction information for the video blocks of the current video slice by parsing motion vectors and other syntax elements, and uses that prediction information to generate prediction blocks for the current video blocks being decoded. For example, the motion compensation unit 72 uses some of the received syntax elements to determine the prediction mode used to code the video blocks of the video slice (e.g., intra or inter-predict), the inter-predict slice type (e.g., B-slice or P-slice), construction information for one or more of the reference picture lists for the slice, motion vectors for each inter-coded video block of the slice, the inter-predict status for each intercoded video block of the slice, and other information for decoding the video blocks in the current video slice.

[0069]

[0088] The motion compensation unit 72 may also perform interpolation based on an interpolation filter. The motion compensation unit 72 may use the interpolation filter used by the video encoder 20 during the encoding of the video block to calculate the interpolated values ​​of the sub-integer pixels of the reference block. In this case, the motion compensation unit 72 may determine the interpolation filter used by the video encoder 20 from the received syntax elements and use that interpolation filter to generate the predicted block.

[0070]

[0089] The video decoder 30, including the motion compensation unit 72, may be configured to perform any of the various techniques of the present disclosure described above with respect to Figure 1 and described in more detail below. For example, the motion compensation unit 72 may be configured to perform motion vector prediction using AMVP mode or merge mode in accordance with the techniques of the present disclosure. Furthermore, the video decoder 30, including the motion compensation unit 72, may be configured to perform any combination of the motion vector candidate list construction techniques of the present disclosure, described in more detail below. The entropy decoding unit 70 may decode one or more syntax elements representing how motion information is currently coded for a block.

[0071]

[0090] Assuming the syntax elements indicate that a merge mode is being performed, the motion compensation unit 72 may form a candidate list containing a set of merge candidates. The motion compensation unit 72 may add candidates to the candidate list based on a specific, predetermined order. In other examples of this disclosure, the motion compensation unit 72 may be configured to dynamically add candidates to the candidate list in a different order based on histogram information of motion vectors from adjacent blocks. The motion compensation unit 72 may also add additional candidates and perform pruning of the candidate list, as described in more detail below. Finally, the motion compensation unit 72 may decode a merge index representing which of those candidates will be used to code the motion information for the current block.

[0072]

[0091] The inverse quantization unit 76 inversely quantizes, i.e., dequantizes, the quantization transformation coefficients given in the bitstream and entropy-decoded by the entropy decoding unit 70. The inverse quantization process determines the degree of quantization and, similarly, the degree of inverse quantization to be applied, which is calculated by the video encoder 30 for each video block in the video slice. Y This may include the use of [the specified term].

[0073]

[0092] The inverse transform unit 78 applies an inverse transform, such as an inverse DCT, an inverse integer transform, or a conceptually similar inverse transform process, to the transform coefficients in order to generate residual blocks in the pixel region.

[0074]

[0093] After the motion compensation unit 72 generates a predicted block for the current video block based on the motion vector and other syntax elements, the video decoder 30 forms a decoded video block by adding the residual block from the inverse unit 78 with the corresponding predicted block generated by the motion compensation unit 72. The adder 80 represents one or more components that perform this addition operation. Deblocking filters may also be applied to filter the decoded blocks to remove blocking artifacts, if desired. Other loop filters may also be used (either during or after the coding loop) to smooth pixel transitions or, optionally, to improve video quality. The decoded video block for a given frame or picture is then stored in a reference picture memory 82 that stores a reference picture to be used for subsequent motion compensation. The reference picture memory 82 also stores the decoded video for later presentation on a display device, such as the display device 32 in Figure 1.

[0075]

[0094] In this way, the video decoder 30 represents an example of a video coder configured to derive a histogram of motion vector information from adjacent blocks to the current block, determine merge candidates for a motion vector candidate list for motion vector prediction for the current block based on the derived histogram, order the motion vector candidate list based on the derived histogram, and perform merge vector prediction using the motion vector candidate list.

[0076]

[0095] The following sections describe several aspects of video coding techniques and standards, with particular consideration to motion vector prediction and related techniques. First, motion information is described. For each block of video data coded using interprediction mode, a set of motion information may be available. The set of motion information includes motion information for the forward and backward prediction directions. Here, the forward and backward prediction directions are two prediction directions corresponding to the current picture or slice's reference picture list 0 (RefPicList0) and reference picture list 1 (RefPicList1). The terms “forward” and “backward” do not necessarily have a geometric meaning. Instead, they are used to distinguish which reference picture list the motion vector is based on. Forward prediction means prediction formed based on reference list 0, and backward prediction means prediction formed based on reference list 1. When both reference list 0 and reference list 1 are used to form predictions for a given block, it is called bidirectional prediction.

[0077]

[0096] If only one reference picture list is used for a given picture or slice, then every block within the picture or slice is predicted forward. If both reference picture lists are used for a given picture or slice, then blocks within the picture or slice may be predicted forward, backward, or bidirectionally.

[0078]

[0097] For each prediction direction, motion information also includes a reference index and a motion vector. The reference index is used to identify the reference picture in the corresponding reference picture list (e.g., RefPicList0 or RefPicList1). The motion vector has both horizontal and vertical components, each component indicating an offset value along the horizontal and vertical directions, respectively. The motion vector indicates the position of the predictor block relative to the position of the currently coded block. The reference index indicates the picture containing the predictor block. In some explanations, for simplicity, the term “motion vector” may be used interchangeably with motion information to refer to both the motion vector and its associated reference index.

[0079]

[0098] In video coding standards, the picture order count (POC) is widely used to identify the display order of pictures. While two pictures within a single coded video sequence may have the same POC value, this generally does not occur within a single coded video sequence. When multiple coded video sequences exist in a bitstream, pictures with the same POC value may be close to each other in terms of decoding order. The POC value of a picture is generally used for constructing the reference picture list, deriving the reference picture set (as in the case of HEVC), and for motion vector scaling.

[0080]

[0099] The following section describes the macroblock (MB) structure in Advanced Video Coding (AVC) (H.264). In H.264 / AVC, each inter-macroblock (MB) (for example, an MB coded using inter-prediction) can be divided in four different ways: • One 16x16MB division • Two 16x8MB divisions • Two 8x16MB divisions • Four 8x8MB divisions

[0100] Different MB segments within a single MB may have different reference index values ​​(RefPicList0 or RefPicList1) for each prediction direction. When an MB is not divided into four 8x8 MB segments, the MB has only one motion vector for each prediction direction for each MB segment.

[0081]

[0101] When MB is divided into four 8x8 MB sections, each 8x8 MB section can be further divided into sub-blocks, each of which may have a different motion vector in each predicted direction. There are four different ways to divide an 8x8 MB section into sub-blocks: • One 8x8 sub-block • Two 8x4 sub-blocks • Two 4x8 sub-blocks • Four 4x4 sub-blocks

[0102] Each subblock may have a different motion vector in each predicted direction. Therefore, the motion vectors exist at a level equal to or higher than that of the subblock.

[0082]

[0103] The Temporal Direct Mode in AVC is not described. In AVC, the Temporal Direct Mode can be enabled at either the MB level or the MB segment level for skips or direct modes in B slices. For each MB segment, the motion vectors of the current MB segment in the current block's RefPicList1[0] and the collated block are used to derive the motion vector. Each motion vector in the collated block is scaled based on the POC distance. In AVC, the direct mode can also predict motion information from spatial neighbors.

[0083]

[0104] Next, the coding unit (CU) structure in HEVC is described. In HEVC, the largest coding unit in a slice is called a coding tree block (CTB) or coding tree unit (CTU). A CTB contains a quadtree whose nodes are coding units. CTBs are described in WJHan et al., "Improved Video Compression Efficiency Through Flexible Unit Representation and Corresponding Extension of Coding Tools," IEEE Transaction on Circuits and Systems for Video Technology, vol.20, no.12, pp. 1709-1720, December 2010, and can be recursively split into CUs in a quadtree manner, as shown in Figure 4. As shown in Figure 4, each level of a slice is a quadtree split into four subblocks. The black blocks are examples of leaf nodes (i.e., blocks that are not further split).

[0084]

[0105] (Technically, an 8x8 CTB size may be supported.) CTB sizes can range from 16x16 to 64x64 in the HEVC main profile. CUs can be the same size as the CTB, but can be smaller, around 8x8. Each CU is coded using one mode (e.g., intra-prediction mode or inter-prediction mode). When a CU is intercoded, it can be further divided into two or four predictive units (PUs), or if no further division is applied, it can be just one PU. When there are two PUs in a single CU, those PUs can be a rectangle that is half the size of the CU, or two rectangles that are one-quarter or three-quarters the size of the CU.

[0085]

[0106] When CUs are interconnected, there exists one set of motion information for each PU (e.g., motion vector, prediction direction, and reference picture). Furthermore, each PU is coded using a unique interconnection prediction mode to derive its set of motion information. However, it should be understood that even if two PUs are uniquely coded, they may still have the same motion information in some situations.

[0086]

[0107] In HEVC, as shown in Figure 5, there are eight partition modes for CUs coded using interpredictive modes: PART_2N×2N, PART_2N×N, PART_N×2N, PART_N×N, PART_2N×nU, PART_2N×nD, PART_nL×2N, and PART_nR×2N. A CU coded using partition mode PART_2N×2N is not further split; that is, the entire CU is treated as a single PU (PU0). A CU coded using partition mode PART_2N×N is symmetrically split horizontally into two PUs (PU0 and PU1). A CU coded using partition mode PART_N×2N is symmetrically split vertically into two PUs. A CU coded using partition mode PART_N×N is symmetrically split into four PUs of equal size (PU0, PU1, PU2, PU3).

[0087]

[0108] A CU coded using partition mode PART_2N×nU is split asymmetrically horizontally into one PU0 (upper PU) having 1 / 4 the size of the CU and one PU1 (lower PU) having 3 / 4 the size of the CU. A CU coded using partition mode PART_2N×nD is split asymmetrically horizontally into one PU0 (upper PU) having 3 / 4 the size of the CU and one PU1 (lower PU) having 1 / 4 the size of the CU. A CU coded using partition mode PART_nL×2N is split asymmetrically vertically into one PU0 (left PU) having 1 / 4 the size of the CU and one PU1 (right PU) having 3 / 4 the size of the CU. A CU coded using partition mode PART_nR×2N is split asymmetrically vertically into one PU0 (left PU) having 3 / 4 the size of the CU and one PU1 (right PU) having 1 / 4 the size of the CU.

[0088]

[0109] HEVC uses a quadtree partitioning structure, but other partitioning structures are being studied for future video coding standards. For example, J. An et al., "Block partitioning structure for next generation video coding," International Telecommunication Union, COM16-C966, September 2015 (hereinafter, "VCEG proposal COM16-C966"), proposed a quadtree-binary (QTBT) partitioning technique for future video coding standards beyond HEVC (e.g., H.266). Simulations showed that the proposed QTBT structure is more efficient than the quadtree structure used in HEVC for several video sequences.

[0089]

[0110] In the proposed QTBT structure of VCEG proposal COM16-C966, the CTB is initially partitioned using a quadtree splitting technique, where the quadtree splitting of a single node may be iterated until the node reaches the minimum allowable quadtree leaf node size. The minimum allowable quadtree leaf node size may be indicated to the video decoder by the value of the syntax element MinQTSize. If the quadtree leaf node size is not greater than the maximum allowable binary root node size (indicated by, for example, the syntax element MaxBTSize), the quadtree leaf node may be further partitioned using binary partitioning. Binary partitioning of a single node may be iterated until the node reaches the minimum allowable binary leaf node size (as indicated by, for example, the syntax element MinBTSize) or the maximum allowable binary depth (as indicated by, for example, the syntax element MaxBTDepth). VCEG proposal COM16-C966 uses the term "CU" to refer to a binary leaf node. In the VCEG proposal COM16-C966, CU is used for prediction (e.g., intra-prediction, inter-prediction, etc.) and transformation without any further division. Generally, according to the QTBT technique, there are two types of splits for binary tree splitting: symmetric horizontal splitting and symmetric vertical splitting. In each case, a block is split by dividing the block in half either horizontally or vertically (down the middle).

[0090]

[0111] In an example of a QTBT partition structure, the CTU size is set to 128x128 (e.g., a 128x128 chroma block and two corresponding 64x64 chroma blocks), MinQTSize is set to 16x16, MaxBTSize is set to 64x64, MinBTSize (for both width and height) is set to 4, and MaxBTDepth is set to 4. The quadtree partition is first applied to the CTU to generate quadtree leaf nodes. Quadtree leaf nodes can have sizes ranging from 16x16 (i.e., MinQTSize is 16x16) to 128x128 (i.e., the CTU size). According to the example of a QTBT partition, if a leaf quadtree node is 128x128, the leaf quadtree node cannot be further split by a binary tree because its size exceeds MaxBTSize (i.e., 64x64). Otherwise, a leaf quad tree node is further partitioned by a binary tree. Thus, a quad tree leaf node is also the root node for the binary tree and has a binary tree depth of 0. A binary tree depth reaching MaxBTDepth (e.g., 4) implies no further splitting. A binary tree node having a width equal to MinBTSize (e.g., 4) implies no further horizontal splitting. Similarly, a binary tree node having a height equal to MinBTSize implies no further vertical splitting. A leaf node (CU) of a binary tree is further processed (e.g., by performing prediction and transformation processes) without any further partitioning.

[0091]

[0112] Figure 6A shows an example of 150 blocks (e.g., CTB) partitioned using the QTBT partitioning technique. As shown in Figure 6A, using the QTBT partitioning technique, each of the resulting blocks is split symmetrically through the center of each block. Figure 6B shows the tree structure corresponding to the block partitioning in Figure 6A. In Figure 6B, solid lines represent quadrutree splitting and dotted lines represent binary tree splitting. In one example, each splitting (i.e., non-leaf) node in the binary tree is signaled with a syntax element (e.g., a flag) to indicate the type of splitting being performed (e.g., horizontal or vertical), where 0 indicates horizontal splitting and 1 indicates vertical splitting. In the case of quadrutree splitting, there is no need to indicate the splitting type because quadrutree splitting always splits a block horizontally and vertically into four subblocks of equal size.

[0092]

[0113] As shown in Figure 6B, at node 170, block 150 is split into four blocks 151, 152, 153, and 154, shown in Figure 6A, using a QT division. Block 154 is not further split and is therefore a leaf node. At node 172, block 151 is further split into two blocks using a BT division. As shown in Figure 6B, node 172 is marked with a 1 indicating a vertical split. Thus, the splitting at node 172 results in block 157 and a block containing both blocks 155 and 156. Blocks 155 and 156 are created at node 174 by further vertical splitting. At node 176, block 152 is further split into two blocks 158 and 159 using a BT division. As shown in Figure 6B, node 176 is marked with a 1 indicating a horizontal split.

[0093]

[0114] At node 178, block 153 is split into four equally sized blocks using a QT partition. Blocks 163 and 166 are created from this QT partition and are not split further. At node 180, the top-left block is first split using vertical binary splitting, producing block 160 and the right vertical block. The right vertical block is then split into blocks 161 and 162 using horizontal binary splitting. The bottom-right block, created from a quad tree split at node 178, is split into blocks 164 and 165 at node 184 using horizontal binary splitting.

[0094]

[0115] The motion vector candidate list construction technique described in more detail below can be used in conjunction with any video block partitioning technique, including the MB partitioning structure for H.264 / AVC, the quadtree partitioning structure for HEVC, or the QTBT partitioning structure such as the QTBT structure proposed for H.266.

[0095]

[0116] Next, motion prediction in HEVC is described. The HEVC standard has two interpretation modes for the PU, called merge mode (skipping is considered a special case of merge) and Advanced Motion Vector Prediction (AMVP) mode, respectively. In either AMVP mode or merge mode, the video encoder 20 and video decoder 30 are configured to construct a motion vector (MV) candidate list for multiple motion vector predictors. Motion vector predictors can be motion vectors from adjacent blocks or artificially generated motion vectors, and they can be used to predict the motion vector for the currently coded block of video data. In merge mode, instead of currently encoding the motion vector of the block itself, the video encoder 20 selects one motion vector from the motion vector candidate list, as well as a reference index associated with that candidate, and uses that candidate motion vector for interpretation. The video encoder 20 may signal the index of the selected motion vector candidate (e.g., a merge index) to the video decoder 30. The video decoder 30 may construct a motion vector candidate list for merge mode in the same manner as the video encoder 20. The video decoder 30 may use a signaled index into the motion vector candidate list to identify the selected candidate, and then retrieve the motion vector and its associated reference index to use as the motion vector for the current block.

[0096]

[0117] The MV candidate list contains up to five candidates for merge mode and two candidates for AMVP mode. A merge candidate may contain motion vectors corresponding to both a set of motion information, such as a reference picture list (lists 0 and 1) and a reference index. When a merge candidate is identified by a merge index, the reference picture is used for prediction of the current block, and the associated motion vector is determined. However, under AMVP mode for each potential prediction direction from either list 0 or list 1, the reference index is explicitly signaled along with the motion vector predictor (MVP) index to the MV candidate list, since the AMVP candidate contains only motion vectors. In AMVP mode, the predicted motion vector can be further refined. In some examples of AMVP, the video encoder 20 may also signal a motion vector difference (MVD). The MVD is the difference between the selected MVP for the current block and the actual determined motion vector. As can be seen above, a merge candidate corresponds to a full set of motion information, while an AMVP candidate contains only one motion vector for a particular prediction direction and reference index.

[0097]

[0118] Figure 7 is a conceptual diagram showing spatial adjacency candidates in HEVC. Spatial MV candidates are derived from the adjacency blocks shown in Figure 7 for a given PU (PU0), but the method of generating candidates from the blocks differs between merge mode and AMVP mode.

[0098]

[0119] In merge mode, up to four spatial MV candidates can be derived in the order shown in Figure 7(a). The order is left (0,A1), top (1,B1), top right (2,B0), bottom left (3,A0), and top left (4,B2), as shown in Figure 7(a). That is, in Figure 7(a), block 200 contains PU0 204A and PU1 204B. When a video coder (e.g., video encoder 20 and / or video decoder 30) should code motion information for PU0 204A using merge mode, the video coder adds motion information from spatially adjacent blocks 208A, 208B, 208C, 208D, and 208E to the candidate list in that order. Blocks 208A, 208B, 208C, 208D, and 208E are sometimes referred to as blocks A1, B1, B0, A0, and B2, respectively, in HEVC.

[0099]

[0120] In AVMP mode, adjacent blocks are divided into two groups: the left group, which includes blocks 0 and 1, and the upper group, which includes blocks 2, 3, and 4, as shown above in Figure 7(b). These blocks are labeled as blocks 210A, 210B, 210C, 210D, and 210E in Figure 7(b), respectively. Block 202 includes PU0 206A and PU1 206B, and blocks 210A, 210B, 210C, 210D, and 210E represent spatial neighbors to PU0 206A. For each group, potential candidates in adjacent blocks that reference the same reference picture as the reference picture indicated by the signaled reference index have the highest priority to be selected to form the final candidate for the group. Not all adjacent blocks may contain motion vectors pointing to the same reference picture. Therefore, if no such candidate can be found, the first available candidate will be scaled to form the final candidate, and thus the time-distance difference can be compensated for.

[0100]

[0121] Figure 8 is a conceptual diagram illustrating time-motion vector prediction (TMVP) in HEVC. In particular, Figure 8(a) shows an exemplary CU220 including PU0 222A and PU1 222B. PU0 222A includes a central block 226 for PU222A and a lower-right block 224 for PU0 122A. Figure 8(a) also shows an outer block 228 from which motion information can be predicted from the motion information of PU0 222A, as described below. Figure 8(b) shows a current picture 230 including a current block 238 from which motion information should be predicted. In particular, Figure 8(b) shows a collated picture 234 for the current picture 230 (including a collated block 240 for the current block 238), a current reference picture 232, and a collated reference picture 236. The collated block 240 is predicted using motion vectors 244, which are used as a temporal motion vector predictor (TMVP) 242 for the motion information of block 238.

[0101]

[0122] A video coder (e.g., video encoder 20 and / or video decoder 30) may add a TMVP candidate (e.g., TMVP candidate 242) to the MV candidate list after any spatial motion vector candidate, if TMVP is enabled and TMVP candidates are available. The process of deriving motion vectors for TMVP candidates is the same for both merge mode and AMVP mode. However, the target reference index for TMVP candidates in merge mode is set to 0, according to HEVC.

[0102]

[0123] The primary block location for TMVP candidate derivation is the lower right block outside the collated PU, as shown in Figure 8(a) as block 224 for PU0 222A, to compensate for the bias to the upper and left blocks used to generate spatial adjacency candidates. However, if block 224 is currently located outside the CTB row or motion information is not available for block 224, the block is replaced with the central block 226 of the PU, as shown in Figure 8(a). The motion vector for TMVP candidate 242 is derived from the collated block 240 of the collated picture 234, as shown in the slice level information.

[0103]

[0124] Similar to the time-direct mode in AVC, the motion vector of a TMVP candidate may undergo motion vector scaling, which is performed to compensate for the POC distance difference between the current picture 230 and the current reference picture 232, and between the colocated picture 234 and the colocated reference picture 236. That is, the motion vector 244 may be scaled based on these POC differences to generate a TMVP candidate 242. Other aspects of motion prediction in HEVC are described below.

[0104]

[0125] Motion vector scaling can be performed in HEVC. The value of the motion vector is assumed to be proportional to the distance between pictures at presentation time. The motion vector relates two pictures, namely a reference picture and a picture containing the motion vector (i.e., a containing picture). When the motion vector is used to predict other motion vectors, the distance between the containing picture and the reference picture is calculated based on the picture order count (POC) value.

[0105]

[0126] For a motion vector to be predicted, both its associated containing picture and reference picture can be different. Therefore, a new distance (based on POC) is calculated. The motion vector is also scaled based on these two POC distances. For spatially adjacent candidates, the containing pictures for two motion vectors are the same, but the reference pictures are different. In HEVC, motion vector scaling is applied to both TMVP and AMVP for spatially and temporally adjacent candidates.

[0106]

[0127] Furthermore, pseudo-motion vector candidate generation may be performed in HEVC. If the motion vector candidate list is incomplete (for example, containing fewer candidates than the prescribed number), the video encoder 20 and / or video decoder 30 may generate pseudo-motion vector candidates. The video encoder 20 and / or video decoder 30 generate pseudo-motion vector candidates and insert them at the end of the motion vector candidate list until the motion vector candidate list contains the prescribed number of candidates.

[0107]

[0128] In merge mode, there can be two types of pseudo-MV candidates: bidirectional combinatorial pseudo-motion vector candidates derived for B slices, and zero motion vector candidates used only in AMVP when the first type (i.e., combinatorial pseudo-motion vector candidates) does not provide enough pseudo-candidates to fill the candidate list.

[0108]

[0129] For each pair of candidates already in the candidate list that possess the required motion information, a bidirectional combined motion vector candidate is derived by combining the motion vector of a first candidate that references a picture in List 0 and the motion vector of a second candidate that references a picture in List 1. A 0 motion vector candidate simply points to a collated block in another picture (for example, a 0 motion vector candidate is (0,0)).

[0109]

[0130] Furthermore, a video coder configured according to HEVC may perform a pruning process for candidate insertion. Motion vector candidates from different blocks may coincidentally be the same, which reduces the efficiency of the merge / AMVP candidate list. That is, having multiple motion vector candidates with the same value reduces the actual number of motion vectors to test and select. To solve this problem, a pruning process may be applied. The pruning process compares one motion vector candidate in the current candidate list with other motion vector candidates to avoid inserting equivalent candidates in some situations. To reduce complexity, only a limited number of pruning processes are applied instead of comparing each potential motion vector candidate (one) with all other existing motion vector candidates (one).

[0110]

[0131] Figure 9 shows an exemplary prediction structure for 3D-HEVC. 3D-HEVC is a 3D video extension of HEVC under development by JCT-3V. Several techniques related to the techniques of this disclosure are described with respect to Figures 9 and 10 below. Figure 9 shows a multi-view prediction structure for the case of three views. V3 represents the base view, and pictures in the non-base view (V1 or V5) can be predicted from pictures in the dependent (base) view of the same time instance. Interview sample prediction (from reconstructed samples) is supported in multi-view HEVC (MV-HEVC), and its general prediction structure is shown in Figure 10.

[0111]

[0132] Both MV-HEVC and 3D-HEVC conform to HEVC so that the base (texture) view can be decoded by an HEVC (version 1) decoder. Test models for MV-HEVC and 3D-HEVC are described in Zhang et al., "Test Model 6 of 3D-HEVC and MV-HEVC," JCT-3V document ISO / IEC JTC1 / SC29 / WG11 N13940, available as of January 26, 2015, at the website mpeg.chiariglione.org / standards / mpeg-h / high-efficiency-video-coding / test-model-6-3d-hevc-and-mv-hevc.

[0112]

[0133] In MV-HEVC, the current picture in a non-base view can be predicted by these pictures by including all of the pictures in the same view and all of the pictures in the reference view of the same time instance in the picture's reference picture list. Thus, the reference picture list of the current picture includes both time-referenced pictures and interview-referenced pictures. The motion vector associated with the reference index corresponding to a time-referenced picture is denoted as the time-motion vector. The motion vector associated with the reference index corresponding to an interview-referenced picture is denoted as the disparity-motion vector. 3D-HEVC supports all the features in MV-HEVC. Therefore, interview sample prediction as described above is enabled.

[0113]

[0134] Additionally, 3D-HEVC supports more advanced texture-only coding tools and depth-related / dependent coding tools. Texture-only coding tools often use the identification of corresponding blocks (between views) that may belong to the same object. Therefore, disparity vector derivation is one technique used in 3D-HEVC.

[0114]

[0135] Figure 10 is a conceptual diagram illustrating sub-PU-based interview motion prediction in 3D-HEVC. Figure 10 shows the current picture 360 ​​in the current view (V1) and the collated picture 362 in the reference view (V0). The current picture 360 ​​contains the current PU364, which includes four sub-PU366A-366D (sub-PU366). Each of the disparity vectors 374A-374D (disparity vector 374) identifies the corresponding sub-PU368A-368D in the collated picture 362 to the sub-PU366. In 3D-HEVC, the sub-PU-level interview motion prediction method is used for interview merge candidates, i.e., candidates derived from the reference block in the reference view.

[0115]

[0136] When such a mode is enabled, the current PU364 may correspond to a reference area in the reference view (which has the same size as the current PU, identified by the disparity vector), and the reference area may have richer motion information (e.g., many different related motion vectors) than required for generating one set of motion information for the PU. Thus, the sub-PU level inter-view motion prediction (SPIVMP) method may be used, as shown in Figure 10. This mode may also be signaled as a special merge candidate. Each sub-PU contains the full set of motion information. Therefore, a PU may contain multiple sets of motion information.

[0116]

[0137] Furthermore, in 3D-HEVC, sub-PU-based motion parameter inheritance (MPI) can be used. In 3D-HEVC depth coding, MPI candidates derived from texture views can also be extended in a similar manner to sub-PU level interview motion prediction. For example, if the current depth PU has a collated region containing multiple PUs, the current depth PU can be separated into sub-PUs, each sub-PU having a different set of motion information. This method is called sub-PU MPI.

[0117]

[0138] An exemplary sub-PU relational technique for 2D video coding is described in U.S. Patent Application No. 14 / 497,128, filed September 25, 2014, which is incorporated herein by reference in its entirety. In U.S. Patent Application No. 14 / 497,128, a sub-PU-based advanced TMVP (ATMVP) design is proposed.

[0118]

[0139] In single-layer coding, a two-stage advanced temporal motion vector prediction (ATMPV) design may be used. The first stage is used to derive a vector that identifies the corresponding block of the current predictor unit (PU) in the reference picture, and the second stage is to extract multiple sets of motion information from the corresponding block and assign them to sub-PUs of the PU. Each sub-PU of the PU is therefore motion compensated separately. The ATMVP concept can be summarized as follows: (1) The vector in the first stage can be derived from the spatial and temporal adjacent blocks of the current PU. (2) This process can be achieved by activating the merge candidate among all other merge candidates. Applicable to single-layer coding and sub-PU temporal motion vector prediction, the PU or CU may have motion improvement data to be transmitted over the predictor.

[0119]

[0140] Several aspects of U.S. Patent Application No. 14 / 497,128 are highlighted below: 1. The first stage of vector derivation may also be simplified by simply using a zero vector. 2. The first stage of vector derivation may include identifying the motion vector and its associated picture together. Various methods have been proposed for selecting the associated picture and further determining that the motion vector is the first stage vector. 3. If motion information is unavailable during the above process, the “first stage vector” is used for substitution. 4. The motion vector identified from the time neighbor may be scaled to be used for the sub-PU in a manner similar to motion vector scaling in TMVP. However, to which reference picture such a motion vector can be scaled may be designed using one of the following methods.

[0120] a. The picture is now identified by its fixed reference index.

[0121] b. If the picture is also available in the current picture's list of referenced pictures, it is identified as the referenced picture of the corresponding time neighbor.

[0122] c. The picture is set to be a collated picture, identified in the first stage and from the location where the motion vector was captured.

[0123]

[0141] To address several design issues in U.S. Patent No. 14 / 497,128, the following technique was proposed in U.S. Patent No. 15 / 005,564, filed on January 25, 2016, the entire content of which is incorporated herein by reference.

[0124] 1. For example, the position of ATMVP candidates when inserted as a merge candidate list a. Assume that spatial candidates and TMVP candidates are inserted into the merge candidate list in a certain order. ATMVP candidates can be inserted in any relatively fixed position among those candidates.

[0125] i. In one alternative, for example, the ATMVP candidate may be inserted into the merge candidate list after the first two spatial candidates, e.g., A1 and B1.

[0126] ii. In one alternative, for example, the ATMVP candidate may be inserted after the first three spatial candidates, for example, A1, B1, and B0.

[0127] iii. In one alternative, for example, the ATMVP candidate may be inserted after the first four candidates, for example, A1, B1, B0, and A0.

[0128] iv. As an alternative, for example, the ATMVP candidate may be inserted immediately before the TMVP candidate.

[0129] v. Alternatively, for example, the ATMVP candidate may be inserted immediately after the TMVP candidate.

[0130] b. Alternatively, the position of the ATMVP candidate in the candidate list may be signaled in the bitstream. Furthermore, the positions of other candidates, including the TMVP candidate, may also be signaled.

[0131] 2. An availability check for ATMVP candidates may be applied by accessing only one set of motion information. If such a set of information is unavailable, for example, when one block is intracoded, all ATMVP candidates are considered unavailable. In that case, the ATMVP is not inserted into the merge list.

[0132] a. A central position or central sub-PU is used purely to examine the availability of ATMVP candidates. When a central sub-PU is used, it is selected to cover a central position (for example, a central 3 position with coordinates (W / 2, H / 2) relative to the top-left sample of the PU, where W×H is the size of the PU). Such a position or central sub-PU may be used with a time vector to identify the corresponding block in the motion source picture. A set of motion information from the block covering the central position of the corresponding block is identified.

[0133] 3. A representative set of motion information for the ATMVP coded PU from the sub-PU.

[0134] a. To form ATMVP candidates, a representative set of motion information is first formed.

[0135] b. Such a representative set of motion information may be derived from fixed positions or fixed sub-PUs. It may be selected in a similar manner to that of the set of motion information used to determine the availability of ATMVP candidates, as described in Clause 2.

[0136] c. When a sub-PU identifies its own set of motion information and it is unavailable, it is set to be equal to a representative set of motion information.

[0137] d. If the representative set of motion information is configured to be that of the sub-PU, additional motion memory is not currently required on the decoder side for the CTU or slice in the worst-case scenario.

[0138] e. Such a representative set of motion information is used in all scenarios, including pruning, so that the decoding process is used to generate combinatorial bipredictive merging candidates when the decoding process requires that the entire PU be represented by one set of motion information.

[0139] 4. ATMVP candidates are pruned using TMVP candidates, and the interaction between TMVP and ATMVP may be considered. Detailed techniques are described below.

[0140] a. Pruning of sub-PU-based candidates, such as ATMVP candidates, using normal candidates may be done by using a representative set of motion information for such sub-PU-based candidates (as in the case of Clause 3). If such a set of motion information is the same as that for the normal merge candidate, the two candidates are considered the same.

[0141] b. Alternatively, as an addition, a check is performed to determine whether the ATMVP contains multiple different sets of motion information for multiple sub-PUs. If at least two different sets are identified, the sub-PU base candidate is not used for pruning, i.e., is considered different from the other candidates. Otherwise, it may be used for pruning (for example, it may be pruned during the pruning process).

[0142] c. Alternatively, as an addition, ATMVP candidates may be pruned using only spatial candidates, e.g., left and top spatial candidates, having positions indicated as A1 and B1.

[0143] d. Alternatively, only one candidate, either an ATMVP candidate or a TMVP candidate, is formed from the time reference. When an ATMVP is available, the candidate is an ATMVP; otherwise, the candidate is a TMVP. Such a candidate is inserted into the merge candidate list at a position similar to that of a TMVP. In this case, the maximum number of candidates can be kept immutable.

[0144] i. Alternatively, TMVP is always disabled, even when ATMVP is unavailable.

[0145] ii. Alternatively, TMVP should only be used when ATMVP is unavailable.

[0146] e. Alternatively, when ATMVP is available and TMVP is unavailable, one set of motion information from one sub-PU is used as a candidate for TMVP. In this case, furthermore, no pruning process is applied between ATMVP and TMVP.

[0147] f. Alternatively or additionally, the time vector used for ATMVP may be used for TMVP so that the lower-right position or center 3 position, as currently used for TMVP in HEVC, does not need to be used.

[0148] i. Alternatively, the positions identified by the time vector, along with the bottom-right and center three positions, are considered together to give available TMVP candidates.

[0149] 5. Multiple availability checks for ATMVPs are supported to give a higher probability that ATMVP candidates will be more accurate and efficient. When the current ATMVP candidate from a motion source picture is unavailable, such as identified by a first time vector (as shown in Figure 9), another picture may be considered a motion source picture. When another picture is considered, it may be associated with a different second time vector, or simply with a second time vector scaled from the first time vector pointing to the unavailable ATMVP candidate.

[0150] a. The second time vector can identify ATMVP candidates in the second motion source picture, and the same availability check can be applied. If an ATMVP candidate such as that derived from the second motion source picture is available, the ATMVP candidate is derived and no other picture needs to be checked; otherwise, other pictures need to be checked as motion source pictures.

[0151] b. The pictures to be examined can be any pictures in the current picture's reference picture list, in a given order. For each list, the pictures are examined in ascending order of their reference index. List X is examined first, followed by the pictures in list Y (which is 1-X).

[0152] i. List X is selected so that List X is a list containing the collated pictures to be used for TMVP.

[0153] ii. Alternatively, X is simply set to be 1 or 0.

[0154] c. The pictures to be examined are those identified by the motion vectors of spatial neighbors, in a given order.

[0155] 6. Currently, ATMVP can be applied to PU classifications that are asymmetric motion partitions (AMPs), such as 2N×2N, N×N, 2N×N, N×2N, or 2N×N / 2.

[0156] a. Alternatively, if other partition sizes are possible, ATMVP may also be supported, such as 64x8.

[0157] b. Alternatively, the mode may only apply to a few divisions, for example, 2N × 2N.

[0158] 7. ATMVP candidates are marked as a different type of merge candidate.

[0159] 8. When identifying a vector (a time vector, as in the case of the first stage) from a neighbor, multiple adjacent positions, for example, adjacent positions used in the construction of the merge candidate list, may be examined in order. For each neighbor, the motion vectors corresponding to reference picture list 0 (list 0) or reference picture list 1 (list 1) may be examined in order. When two motion vectors are available, the motion vector in list X may be examined first, followed by list Y (where Y is equal to 1-X), so that list X becomes a list containing the collated pictures used for TMVP. In ATMVP, a time vector is also used as a shift of the center position of the sub-PU, where the components of the time vector may need to be shifted to an integer. Such a shifted center position is used to identify the smallest unit to which the motion vector can be assigned, for example, having a size of 4x4 covering the current center position.

[0160] a. Alternatively, the motion vector corresponding to list 0 may be checked before the motion vector corresponding to list 1.

[0161] b. Alternatively, the motion vector corresponding to List 1 may be checked before the motion vector corresponding to List 0.

[0162] c. Alternatively, all motion vectors corresponding to list X in all spatial neighbors are examined in order, followed by motion vectors corresponding to list Y (where Y is equal to 1-X), where list "X" can be a list indicating where the collated picture belongs, or it can simply be set to 0 or 1.

[0163] d. The order of spatial neighbors may be the same as the order used in HEVC merge mode.

[0164] 9. In the first stage of identification, when the time vector does not contain information to identify the reference picture, the motion source picture, as shown in Figure 9, may simply be set to be a fixed picture, for example, a colocated picture used for TMVP.

[0165] a. In such cases, the vector can only be identified from the motion vector pointing to such a fixed picture.

[0166] b. In such cases, the vector is identified only from the motion vector pointing to any picture, but can further be scaled toward a stationary picture.

[0167] 10. When the first stage is to identify the vector, which will identify the reference picture, one or more of the following additional checks may be applied to the candidate motion vector, such as the motion source picture shown in Figure 9.

[0168] a. If a motion vector is associated with a picture or slice that is being intracoded, such motion vector may be considered unavailable and may not be used to convert to that vector.

[0169] b. If a motion vector identifies an intrablock in the associated picture (for example, by adding the current center coordinates with the motion vector), such a motion vector may be considered unavailable and not used for conversion to that vector.

[0170] 11. When in the first stage of identifying a vector, the components of the vector may be set to (current PU width 1 / 2, current PU height 1 / 2) so that it identifies the bottom-right pixel position in the motion source picture, where (x,y) represents the horizontal and vertical components of a motion vector.

[0171] a. Alternatively, the components of the vector may be set to (sum(width of 1 / 2 of the current PU, M), sum(height of 1 / 2 of the current PU, N)), where the function sum(a, b) returns the sum of a and b. In one example, when motion information is stored in a 4x4 unit, both M and N are set to be equal to 2. In another example, when motion information is stored in an 8x8 unit, both M and N are set to be equal to 4.

[0172] 12. When ATMVP is applied, the subblock / subPU size is signaled in the parameter set, for example, the sequence parameter set of the picture parameter set. The size ranges from the minimum PU size to the CTU size. The size can also be predefined or signaled. The size can be as small as, for example, 4x4. Alternatively, the subblock / subPU size can be derived based on the size of the PU or CU. For example, the subblock / subPU can be set to equal max(4x4, (CU width)>>M). The value of M can be predefined or signaled in the bitstream.

[0173] 13. The maximum number of merge candidates can be increased by 1, as ATMVP may be considered a new merge candidate. For example, compared to HEVC, which requires a maximum of 5 candidates in the merge candidate list after pruning, the maximum number of merge candidates can be increased to 6.

[0174] a. Alternatively, pruning with or unification with conventional TMVP candidates may be performed on ATMVP such that the maximum number of merge candidates can be kept constant.

[0175] b. Alternatively, when ATMVP is identified as available, the spatial adjacency candidate is excluded from the merge candidate list, for example, the last spatial adjacency candidate in the fetching order is excluded.

[0176] 14. When multiple spatially adjacent motion vectors are considered to derive a time vector, the similarity of the motion vectors can be calculated based on the adjacent motion vectors of the current PU, as well as adjacent motion vectors identified by a specific time vector set to be equal to the motion vector. The one that yields the highest motion similarity can be selected as the final time vector.

[0177] a. In one alternative, for each motion vector from an adjacent position N, the motion vector identifies a block in the motion source picture (currently the same size as the PU), where its adjacent position N contains a set of motion information. This set of motion vectors is compared to the set of motion information, as in the case of the adjacent position N of the current block.

[0178] b. In another alternative, for each motion vector from an adjacent position N, the motion vector identifies a block in the motion source picture, where its adjacent position contains multiple sets of motion information. These multiple sets of motion vectors are compared with multiple sets of motion information from the adjacent positions of the current PU at the same relative position. The motion information similarity is calculated. For example, the current PU has the following sets of motion information from A1, B1, A0, and B0, represented as MIA1, MIB1, MIA0, and MIB0. For the time vector TV, it identifies the block in the motion source picture corresponding to the PU. Such a block has motion information from the same relative A1, B1, A0, and B0 positions, represented as TMIA1, TMIB1, TMIA0, and TMIB0. The motion similarity determined by TV is:

[0179]

number

[0180] It is calculated as follows, where MVSim defines the similarity between two sets of motion information.

[0181] c. In both of the above cases, motion similarity MVSim may be used, where the two input parameters are two sets of motion information, each containing up to two motion vectors and two reference indices. Each pair of motion vectors in list X is actually related to a different picture, i.e., a reference picture in a different list X of the current picture and the motion source picture. For each of the two motion vectors MVXN and TMVXN (where X is equal to 0 or 1), the motion vector difference MVDXN can be calculated as MVXN-TMVXN. Then the difference MVSimX is calculated, for example,

[0182]

number

[0183] or

[0184]

number

[0185] i. In order to have a unified calculation of motion differences, both motion vectors need to be scaled to the same fixed picture, which may be the first reference picture RefPicListX[0] of the current picture list X.

[0186] ii. If the availability of motion vectors in list X from the first set is different from the availability of motion vectors in list X from the second set, that is, if the reference index of one is -1 and the reference index of the other is not -1, then such two sets of motion information are considered dissimilar in direction X. If the two sets are dissimilar in both sets, the final MVSim function may return a large value T, which may be considered, for example, infinity.

[0187] iii. Alternatively, for pairs of motion information sets, if one is predicted from list X (where X is equal to 0 or 1) rather than list Y (where Y is equal to 1-X), and the other has the same status, a weighting between 1 and 2 (for example, MVSim is equal to MVSimX * 1.5) may be used. When one set is predicted from list X only and the other set is predicted from list Y only, MVSim is set to a large value T.

[0188] iv. Alternatively, for any set of motion information, both motion vectors are generated as long as one motion vector is available. If only one motion vector is available (corresponding to list X), it is scaled to form the motion vector corresponding to the other list Y.

[0189] d. Alternatively, the motion vector can be measured based on the difference between the adjacent pixels of the current PU and the adjacent pixels of the block identified by the motion vector (of the same size as the current PU). The motion vector that yields the smallest difference can be selected as the final time vector.

[0190] 15. When deriving the time vector of the current block, motion vectors and / or time vectors from adjacent blocks coded using ATMVP may have a higher priority than motion vectors from other adjacent blocks.

[0191] a. In one example, only the time vectors of adjacent blocks are checked first, and the first available one may be set as the time vector of the current block. Only if no such time vector exists is the normal motion vector checked further. In this case, the time vector for the ATMVP coded block needs to be stored.

[0192] b. In another example, only motion vectors from ATMVP-coded adjacent blocks are checked first, and the first available one may be set as the time vector of the current block. Only if no such time vector exists, are the normal motion vectors checked next.

[0193] c. In another example, only motion vectors from ATMVP-coded adjacent blocks are checked first, and the first available one may be set as the time vector of the current block. If no such motion vector is available, the check for time vectors proceeds as in the case of clause 15a.

[0194] d. In another example, time vectors from adjacent blocks are checked first, and the first available one may be set as the time vector for the current block. If no such motion vector is available, the time vector check continues as in the case of clause 15b.

[0195] e. In another example, the time vector and motion vector of an ATMVP-coded adjacent block are checked first, and the first available one may be set as the time vector of the current block. Only if no such time vector and motion vector exist, the normal motion vector is then checked.

[0196] 16. When multiple spatially adjacent motion vectors are considered to derive a time vector, the motion vectors may be selected so as to minimize the distortion calculated from the pixel region, for example, template matching may be used to derive the time vector so that the one that yields the minimum matching cost is selected as the final time vector.

[0197] 17. The derivation of the set of motion information from the corresponding block (in the motion source picture) is performed such that, for the current sub-PU of an ATMVP candidate, the motion vector is considered available for list X (by scaling the MVX) when the motion vector is available in the corresponding block for any list X (we indicate that the motion vector is MVX). If the motion vector is not available in the corresponding block for any list X, the motion vector is considered unavailable for list X.

[0198] a. Alternatively, when the motion vector in the corresponding block is unavailable for list X but available for list 1-X (where 1-X is denoted by Y and the motion vector is denoted as MVY), the motion vector is still considered available for list X (by scaling MVY toward the target reference picture in list X).

[0199] b. Alternatively, or additionally, when both motion vectors in the corresponding blocks for list X and list (equal to 1-X)Y are available, the motion vectors from list X and list Y are not used as they are currently needed to directly scale and generate the two motion vectors of the subPU by scaling.

[0200] i. In one example, when constructing an ATMVP candidate, a low-latency check, similar to that performed in TMVP, is applied to each sub-PU. For every picture (indicated by refPic) in every reference picture list of the current slice, if the picture order count (POC) value of refPic is less than the POC of the current slice, the current slice is considered in low-latency mode. In this low-latency mode, motion vectors from lists X and Y are scaled to generate motion vectors for the current sub-PU for lists X and Y, respectively. When not in low-latency mode, only one motion vector MVZ from MVX or MVY is selected and scaled to generate two motion vectors for the current sub-PU. Similar to TMVP, in such cases, Z is set to collocated_from_l0_flag, meaning it depends on whether a collocated picture, as in the case of TMVP, is in list X or list Y of the current pictures. Alternatively, Z is set to X, i.e., if the motion source picture is identified from list X, then Z is set to X. Alternatively, as an addition, if the motion source picture belongs to both reference picture lists, and RefPicList0[idx0] is the first motion source picture in list 0, and RefPicList(1)[idx1] is the first motion source picture in list 1, then Z is set to 0 if idx0 is less than or equal to idx1, and to 1 otherwise.

[0201] 18. Motion source pictures may be signaled by the video encoder 20 in the encoded bitstream, for example, to be generated. In detail, a flag indicating whether the motion source picture is from list 0 or list 1 is signaled for the B slice. Alternatively, as an addition, a reference index of the current picture to list 0 or list 1 may be signaled to identify the motion source picture.

[0202]

[0142] When identifying a time vector, if it points to an intra-coded block in the associated motion source picture, the vector is considered unavailable (and therefore other vectors may be considered).

[0203]

[0143] Figure 11 is a conceptual diagram showing sub-PU motion prediction from a reference picture. In this example, the current picture 380 contains the current PU 384 (e.g., PU). In this example, motion vector 392 identifies PU 386 of reference picture 382 relative to PU 384. PU 386 is divided into sub-PUs 388A to 388D, each having its own motion vector 390A to 390D. Therefore, although the current PU 384 is not actually divided into separate sub-PUs, in this example, the current PU 384 can be predicted using motion information from sub-PUs 388A to 388D. In particular, the video coder can code the sub-PUs of the current PU 384 using their respective motion vectors 390A to 390D. However, the video coder does not need to code a syntax element indicating that the current PU 384 is split into sub-PUs. In this way, the current PU384 can be effectively predicted using multiple motion vectors 390A-390D, inherited from each of the sub-PU388A-388D, without the signaling overhead of the syntax elements used to split the current PU384 into multiple sub-PUs.

[0204]

[0144] Figure 12 is a conceptual diagram showing the relevant pictures in ATMVP (similar to TMVP). In particular, Figure 12 shows the current picture 404, the motion source picture 406, and the reference pictures 400 and 402. More specifically, the current picture 404 includes the current block 408. The time motion vector 412 identifies the corresponding block 410 of the motion source picture 406 relative to the current block 408. The corresponding block 410 then includes the motion vector 414, which references the reference picture 402 and acts as an advanced time motion vector predictor for at least a portion of the current block 408, for example, for a sub-PU of the current block 408. That is, the motion vector 414 may be added as a candidate motion vector predictor for the current block 408. If selected, at least a portion of the current block 408 may be predicted using the corresponding motion vector, i.e., the motion vector 416, which references the reference picture 400.

[0205]

[0145] Furthermore, a sub-PU relation technique for HEVC is described in U.S. Patent Application No. 15 / 176,790, filed July 9, 2016, the entirety of which is incorporated herein by reference. To improve performance using sub-PU motion prediction, spatial-temporal motion information of neighboring sub-PUs (ATMVP_EXT) is exploited. In this example, the motion vector for each sub-PU is derived from information of neighboring blocks in a three-dimensional domain. This means that neighboring blocks can be spatial neighbors in the current picture or temporal neighbors in previous coded pictures. Figure 13 shows a flowchart of the spatial-temporal motion vector predictor (STMVP) derivation process. In addition to what is described below, the methods described above for ATMVP (e.g., Clauses 1, 2, 3, 4, 6, 7, 12, 13) can be directly extended to STMVP.

[0206]

[0146] As shown in Figure 13, the video encoder 20 and / or video decoder 30 may be configured to acquire a motion field available from spatially or temporally adjacent blocks for the currently sub-PU (430). In this context, the motion field is a collection of motion vectors selected as the best for spatially / temporally adjacent blocks. For example, blocks located to the left or above the current block have already been coded and the best motion vectors are available before coding the currently sub-PU. The motion information available from adjacent blocks is equivalent in both the video encoder 20 and the video decoder 30. The motion information includes one or two three-dimensional vectors (MVx, Mvy, in the temporal direction), i.e., one vector for single prediction and two vectors for biprediction. The video encoder 20 and / or video decoder 30 then derive motion information from the acquired adjacent motion field (432). The video encoder 20 and / or video decoder 30 then determine whether all of the sub-PUs have been processed (434). If no, the video encoder 20 and / or video decoder 30 move to the next sub-PU. If yes, the video encoder 20 and / or video decoder 30 may determine the availability of the space-time sub-PU motion predictor (436). If available, the video encoder 20 and / or video decoder 30 insert the space-time sub-PU motion predictor into the merge list.

[0207]

[0147] In the following description, the term “block” is used to refer to a block unit for storing predictive relational information, such as inter or intra predictive, intra predictive mode, motion information, etc. Such predictive information may be stored and used to code future blocks, for example, to predict predictive mode information for future blocks. In AVC and HEVC, the size of such blocks is 4x4. Note that in the following description, “PU” is used to indicate an intercoded block unit and sub-PU to indicate a unit that derives motion information from adjacent blocks. Any combination of the following techniques may be applied.

[0208]

[0148] In one example, the video encoder 20 and / or video decoder 30 may be configured to acquire motion information from adjacent blocks. Sub-PUs and adjacent blocks may have different sizes. Consider a PU with multiple sub-PUs. The size of a sub-PU is usually equal to or larger than the size of its adjacent block. In one example, as shown in Figure 14, the hashed squares represent adjacent blocks (a, b, ..., i) that are currently outside the PU, and the remaining unhashed squares (A, B, ..., P) represent sub-PUs that are currently in the PU. As shown in Figure 14, the size of a sub-PU is the same as the size of its adjacent block. In one example, the size of a sub-PU is equal to 4x4, but sub-PUs of different sizes may be used. Figure 15 shows another example where a sub-PU is larger than its adjacent block. In other examples, sub-PUs may take non-square shapes such as rectangles or triangles. In some examples, the size of a sub-PU may be signaled in the slice header.

[0209]

[0149] In other examples, the process in clause 12 of the above description relating to ATMPV can be extended to STMVP. For example, when STMVP is applied, the subblock / subPU size is signaled in the parameter set, e.g., the sequence parameter set of the picture parameter set. The size ranges from the minimum PU size to the CTU size. The size can also be predefined or signaled. The size can be as small as, for example, 4x4. Alternatively, the subblock / subPU size can be derived based on the size of the PU or CU. For example, the subblock / subPU can be set to equal max(4x4, (CU width)>>M). The value of M can be predefined or signaled in the bitstream.

[0210]

[0150] In STMVP, different inspection sequences of sub-PUs may be used. In the example in Figure 14, we assume in the following description that a raster scan sequence (A, B, C, D, E...) is applied to the sub-PUs for the purpose of deriving their motion predictions. However, it should be noted that other scan sequences may also be applied, and the techniques of this disclosure are not limited to raster scan sequences.

[0211]

[0151] In STMVP, adjacent blocks can be classified into two different types: spatial and temporal. Spatial adjacent blocks are already coded blocks or already traversed sub-PUs that are currently in a picture or slice and are adjacent to a sub-PU. Temporal adjacent blocks are blocks in a previous coded picture that are adjacent to a collated block in the current sub-PU. In one example, all reference pictures related to the current PU are used to obtain temporal adjacent blocks. In another example, a subset of reference pictures is used for STMVP derivation. For example, only the first entry of each reference picture list is used.

[0212]

[0152] Following this definition, referring to Figure 14, in the case of subPU(A), all adjacent blocks (a, b, ... i) in the previous coded picture and their collated blocks are spatially and temporally adjacent blocks that are treated as available. According to the raster scan order, blocks B, C, D, E... P are not spatially available. Nevertheless, all subPUs (from A to P) are temporally available adjacent blocks for subPU(A) because their motion information can be found in their collated blocks in the previous coded picture. As another example, subPU(G) has spatially adjacent blocks that are available, including a, b... through i and also A through F. In some examples, some restrictions may apply to spatially adjacent blocks, such as assuming that spatially adjacent blocks (i.e., a, b... through i) are in the same LCU / slice / tile.

[0213]

[0153] The video encoder 20 and / or video decoder 30 select a subset of all available neighboring blocks to derive motion information or motion fields for each sub-PU. The subset used for the derivation of each PU may be predefined. In other examples, the subset used for derivation may be signaled as high-level syntax in the slice header, picture parameter set (PPS), and / or sequence parameter set (SPS). To optimize coding performance, the subset may differ for each sub-PU. In practice, a fixed pattern of locations for the subset is preferred for simplicity. For example, each sub-PU may use as a subset its immediately above spatial neighbor, its immediately to its left spatial neighbor, and its immediately below-right temporal neighbor. As shown in Figure 14, when considering sub-PU (J), the above block (F) and the left block (I) are the spatially available neighboring blocks, and the below-right block (O) is the temporally available neighboring block. In the case of such a subset, the sub-PUs currently in the PU are processed sequentially due to processing dependencies.

[0214]

[0154] Different subsets of neighboring blocks can be defined and used to enable parallel processing of each sub-PU in the current PU. In one example, the subset includes only spatial neighbor blocks that do not belong to the current PU, e.g., blocks a, b, ... i. In this case, parallel processing would be possible. In another example, for a given sub-PU, if its spatial neighboring blocks are in the current PU, the colocated blocks of those spatial neighboring blocks can be placed in a subset and used to derive motion information for the current sub-PU. For example, when considering sub-PU(J), the temporally colocated blocks of the upper block (F), left block (I), and lower right block (O) are selected as a subset to derive motion for sub-PU(J). In this case, the subset for sub-PU(J) includes three temporal neighboring blocks. In another example, partial parallel processing can be enabled, where one PU is split into several regions, and each region (covering several sub-PUs) can be processed independently.

[0215]

[0155] Sometimes adjacent blocks are intracoded, and it is desirable to have rules for determining alternative motion information for those blocks for better motion prediction and coding efficiency. For example, in subPU(A), blocks b, c, and f may be intracoded, and blocks a, d, e, g, h, and i may be interconnected.

[0216]

[0156] For spatial neighbors, a predefined order may be used to populate the motion information of the intra-coded block with the motion information of the first intercoded block found. For example, the search order for upper neighbors may be set to start from the immediately above neighbor, moving to the rightmost neighbor, which means the order b, c, d, and e. The search order for left neighbors may be set to start from the immediately to the left neighbor, moving downwards to the lowest neighbor. In this example, the order is f, g, h, and then i. If an intercoded block is not found throughout the search process, the upper or left spatial neighbor is considered unavailable.

[0217]

[0157] The same rules specified in the TMVP derivation may be used for time neighbors. However, note that other rules, such as those based on direction of movement, time distance (searching in different reference pictures), and spatial location, may also be used.

[0218]

[0158] The video encoder 20 and / or video decoder 30 may then derive motion information for a given subPU. As part of this process, the video encoder 20 and / or video decoder 30 may perform target reference picture determination and motion vector scaling. For adjacent blocks, motion vector scaling may be applied to the motion vectors associated with the adjacent blocks based on each reference picture list in order to map the motion vectors of all adjacent blocks to the same reference picture in each list. In this example, there may be two steps: firstly, to determine the source motion vector to be used for scaling, and secondly, to determine the target reference picture on which the source motion vector is projected.

[0219]

[0159] In the first step, several methods may be used.

[0220] (a) For each reference list, motion vector scaling is independent of motion vectors in other reference lists. If there are no motion vectors in a reference list for the motion information of a given block (for example, in single-prediction mode instead of bi-prediction mode), motion vector scaling is not performed for that list.

[0221] (b) Motion vector scaling is not independent of motion vectors in other reference lists. For motion information of a given block, if motion vectors are available in a reference list, motion vectors can be scaled from motion vectors in another reference list.

[0222] (c) Both motion vectors are scaled from a single predefined reference list (as in the case of TMVP described above).

[0223]

[0160] As an example, method (a) is used to scale the motion vectors of spatially adjacent blocks, and method (c) is used to scale the motion vectors of temporally adjacent blocks.

[0224]

[0161] With respect to the second step, the video encoder 20 and / or video decoder 30 may select a target reference picture according to a rule based on motion information (e.g., reference pictures) of available spatially adjacent blocks. One example of such a rule is a majority rule, i.e., selecting a reference picture shared by the majority of the block. In this case, there is no signaling required from the encoder to the decoder for the target reference picture, since the same information can be inferred on the decoder side using the same rule. Alternatively, such a reference picture may also be explicitly specified in the slice header or signaled to the decoder in some other way. The target reference picture is determined as the first reference picture (refidx=0) in each reference list.

[0225]

[0162] The video encoder 20 and / or video decoder 30 may be configured to derive motion information for a given sub-PU. As shown in the previous section, motion information is taken from adjacent blocks and (if necessary) a motion scaling process is performed to derive motion information for the current sub-PU. Assume there are N available adjacent blocks, each having motion information for one given sub-PU. First, a prediction instruction (InterDir) is determined. An exemplary method is as follows:

[0226] a.InterDir is initialized to 0, and then loops through the movement information of N available adjacent blocks. b. If there is at least one motion vector in list 0, then InterDir = (InterDir bitwiseOR1). c. If there is at least one motion vector in List 1, then InterDir = (InterDir bitwise OR 2).

[0227] Here, "bitwiseOR" represents a bitwise OR operation. The values ​​of InterDir are defined as 0 (no inter prediction), 1 (inter prediction based on list 0), 2 (inter prediction based on list 1), and 3 (inter prediction based on both list 0 and list 1).

[0228]

[0163] In another example, similar to the decision regarding the target reference picture for motion vector scaling described above, a majority rule may be used to determine the value of InterDir for a given subPU based on the motion information of all available adjacent blocks.

[0229]

[0164] After InterDir is determined, motion vectors can be derived. For each reference list based on the derived InterDir, there can be M motion vectors (M≦N) available through motion vector scaling with respect to the target reference picture, as described above. The motion vectors for a reference list can be derived as follows:

[0230]

number

[0231] Here, w i and w j These are weighting factors for the horizontal and vertical motion components, respectively, O i and O j This is an offset value that depends on the weighting factor.

[0232]

[0165] The weighting factor can be determined based on various factors. For example, the same rule can be applied to all sub-PUs within a single PU. The rule can be defined as follows: For example, the weighting factor can be determined based on the location distance between the current sub-PU and the corresponding adjacent block. In another example, the weighting factor can also be determined based on the POC distance between the target reference picture and the reference picture related to the motion vector of the corresponding adjacent block before scaling. In yet another example, the weighting factor can be determined based on the motion vector difference or consistency. Also, for simplicity, all weighting factors can be set to 1.

[0233]

[0166] In another example, different rules may apply to sub-PUs within a single PU. For example, the above rules may apply, and furthermore, for sub-PUs located in the first row / first column, the weighting factor for motion vectors derived from time-adjacent blocks is set to 0, and for the remaining blocks, the weighting factor for motion vectors derived from spatial-adjacent blocks is set to 0.

[0234]

[0167] It should be noted that in practice, the above formula may be implemented as is or simplified for easier implementation. For example, fixed-point arithmetic may be used to approximate the above formula to avoid division or floating-point arithmetic. One example is that to avoid dividing by 3, one may choose to multiply by 43 / 128 instead, replacing the division operation with multiplication and bit shifts. These variations in implementation should be considered covered under the same intent as the techniques of this disclosure. Alternatively, nonlinear operations, such as median filters, may also be applied to derive motion vectors.

[0235]

[0168] The video encoder 20 and / or video decoder 30 may also be configured to perform availability checks during the candidate list construction process for STMVP. It is proposed that the STMVP mode may be reset to be unavailable for one PU even if the motion vector predictor for each sub-PU is available. For example, once the motion vector predictor for each sub-PU is derived for a given PU, several availability checks are performed to determine whether the STMVP mode should be made available for the given PU. Such calculations are used to eliminate cases where it is extremely unlikely that the STMVP mode will ultimately be selected for a given PU. When the STMVP mode is unavailable, mode signaling does not include STMVP. If the STMVP mode is implemented by inserting STMVP into a merge list, the merge list does not include this STMVP candidate when it is determined that the STMVP mode is unavailable. As a result, signaling overhead may be reduced.

[0236]

[0169] Consider a case where one PU is divided into M sub-PUs. In one example, if N1 (N1 ≤ M) of the M sub-PUs have the same motion vector predictor (i.e., the same motion vector and the same reference picture index), STMVP is made available only when N1 is less than a threshold or the predictor is different from other motion vector predictors in the merge list (which have a smaller merge index). In another example, if N2 (N2 ≤ M) sub-PUs under STMVP mode share the same motion vector predictor as the corresponding sub-PU under ATMVP, STMVP is made available only when N2 is less than another threshold. In the example of this disclosure, both the threshold for N1 and the threshold for N2 are set to be equal to M.

[0237]

[0170] If STMVP is available, the video encoder 20 and / or video decoder 30 insert the STMVP candidate into the merge list. The process in Clause 1 above for ATMVP may be extended so that the STMVP candidate is inserted either before or after the ATMVP candidate. In one example, the STMVP candidate is inserted right after the ATMVP candidate in the merge list.

[0238]

[0171] The POC-based MV pruning technique is described in U.S. Patent Application No. 15 / 431,321, filed February 13, 2017, the entirety of which is incorporated herein by reference. To maximize the efficiency of MV prediction, the uniqueness of available MVs may be examined. Otherwise, redundant MVs can lead to inefficient resource use, such as wasting the bit budget or resources of the target device. Therefore, eliminating redundancy in MV candidates, so-called pruning, can be an important step in keeping MVs as unique and diverse as possible in order to give more meaningful MV candidates in MV prediction.

[0239]

[0172] This disclosure describes a POC-based pruning method that has three main advantages: (1) higher accuracy, (2) simplicity, and (3) universality. The proposed technique has higher pruning accuracy because it can detect redundant MVs that were not captured by existing pruning methods. Furthermore, it is simple because no further complexity is required. Finally, POC-based pruning is universal in that it can be applied to a wide range of situations, such as spatial MVs for ATMVP / merge candidates, sub-PU (ATMVP and STMVP) MVs, TMVP, combined MVs, and even 0 MVs.

[0240] [Table 1]

[0241]

[0173] Table 1 summarizes the types of MV pairs that can be pruned using the POC-based method. In category C1, MVs from a regular PU (not a sub-PU) are compared. The comparison can be between two single MVs (e.g., motion vectors for a single prediction) or two bi-MVs (e.g., motion vectors for a bi-prediction). From C2 to C4, (one or more) sub-PU MVs are included in the comparison. In C2, POC-based pruning is used to determine whether all sub-PU MVs within the PU are equal. This can be handled by applying the same technique as in C1 to MVs from sub-PUs. The situation where all sub-PU MVs are equal falls into C3, where MV represents all MVs from sub-PUs, and therefore the same comparison as in C1 applies. However, if not all of a candidate's sub-PU MVs are equal, and there is another candidate with sub-PU MVs, i.e., C4, then POC-based pruning is applied to each pair of MVs from sub-PUs located at the same position within the PU. Both C5 and C6 involve dual MV construction by combining two single MVs, namely a single MV from L0 and another single MV from L1. If the two single MVs are equivalent (e.g., the same MV from the same reference picture), the resulting dual MV will be the same as the single MV, so dual MV construction is not necessary. Therefore, POC-based pruning can help save resources by more accurately detecting equivalent MVs, especially when L0 and L1 have the same reference picture.

[0242]

[0174] Given a given list of candidates, two factors that can determine the efficiency of the merge candidate list are (1) ordering of the candidate list (e.g., how the order of the candidates in the list should be assigned), and (2) pruning (e.g., removing redundancy between those candidates). Generally, it is preferred to have the selected candidate that is most likely to be first in order in the candidate list, as the index to the first candidate can be signaled with fewer steps. Also, having a greater variety of candidates in the list (e.g., less redundancy) increases the chances that a more accurate motion vector exists among the candidates in the list.

[0243]

[0175] The techniques of the present disclosure include techniques for determining a set of merge candidates from a larger group of possible candidates. Furthermore, the present disclosure describes techniques for adaptive selection, ordering, and pruning of merge candidates to achieve greater efficiency of motion vector candidate lists. In the case of adaptive ordering, the proposed techniques leverage additional MV information to assign higher priority (resulting in a smaller index in the list) to candidates that have a higher probability of having more accurate motion information. In the case of adaptive pruning, motion vector differences (MVDs) may be used to determine whether two MVs are equivalent (or very close) by comparing the MVDs with an adaptive threshold.

[0244]

[0176] Due to the flexibility of the proposed techniques, the techniques of this disclosure can be applied to most existing state-of-the-art codecs, such as H.264, HEVC, or H.266, and can be easily extended to different segmented frameworks, such as the QTBT structure described above. Furthermore, different combinations of the proposed techniques can be combined to form a desired solution for a particular application. That is, the following techniques can be applied independently or in any non-mutually exclusive combination.

[0245]

[0177] Furthermore, the following proposed techniques can be performed without additional signaling other than the merge index, as in the case of HEVC or H.266 reference software. That is, in some examples, the video encoder 20 and video decoder 30 may be configured to perform the following techniques based on a given set of rules and without using explicit signaling. The video encoder 20 may be configured to signal the merge index for the current block, and the video decoder 30 may be configured to perform the same procedure that the video encoder 20 performs to derive merge candidates. Thus, using the received merge index, the video decoder 30 may be configured to determine equivalent MV information without any mismatch.

[0246]

[0178] Figure 16 shows an exemplary set of adjacent blocks for block 450 now. As shown in Figure 16, the shaded adjacent blocks a, e, f, j, and k are the same as those used as spatial merge candidates in HEVC. The disclosure proposes using motion information from additional adjacent blocks coded before block 450 now. Such additional adjacent blocks may include adjacent blocks b, c, d, g, h, and i. By deriving the final motion vector candidate list from more adjacent blocks, the probability that a more accurate motion vector is among the motion vector candidate list is increased.

[0247]

[0179] In the example in Figure 16, the current block 450 is 16x16, and each of the adjacent blocks is a 4x4 block. However, it should be noted that the adjacent blocks may be of different sizes based on the size of the current block. Generally, the video encoder 20 and video decoder 30 may be configured to construct a list of candidate motion vectors for the current block 450, where the list of candidate motion vectors contains motion vector information from a number of adjacent blocks for the current block, where the number of adjacent blocks is greater than 5.

[0248]

[0180] In another example of the present disclosure, the video encoder 20 and video decoder 30 may be configured to construct a list of motion vector candidates using motion vector histogram-based ordering of motion information from adjacent blocks. Based on the assumption that motion is spatially homogeneous (for example, that it may be the same or nearly the same at a given spatial location in a picture), the dominant motion information of an adjacent block is likely to be the motion information selected for the current block. Thus, the video encoder 20 and video decoder 30 may be configured to derive a motion vector histogram from the motion vector distribution of adjacent blocks. As described above, the motion vector information includes a three-dimensional vector (MVx, MVy, direction), where MVx is the horizontal component of the motion vector and MVy is the vertical component of the motion vector, where the direction refers to either a past (reference list L0) predicted direction or a future (reference list L1) predicted direction. Referring to Figure 13, the video encoder 20 and video decoder 30 may determine how often a particular motion vector is the same for each of adjacent blocks a to k.

[0249]

[0181] Video encoder 20 and video decoder 30 can use histogram information in a plurality of different ways. In one example, video encoder 20 and video decoder 30 can use histogram information to determine which motion vectors, and thus which adjacent blocks, can be used as spatial merge candidates in the candidate list. In another example, video encoder 20 and video decoder 30 can use the histogram to determine in what order several (certain) spatial merge candidates should be added to the list.

[0250]

[0182] Generally, video encoder 20 and video decoder 30 can be configured to derive a motion vector histogram from adjacent pixels or blocks. As explained above, FIG. 16 shows an example of 4×4 adjacent blocks (a - k) to be used for the motion vector histogram of 16×16 current block 450. The highlighted blocks (a, e, f, j, and k) are the locations of spatial merge candidates in HEVC.

[0251]

[0183] In some examples, video encoder 20 and video decoder 30 derive a motion vector histogram from the motion vector distribution of adjacent blocks having a certain size. FIG. 16 shows which adjacent blocks (a - k) are used to construct the MV histogram. The unit size of the adjacent blocks can be a specific size, e.g., 4×4, or some predefined minimum size for motion compensation. If blocks do not have relevant motion information (e.g., intra - predicted blocks), they can be ignored or filled with motion information from other adjacent blocks. For example, if adjacent block h is an intra - predicted block, video encoder 20 and video decoder 30 may simply not use that adjacent block. In other examples, if adjacent block h is an intra - predicted block, video encoder 20 and video decoder 30 can use the motion information from the block to the left of adjacent block h.

[0252]

[0184] As shown in one example in FIG. 16, to construct the MV histogram of a 16×16 current block, video encoder 20 and video decoder 30 can examine 11 adjacent blocks (from block a to block k) having a size of 4×4. It should be noted that the adjacent blocks can be predefined as in the case of FIG. 16 (including the top row / left column), or can depend on the size and / or shape of the current block.

[0253]

[0185] In another example, the histogram can be constructed using a certain weight proportional to the size of the adjacent blocks. For example, the number of pixels (or unit blocks, i.e., 4×4 blocks) belonging to the adjacent blocks can be used as the weight for the histogram. That is, motion vectors from larger blocks (specifically, blocks containing more pixels) have a higher weight proportional to the number of pixels in those blocks. In another example, the weight for the histogram can be determined by a combination of the two factors described above, i.e., the number of pixels (or unit blocks) in the adjacent blocks and the number of pixels (or unit blocks) adjacent to the current block.

[0254]

[0186] It should be noted that both video encoder 20 and video decoder 30 should follow equivalent rules to construct the histogram to avoid inconsistencies. Assuming equivalent histograms in both video encoder 20 and video decoder 30, all the following adaptation methods for merge candidates will lead to an equivalent merge list.

[0255]

[0187] After determining the motion vector histogram, video encoder 20 and video decoder 30 can then use the histogram to determine the order of spatial merge candidates in the motion vector candidate list. In some examples, the constructed histogram can be used to determine the order of a given (fixed) N f number of spatial merge candidates, where N fThis is the number of candidate fixed spaces. As an example, N fixed spaces f The spatial candidates can be adjacent blocks a, e, f, j, and k, as used in HEVC. However, any subset of the total number of candidates can be used. For example, referring to Figure 16, any fixed subset of adjacent blocks a-k can be used as spatial merge candidates.

[0256]

[0188] Depending on the frequency of each motion vector in the available adjacent blocks, the most frequent motion vector from the histogram is inserted first into the merge list, and the least frequent motion vector from the histogram is the last of the spatial merge candidates to be inserted into the list. For example, Figure 16 shows five spatial merge candidates (a, e, f, j, k) used in HEVC. Instead of following a fixed order of those candidates (the jefka order in HEVC), both the video encoder 20 and the video decoder 30 may be configured to adaptively determine the order from the MV histogram. In another example, instead of examining each adjacent block (e.g., a 4x4 spatial merge candidate), the sorting of spatial merging candidates is based on the size of the prediction block (e.g., PU in HEVC) that contains the blocks used to derive the spatial merging candidates.

[0257] Consider the example of FIG. 17. As shown in FIG. 17, three adjacent blocks (e, g, and h) have motion vector 0 (MV0), four different adjacent blocks (a, b, c, and d) have motion vector 1 (MV1), one adjacent block (f) has motion vector 2 (MV2), two different adjacent blocks (i and j) have motion vector 3 (MV3), and one adjacent block (k) has motion vector 4 (MV4). Thus, video encoder 20 and video decoder 30 are configured to order the motion vector candidate list using the fixed candidates a, e, f, j, and k as MV1-candidate a (index 0), MV0-candidate e (index 0), MV3-candidate j (index 0), MV2-candidate f (index 0), MV4-candidate k (index 0). The example of FIG. 17 assumes that all adjacent blocks have the same weight. In some examples, if two or more of the fixed candidates have the same associated motion vector having the same number of occurrences in a histogram (e.g., MV2 and MV4 in FIG. 17), a predetermined order may be used for examining the candidates. In the example of FIG. 17, candidate f is placed in the list before candidate k. However, any predetermined order may be used.

[0258]

[0190] In another example of the present disclosure, after determining the motion vector histogram, video encoder 20 and video decoder 30 may then use the histogram to determine which of the adjacent blocks, regardless of order, should be used as spatial merge candidates in the motion vector candidate list. That is, instead of using a fixed number of spatial merging candidates, video encoder 20 and video decoder 30 may determine which of all possible adjacent blocks should be used as spatial merge candidates in the list. In this example, with respect to FIG. 16, all adjacent blocks a - k may be considered for inclusion as spatial merge candidates in the motion vector candidate list. f Rather than using a fixed number of spatial merging candidates, video encoder 20 and video decoder 30 may determine which of all possible adjacent blocks should be used as spatial merge candidates in the list. In this example, with respect to FIG. 16, all adjacent blocks a - k may be considered for inclusion as spatial merge candidates in the motion vector candidate list.

[0259]

[0191] The video encoder 20 and video decoder 30 determine which of the total number of available adjacent blocks has a predetermined number (N) in the motion vector candidate list. h A motion vector histogram can be used to determine which candidates will be made up. Instead of simply changing the order of a given list of candidates as described above, both the location (e.g., which actual adjacent blocks) and order of the determined spatial merge candidates can be adaptively derived from the adjacent motion vector distribution in the determined histogram. For example, N h If = 2, the two most frequent motion vectors from adjacent blocks are placed in the merge list in order of frequency. If more than one adjacent block is associated with the most frequent motion vector in the histogram, the video encoder 20 and video decoder 30 may use predetermined rules to determine which of the adjacent blocks should be placed in the candidate list. However, it should be noted that regardless of the rule used, the adjacent block associated with the most frequently occurring motion vector in the histogram will be added to the motion vector candidate list. In this example, referring to Figure 17, the adjacent block associated with MV0 and the adjacent block associated with MV1 will be added to the motion vector candidate list.

[0260]

[0192] In some examples, the video encoder 20 and the video decoder 30 may be configured to use both of the above techniques for constructing a merge list using a determined histogram. That is, both the video encoder 20 and the video decoder 30 may use a histogram to order a fixed set of candidates, as well as a certain number (N) based on the histogram. h ) can be added as non-fixed candidates. As mentioned above, N f The location of each spatial merge candidate is fixed across all blocks, for example, blocks a, e, f, j, and k in Figure 16. Furthermore, N from adjacent blocks hThe motion information that appears most frequently is added to the list as a candidate for spatial merge, and then (N f +N h The order of the candidates is determined based on the frequency of occurrence of the relevant motion vectors in the determined histogram.

[0261]

[0193] In another example, N f The locations of the N spatial merge candidates are fixed across all blocks, for example, blocks a, e, f, j, and k in Figure 16, and the video encoder 20 and video decoder 30 use the determined histogram to determine the order of the fixed candidates. Furthermore, N from adjacent blocks h The most frequently occurring motion information is added to the list, but further (additional)N h Each candidate is inserted at a predetermined position (for example, before or after the motion vector from block e in Figure 16).

[0262]

[0194] In another example, the disclosure describes the adaptive ordering of sub-PU merge candidates, for example, the ATMVP and ATMVP_EXT candidates described above. In an example of JEM2.0 software, ATMVP and ATMVP_EXT are always placed between candidate k and candidate a (for example, as shown in Figure 16). Instead of placing ATMVP / ATMVP_EXT in a fixed location in the merge list, the video encoder 20 and video decoder 30 may be configured to adaptively place ATMVP / ATMVP_EXT candidates depending on the state of other available merge candidates, ATMVP / ATMVP_EXT, or combinations thereof.

[0263]

[0195] In some examples, the motion vector difference (MVD) between two spatial merge candidates may be used to determine the location of the ATMVP / ATMVP_EXT candidate. The video encoder 20 and video decoder 30 may be configured to compute the MVD as a function of the motion vectors. In one example, the absolute difference sum between the two MVs is MVD = abs(MV x [1]-MV x [0])+abs(MV y [1]-MV y [0]) is the case. In another example, the function is MVD=(MV x [1]-MV x [0])*(MV x [1]-MV x [0])+(MV y [1]-MV y [0])*(MV y [1]-MV y [0]) is defined as . Note that the function for calculating MVD may differ based on the motion vector precision, such as integer, 1 / 2, 1 / 4, 1 / 8, or 1 / 16 pixel precision.

[0264]

[0196] For example, unlike modern JEM software which always places ATMVP / ATMVP_EXT between candidate k and candidate a in Figure 16, the video encoder 20 and video decoder 30 place MVD(MVD) between candidate j and candidate k. jk Depending on the configuration, the ATMVP / ATMVP_EXT candidate may be positioned before candidate k. MVD jk If it is less than threshold TH1 or greater than another threshold TH2, i.e., MVD jk <TH1またはMVD jkIf >TH2, ATMVP / ATMVP_EXT is placed before candidate k. For example, the adaptive threshold can be used to calculate TH1 by taking the minimum MVD between all or some spatially adjacent motion vectors and to calculate TH2 by taking the maximum MVD, where the calculation excludes equivalent MV pairs as well as pairs of (MVj and MVk). Since both video encoder 20 and video decoder 30 have access identical neighboring motion vectors, the calculation will yield the same TH1 and TH2. Alternatively, both TH1 and TH2 can be determined experimentally, for example, TH1=2 and TH2=8 with 1 / 16 pixel MV precision. Otherwise, candidate k is placed before ATMVP / ATMVP_EXT in the list. Similarly, video encoder 20 and video decoder 30 can calculate the MVD between candidate e and candidate f (MVD ef The order of ATMVP / ATMVP_EXT candidates and candidate f can be determined by examining the following. If one or neither of the candidates used for MVD computation, i.e., candidate k or candidate a in the above example, is available in the merge list, then the ATMVP / ATMVP_EXT candidates can be placed in the move vector candidate list in the default order.

[0265]

[0197] In another example, the video encoder 20 and video decoder 30 may be configured to analyze the characteristics of ATMVP / ATMVP_EXT candidates, such as the variance of subblock MV or the spatial distribution of subblock MV, in order to determine where those candidates will be located in the motion vector candidate list. If the variance falls within the range [TH1, TH2], a higher priority, i.e., a smaller index in the list, is assigned. The range [TH1, TH2] may be determined by the mean variance of subblock MV of a previously coded block that selected ATMVP or ATMVP_EXT as the best merge candidate, such as TH1=C1*Var1 and TH2=C2*Var2, where Var1 and Var2 are calculated and stored from the previously coded block. The coefficients C1 and C2 may be fixed as constants or may depend on the size and / or shape of the current block. The range may depend on the size and / or shape of the current block. For larger blocks, both TH1 and TH2 increase, and the range becomes wider. The range may depend on the accuracy of the motion vector.

[0266]

[0198] In another example, the video encoder 20 and video decoder 30 may be configured to analyze the status of both the spatial merge candidate and the ATMVP / ATMVP_EXT candidate in order to determine the order among those candidates. For example, the most frequent subblock motion vector, or average motion vector, from the ATMVP or ATMVP_EXT candidate may be considered the delegate MV of the subblock. The delegate motion vector may be used to calculate the MVD for the spatial candidate, for example, block f in Figure 16. If the MVD is greater than TH1 but less than TH2, the video encoder 20 and video decoder 30 may be configured to place the ATMVP / ATMVP_EXT candidate before the spatial candidate.

[0267]

[0199] In HEVC, a combination motion vector (combi-mv) candidate is derived using two available bidirectional merge candidates, namely C1 and C2, which contain two motion vectors for both prediction directions, i.e., for reference list L0 and reference list L1. Both C1 and C2 have bidirectional MVs, i.e., MV for candidate C1. L0C1 and MV L1C1 , and MV for candidate C2 L0C2 and MV L1C2 Let us assume that the video encoder 20 and video decoder 30 may be configured to derive a new combi-mv, i.e., (MV0,MV1), by taking L0 MV from C1 and L1 MV from C2, where (MV0,MV1)=(MV L0C1 MV L1C2 Similarly, the video encoder 20 and video decoder 30 may be configured to derive another combi-mv by taking the remaining MV, so (MV0',MV1')=(MV L0C2 MV L1C1 ) is possible.

[0268]

[0200] In some cases, the maximum number of combi-mv candidates is fixed. In HEVC, if the number of available merge candidates is less than the maximum number of merge candidates, for example 5 in HEVC, and there are two or more bidirectional merge candidates available in the list, then at most 12 combi-mv candidates can be considered merge candidates. Extensions to HEVC add more merge candidates, such as ATMVP and ATMVP_EXT candidates, and therefore it is a possible extension to increase the maximum number of combi-mv candidates from 12 to a certain large number, for example 30.

[0269]

[0201] In another example of the present disclosure, the video encoder 20 and video decoder 30 may be configured to consider more combi-mvs if necessary (for example, if the maximum number of merge candidates has not yet been reached). For example, if the similarity between available merge candidates is higher than a certain threshold, the combi-mv candidates will also be similar to existing candidates, and thus the maximum number of combi-mvs will be suppressed. Similarity may be measured by the sum of absolute differences (SAD), SATD, mean luminance or chrominance value, pixel variance, and / or MV orbital.

[0270]

[0202] When more combi-mvs are considered, adaptive ordering may be considered to maximize the benefit from spare combi-mv candidates. Assuming an order for combi-mv candidates, the following technique sorts the candidates with respect to a certain criterion. Candidates that do not meet that criterion follow the default order.

[0271]

[0203] In some examples, the video encoder 20 and video decoder 30 may be configured to reorder the combi-mv candidates with respect to the similarity between the derived combi-mv and the existing mv from the available candidates. Both C1 and C2 are configured to be bidirectional MV, i.e., MV C1 =(MV L0C1 MV L1C1 ) and MV C2 =(MV L0C2 MV L1C2 ) has two combi-mvs MV combi-1 =(MV L0C1 MV L1C2 ) and MV combi-2 =(MV L0C2 MV L1C1 Assume that it can be derived as follows: MV L0C1 and MV L0C2 (and / or MV L1C2 and MV L1C1 ) refers to the same picture, MV L0C1 and MV L0C2and (and / or MV L1C2 and MV L1C1 and) the MVD between them is calculated. Then, video encoder 20 and video decoder 30 are configured to add the derived combi-mv to the motion vector candidate list after pruning if one of the following conditions is satisfied, i.e., (1) the MVD is between two thresholds, i.e., TH1 < MVD < TH2, or (2) MV L0C1 and MV L0C2 and point to different pictures. In other cases, the combi-mv is left behind. In one example for 1 / 16 pixel motion vector accuracy, TH1 = 2 and TH2 = 8 when both the width and height of the current block are less than 8. When the width and height of the current block are greater than 8 and less than 32, TH1 = 8 and TH2 = 32. When the width and height are greater than 32, TH1 = 16 and TH2 = 64. Once all combi-mv candidates that meet those conditions are first added to the merge list, video encoder 20 and video decoder 30 can be configured to add the remaining combi-mv to the list after pruning. The thresholds, TH1 and TH2, can be adaptively selected according to the size or shape of the current block, e.g., max(width, height).

[0272]

[0204] In another example, video encoder 20 and video decoder 30 can be configured to sort the combi-mv candidates with respect to the MVD described above. For simplicity, the MVD of a combi-mv candidate having MV L0C1 and MV L0C2 (or MV L1C1 and MV L1C2 ) pointing to different pictures is set to 0. If combi-mv candidates have equal MVD values, they follow the default order. Once sorted, video encoder 20 and video decoder 30 can be configured to add the candidates to the motion vector candidate list after pruning.

[0273]

[0205] In addition to POC-based pruning, as described above, the video encoder 20 and the video decoder 30 may be configured to further prune merge candidates using an adaptive criterion to be determined by additional information other than the motion vector itself (e.g., the size and / or shape of the current block, the type of target merge candidate, and / or the location of the merge candidate if spatial).

[0274]

[0206] In some examples, the video encoder 20 and the video decoder 30 may be configured to consider a pair of MVs with an MVD smaller than an adaptive threshold as identical, and thus consider them to be pruned to further diversify the motion vector candidate list (consider a pair of MVs with smaller MVD than an adaptive threshold as identical, and thus pruned to diversify the motion vector candidate list further). The threshold may be adaptively selected according to the size and / or shape of the current block.

[0275]

[0207] In some examples, pruning methods such as MVD-based pruning using the above adaptive threshold may be applied to all types of merge candidates, i.e., spatial candidates, temporal candidates, sub-block candidates, or combi-mv candidates. In another example, different criteria may be considered for different types of candidates. As an example of spatial candidates, the distance between the locations from which they are derived may be used as a metric for determining the adaptive threshold. For example, if two motion vectors are derived from adjacent blocks, e.g., blocks e and f in FIG. 16, the motion vectors are pruned by a threshold smaller than the threshold derived from blocks that are distant, e.g., blocks f and k in FIG. 16.

[0276]

[0208] In some examples, for bidirectional merge candidates, the similarity of two uni-predicted blocks (one from L0 and another from L1) could indicate how reliable the merge candidate is. Based on this observation, the video encoder 20 and video decoder 30 may be configured to use the measurement to differentiate bidirectional merge candidates by using the similarity of two uni-predicted blocks and to sort the bidirectional merge candidates accordingly. For example, the video encoder 20 and video decoder 30 may be configured to use absolute difference sum (SAD), SSE, SATD, mean luminance or chrominance value, pixel variance, and / or MV orbital to determine similarity. More complex metrics may give higher precision to measure predictive performance. The determination of the metric may depend on the requirements of the target application.

[0277]

[0209] When SAD is used, for two bidirectional merge candidates, C1 and C2, for each bidirectional candidate, two SADs are used between the L0 direction and the L1 direction, i.e., SAD C1 and SAD C2 The following is calculated. The video encoder 20 and video decoder 30 may be configured to place, in the merge list, candidates with smaller final SADs, i.e., SADC1 or SADC2, ahead of the other.

[0278]

[0210] In some examples, all the proposed techniques described above can be combined to construct a list of motion vector candidates. In other examples, a set or subset of the proposed techniques may be incorporated.

[0279]

[0211] Figure 18 is a flowchart illustrating an exemplary encoding method of the present disclosure. The technique of Figure 18 may be performed by one or more hardware units of a video encoder 20, including a motion estimation unit 42 and a motion compensation unit 44.

[0280]

[0212] In one example of the present disclosure, the video encoder 20 may be configured to receive the current block of video data (500). The video encoder 20 may derive a histogram of motion vector information for adjacent blocks relative to the current block (502). In one example of the present disclosure, the number of adjacent blocks considered for the motion vector candidate list is greater than 5, based on the size of the current block. In this context, the term “considered” may include the video encoder 20 analyzing the adjacent blocks, determining whether the adjacent blocks have relevant motion information, and, if the adjacent blocks have relevant motion information, using the motion information to construct a motion vector candidate list. As described above, the motion information may be added directly to the motion vector candidate list, or it may be used to construct a histogram which can be used to determine the order and / or location of adjacent blocks that should be used as spatial merge candidates in the motion vector candidate list. The video encoder 20 may be further configured to construct a motion vector candidate list of merge candidates for the current block based on the motion information from the above number of adjacent blocks relative to the current block of video data. In some examples, the motion information considered is a derived histogram (504). The video encoder 20 then determines the current motion vector from a list of motion vector candidates (506) and uses the current motion vector to encode the current block of video data (508).

[0281]

[0213] In another example of the present disclosure, the video encoder 20 may be configured to order a predetermined fixed subset of spatial merge candidates in the motion vector candidate list based on the derived histogram.

[0282]

[0214] In another example of the present disclosure, the video encoder 20 may be configured to determine a fixed number of spatial merge candidates to be added to the motion vector candidate list from the total number of adjacent blocks, based on the derived histogram.

[0283]

[0215] In another example of the present disclosure, the video encoder 20 may be configured to determine a fixed number of spatial merge candidates to be added to a motion vector candidate list from the total number of adjacent blocks based on a derived histogram, and to order a predetermined fixed subset of spatial merge candidates and the determined fixed number of spatial merge candidates in the motion vector candidate list based on a derived histogram.

[0284]

[0216] In another example of the present disclosure, the video encoder 20 may be configured to order a predetermined fixed subset of spatial merge candidates in a motion vector candidate list based on a derived histogram, to determine a fixed number of spatial merge candidates to be added to the motion vector candidate list from the total number of adjacent blocks based on the derived histogram, and to insert the determined fixed number of spatial merge candidates at predetermined locations in the motion vector candidate list.

[0285]

[0217] In another example of the present disclosure, the video encoder 20 may be configured to add ATMVP candidates to a motion vector candidate list based on a function of motion vectors for one or more advanced time motion vector prediction (ATMVP) candidates. In another example of the present disclosure, the video encoder 20 may be configured to determine a location in the motion vector candidate list for adding an ATMVP candidate based on a function of motion vectors for one or more ATMVP candidates.

[0286]

[0218] In another example of the present disclosure, the video encoder 20 may be configured to determine a combined motion vector candidate by combining motion vector information from two bidirectional motion vector candidates, and to add the combined motion vector candidate to a list of motion vector candidates.

[0287]

[0219] In another example of the present disclosure, the video encoder 20 may be configured to determine a location in a motion vector candidate list for adding a combination motion vector candidate based on a function of motion vectors for one or more combination motion vector candidates.

[0288]

[0220] In another example of the present disclosure, the video encoder 20 may be configured to prune the motion vector candidate list based on motion vector difference information of the motion vector candidates in the motion vector candidate list.

[0289]

[0221] In another example of the present disclosure, the video encoder 20 may be configured to order bidirectional candidates based on motion vector difference information of bidirectional candidates in the motion vector candidate list.

[0290]

[0222] Figure 19 is a flowchart illustrating an exemplary decoding method of the present disclosure. The technique of Figure 19 may be performed by one or more hardware units of a video decoder 30, including a motion compensation unit 72.

[0291]

[0223] In one example of the present disclosure, the video decoder 30 may be configured to receive the current block of video data encoded using merge mode (550). The video decoder 30 derives a histogram of motion vector information for adjacent blocks relative to the current block (552). In one example of the present disclosure, the number of adjacent blocks considered for the motion vector candidate list is greater than 5, based on the size of the current block. In this context, the term “considered” may include the video decoder 30 analyzing the adjacent blocks, determining whether the adjacent blocks have relevant motion information, and, if the adjacent blocks have relevant motion information, using the motion information to construct the motion vector candidate list. As described above, the motion information may be added directly to the motion vector candidate list, or it may be used to construct a histogram which can be used to determine the order and / or location of adjacent blocks that should be used as spatial merge candidates in the motion vector candidate list. The video decoder 30 may be further configured to construct a motion vector candidate list of merge candidates for the current block based on the motion information from the above number of adjacent blocks relative to the current block of video data. In some examples, the motion information considered is a derived histogram (554). The video decoder 30 then determines the current motion vector from the motion vector candidate list (556) and uses the current motion vector to decode the current block of video data (558).

[0292]

[0224] In another example of the present disclosure, the video decoder 30 may be configured to order a predetermined fixed subset of spatial merge candidates in a motion vector candidate list based on a derived histogram.

[0293]

[0225] In another example of the present disclosure, the video decoder 30 may be configured to determine a fixed number of spatial merge candidates to be added to the motion vector candidate list from the total number of adjacent blocks, based on the derived histogram.

[0294]

[0226] In another example of the present disclosure, the video decoder 30 may be configured to determine a fixed number of spatial merge candidates to be added to the motion vector candidate list from the total number of adjacent blocks based on the derived histogram, and to order a predetermined fixed subset of spatial merge candidates and the determined fixed number of spatial merge candidates in the motion vector candidate list based on the derived histogram.

[0295]

[0227] In another example of the present disclosure, the video decoder 30 may be configured to order a predetermined fixed subset of spatial merge candidates in a motion vector candidate list based on a derived histogram, to determine a fixed number of spatial merge candidates to be added to the motion vector candidate list from the total number of adjacent blocks based on the derived histogram, and to insert the determined fixed number of spatial merge candidates at predetermined locations in the motion vector candidate list.

[0296]

[0228] In another example of the present disclosure, the video decoder 30 may be configured to add ATMVP candidates to the motion vector candidate list based on a function of motion vectors for one or more advanced time motion vector prediction (ATMVP) candidates. In another example of the present disclosure, the video encoder 20 may be configured to determine a location in the motion vector candidate list for adding an ATMVP candidate based on a function of motion vectors for one or more ATMVP candidates.

[0297]

[0229] In another example of the present disclosure, the video decoder 30 may be configured to determine a combined motion vector candidate by combining motion vector information from two bidirectional motion vector candidates, and to add the combined motion vector candidate to a list of motion vector candidates.

[0298]

[0230] In another example of the present disclosure, the video decoder 30 may be configured to determine a location in the motion vector candidate list for adding a combination motion vector candidate based on a function of motion vectors for one or more combination motion vector candidates.

[0299]

[0231] In another example of the present disclosure, the video decoder 30 may be configured to prune the motion vector candidate list based on motion vector difference information of the motion vector candidates in the motion vector candidate list.

[0300]

[0232] In another example of the present disclosure, the video decoder 30 may be configured to order bidirectional candidates based on motion vector difference information of bidirectional candidates in the motion vector candidate list.

[0301]

[0233] As an example, the proposed combination of techniques shows a 0.4% BD rate improvement in a random access configuration on JEM2.0 software, as shown in the table below. The gains in the following example come from a combination of tools: (1) histogram-based spatial merge candidate ordering, (2) MVD-based combination merge candidate ordering and pruning, (3) ATMVP, combination, pruning on 0mv candidates, and (4) an increased number of merge candidates and combi merge candidates.

[0302] [Table 2]

[0303]

[0234] In accordance with the above examples, it should be noted that some of the actions or events among the techniques described herein may be performed in different sequences, added, merged, or completely excluded (for example, not all described actions or events are necessary for the implementation of the techniques). Furthermore, in some examples, the actions or events may not be performed sequentially, but rather simultaneously, for example, through multithreading, interrupt handling, or across multiple processors.

[0304]

[0235] In one or more examples, the described functions may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted via computer-readable media as one or more instructions or codes and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media corresponding to tangible media such as data storage media, or communication media including any media that enables the transfer of computer programs from one place to another according to a communication protocol, for example. Thus, computer-readable media may generally correspond to (1) non-transient tangible computer-readable storage media, or (2) communication media such as signals or carrier waves. Data storage media may be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, codes and / or data structures for implementation of the techniques described herein. Computer program products may include computer-readable media.

[0305]

[0236] As an example, and not an limitation, such computer-readable storage media may include RAM, ROM, EEPROM®, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory, or any other media that can be used to store desired program code in the form of instructions or data structures and can be accessed by a computer. Any connection is also appropriately called a computer-readable medium. For example, if instructions are transmitted from a website, server or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of a medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carriers, signals or other temporary media, but instead cover non-temporary tangible storage media. As used herein, the terms "disk" and "disc" include Compact Disc (CD), LaserDisc® (disc), Optical Disc (disc), Digital Multipurpose Disc (disc) (DVD), Floppy Disk (disk), and Blu-ray Disc (disc), where a disk typically reproduces data magnetically and a disc reproduces data optically using a laser. Any combination of the above should also be included within the scope of computer-readable media.

[0306]

[0237] Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other equivalent integrated circuits or discrete logic circuits. Thus, the term “processor” as used herein may refer to any of the above-described structures or any other structure suitable for implementing the techniques described herein. Furthermore, in some embodiments, the functions described herein may be provided in dedicated hardware and / or software modules configured for encoding and decoding, or incorporated into a composite codec. The techniques may also be fully implemented by one or more circuits or logic elements.

[0307]

[0238] The techniques of the Disclosure may be implemented in a wide variety of devices or apparatus, including wireless handsets, integrated circuits (ICs) or sets of ICs (e.g., chipsets). While various components, modules, or units have been described in this Disclosure to highlight the functional aspects of devices configured to perform the techniques disclosed, these components, modules, or units do not necessarily require implementation by different hardware units. Rather, as described above, the various units may be combined in a codec hardware unit, including one or more processors described above, along with suitable software and / or firmware, or provided by a set of interoperable hardware units.

[0308]

[0239] Various examples have been described. These and other examples fall within the scope of the following claims. The invention described in the original claims of this application is listed below. [C1] A method for decoding video data, wherein the method is Receiving the current block of video data encoded in merge mode, Based on motion information from a certain number of adjacent blocks relative to the current block in the video data, a list of candidate motion vectors for merge candidates for the current block is constructed, wherein the number of adjacent blocks considered for the candidate motion vector list is based on the size of the current block, wherein the number of adjacent blocks is greater than 5. The current motion vector is determined from the aforementioned list of motion vector candidates, A method comprising decoding the current block of video data using the current motion vector. [C2] To derive a histogram of motion vector information for the adjacent block, The method of C1, further comprising constructing the motion vector candidate list based on the derived histogram. [C3] The method of C2, further comprising ordering a predetermined fixed subset of spatial merge candidates in the motion vector candidate list based on the derived histogram. [C4] The method of C2, further comprising determining a fixed number of spatial merge candidates to be added to the motion vector candidate list from the total number of adjacent blocks based on the derived histogram. [C5] Based on the derived histogram, a fixed number of spatial merge candidates to be added to the motion vector candidate list are determined from the total number of adjacent blocks. The method of C2, further comprising ordering a predetermined fixed subset of spatial merge candidates and the determined fixed number of spatial merge candidates in the motion vector candidate list based on the derived histogram. [C6] Based on the derived histogram, the spatial merge candidates of a predetermined fixed subset are ordered in the motion vector candidate list. Based on the derived histogram, a fixed number of spatial merge candidates to be added to the motion vector candidate list are determined from the total number of adjacent blocks. The method of C2, further comprising inserting a fixed number of determined spatial merge candidates at predetermined locations in the motion vector candidate list. [C7] The method of C2, further comprising adding ATMVP candidates to the list of motion vector candidates based on a function of motion vectors for one or more advanced time motion vector prediction (ATMVP) candidates. [C8] The method of C7, further comprising determining a location in the motion vector candidate list for adding the ATMVP candidate based on the function of motion vectors for one or more ATMVP candidates. [C9] By combining motion vector information from two bidirectional motion vector candidates, a combined motion vector candidate is determined. The method of C2, further comprising adding the combined motion vector candidates to the motion vector candidate list. [C10] The method of C9, further comprising determining a location in the motion vector candidate list for adding the combination motion vector candidate based on a function of motion vectors for one or more combination motion vector candidates. [C11] The method of C2, further comprising pruning the motion vector candidate list based on motion vector difference information of the motion vector candidates in the motion vector candidate list. [C12] The method of C2, further comprising ordering the bidirectional candidates based on the motion vector difference information of the bidirectional candidates in the motion vector candidate list. [C13] A device configured to decode video data, wherein the device is Memory configured to store the current block of video data, It comprises one or more processors, and the one or more processors are Receiving the current block of video data encoded in merge mode; constructing a list of merge candidate motion vectors for the current block based on motion information from a certain number of adjacent blocks to the current block of video data, wherein the number of adjacent blocks considered for the motion vector candidate list is based on the size of the current block, wherein the number of adjacent blocks is greater than 5. The current motion vector is determined from the aforementioned list of motion vector candidates, A device configured to decode the current block of video data using the current motion vector. [C14] The one or more processors described above To derive a histogram of motion vector information for the adjacent block, The apparatus according to C13, further configured to construct the motion vector candidate list based on the derived histogram. [C15] The one or more processors described above The apparatus according to C14, further configured to order a predetermined fixed subset of spatial merge candidates in the motion vector candidate list based on the derived histogram. [C16] The one or more processors described above The apparatus according to C14, further configured to determine a fixed number of spatial merge candidates to be added to the motion vector candidate list from the total number of adjacent blocks, based on the derived histogram. [C17] The one or more processors described above Based on the derived histogram, a fixed number of spatial merge candidates to be added to the motion vector candidate list are determined from the total number of adjacent blocks. The apparatus according to C14, further configured to order a predetermined fixed subset of spatial merge candidates and a determined fixed number of spatial merge candidates in the motion vector candidate list based on the derived histogram. [C18] The one or more processors described above Based on the derived histogram, the spatial merge candidates of a predetermined fixed subset are ordered in the motion vector candidate list. Based on the derived histogram, a fixed number of spatial merge candidates to be added to the motion vector candidate list are determined from the total number of adjacent blocks. The apparatus according to C14, further configured to insert a fixed number of determined spatial merge candidates at predetermined locations in the motion vector candidate list. [C19] The one or more processors described above The apparatus according to C14, further configured to add ATMVP candidates to the list of motion vector candidates based on a function of motion vectors for one or more advanced time motion vector prediction (ATMVP) candidates. [C20] The one or more processors described above The apparatus according to C19, further configured to determine a location in the motion vector candidate list for adding the ATMVP candidate based on the function of motion vectors for one or more ATMVP candidates. [C21] The one or more processors described above By combining motion vector information from two bidirectional motion vector candidates, a combined motion vector candidate is determined. The apparatus according to C14, further configured to add the combination motion vector candidates to the motion vector candidate list. [C22] The one or more processors described above The apparatus according to C14, further configured to determine a location in the motion vector candidate list for adding the combination motion vector candidate based on a function of motion vectors for one or more combination motion vector candidates. [C23] The one or more processors described above The apparatus according to C14, further configured to prune the motion vector candidate list based on motion vector difference information of the motion vector candidates in the motion vector candidate list. [C24] The one or more processors described above The apparatus according to C14, further configured to order the bidirectional candidates based on the motion vector difference information of the bidirectional candidates in the motion vector candidate list. [C25] A computer-readable storage medium for storing instructions, wherein, when the instructions are executed, one or more processors configured to decode video data, Receiving the current block of video data encoded in merge mode, Based on motion information from a certain number of adjacent blocks relative to the current block in the video data, a list of candidate motion vectors for merge candidates for the current block is constructed, wherein the number of adjacent blocks considered for the candidate motion vector list is based on the size of the current block, wherein the number of adjacent blocks is greater than 5. The current motion vector is determined from the aforementioned list of motion vector candidates, A computer-readable storage medium that allows decoding of the current block of video data using the current motion vector. [C26] A device configured to encode video data, wherein the device is Memory configured to store the current block of video data, It comprises one or more processors, and the one or more processors are Receiving the current block of video data, Based on motion information from a certain number of adjacent blocks relative to the current block in the video data, a list of candidate motion vectors for merge candidates for the current block is constructed, wherein the number of adjacent blocks considered for the candidate motion vector list is based on the size of the current block, wherein the number of adjacent blocks is greater than 5. The current motion vector is determined from the aforementioned list of motion vector candidates, A device configured to encode the current block of video data using the current motion vector. [C27] The one or more processors described above To derive a histogram of motion vector information for the adjacent block, The apparatus according to C26, further configured to construct the motion vector candidate list based on the derived histogram. [C28] The one or more processors described above The apparatus according to C27, further configured to order a predetermined fixed subset of spatial merge candidates in the motion vector candidate list based on the derived histogram. [C29] The one or more processors described above The apparatus according to C27, further configured to determine a fixed number of spatial merge candidates to be added to the motion vector candidate list from the total number of adjacent blocks, based on the derived histogram. [C30] The one or more processors described above Based on the derived histogram, a fixed number of spatial merge candidates to be added to the motion vector candidate list are determined from the total number of adjacent blocks. The apparatus according to C27, further configured to order a predetermined fixed subset of spatial merge candidates and a determined fixed number of spatial merge candidates in the motion vector candidate list based on the derived histogram.

Claims

1. A method for decoding video data, wherein the method is Receiving the current block of video data encoded in merge mode, Constructing a list of candidate motion vectors for merge candidates for the current block of video data based on motion information from a subset of multiple adjacent blocks relative to the current block, wherein the multiple adjacent blocks include all adjacent blocks adjacent to the top row of the current block or adjacent to the left column of the current block, the upper right adjacent block adjacent to the current block, the upper left adjacent block adjacent to the current block, and the lower left adjacent block adjacent to the current block, wherein each adjacent block has the number of adjacent unit blocks adjacent to the current block, and wherein the number of adjacent blocks in the subset of adjacent blocks is greater than 5. A histogram of motion vector information is derived from the motion vector distribution of the subset of adjacent blocks, wherein the motion vector distribution represents the number of occurrences of a particular motion vector for the subset of adjacent blocks, and each occurrence is weighted by the number of adjacent unit blocks in each adjacent block. Ordering the spatial merge candidates in the motion vector candidate list according to the frequency of the associated motion vectors in the histogram, wherein if two or more spatial merge candidates have associated motion vectors with the same frequency in the histogram, a predetermined order is used. The current motion vector is determined from the aforementioned list of motion vector candidates, A method comprising decoding the current block of video data using the current motion vector.

2. The method according to claim 1, wherein the unit block has a predefined minimum size for motion compensation.

3. The method according to claim 1, wherein the unit block has a size of 4 x 4.

4. The method according to any one of claims 1 to 3, wherein the subset of adjacent blocks is predefined.

5. A device configured to decode video data, wherein the device is Memory configured to store the current block of video data, It comprises one or more processors, and the one or more processors are Receiving the current block of video data encoded in merge mode, Constructing a list of candidate motion vectors for merge candidates for the current block of video data based on motion information from a subset of multiple adjacent blocks relative to the current block, wherein the multiple adjacent blocks include all adjacent blocks adjacent to the top row of the current block or adjacent to the left column of the current block, the upper right adjacent block adjacent to the current block, the upper left adjacent block adjacent to the current block, and the lower left adjacent block adjacent to the current block, wherein each adjacent block has the number of adjacent unit blocks adjacent to the current block, and wherein the number of adjacent blocks in the subset of adjacent blocks is greater than 5. A histogram of motion vector information is derived from the motion vector distribution of the subset of adjacent blocks, wherein the motion vector distribution represents the number of occurrences of a particular motion vector for the subset of adjacent blocks, and each occurrence is weighted by the number of adjacent unit blocks in each adjacent block. Ordering the spatial merge candidates in the motion vector candidate list according to the frequency of the associated motion vectors in the histogram, wherein if two or more spatial merge candidates have associated motion vectors with the same frequency in the histogram, a predetermined order is used. The current motion vector is determined from the aforementioned list of motion vector candidates, A device configured to decode the current block of video data using the current motion vector.

6. The apparatus according to claim 5, wherein the unit block has a predefined minimum size for motion compensation.

7. The apparatus according to claim 5, wherein the unit block has a size of 4 x 4.

8. The apparatus according to any one of claims 5 to 7, wherein the subset of adjacent blocks is predefined.

9. A computer-readable storage medium for storing instructions, wherein, when executed, the instructions cause one or more processors configured to decode video data to perform the method according to any one of claims 1 to 4.

10. A device configured to encode video data, wherein the device is Memory configured to store the current block of video data, It comprises one or more processors, and the one or more processors are Receiving the current block of video data, Constructing a list of candidate motion vectors for merge candidates for the current block of video data based on motion information from a subset of multiple adjacent blocks relative to the current block, wherein the multiple adjacent blocks include all adjacent blocks adjacent to the top row of the current block or adjacent to the left column of the current block, the upper right adjacent block adjacent to the current block, the upper left adjacent block adjacent to the current block, and the lower left adjacent block adjacent to the current block, wherein each adjacent block has the number of adjacent unit blocks adjacent to the current block, and wherein the number of adjacent blocks in the subset of adjacent blocks is greater than 5. A histogram of motion vector information is derived from the motion vector distribution of the subset of adjacent blocks, wherein the motion vector distribution represents the number of occurrences of a particular motion vector for the subset of adjacent blocks, and each occurrence is weighted by the number of adjacent unit blocks in each adjacent block. Ordering the spatial merge candidates in the motion vector candidate list according to the frequency of the associated motion vectors in the histogram, wherein if two or more spatial merge candidates have associated motion vectors with the same frequency in the histogram, a predetermined order is used. The current motion vector is determined from the aforementioned list of motion vector candidates, A device configured to encode the current block of video data using the current motion vector.

11. The apparatus according to claim 10, wherein the unit block has a predefined minimum size for motion compensation.

12. The apparatus according to claim 10, wherein the unit block has a size of 4 x 4.

13. The apparatus according to any one of claims 10 to 12, wherein the subset of adjacent blocks is predefined.

14. A method for encoding video data, wherein the method is: Receiving the current block of video data, Constructing a list of candidate motion vectors for merge candidates for the current block of video data based on motion information from a subset of multiple adjacent blocks relative to the current block, wherein the multiple adjacent blocks include all adjacent blocks adjacent to the top row of the current block or adjacent to the left column of the current block, the upper right adjacent block adjacent to the current block, the upper left adjacent block adjacent to the current block, and the lower left adjacent block adjacent to the current block, wherein each adjacent block has the number of adjacent unit blocks adjacent to the current block, and wherein the number of adjacent blocks in the subset of adjacent blocks is greater than 5. A histogram of motion vector information is derived from the motion vector distribution of the subset of adjacent blocks, wherein the motion vector distribution represents the number of occurrences of a particular motion vector for the subset of adjacent blocks, and each occurrence is weighted by the number of adjacent unit blocks in each adjacent block. Ordering the spatial merge candidates in the motion vector candidate list according to the frequency of the associated motion vectors in the histogram, wherein if two or more spatial merge candidates have associated motion vectors with the same frequency in the histogram, a predetermined order is used. The current motion vector is determined from the aforementioned list of motion vector candidates, A method comprising encoding the current block of video data using the current motion vector.

15. The method according to claim 14, wherein the unit block has a predefined minimum size for motion compensation.

Images