Merge candidate for motion vector prediction for video coding

Abstract

A method of decoding video data, the method comprising constructing a motion vector candidate list of merge candidates for a current block of video data based on motion information from a number of neighboring blocks relative to the current block, wherein the number of neighboring blocks considered for the motion vector candidate list is based on a size of the current block, and wherein the number of neighboring blocks is more than 5. In some examples, the method includes deriving a histogram of motion vector information for the neighboring blocks, and constructing the motion vector candidate list based on the derived histogram.

Description

[0001] This application claims the benefit of U.S. Provisional Application No. 62 / 336,449, filed May 13, 2016, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This disclosure relates to video decoding. Background Technology

[0003] Digital video capabilities can be incorporated into a wide variety of devices, including digital television, digital live broadcast systems, wireless broadcasting systems, personal digital assistants (PDAs), laptops or desktop computers, tablets, e-book readers, digital cameras, digital recording devices, digital media players, video game devices, video game consoles, cellular or satellite radio phones, so-called "smartphones," video conferencing devices, video streaming devices, and more. Digital video devices implement video decoding technologies, such as those described in the following standards: standards defined by MPEG-2, MPEG-4, ITU-T H.263, ITU-T H.264 / MPEG-4 Part 10 Advanced Video Decoding (AVC), ITU-T H.265 (also known as High Efficiency Video Decoding (HEVC)), and extensions to these standards. Video devices can more efficiently transmit, receive, encode, decode, and / or store digital video information by implementing these video decoding technologies.

[0004] Video decoding techniques involve spatial (intra-picture) prediction and / or temporal (inter-picture) prediction to reduce or remove inherent redundancy in video sequences. For block-based video decoding, video slices (e.g., video frames or portions of video frames) can be segmented into video blocks, which, in some techniques, may also be referred to as tree blocks, decoding units (CUs), and / or decoding nodes. Video blocks in intra-frame decoded (I) slices of a picture are encoded using spatial predictions about reference samples in adjacent blocks within the same picture. Video blocks in inter-frame decoded (P or B) slices of a picture can use spatial predictions about reference samples in adjacent blocks within the same picture, or temporal predictions about reference samples in other reference pictures. A picture may be referred to as a frame, and a reference picture may be referred to as a reference frame.

[0005] Spatial or temporal prediction produces a prediction block for the block to be decoded. Residual data represents the pixel difference between the original block to be decoded and the prediction block. Inter-frame decoded blocks are encoded based on motion vectors pointing to reference sample blocks forming the prediction blocks and residual data indicating the difference between the decoded blocks and the prediction blocks. Intra-frame decoded blocks are encoded based on intra-frame decoding patterns and residual data. For further compression, the residual data can be transformed from the pixel domain to the transform domain to produce residual transform coefficients, which can then be quantized. The quantized transform coefficients, initially arranged in a two-dimensional array, can be scanned to produce a one-dimensional vector of transform coefficients, and entropy decoding can be applied to achieve even more compression. Summary of the Invention

[0006] Generally speaking, this disclosure describes techniques related to the decoding (e.g., encoding or decoding) of motion information in video data blocks. In various instances of this disclosure, a list of motion vector candidates (also referred to as a merge candidate list, or simply a candidate list) can be constructed using motion information from multiple adjacent blocks. A histogram of the motion information can be derived and then used to determine the order and / or position of the spatial merge candidates in the motion vector candidate list.

[0007] In one instance, this disclosure describes a method for decoding video data, the method comprising: receiving a current video data block encoded in a merging mode; constructing a motion vector candidate list for merging candidates of the current block based on motion information from a number of neighboring blocks relative to the current video data block, wherein the number of neighboring blocks considered for the motion vector candidate list is based on the size of the current block and wherein the number of neighboring blocks is greater than 5; determining a current motion vector from the motion vector candidate list; and decoding the current video data block using the current motion vector.

[0008] In another example, this disclosure describes an apparatus configured to decode video data, the apparatus including a memory configured to store a current video data block and one or more processors configured to: receive the current video data block encoded in a merging mode; construct a motion vector candidate list for merging candidates of the current block based on motion information from a number of neighboring blocks relative to the current video data block, wherein the number of neighboring blocks considered for the motion vector candidate list is based on the size of the current block and wherein the number of neighboring blocks is more than 5; determine a current motion vector from the motion vector candidate list; and decode the current video data block using the current motion vector.

[0009] In another instance, this disclosure describes a computer-readable storage medium storing instructions that, when executed, cause one or more processors configured to decode video data to: receive a current video data block encoded in a merge mode; construct a motion vector candidate list for merging candidates of the current block based on motion information from a number of neighboring blocks relative to the current video data block, wherein the number of neighboring blocks considered for the motion vector candidate list is based on the size of the current block and wherein the number of neighboring blocks is greater than 5; determine a current motion vector from the motion vector candidate list; and decode the current video data block using the current motion vector.

[0010] In another example, this disclosure describes an apparatus configured to encode video data, the apparatus including a memory configured to store a current video data block and one or more processors configured to receive the current video data block; construct a motion vector candidate list for merging candidates of the current block based on motion information from a number of neighboring blocks relative to the current video data block, wherein the number of neighboring blocks considered for the motion vector candidate list is based on the size of the current block and wherein the number of neighboring blocks is greater than 5; determine a current motion vector from the motion vector candidate list; and decode the current video data block using the current motion vector.

[0011] Details of one or more examples are set forth in the accompanying drawings and the following description. Other features, objectives, and advantages will become apparent from the description and drawings, as well as from the claims. Attached Figure Description

[0012] Figure 1 This is a block diagram illustrating an example video encoding and decoding system that can be configured to perform the techniques of this disclosure.

[0013] Figure 2 This is a block diagram illustrating an example of a video encoder that can be configured to perform the techniques of this disclosure.

[0014] Figure 3 This is a block diagram illustrating an example of a video decoder that can be configured to perform the techniques of this disclosure.

[0015] Figure 4 A conceptual diagram showing the structure of the decoding unit (CU) in High Efficiency Video Decoding (HEVC) is presented.

[0016] Figure 5 This is a conceptual diagram illustrating the instance segmentation type used for inter-frame prediction modes.

[0017] Figure 6A This is a conceptual diagram illustrating an example of block partitioning using a quadtree-binary tree (QTBT) structure.

[0018] Figure 6B It shows the corresponding Figure 6A A conceptual diagram of an instance tree structure using QTBT structure for block partitioning.

[0019] Figure 7 This is a conceptual diagram illustrating spatial adjacency candidates in HEVC.

[0020] Figure 8 This is a conceptual diagram illustrating the Time Motion Vector Prediction (TMVP) in HEVC.

[0021] Figure 9 This is a conceptual diagram illustrating the instance prediction structure used in 3D-HEVC.

[0022] Figure 10 This is a conceptual diagram illustrating view-to-view motion prediction based on sub-PUs in 3D-HEVC.

[0023] Figure 11 This is a conceptual diagram illustrating sub-PU motion predictions from a reference image.

[0024] Figure 12 This is a concept diagram showing related images in ATMVP.

[0025] Figure 13 This is a flowchart illustrating an example method according to the technology disclosed herein.

[0026] Figure 14 This is a conceptual diagram showing an instance of a PU and its adjacent blocks.

[0027] Figure 15 This is a conceptual diagram showing another instance of PU and adjacent blocks.

[0028] Figure 16 This is a conceptual diagram showing another instance of PU and adjacent blocks.

[0029] Figure 17 This is a conceptual diagram showing another instance of PU and adjacent blocks.

[0030] Figure 18 This is a flowchart illustrating an example encoding method of this disclosure.

[0031] Figure 19 This is a flowchart illustrating an example decoding method of this disclosure. Detailed Implementation

[0032] This disclosure describes techniques for improving the efficiency of merging-based motion vector prediction. This disclosure describes techniques for determining merging candidates for motion vector prediction. Example techniques of this disclosure may include adaptive ranking and simplification of merging candidates. Example adaptive ranking techniques of this disclosure may include adaptive ranking of spatial sub-prediction units (PUs) and combined motion vector (combi-mv) candidates. In some instances, the proposed adaptive simplification techniques of this disclosure can be applied to all merging candidates, including temporal motion vector prediction (TMVP) candidates, zero-mv candidates, and the aforementioned candidates.

[0033] The techniques disclosed herein can provide one or more of the following advantages: (1) higher efficiency; (2) simplicity (e.g., lower decoder implementation complexity); and (3) flexibility. The techniques described herein can provide greater bit savings by assigning higher priority to merging candidates whose values ​​are closer (or potentially closer) to the actual motion vector. Additionally, the complexity of the video encoder and video decoder, as well as memory requirements, are relatively lower. Finally, the proposed techniques can be applied to a variety of codecs, such as H.266-based and quadtree plus binary tree (QTBT)-based video codecs. Furthermore, the proposed techniques offer flexibility because the techniques described herein can be used independently or together, and therefore can be combined in any way.

[0034] Figure 1 This is a block diagram illustrating an example video encoding and decoding system 10 that can be configured to perform motion vector prediction using techniques of this disclosure. Figure 1 As shown, system 10 includes a source device 12 that provides encoded video data to be decoded later by a destination device 14. Specifically, source device 12 provides video data to destination device 14 via computer-readable medium 16. Source device 12 and destination device 14 can include any of a variety of devices, including desktop computers, laptop computers, tablet computers, set-top boxes, mobile phones such as so-called "smart" phones, so-called "smart" mats, televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, etc. In some cases, source device 12 and destination device 14 may be equipped for wireless communication.

[0035] Destination device 14 can receive encoded video data to be decoded via computer-readable medium 16. Computer-readable medium 16 may include any type of media or device capable of moving encoded video data from source device 12 to destination device 14. In one example, computer-readable medium 16 may include communication media enabling source device 12 to transmit encoded video data directly to destination device 14 in real time. The encoded video data may be modulated according to communication standards (e.g., wireless communication protocols) and transmitted to destination device 14. Communication media may include any wireless or wired communication media, such as radio frequency (RF) spectrum or one or more physical transmission lines. Communication media may form part of a packet-based network (e.g., a local area network, a wide area network, or a global network, such as the Internet). Communication media may include routers, switches, base stations, or any other devices that can be used to facilitate communication from source device 12 to destination device 14.

[0036] In some instances, encoded data can be output from output interface 22 to a storage device. Similarly, encoded data can be accessed from a storage device via an input interface. The storage device can comprise any of a variety of distributed or locally accessed data storage media, such as hard disk drives, Blu-ray discs, DVDs, CD-ROMs, flash memory, volatile or non-volatile memory, or any other suitable digital storage media for storing encoded video data. In other instances, the storage device can correspond to a file server or another intermediate storage device capable of storing the encoded video generated by source device 12. Destination device 14 can access the stored video data from the storage device via streaming or downloading. The file server can be any type of server capable of storing encoded video data and transferring encoded video data to destination device 14. Example file servers include web servers (e.g., for websites), FTP servers, network attached storage (NAS) devices, and local disk drives. Destination device 14 can access the encoded video data via any standard data connection including an Internet connection. This can include wireless channels (e.g., Wi-Fi connections), wired connections (e.g., DSL, cable modems, etc.), or a combination of both suitable for accessing encoded video data stored on a file server. Transfer of encoded video data from the storage device can be streaming, downloading, or a combination thereof.

[0037] The technology disclosed herein is not necessarily limited to wireless applications or setups. The technology can be applied to video decoding supporting any of a variety of multimedia applications, such as over-the-air television broadcasting, cable television transmission, satellite television transmission, Internet streaming video transmission (e.g., HTTP Dynamic Adaptive Streaming (DASH)), digital video encoded onto data storage media, decoding digital video stored on data storage media, or other applications. In some instances, system 10 can be configured to support one-way or two-way video transmission to support applications such as video streaming, video playback, video broadcasting, and / or video telephony.

[0038] exist Figure 1 In one example, source device 12 includes a video source 18, a video encoder 20, and an output interface 22. Destination device 14 includes an input interface 28, a video decoder 30, and a display device 32. According to this disclosure, the video encoder 20 of source device 12 can be configured to apply the techniques of this disclosure to motion vector prediction. In other examples, the source and destination devices may include other components or arrangements. For example, source device 12 may receive video data from an external video source 18 (e.g., an external camera). Similarly, destination device 14 may interface with an external display device instead of including an integrated display device.

[0039] Figure 1 The system 10 shown is merely one example. The techniques of motion vector prediction disclosed herein can be implemented by any digital video encoding and / or decoding apparatus. Although the techniques of this disclosure are generally implemented by video encoding apparatuses, they can also be implemented by video encoders / decoders (commonly referred to as "codecs"). Furthermore, the techniques of this disclosure can also be implemented by video preprocessors. Source apparatus 12 and destination apparatus 14 are merely examples of such decoding apparatuses, wherein source apparatus 12 generates decoded video data for transmission to destination apparatus 14. In some instances, apparatuses 12, 14 can operate in a substantially symmetrical manner, such that each of apparatuses 12, 14 includes video encoding and decoding components. Therefore, system 10 can support one-way or two-way video transmission between video apparatuses 12, 14 for, for example, video streaming, video playback, video broadcasting, or video telephony.

[0040] The video source 18 of source device 12 may include a video capture device, such as a camera, a video archive containing previously captured video, and / or a video feed interface for receiving video from a video content provider. Alternatively, video source 18 may generate computer graphics-based data as source video, or a combination of live video, archived video, and computer-generated video. In some cases, if video source 18 is a camera, then source device 12 and destination device 14 may form a so-called camera phone or video phone. However, as mentioned above, the techniques described in this disclosure are generally applicable to video decoding and can be applied to wireless and / or wired applications. In each case, captured, pre-captured, or computer-generated video may be encoded by video encoder 20. The encoded video information may then be output from output interface 22 to computer-readable media 16.

[0041] Computer-readable media 16 may comprise transient media, such as wireless broadcasting or wired network transmissions, or storage media (i.e., non-transitory storage media), such as hard disks, flash drives, compressed optical discs, digital video optical discs, Blu-ray discs, or other computer-readable media. In some instances, a network server (not shown) may receive encoded video data from source device 12 and, for example, provide the encoded video data to destination device 14 via network transmission. Similarly, a computing device in a media production facility (e.g., an optical disc stamping facility) may receive encoded video data from source device 12 and produce optical discs containing the encoded video data. Therefore, in various instances, computer-readable media 16 can be understood to comprise one or more computer-readable media of various forms.

[0042] The input interface 28 of the destination device 14 receives information from the computer-readable medium 16. The information on the computer-readable medium 16 may include grammatical information defined by the video encoder 20, which is also used by the video decoder 30, including descriptive blocks and other characteristics and / or processed grammatical elements of decoding units (e.g., GOPs). The display device 32 displays the decoded video data to the user and may include any of a variety of display devices, such as a cathode ray tube (CRT), liquid crystal display (LCD), plasma display, organic light-emitting diode (OLED) display, or another type of display device.

[0043] Video encoder 20 and video decoder 30 can operate according to video decoding standards, such as the High Efficiency Video Decoding (HEVC) standard, extensions to or successors to the HEVC standard, such as ITU-T H.266. Alternatively or additionally, video encoder 20 and video decoder 30 can also operate according to other proprietary or industry standards (such as ITU-T H.264, alternatively referred to as MPEG-4 Part 10, Advanced Video Decoding (AVC)) or extensions of such standards. However, the techniques disclosed herein are not limited to any particular decoding standard. Other examples of video decoding standards include MPEG-2 and ITU-T H.263. Although Figure 1 Not shown, but in some respects, the video encoder 20 and video decoder 30 may each be integrated with the audio encoder and decoder, and may include appropriate multiplexer-demultiplexer units or other hardware and software to process the encoding of both audio and video in a common data stream or separate data streams. Where applicable, the multiplexer-demultiplexer units may conform to the ITU H.223 multiplexer protocol, or other protocols such as User Datagram Protocol (UDP).

[0044] The video encoder 20 and video decoder 30 can each be implemented as any of a variety of suitable encoder or decoder circuits, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combination thereof. When the technology is implemented in part in software, the device may store instructions for the software in a suitable non-transitory computer-readable medium and use one or more processors to execute the instructions in hardware to perform the technology of this disclosure. Each of the video encoder 20 and video decoder 30 may be included in one or more encoders or decoders, any of which may be integrated as part of a combined encoder / decoder (codec) in the respective device.

[0045] As will be explained in more detail below, the video encoder 20 and the video decoder 30 may be configured to receive a current video data block; construct a motion vector candidate list for merging candidates of the current block based on motion information from a number of neighboring blocks relative to the current video data block, wherein the number of neighboring blocks considered for the motion vector candidate list is based on the size of the current block and wherein the number of neighboring blocks is more than 5; determine a current motion vector from the motion vector candidate list; and decode (e.g., encode or decode) the current video data block using the current motion vector.

[0046] Video decoding standards include ITU-T H.261, ISO / IEC MPEG-1 Visual, ITU-T H.262 or ISO / IEC MPEG-2 Visual, ITU-T H.263, ISO / IEC MPEG-4 Visual, and ITU-T H.264 (also known as ISO / IEC MPEG-4 AVC), which include their Scalable Video Coding (SVC) and Multi-View Video Coding (MVC) extensions. A joint draft of MVC is described in "Advanced video coding for generic audiovisual services" (ITU-T Recommendation H.264) in March 2010.

[0047] In addition, there is a newly developed video decoding standard, HEVC, which was developed by the ITU-T Video Decoding Experts Group (VCEG) and the Joint Collaborative Video Decoding Group (JCT-VC) of the ISO / IEC Animation Experts Group (MPEG). The latest draft of HEVC is available at http: / / phenix.int-evry.fr / jct / doc_end_user / documents / 12_Geneva / wg11 / JCTVC-L1003-v34.zip. The HEVC standard is also jointly presented in ITU-T H.265 and the international standard ISO / IEC 23008-2, both titled "High-Efficiency Video Decoding" and both published in October 2014.

[0048] JCT-VC developed the HEVC standard. HEVC standardization work is based on an evolutionary model of video decoding devices called the HEVC Test Model (HM). The HM assumes that the video decoding device possesses several additional capabilities relative to existing devices, such as those provided by ITU-T H.264 / AVC. For example, while H.264 provides nine intra-frame prediction coding modes, the HEVC HM can provide up to thirty-three intra-frame prediction coding modes. Although some HEVC terminology may be used in this disclosure for illustrative purposes, the techniques of this disclosure are not limited to HEVC, and it is explicitly contemplated that the techniques of this disclosure can be implemented in successor standards to HEVC.

[0049] Generally, the working model of HM describes how video frames or images can be divided into sequences of tree blocks or maximum decoding units (LCUs) containing both luma and chroma samples. The syntax data within the bitstream defines the size of the LCU (the maximum decoding unit in terms of pixel count). A slice contains several consecutive tree blocks in decoding order. Video frames or images can be segmented into one or more slices. Each tree block can be split into decoding units (CUs) according to a quadtree. Generally, the quadtree data structure contains one node for each CU, where the root node corresponds to the tree block. If a CU splits into four sub-CUs, then the node corresponding to the CU contains four leaf nodes, where each leaf node corresponds to one of the sub-CUs.

[0050] Each node in a quadtree data structure provides syntax data for the corresponding CU. For example, a node in a quadtree can contain a split flag to indicate whether the CU corresponding to the node has split into subCUs. The syntax elements of a CU can be defined recursively and can depend on whether the CU has been divided into several subCUs. If a CU is not further split, it is called a leaf CU. In this disclosure, the four subCUs of a leaf CU will also be called leaf CUs, even if there is no explicit split of the original leaf CU. For example, if a 16×16 CU is not further split, then although the 16×16 CU has never split, the four 8×8 subCUs will also be called leaf CUs.

[0051] A CU serves a similar purpose to a macroblock in the H.264 standard, except that CUs are not size-distinguished. For example, a tree block can be split into four child nodes (also called child CUs), and each child node can be a parent node and split into four more child nodes. The last unsplit child node (called the leaf node of the quadtree) includes the decoding node, also called the leaf CU. The syntax data associated with the decoded bitstream can define the maximum number of times a tree block can be split, called the maximum CU depth, and can also define the minimum size of the decoding node. Therefore, the bitstream can also define the minimum decoding unit (SCU). This disclosure uses the term "block" to refer to any of the CU, PU, ​​or TU in the context of HEVC, or similar data structures in the context of other standards (e.g., macroblocks and subblocks in H.264 / AVC).

[0052] A CU comprises a decoding node and associated prediction units (PUs) and transform units (TUs). The size of the CU corresponds to the size of the decoding node and its shape must be square. The size of the CU can range from 8×8 pixels to a tree block size of up to 64×64 pixels or larger. Each CU may contain one or more PUs and one or more TUs. For example, the syntax data associated with the CU may describe the segmentation of the CU into one or more PUs. The segmentation mode may vary between CUs being skipped or encoded in direct mode, intra-predictive mode, or inter-predictive mode. PUs may be segmented into non-square shapes. For example, the syntax data associated with the CU may also describe the segmentation of the CU into one or more TUs according to a quadtree. The shape of the TU may be square or non-square (e.g., rectangular).

[0053] The HEVC standard allows transformations based on Transform Units (TUs), which can vary for different CUs. A TU is typically sized based on the size of a PU within a given CU defined by a segmented LCU, but this may not always be the case. A TU is usually the same size as or smaller than the PU. In some instances, the residual samples corresponding to a CU can be subdivided into smaller units using a quadtree structure called a "Residual Quadtree" (RQT). The leaf nodes of the RQT are called Transform Units (TUs). The pixel differences associated with the TU can be transformed to produce transform coefficients, which can be quantized.

[0054] A leaf CU can contain one or more prediction units (PUs). Generally, a PU represents all or part of a spatial region corresponding to a given CU and may contain data for retrieving reference samples for the PU. Furthermore, the PU contains prediction-related data. For example, when the PU is coded in intra-frame mode, the data for the PU may be contained in a residual quadtree (RQT), which may contain data describing the intra-frame prediction mode of the TU corresponding to the PU. As another example, when the PU is coded in inter-frame mode, the PU may contain data defining one or more motion vectors of the PU. The data defining the motion vectors of the PU may describe, for example, the horizontal component of the motion vector, the vertical component of the motion vector, the resolution of the motion vector (e.g., quarter-pixel accuracy or eighth-pixel accuracy), the reference picture to which the motion vector points, and / or a list of reference pictures of the motion vector (e.g., list 0, list 1, or list C).

[0055] A leaf CU with one or more PUs can also contain one or more transform units (TUs). Transform units can be specified using an RQT (also known as a TU quadtree structure), as discussed above. For example, a split flag can indicate whether a leaf CU is split into four transform units. Each transform unit can then be further split into more sub-TUs. When a TU is not further split, it can be called a leaf TU. Typically, for intra-frame decoding, all leaf TUs belonging to a leaf CU share the same intra-frame prediction mode. That is, the same intra-frame prediction mode is generally applied to compute the prediction values ​​for all TUs of the leaf CU. For intra-frame decoding, the video encoder can use the intra-frame prediction mode to compute the residual value for each leaf TU as the difference between the TU-corresponding portion of the CU and the original block. A TU is not necessarily limited to the size of a PU. Therefore, a TU can be larger or smaller than a PU. For intra-frame decoding, a PU can be juxtaposed with the corresponding leaf TU of the same CU. In some instances, the maximum size of a leaf TU can correspond to the size of the corresponding leaf CU.

[0056] Furthermore, the TU of a leaf CU can also be associated with a corresponding quadtree data structure (called a residual quadtree (RQT)). That is, a leaf CU can contain a quadtree indicating how the leaf CU is partitioned into TUs. The root node of the TU quadtree generally corresponds to a leaf CU, while the root node of the CU quadtree generally corresponds to a tree block (or LCU). The unsplit TU of the RQT is called a leaf TU. Generally, this disclosure uses the terms CU and TU to refer to leaf CU and leaf TU respectively, unless otherwise indicated.

[0057] A video sequence typically comprises a series of video frames or pictures. A group of pictures (GOP) generally consists of a series of one or more video pictures. A GOP may contain a header for the GOP, headers for one or more pictures, or syntax data elsewhere describing the multiple pictures contained in the GOP. Each slice of a picture may contain slice syntax data describing the encoding mode used for the corresponding slice. The video encoder 20 typically operates on video blocks within individual video slices to encode video data. Video blocks may correspond to decoding nodes within the CU. Video blocks may have a fixed or variable size and may vary in size depending on a specified decoding standard.

[0058] As an example, HM supports prediction for various PU sizes. Assuming a specific CU size is 2N×2N, HM supports intra-frame prediction for PU sizes of 2N×2N or N×N, and inter-frame prediction for symmetrical PU sizes of 2N×2N, 2N×N, N×2N, or N×N. HM also supports asymmetric segmentation for inter-frame prediction in PU sizes of 2N×nU, 2N×nD, nL×2N, and nR×2N. In asymmetric segmentation, one direction of the CU is not segmented, but the other direction is segmented into 25% and 75%. The portion of the CU corresponding to the 25% segmentation is indicated by "n" followed by "up (U)," "down (D)," "left (L)," or "right (R)." Therefore, for example, "2N×nU" refers to a horizontally segmented 2N×2N CU, where the top is a 2N×0.5N PU and the bottom is a 2N×1.5N PU.

[0059] In this disclosure, "N×N" and "N by N" are used interchangeably to refer to the pixel size of a video block in both the vertical and horizontal dimensions, for example, 16×16 pixels or 16 by 16 pixels. Generally, a 16×16 block will have 16 pixels vertically (y = 16) and 16 pixels horizontally (x = 16). Similarly, an N×N block will have N pixels vertically and N pixels horizontally, where N represents a non-negative integer value. Pixels in a block can be arranged in rows and columns. Furthermore, a block does not necessarily need to have the same number of pixels horizontally and vertically. For example, a block may include N×M pixels, where M is not necessarily equal to N.

[0060] After performing intra-frame or inter-frame prediction decoding using the PU of the CU, the video encoder 20 can compute residual data of the TU of the CU. The PU may include syntax data describing the method or pattern for generating predicted pixel data in the spatial domain (also known as the pixel domain), and the TU may include coefficients in the transform domain after applying a transform (e.g., discrete cosine transform (DCT), integer transform, wavelet transform, or conceptually similar transform) to the residual video data. The residual data may correspond to the pixel difference between pixels in the uncoded image and the predicted values ​​corresponding to the PU. The video encoder 20 can form a TU containing the residual data for the CU and then transform the TU to produce transform coefficients for the CU.

[0061] Following any transform used to generate the transform coefficients, the video encoder 20 can perform quantization of the transform coefficients. Quantization generally refers to the process of quantizing the transform coefficients to potentially reduce the amount of data used to represent them, thereby providing further compression. The quantization process can reduce the bit depth associated with some or all of the coefficients. For example, an n-bit value can be rounded down to an m-bit value during quantization, where n is greater than m.

[0062] After quantization, the video encoder can scan the transform coefficients to generate a one-dimensional vector from a two-dimensional matrix containing the quantized transform coefficients. The scan can be designed to place higher-energy (and therefore lower-frequency) coefficients at the front of the array and lower-energy (and therefore higher-frequency) coefficients at the back. In some instances, the video encoder 20 can utilize a predefined scan order to scan the quantized transform coefficients to produce a serialized vector that can be entropy-encoded. In other instances, the video encoder 20 can perform adaptive scanning. After scanning the quantized transform coefficients to form a one-dimensional vector, the video encoder 20 can entropy-encode the one-dimensional vector, for example, using context-adaptive variable-length decoding (CAVLC), context-adaptive binary arithmetic decoding (CABAC), syntax-based context-adaptive binary arithmetic decoding (SBAC), probabilistic interval partitioning entropy (PIPE) decoding, or another entropy encoding method. The video encoder 20 can also entropy-encode the syntax elements associated with the encoded video data for use by the video decoder 30 when decoding the video data.

[0063] To perform CABAC, the video encoder 20 can assign context within a context model to the symbols to be transmitted. For example, the context can be related to whether the neighboring values ​​of the symbol are non-zero. To perform CAVLC, the video encoder 20 can select variable-length codes for the symbols to be transmitted. Codewords in VLC can be constructed such that relatively shorter codes correspond to more likely symbols, while longer codes correspond to less likely symbols. In this way, using VLC can achieve bit savings compared to, for example, using a fixed-length codeword for each symbol to be transmitted. Probability determination can be based on the context assigned to the symbol.

[0064] Figure 2 This is a block diagram illustrating an example of a video encoder 20 that can be configured to perform motion vector prediction for the techniques of this disclosure, as will be explained in more detail below. The video encoder 20 can perform intra-frame and inter-frame decoding of video blocks within a video slice. Intra-frame decoding relies on spatial prediction to reduce or remove spatial redundancy in the video within a given video frame or picture. Inter-frame decoding relies on temporal prediction to reduce or remove temporal redundancy in the video within neighboring frames or pictures of a video sequence. An intra-frame mode (I-mode) can refer to any of several spatial-based decoding modes. For example, an inter-frame mode with unidirectional prediction (P-mode) or bidirectional prediction (B-mode) can refer to any of several temporal-based decoding modes.

[0065] like Figure 2 As shown, the video encoder 20 receives the current video block within the video frame to be encoded. Figure 2In this example, the video encoder 20 includes a video data memory 41, a mode selection unit 40, a reference image memory 64, a summer 50, a transform processing unit 52, a quantization unit 54, and an entropy coding unit 56. The mode selection unit 40 further includes a motion compensation unit 44, a motion estimation unit 42, an intra-frame prediction unit 46, and a segmentation unit 48. For video block reconstruction, the video encoder 20 also includes an inverse quantization unit 58, an inverse transform unit 60, and a summer 62. It may also include a deblocking filter (…). Figure 2 (Not shown) Filters block boundaries to remove blocking artifacts from the reconstructed video. When necessary, a deblocking filter typically filters the output of summer 62. In addition to the deblocking filter, additional filters (in-loop or after-loop) may be used. These filters are not illustrated for simplicity, but they can be used to filter the output of summer 50 (as in-loop filters) when necessary.

[0066] Video data memory 41 can be configured to store video data to be encoded by components of video encoder 20. The video data stored in video data memory 41 can be obtained, for example, from video source 18. Reference picture memory 64 (sometimes referred to as a decoded picture buffer) can be a reference picture memory that stores reference video data for the video encoder 20 to use, for example, to encode the video data in intra-frame or inter-frame decoding modes. Video data memory 41 and reference picture memory 64 can be formed of any of a variety of memory devices, such as dynamic random access memory (DRAM) (including synchronous DRAM (SDRAM)), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. Video data memory 41 and reference picture memory 64 can be provided by the same memory device or separate memory devices. In various instances, video data memory 41 can be on-chip along with other components of video encoder 20, or off-chip relative to those components.

[0067] During the encoding process, video encoder 20 receives video frames or slices to be decoded. These frames or slices may be divided into multiple video blocks. Motion estimation unit 42 and motion compensation unit 44 perform inter-frame prediction decoding of the received video blocks relative to one or more blocks in one or more reference frames to provide temporal prediction. Intra-frame prediction unit 46 may alternatively perform intra-frame prediction decoding of the received video blocks relative to one or more adjacent blocks in the same frame or slice as the block to be decoded to provide spatial prediction. Video encoder 20 may perform, for example, multiple decoding passes to select an appropriate decoding mode for each video data block.

[0068] Furthermore, segmentation unit 48 can segment video data blocks into sub-blocks based on the evaluation of the aforementioned segmentation scheme in the aforementioned decoding passes. For example, segmentation unit 48 can initially segment frames or slices into LCUs, and then segment each of the LCUs into sub-CUs based on rate distortion analysis (e.g., rate distortion optimization). Mode selection unit 40 can further generate a quadtree data structure indicating the segmentation of LCUs into several sub-CUs. The leaf nodes of the quadtree, CUs, can contain one or more PUs and one or more TUs.

[0069] The mode selection unit 40 may select one of the coding modes, intra-frame or inter-frame, for example, based on the error result, and provide the resulting intra-frame decoded block or inter-frame decoded block to the summer 50 to generate residual block data, which is then provided to the summer 62 to reconstruct the coded block for use as a reference frame. The mode selection unit 40 also provides syntax elements (e.g., motion vectors, intra-frame mode indicators, segmentation information, and other such syntax information) to the entropy coding unit 56.

[0070] The motion estimation unit 42 and the motion compensation unit 44 can be highly integrated, but are described separately for conceptual purposes. Motion estimation performed by the motion estimation unit 42 is the process of generating motion vectors that estimate the motion of video blocks. For example, the motion vector can indicate the displacement of the PU of a video block within the current video frame or picture relative to the displacement of a predicted block within a reference frame (or other decoded unit) relative to the current block being decoded within the current frame (or other decoded unit). The predicted block is the block found to closely match the block to be decoded in terms of pixel difference, which can be determined by the sum of absolute differences (SAD), sum of squared differences (SSD), or other difference metrics. In some instances, the video encoder 20 can calculate the values ​​of sub-integer pixel positions of a reference picture stored in the reference picture memory 64. For example, the video encoder 20 can interpolate the values ​​of quarter-pixel positions, eighth-pixel positions, or other fractional pixel positions of the reference picture. Therefore, the motion estimation unit 42 can perform motion search relative to full-pixel positions and fractional pixel positions and output motion vectors with fractional-pixel accuracy.

[0071] The motion estimation unit 42 calculates the motion vector of the PU by comparing the position of the PU in the video block in the inter-frame decoded slice with the position of the predicted block in the reference image. The reference image can be selected from a first reference image list (list 0) or a second reference image list (list 1), each of which identifies one or more reference images stored in the reference image memory 64. The motion estimation unit 42 sends the calculated motion vector to the entropy coding unit 56 and the motion compensation unit 44.

[0072] Motion compensation performed by motion compensation unit 44 may involve capturing or generating prediction blocks based on motion vectors determined by motion estimation unit 42. Again, in some instances, motion estimation unit 42 and motion compensation unit 44 may be functionally integrated. After receiving the motion vector of the PU for the current video block, motion compensation unit 44 may locate the prediction block to which the motion vector points in one of the reference image lists. Summer 50 forms a residual video block by subtracting the pixel values ​​of the prediction block from the pixel values ​​of the current video block being decoded, as discussed below. Generally, motion estimation unit 42 performs motion estimation relative to the luminance component, and motion compensation unit 44 uses motion vectors calculated based on the luminance component for both chroma and luminance components. Mode selection unit 40 may also generate syntax elements associated with video blocks and video slices for use by video decoder 30 when decoding video blocks of video slices.

[0073] The video encoder 20, which includes a motion estimation unit 42 and a motion compensation unit 44, can be configured to perform the above-mentioned functions. Figure 1 Any of the various techniques of this disclosure discussed and will be described in more detail below. For example, motion compensation unit 44 may be configured to decode motion information of video data blocks using AMVP or merging mode according to the techniques of this disclosure. Additionally, video encoder 20, which includes motion estimation unit 42 and motion compensation unit 44, may be configured to perform any combination of the motion vector candidate list construction techniques of this disclosure described in more detail below. In the context of this disclosure, the terms motion vector candidate list, merge candidate list, and candidate list are used interchangeably.

[0074] Assuming motion compensation unit 44 selects to execute the merging mode, it can form a candidate list containing a set of merging candidates. Motion compensation unit 44 can add candidates to the candidate list based on a specific predetermined order. In other embodiments of this disclosure, motion compensation unit 44 can be configured to add candidates to the candidate list in dynamically different orders based on histogram information from motion vectors of neighboring blocks. Motion compensation unit 44 can also add additional candidates and perform simplification of the candidate list, as discussed in more detail below. Finally, mode selection unit 40 can determine which candidates will be used to encode the motion information of the current block and encode the merging index representing the selected candidate.

[0075] As an alternative to the inter-frame prediction performed by the motion estimation unit 42 and motion compensation unit 44 as described above, the intra-frame prediction unit 46 may perform intra-frame prediction for the current block. Specifically, the intra-frame prediction unit 46 may determine the intra-frame prediction mode to be used for encoding the current block. In some instances, the intra-frame prediction unit 46 may encode the current block using various intra-frame prediction modes, for example, during separate encoding passes, and the intra-frame prediction unit 46 (or, in some instances, the mode selection unit 40) may select an appropriate intra-frame prediction mode from the test modes for use.

[0076] For example, intra-prediction unit 46 can use rate distortion analysis to calculate rate distortion values ​​for various tested intra-prediction modes and select the intra-prediction mode with the best rate distortion characteristics from the tested modes. Rate distortion analysis generally determines the amount of distortion (or error) between the coded block and the original uncoded block encoded to produce the coded block, as well as the bit rate (i.e., the number of bits) used to generate the coded block. Intra-prediction unit 46 can calculate a ratio based on the distortion and rate of various coded blocks to determine which intra-prediction mode exhibits the best rate distortion value for the block.

[0077] After selecting an intra-prediction mode for a block, the intra-prediction unit 46 can provide information indicating the selected intra-prediction mode for the block to the entropy coding unit 56. The entropy coding unit 56 can encode the information indicating the selected intra-prediction mode. The video encoder 20 can include configuration data in the transmitted bitstream, which may include multiple intra-prediction mode index tables and multiple modified intra-prediction mode index tables (also called codeword mapping tables), definitions for encoding the context of various blocks, and indications of the most probable intra-prediction mode for each of the contexts, the intra-prediction mode index tables, and the modified intra-prediction mode index tables.

[0078] The video encoder 20 forms a residual video block by subtracting the prediction data from the mode selection unit 40 from the original video block being decoded. The summer 50 represents one or more components performing this subtraction. The transform processing unit 52 applies a transform, such as a discrete cosine transform (DCT) or a conceptually similar transform, to the residual block, thereby producing a video block that includes the residual transform coefficient values. The transform processing unit 52 can perform other transforms conceptually similar to DCT. Wavelet transform, integer transform, subband transform, or other types of transforms can also be used.

[0079] In any case, the transform processing unit 52 applies the transform to the residual block, thereby producing a block of residual transform coefficients. The transform converts the residual information from the pixel value domain to the transform domain (e.g., the frequency domain). The transform processing unit 52 can then send the resulting transform coefficients to the quantization unit 54. The quantization unit 54 quantizes the transform coefficients to further reduce the bit rate. The quantization process can reduce the bit depth associated with some or all of the coefficients. The degree of quantization can be modified by adjusting the quantization parameters. In some instances, the quantization unit 54 may then perform a scan on the matrix containing the quantized transform coefficients. Alternatively, the entropy coding unit 56 may perform the scan.

[0080] After quantization, entropy coding unit 56 performs entropy decoding on the quantized transform coefficients. For example, entropy coding unit 56 can perform context-adaptive variable-length decoding (CAVLC), context-adaptive binary arithmetic decoding (CABAC), syntax-based context-adaptive binary arithmetic decoding (SBAC), probabilistic interval partitioning entropy (PIPE) decoding, or another entropy decoding technique. In the case of context-based entropy decoding, the context can be based on adjacent blocks. After entropy decoding by entropy coding unit 56, the encoded bitstream can be transmitted to another device (e.g., video decoder 30) or archived for later transmission or retrieval.

[0081] Inverse quantization unit 58 and inverse transform unit 60 apply inverse quantization and inverse transform, respectively, to reconstruct the residual block in the pixel domain, for example, to be used later as a reference block. Motion compensation unit 44 can compute the reference block by adding the residual block to the prediction block of a frame in reference image memory 64. Motion compensation unit 44 can also apply one or more interpolation filters to the reconstructed residual block to compute sub-integer pixel values ​​for motion estimation. Summer 62 adds the reconstructed residual block to the motion-compensated prediction block generated by motion compensation unit 44 to produce a reconstructed video block for storage in reference image memory 64. The reconstructed video block can be used as a reference block by motion estimation unit 42 and motion compensation unit 44 for inter-frame decoding of blocks in subsequent video frames.

[0082] In this way, Figure 2 The video encoder 20 represents an instance of a video decoder configured to derive a histogram of motion vector information from neighboring blocks relative to the current block, determine merged candidates for a motion vector candidate list for motion vector prediction of the current block based on the derived histogram, sort the motion vector candidate list based on the derived histogram, and perform merged vector prediction using the motion vector candidate list.

[0083] Figure 3 This is a block diagram illustrating an example of a video decoder 30 that can be configured to perform the motion vector prediction techniques of this disclosure. Figure 3 In one example, the video decoder 30 includes a video data memory 71, an entropy decoding unit 70, a motion compensation unit 72, an intra-frame prediction unit 74, an inverse quantization unit 76, an inverse transform unit 78, a reference image memory 82, and a summer 80. In some instances, the video decoder 30 can perform a decoding pass that is generally the inverse of the encoding pass described with respect to the video encoder 20 (FIG. 2). The motion compensation unit 72 can generate prediction data based on motion vectors received from the entropy decoding unit 70, while the intra-frame prediction unit 74 can generate prediction data based on intra-frame prediction mode indicators received from the entropy decoding unit 70.

[0084] Video data memory 71 can store encoded video data, such as encoded video bitstreams, to be decoded by components of video decoder 30. The video data stored in video data memory 71 can be obtained, for example, from computer-readable media 16, such as from a local video source (e.g., a camera), via wired or wireless network communication of video data, or by accessing physical data storage media. Video data memory 71 can form a decoded picture buffer (CPB) storing encoded video data from the encoded video bitstream. Reference picture memory 82 (also referred to as decoded picture buffer (DPB)) can be a reference picture memory storing reference video data for the video decoder 30 to use, for example, for decoding the video data in intra-frame or inter-frame decoding modes or for output. Video data memory 71 and reference picture memory 82 can be formed from any of various memory devices, such as DRAM, SDRAM, MRAM, RRAM, or other types of memory devices. Video data memory 71 and reference picture memory 82 can be provided by the same memory device or separate memory devices. In various instances, the video data memory 71 may be on-chip along with other components of the video decoder 30, or off-chip relative to those components.

[0085] During the decoding process, video decoder 30 receives an encoded video bitstream representing video blocks and associated syntax elements from video encoder 20. Entropy decoding unit 70 of video decoder 30 performs entropy decoding on the bitstream to generate quantized coefficients, motion vectors or intra-frame prediction mode indicators, and other syntax elements. Entropy decoding unit 70 forwards the motion vectors and other syntax elements to motion compensation unit 72. Video decoder 30 may receive syntax elements at the video slice level and / or video block level.

[0086] When a video slice is decoded into an intra-decoded (I) slice, the intra-prediction unit 74 can generate prediction data for the video block of the current video slice based on the intra-prediction mode represented by a signal and data from the previously decoded block from the current frame or picture. When a video frame is decoded into an inter-decoded (e.g., BB or P) slice, the motion compensation unit 72 generates a prediction block for the video block of the current video slice based on motion vectors and other syntax elements received from the entropy decoding unit 70. Prediction blocks can be generated from one of the reference pictures in one of the reference picture lists. The video decoder 30 can construct a reference frame list, i.e., list 0 and list 1, based on the reference pictures stored in the reference picture memory 82 using a default construction technique.

[0087] Motion compensation unit 72 determines prediction information for video blocks in the current video slice by analyzing motion vectors and other syntax elements, and uses the prediction information to generate prediction blocks for the current video slice being decoded. For example, motion compensation unit 72 uses some of the received syntax elements to determine the prediction mode (e.g., intra-frame prediction or inter-frame prediction) for decoding video blocks in the video slice, the inter-frame prediction slice type (e.g., B-slice or P-slice), one or more construction information from the list of reference pictures for the slice, the motion vectors of each inter-frame coded video block in the slice, the inter-frame prediction state of each inter-frame decoded video block in the slice, and other information for decoding video blocks in the current video slice.

[0088] The motion compensation unit 72 can also perform interpolation based on an interpolation filter. The motion compensation unit 72 can use the interpolation filter used by the video encoder 20 during encoding of the video block to calculate the interpolated values ​​of sub-integer pixels of the reference block. In this case, the motion compensation unit 72 can determine the interpolation filter used by the video encoder 20 based on the received syntax elements, and use the interpolation filter to generate the prediction block.

[0089] The video decoder 30, which includes the motion compensation unit 72, can be configured to perform the above-mentioned functions. Figure 1 Any of the various techniques discussed herein, as will be described in more detail below. For example, motion compensation unit 72 may be configured to perform motion vector prediction using AMVP or merging mode according to the techniques of this disclosure. Additionally, video decoder 30, which includes motion compensation unit 72, may be configured to perform any combination of the motion vector candidate list construction techniques of this disclosure, as described in more detail below. Entropy decoding unit 70 may decode one or more syntax elements representing how motion information is decoded for the current block.

[0090] Assuming the syntax element indicates the execution of a merge mode, motion compensation unit 72 can form a candidate list containing a set of merge candidates. Motion compensation unit 72 can add candidates to the candidate list based on a specific predetermined order. In other embodiments of this disclosure, motion compensation unit 72 can be configured to add candidates to the candidate list in dynamically different orders based on histogram information of motion vectors from neighboring blocks. Motion compensation unit 72 can also add additional candidates and perform a reduction of the candidate list, as discussed in more detail below. Finally, mode selection unit 72 can decode the merge index, which indicates which candidates are used to decode the motion information for the current block.

[0091] The inverse quantization unit 76 performs inverse quantization, or dequantization, on the quantized transform coefficients provided in the bitstream and decoded by the entropy decoding unit 70. The inverse quantization process may involve using a quantization parameter QP calculated by the video decoder 30 for each video block in the video slice to determine the degree of quantization and the same degree of inverse quantization to be applied. Y .

[0092] The inverse transform unit 78 applies an inverse transform (e.g., inverse DCT, inverse integer transform, or a conceptually similar inverse transform process) to the transform coefficients in order to produce a residual block in the pixel domain.

[0093] After the motion compensation unit 72 generates a prediction block for the current video block based on motion vectors and other syntax elements, the video decoder 30 forms a decoded video block by summing the residual block from the inverse transform unit 78 with the corresponding prediction block generated by the motion compensation unit 72. The summer 80 represents one or more components performing this summation operation. If necessary, a deblocking filter can also be applied to filter the decoded block to remove blocking artifacts. Other loop filters (in or after the decoding loop) can also be used to smooth pixel transitions or otherwise improve video quality. The decoded video block in a given frame or image is then stored in a reference image memory 82, which stores reference images for subsequent motion compensation. The reference image memory 82 also stores the decoded video for later presentation on a display device (e.g., Figure 1 On the display device 32).

[0094] In this way, video decoder 30 represents an instance of video decoder configured to derive histograms of motion vector information from neighboring blocks relative to the current block, determine merged candidates for a motion vector candidate list for motion vector prediction of the current block based on the derived histograms, sort the motion vector candidate list based on the derived histograms, and perform merged vector prediction using the motion vector candidate list.

[0095] The following sections describe certain aspects of video decoding technologies and standards, particularly regarding motion vector prediction and related techniques. First, motion information is discussed. For each video data block decoded using inter-frame prediction mode, a set of motion information is obtained. This set of motion information contains motion information for the forward and backward prediction directions. Here, the forward and backward prediction directions are two prediction directions corresponding to reference image list 0 (RefPicList0) and reference image list 1 (RefPicList1) for the current image or slice. The terms "forward" and "backward" do not necessarily have a geometric meaning. Instead, they are used to distinguish the reference image lists on which the motion vectors are based. Forward prediction means a prediction formed based on reference list 0, while backward prediction means a prediction formed based on reference list 1. When both reference list 0 and reference list 1 are used to form a prediction for a given block, it is called bidirectional prediction.

[0096] For a given image or slice, if only one list of reference images is used, then every block within the image or slice is forward-predicted. If the entire list of reference images is used for a given image or slice, then the blocks within the image or slice can be forward-predicted, backward-predicted, or bidirectionally predicted.

[0097] For each prediction direction, the motion information also includes a reference index and a motion vector. The reference index is used to identify the reference image in the corresponding list of reference images (e.g., RefPicList0 or RefPicList1). The motion vector has both horizontal and vertical components, each indicating an offset value along the horizontal and vertical directions, respectively. The motion vector indicates the position of the predictor block relative to the current decoded block. The reference index indicates the image containing the predictor block. In some descriptions, for simplicity, the term "motion vector" may be used interchangeably with "motion information" to refer to both the motion vector and its associated reference index.

[0098] Picture order count (POC) is widely used in video decoding standards to identify the display order of pictures. While it's possible for two pictures within a decoded video sequence to have the same POC value, this is generally not the case within a decoded video sequence. When multiple decoded video sequences exist in a bitstream, pictures with the same POC value are likely closer to each other in terms of decoding order. Picture POC values ​​are commonly used for constructing reference picture lists and deriving reference picture sets, as in HEVC and motion vector scaling.

[0099] The next section describes the macroblock (MB) structure in Advanced Video Decoding (AVC) (H.264). In H.264 / AVC, each inter-frame macroblock (MB) (e.g., an MB using inter-frame prediction decoding) can be segmented in four different ways:

[0100] • A 16x16MB partition

[0101] Two 16×8MB partitions

[0102] Two 8×16MB partitions

[0103] Four 8x8MB partitions

[0104] Different MB partitions within an MB can have different reference index values ​​(RefPicList0 or RefPicList1) for each prediction direction. When an MB is not divided into four 8×8 MB partitions, the MB has only one motion vector for each MB partition in each prediction direction.

[0105] When the MB is divided into four 8×8 MB partitions, each 8×8 MB partition can be further divided into sub-blocks, each of which can have a different motion vector in each prediction direction. There are four different methods for dividing an 8×8 MB partition into sub-blocks:

[0106] • An 8×8 sub-block

[0107] Two 8×4 sub-blocks

[0108] Two 4×8 sub-blocks

[0109] • Four 4×4 sub-blocks

[0110] Each sub-block can have a different motion vector in each predicted direction. Therefore, motion vectors exist at levels equal to or higher than the sub-block.

[0111] The temporal direct mode in AVC will not be described further. In AVC, temporal direct mode can be enabled at the MB level or MB partition level for use in skip or direct mode in B slices. For each MB partition, motion vectors are derived using motion vectors from blocks that are co-located with the current MB partition in RefPicList1[0] of the current block. Each motion vector in a co-located block is scaled based on the POC distance. In AVC, direct mode can also predict motion information from spatial neighbors.

[0112] The structure of the decoding unit (CU) in HEVC will now be described. In HEVC, the largest decoding unit in a slice is called a decoding tree block (CTB) or decoding tree unit (CTU). A CTB contains a quadtree whose nodes are decoding units. A CTB can be recursively split into CUs in a quadtree manner, as described in WJHan et al.'s "Improved Video Compression Efficiency Through Flexible Unit Representation and Corresponding Extension of Coding Tools" (IEEE Transactions on Video Technology Circuits and Systems, Vol. 20, No. 12, pp. 1709-1720, December 2010), and in Figure 4 As shown in the image. Figure 4 As shown, each partition level is a quadtree that splits into four sub-blocks. Black blocks are instances of leaf nodes (i.e., blocks that do not split further).

[0113] The size of the CTB in the HEVC master specification can range from 16×16 to 64×64 (although an 8×8 CTB size is technically supported). However, the size of the CU can be the same as the CTB and as small as 8×8. Each CU is decoded in one mode (e.g., intra-prediction mode or inter-prediction mode). When a CU is inter-decoded, it can be further divided into 2 or 4 prediction units (PUs), or become a single PU without further division. When there are two PUs in a CU, the PUs can be half-size rectangles, or two rectangles that are 1 / 4 or 3 / 4 the size of the CU.

[0114] When the CU is inter-frame decoded, there is a set of motion information (e.g., motion vectors, predicted direction, and reference image) for each PU. Furthermore, each PU is decoded using a unique inter-frame prediction mode to derive this set of motion information. However, it should be understood that even if two PUs are uniquely decoded, they may still have the same motion information in some cases.

[0115] In HEVC, there are eight partitioning modes for CUs decoded in inter-frame prediction mode: PART_2N×2N, PART_2N×N, PART_N×2N, PART_N×N, PART_2N×nU, PART_2N×nD, PART_nL×2N, and PART_nR×2N. Figure 5As shown. The CU decoded using the PART_2N×2N splitting pattern is not further split. That is, the entire CU is processed into a single PU (PU0). The CU decoded using the PART_2N×N splitting pattern is horizontally symmetrically split into two PUs (PU0 and PU1). The CU decoded using the PART_N×2N splitting pattern is vertically symmetrically split into two PUs. The CU decoded using the PART_N×N splitting pattern is symmetrically split into four PUs of equal size (PU0, PU1, PU2, PU3).

[0116] The CU decoded in PART_2N×nU splitting mode is asymmetrically horizontally split into a PU0 (upper PU), which is 1 / 4 the size of the CU, and a PU1 (lower PU), which is 3 / 4 the size of the CU. The CU decoded in PART_2N×nD splitting mode is asymmetrically horizontally split into a PU0 (upper PU), which is 3 / 4 the size of the CU, and a PU1 (lower PU), which is 1 / 4 the size of the CU. The CU decoded in PART_nL×2N splitting mode is asymmetrically vertically split into a PU0 (left PU), which is 1 / 4 the size of the CU, and a PU1 (right PU), which is 3 / 4 the size of the CU. The CU decoded in PART_nR×2N splitting mode is asymmetrically vertically split into a PU0 (left PU), which is 3 / 4 the size of the CU, and a PU1 (right PU), which is 1 / 4 the size of the CU.

[0117] While HEVC uses a quadtree partitioning structure, other partitioning structures are being researched for future video decoding standards. For example, in J. An et al.'s "Block partitioning structure for next generation video coding," International Telecommunication Union (ITU), COM16-C966, September 2015 (hereinafter referred to as "VCEG Proposal COM16-C966"), a quadtree-binary tree (QTBT) partitioning technique was proposed for future video decoding standards other than HEVC (e.g., H.266). Simulations show that for certain video sequences, the proposed QTBT structure is more efficient than the quadtree structure used in HEVC.

[0118] In the QTBT structure of the proposed VCEG proposal COM16-C966, the CTB is first segmented using a quadtree segmentation technique, where the quadtree split of a node can be iterated until the node reaches the minimum allowed quadtree leaf node size. The minimum allowed quadtree leaf node size can be indicated to the video decoder by the value of the syntax element MinQTSize. If the quadtree leaf node size is not greater than the maximum allowed binary tree root node size (e.g., as indicated by the syntax element MaxBTSize), binary tree segmentation can be used to further segment the quadtree leaf nodes. The binary tree segmentation of a node can be iterated until the node reaches the minimum allowed binary tree leaf node size (e.g., as indicated by the syntax element MinBTSize) or the maximum allowed binary tree depth (e.g., as indicated by the syntax element MaxBTDepth). The VCEG proposal COM16-C966 uses the term "CU" to refer to binary tree leaf nodes. In the VCEG proposal COM16-C966, CUs are used for prediction (e.g., intra-frame prediction, inter-frame prediction, etc.) and transform without any further segmentation. Typically, according to the QTBT technique, binary tree splitting has two types: symmetric horizontal splitting and symmetric vertical splitting. In each case, nodes are split by dividing blocks horizontally or vertically from the middle.

[0119] In one instance of a QTBT partitioning structure, the CTU size is set to 128×128 (e.g., a 128×128 luma block and two corresponding 64×64 chroma blocks), the MinQTSize is set to 16×16, the MaxBTSize is set to 64×64, the MinBTSize (for both width and height) is set to 4, and the MaxBTDepth is set to 4. Quadtree partitioning is first applied to the CTU to generate quadtree leaf nodes. Quadtree leaf nodes can have sizes ranging from 16×16 (i.e., MinQTSize is 16×16) to 128×128 (i.e., the CTU size). According to one instance of QTBT partitioning, if a quadtree leaf node is 128×128, then the quadtree leaf node cannot be further partitioned by the binary tree because the size of the quadtree leaf node exceeds the MaxBTSize (i.e., 64×64). Otherwise, the quadtree leaf node is further partitioned by the binary tree. Therefore, the quadtree leaf node is also the root node of the binary tree, and the binary tree depth is 0. A binary tree depth reaching MaxBTDepth (e.g., 4) implies no further splitting. A binary tree node with a width equal to MinBTSize (e.g., 4) implies no further horizontal splitting. Similarly, a binary tree node with a height equal to MinBTSize implies no further vertical splitting. Leaf nodes (CUs) of the binary tree can be further processed without any further splitting (e.g., by performing prediction and transformation processing).

[0120] Figure 6A An example of block 150 (e.g., CTB) segmented using the QTBT segmentation technique is shown. Figure 6A As shown, the QTBT segmentation technique is used to split each resulting block symmetrically through the center of each block. Figure 6B The tree structure corresponding to the block segmentation in Figure 6A is shown. Figure 6B Solid lines in the diagram indicate quadtree splits, while dashed lines indicate binary tree splits. In one instance, at each split (i.e., non-leaf) node of a binary tree, a signal is used to represent a syntax element (e.g., a flag) to indicate the type of split performed (e.g., horizontal or vertical), where 0 indicates a horizontal split and 1 indicates a vertical split. For quadtree splits, it is not necessary to indicate the split type because a quadtree split always divides a block horizontally and vertically into four sub-blocks of the same size.

[0121] like Figure 6B As shown, at node 170, QT partitioning is used to split block 150 into four blocks 151, 152, 153, and 154, as follows. Figure 6AAs shown. Block 154 does not split further, therefore it is a leaf node. At node 172, BT splitting is used to further split block 151 into two blocks. (See diagram) Figure 6B As shown, node 172 is marked with a 1, indicating a vertical split. Thus, the split at node 172 forms block 157, and this block includes blocks 155 and 156. Blocks 155 and 156 are created by a further vertical split at node 174. At node 176, block 152 is further split into two blocks 158 and 159 using BT partitioning. Figure 6B As shown, node 176 is marked with 1, indicating a horizontal split.

[0122] At node 178, QT splitting is used to divide block 153 into four equal-sized blocks. Blocks 163 and 166 are generated by this QT splitting and are not further split. At node 180, a vertical binary tree split is first used to split the upper left block, resulting in block 160 and a right vertical block. Then, a horizontal binary tree split is used to split the right vertical block into blocks 161 and 162. The lower right block generated by quadtree splitting at node 178 is split into blocks 164 and 165 by horizontal binary tree splitting at node 184.

[0123] The motion vector candidate list construction technique described in more detail below can be used in conjunction with any video block segmentation technique, including the MB segmentation structure for H.264 / AVC, the quadtree segmentation structure for HEVC, or the QTBT segmentation structure, such as the QTBT structure proposed for H.266.

[0124] Motion prediction in HEVC will now be described. In the HEVC standard, there are two inter-frame prediction modes for the PU, referred to as the merge mode (skipping special cases considered as merges) and the Advanced Motion Vector Prediction (AMVP) mode. In either AMVP or merge mode, the video encoder 20 and video decoder 30 are configured to construct a list of motion vector (MV) candidates for multiple motion vector predictors. The motion vector predictors can be motion vectors from neighboring blocks or artificially generated motion vectors that can be used to predict the motion vector of the currently decoded video data block. In merge mode, instead of encoding the motion vector of the current block itself, the video encoder 20 selects a motion vector from the list of motion vector candidates and a reference index associated with that candidate, and uses the candidate motion vector for inter-frame prediction. The video encoder 20 can signal the index of the selected motion vector candidate (e.g., the merge index) to the video decoder 30. The video decoder 30 can construct a list of motion vector candidates for merge mode in the same manner as the video encoder 20. The video decoder 30 can use the index notified by the signal to identify the selected candidate in the motion vector candidate list, and then retrieve the motion vector and reference index associated with the candidate to use as the motion vector for the current block.

[0125] The MV candidate list contains up to five candidates for merge mode and two candidates for AMVP mode. Merge candidates may contain a set of motion information, such as motion vectors and reference indices corresponding to two lists of reference images (list 0 and list 1). If a merge candidate is identified by the merge index, the reference images are used for prediction of the current block, and the associated motion vectors are determined. However, in AMVP mode, for each potential prediction direction starting from either list 0 or list 1, since the AMVP candidate only contains motion vectors, the reference index is explicitly signaled to the MV candidate list along with the motion vector predictor (MVP) index. In AMVP mode, the predicted motion vectors can be further optimized. In some instances of AMVP, the video encoder 20 may also signal the motion vector difference (MVD). MVD is the difference between the selected MVP and the actual determined motion vectors for the current block. As can be seen above, merge candidates correspond to the entire set of motion information, while AMVP candidates contain only a motion vector for a specific prediction direction and a reference index.

[0126] Figure 7 This is a conceptual diagram illustrating spatial adjacency candidates in HEVC. Although Figure 7 This shows the spatial MV candidates derived from adjacent blocks for a specific PU (PU0), but the method of generating candidates from blocks differs for merge mode and AMVP mode.

[0127] In merge mode, it is possible Figure 7 (a) shows the order that yields up to four spatial MV candidates. The order is as follows: left (0,A1), top (1,B1), top right (2,B0), bottom left (3,A0), and top left (4,B2), as shown. Figure 7 As shown in (a). That is to say, in Figure 7 In (a), block 200 includes PU0 204A and PU1 204B. When a video decoder (e.g., video encoder 20 and / or video decoder 30) decodes motion information for PU0 204A using a merging mode, the video decoder adds motion information from spatially adjacent blocks 208A, 208B, 208C, 208D, and 208E to a candidate list in the aforementioned order. Blocks 208A, 208B, 208C, 208D, and 208E may also be referred to as blocks A1, B1, B0, A0, and B2 in HEVC, respectively.

[0128] In AVMP mode, adjacent blocks are divided into two groups: a left group containing blocks 0 and 1, and an upper group containing blocks 2, 3, and 4, as shown below. Figure 7 As shown in (b). These blocks are in Figure 7 (b) are labeled as blocks 210A, 210B, 210C, 210D, and 210E, respectively. Block 202 contains PU0 206A and PU1 206B, and blocks 210A, 210B, 210C, 210D, and 210E represent spatial neighbors of PU0 206A. For each group, potential candidates in adjacent blocks referencing the same reference image, as indicated by a signaled reference index, have the highest priority for selection as the final candidate forming the group. It is possible that all adjacent blocks do not contain motion vectors pointing to the same reference image. Therefore, if such candidates cannot be found, the first available candidates can be scaled to form the final candidates; thus, temporal distance differences can be compensated.

[0129] Figure 8 This is a conceptual diagram illustrating Temporal Motion Vector Prediction (TMVP) in HEVC. Specifically, Figure 8 (a) shows an instance CU 220 containing PU0 222A and PU1 222B. PU0 222A contains a central block 226 for PU 222A and a lower right block 224 for PU01 222A. Figure 8 (a) Also shown is an external block 228 from which motion information can be predicted from the motion information of PU0 222A, as discussed below. Figure 8 (b) Shows the current image 230 containing motion information to be predicted for the current block 238. Specifically, Figure 8(b) Shows the juxtaposed image 234 of the current image 230 (containing the juxtaposed block 240 relative to the current block 238), the current reference image 232, and the juxtaposed reference image 236. The juxtaposed block 240 is predicted using motion vector 244, which serves as a temporal motion vector predictor (TMVP) 242 for motion information of block 238.

[0130] If TMVP is enabled and TMVP candidates are available, the video decoder (e.g., video encoder 20 and / or video decoder 30) can add a TMVP candidate (e.g., TMVP candidate 242) to the MV candidate list after any spatial motion vector candidate. The motion vector derivation process for TMVP candidates is the same for both merge mode and AMVP mode. However, according to HEVC, the target reference index of the TMVP candidate is set to 0 in merge mode.

[0131] The main block position exported by the TMVP candidate is the lower right block outside the PU, such as... Figure 8 (a) shows block 224 of PU0 222A, which is used to compensate for the offset of the upper-left block used to generate spatially adjacent candidates. However, if block 224 is located outside the current CTB row or motion information is not available for block 224, then as Figure 8 The block shown in (a) is replaced by the center block 226 of the PU. The motion vector of the TMVP candidate 242 is derived from the juxtaposed block 240 of the juxtaposed image 234, as indicated by the slice-level information.

[0132] Similar to the direct temporal mode in AVC, the motion vectors of TMVP candidates can undergo motion vector scaling to compensate for the point of convergence (POC) distance differences between the current image 230 and the current reference image 232, and between the juxtaposed image 234 and the juxtaposed reference image 236. In other words, motion vector 244 can be scaled based on these POC differences to generate TMVP candidate 242. Other aspects of motion prediction in HEVC are described below.

[0133] Motion vector scaling can be performed in HEVC. It is assumed that the value of the motion vector is proportional to the distance between the images at the time of rendering. The motion vector associates two images: a reference image and an image containing the motion vector (i.e., the containing image). When a motion vector is used to predict another motion vector, the distance between the containing image and the reference image is calculated based on the Picture Order Count (POC) value.

[0134] For a motion vector to be predicted, its associated containment image and reference image can be different. Therefore, a new distance (based on POC) is calculated. The motion vector is then scaled proportionally based on these two POC distances. For spatially adjacent candidates, the containment images of the two motion vectors are the same, while the reference images are different. In HEVC, motion vector scaling applies to both TMVP and AMVP for spatially and temporally adjacent candidates.

[0135] Artificial motion vector candidate generation can also be performed in HEVC. If the motion vector candidate list is incomplete (e.g., includes fewer than a specified number of candidates), the video encoder 20 and / or video decoder 30 can generate artificial motion vector candidates. The video encoder 20 and / or video decoder 30 generate and insert artificial motion vector candidates at the end of the motion vector candidate list until the motion vector candidate list has a specified number of candidates.

[0136] In the merge mode, there may be two types of artificial motion vector candidates: bidirectional combined artificial motion vector candidates derived for B slices, and zero motion vector candidates using only AMVP if the first type (i.e., combined artificial motion vector candidates) does not provide enough artificial candidates to fill the candidate list.

[0137] For each pair of candidates that is already in the candidate list and has the necessary motion information, a bidirectional combined motion vector candidate is derived by combining the motion vector of the first candidate in the image in reference list 0 and the motion vector of the second candidate in the image in reference list 1. The zero motion vector candidate simply points to the juxtaposed block in the other image (e.g., the zero motion vector candidate is (0, 0)).

[0138] The video decoder configured according to HEVC can also perform a simplification process for candidate insertion. Motion vector candidates from different blocks may happen to be identical, which reduces the efficiency of merging / AMVP candidate lists. That is, multiple motion vector candidates with the same value reduce the actual number of motion vectors to be tested and selected. A simplification process can be used to solve this problem. The simplification process compares a motion vector candidate with other motion vector candidates in the current candidate list to avoid inserting the same candidate in some cases. To reduce complexity, only a limited number of simplification processes are applied, rather than comparing every potential simplification process with all other existing simplification processes.

[0139] Figure 9 This illustrates the instance prediction structure of 3D-HEVC. 3D-HEVC is a 3D video extension of HEVC developed by JCT-3V. The following section discusses... Figure 9 and 10Describes certain technologies related to the technologies disclosed herein. Figure 9 The multi-view prediction structure for a three-view scenario is illustrated. V3 represents the base view, and images in non-base views (V1 or V5) can be predicted from images in subordinate (base) views of the same time instance. Multi-view HEVC (MV-HEVC) supports inter-view sample prediction (from reconstructed samples), and its typical prediction structure is shown in... Figure 10 As shown in the image.

[0140] Both MV-HEVC and 3D-HEVC are compatible with HEVC in a way that the HEVC (version 1) decoder can decode the underlying (texture) view. Test models for MV-HEVC and 3D-HEVC are described in Zhang et al.'s "Test Model 6 of 3D-HEVC and MV-HEVC" (JCT-3V document ISO / IEC JTC1 / SC29 / WG11 N13940, available at mpeg.chiariglione.org / standards / mpeg-h / high-efficiency-video-coding / test-model-6-3d-hevc-and-mv-hevc, January 26, 2015).

[0141] In MV-HEVC, by placing all these images in a reference image list, the current image in a non-base view can be predicted using two images from the same view and images from a reference view at the same time instance. Therefore, the reference image list for the current image contains both a temporal reference image and an inter-view reference image. The motion vector associated with the reference index corresponding to the temporal reference image is represented as a temporal motion vector. The motion vector associated with the reference index corresponding to the inter-view reference image is represented as a disparity motion vector. 3D-HEVC supports all the features of MV-HEVC. Therefore, inter-view sample prediction as described above is possible.

[0142] Additionally, 3D-HEVC supports more advanced texture-only decoding tools and depth-dependent / dependent decoding tools. Texture-only decoding tools typically use the identification of corresponding blocks (between views) that can belong to the same object. Therefore, parallax vector export is a technique used in 3D-HEVC.

[0143] Figure 10 This is a conceptual diagram illustrating view-to-view motion prediction based on sub-PUs in 3D-HEVC. Figure 10The image shows the current image 360 ​​in the current view (V1) and the juxtaposed image 362 in the reference view (V0). The current image 360 ​​contains the current PU 364, which contains four sub-PUs 366A-366D (sub-PUs 366). Each disparity vector 374A-374D (disparity vector 374) identifies the corresponding sub-PU 368A-368D to the sub-PU 366 in the juxtaposed image 362. In 3D-HEVC, a sub-PU-level inter-view motion prediction method is used for inter-view merging candidates, i.e., candidates derived from reference blocks in the reference view.

[0144] When this mode is enabled, the current PU 364 can correspond to a reference region in the reference view (with the same size as the current PU identified by the disparity vector), and the reference region can have richer motion information (e.g., many different related motion vectors) than is required to generate the set of motion information for the PU. Therefore, sub-PU level inter-view motion prediction (SPIVMP) methods, such as... Figure 10 As shown. This mode can also be used as a special merging candidate signal notification. Each of the sub-PUs contains a complete set of motion information. Therefore, a PU can contain multiple sets of motion information.

[0145] Sub-PU-based Motion Parameter Inheritance (MPI) can also be used in 3D-HEVC. In 3D-HEVC depth decoding, MPI candidates derived from the texture view can be extended in a manner similar to inter-view motion prediction at the sub-PU level. For example, if the current depth PU has a juxtaposed region containing multiple PUs, then the current depth PU can be separated into sub-PUs, each with a different set of motion information. This method is called Sub-PU MPI.

[0146] U.S. Patent Application No. 14 / 497,128, filed September 25, 2014, describes an example subPU-related technique for 2D video decoding, the entire contents of which are incorporated herein by reference. An advanced TMVP (ATMVP) design based on a subPU has been proposed in U.S. Patent Application No. 14 / 497,128.

[0147] In single-layer decoding, a two-level high-level temporal motion vector prediction design can be used. The first level is used to derive the vectors of the corresponding block of the current prediction unit (PU) in the identification reference image, and the second level is used to extract multiple sets of motion information from the corresponding block and assign them to the sub-PUs of the PU. Thus, each sub-PU of the PU performs motion compensation separately. The concept of ATMVP is summarized as follows: (1) The vectors in the first level can be derived from the spatial and temporal neighboring blocks of the current PU. (2) This process can be implemented by activating one merging candidate among all other merging candidates. Applicable to single-layer decoding and sub-PU temporal motion vector prediction, the PU or CU can have motion refinement data transmitted on top of the predictor.

[0148] The following highlights several aspects of U.S. Application No. 14 / 497,128:

[0149] 1. The first level of vector-derived vectors can also be simplified by using only zero vectors.

[0150] 2. The first level of vector-derived vectors can contain jointly identified motion vectors and their associated images. Various methods have been proposed for selecting associated images and further determining the motion vectors as first-level vectors.

[0151] 3. If motion information is unavailable during the above process, then use the "first-level vector" instead.

[0152] 4. Motion vectors identified from temporal neighbors must be scaled proportionally for the current sub-PU, similar to motion vector scaling in TMVP. However, the reference image to which such motion vectors can be scaled can be designed using one of the following methods:

[0153] a. Identify the image using a fixed reference index of the current image.

[0154] b. If the image is also available in the list of reference images for the current image, then the image is identified as the reference image for the corresponding time-adjacent item.

[0155] c. The image is set to the juxtaposed image identified in the first level and the position of the motion vector is obtained from it.

[0156] To address certain design issues in U.S. Application No. 14 / 497,128, the following technology is proposed in U.S. Application No. 15 / 005,564, filed January 25, 2016, the entire contents of which are incorporated herein by reference.

[0157] 1. The position of ATMVP candidates (if inserted), for example, as part of the merge candidate list.

[0158] a. Assume that spatial candidates and TMVP candidates are inserted into the merge candidate list in a specific order. ATMVP candidates can be inserted in any relatively fixed position among those candidates.

[0159] i. In an alternative, for example, ATMVP candidates can be inserted into the merge candidate list after the first two space candidates, such as A1 and B1;

[0160] ii. In an alternative, for example, the ATMVP candidate can be inserted after the first three space candidates, such as A1, B1, and B0;

[0161] iii. In an alternative, for example, the ATMVP candidate can be inserted after the first four candidates, such as A1, B1, B0 and A0.

[0162] iv. In an alternative, for example, the ATMVP candidate could be inserted just before the TMVP candidate.

[0163] v. In an alternative, for example, the ATMVP candidate can be inserted exactly after the TMVP candidate.

[0164] b. Alternatively, the position of an ATMVP candidate in the candidate list can be signaled in the bit stream. The positions of other candidates, including TMVP candidates, can also be signaled.

[0165] 2. The availability check for ATMVP candidates can be applied by accessing only one set of motion information. When this set of information is unavailable, for example, if a block is intra-decoded, the entire ATMVP candidate is considered unavailable. In this case, the ATMVP will not be inserted into the merge list.

[0166] a. A center location or a center sub-PU can be used to check the availability of ATMVP candidates. When using a center sub-PU, the center sub-PU is selected as one that covers the center location (e.g., center 3 location, whose relative coordinates to the upper left sample of the PU are (W / 2, H / 2), where WxH is the size of the PU). Such a location or center sub-PU can be used in conjunction with a time vector to identify corresponding blocks in the motion source image. The set of motion information from the block covering the center location of the corresponding block is identified.

[0167] 3. The representative set of motion information used for ATMVP decodes the PU from the sub-PU.

[0168] a. In order to form ATMVP candidates, a representative sports information set is first formed.

[0169] b. Such representative motion information sets can be derived from fixed locations or fixed sub-PUs. These representative motion information sets can be selected in the same manner as using motion information sets to determine the availability of ATMVP candidates, as described in item #2.

[0170] c. When a sub-PU has already identified its own set of motion information and it is unavailable, it is set to be equal to the representative set of motion information.

[0171] d. If the representative motion information set is set as the representative set of motion information of the subPU, then in the worst case scenario, no additional motion storage device is required on the decoder side for the current CTU or slice.

[0172] e. When the decoding process requires the entire PU to be represented by a set of motion information, such a representative set of motion information is used for all scenarios, including simplification, so that the process can be used to generate combined bidirectional prediction merging candidates.

[0173] 4. The ATMVP candidates are streamlined together with the TMVP candidates, and the interaction between TMVP and ATMVP can be considered; detailed technical details are listed below:

[0174] a. Reduction of sub-PU-based candidates (e.g., ATMVP candidates) along with regular candidates can be done by using a representative set of motion information for such sub-PU-based candidates (as in item #3). If such a set of motion information is the same as that for regular merged candidates, the two candidates are considered identical.

[0175] b. Alternatively, an inspection is performed to determine whether the ATMVP contains multiple distinct sets of motion information for multiple sub-PUs; if at least two distinct sets are identified, the sub-PU-based candidate is not used for simplification, i.e., is considered distinct from any other candidate; otherwise, it may be used for simplification (e.g., it may be simplified during the simplification process).

[0176] c. Alternatively, ATMVP candidates can be streamlined along with spatial candidates, for example, only the spatial candidates on the left and top, labeled A1 and B1.

[0177] d. Alternatively, only one candidate is formed from the time reference, which is either an ATMVP candidate or a TMVP candidate. When ATMVP is available, the candidate is ATMVP; otherwise, the candidate is TMVP. A candidate is inserted into the merged candidate list at a position similar to that of the TMVP. In this case, the maximum number of candidates can remain unchanged.

[0178] i. Alternatively, TMVP is always disabled even when ATMVP is unavailable.

[0179] ii. Alternatively, TMVP may be used only if ATMVP is unavailable.

[0180] e. Alternatively, when ATMVP is available and TMVP is unavailable, a set of motion information from a sub-PU is used as a TMVP candidate. Furthermore, in this case, the simplification process between ATMVP and TMVP is not applied.

[0181] f. Alternatively or additionally, the time vector used for ATMVP can also be used for TMVP, so that the lower right or center 3 position of the current TMVP in HEVC does not need to be used.

[0182] i. Alternatively, the locations identified by the time vector, as well as the lower right and center 3 positions, are collectively considered as providing available TMVP candidates.

[0183] 5. Supports multiple availability checks for ATMVP to increase the chances of more accurate and efficient ATMVP candidates. When current ATMVP candidates are identified from motion source images via first-time vector recognition (e.g., ... Figure 9 When an unavailable image (as shown) is not available, other images can be considered as motion source images. When considering another image, it can be associated with a different second time vector, or simply with a second time vector scaled from the first time vector and pointing to an unavailable ATMVP candidate.

[0184] a. The second time vector can identify ATMVP candidates in the second motion source image and can apply the same availability check. If the ATMVP candidates exported from the second motion source image are available, then the ATMVP candidates are exported and no other images need to be checked; otherwise, other images similar to the motion source image can be checked.

[0185] b. The images to be checked can be those in the reference image list for the current image, which have a given order. For each list, the images are checked in ascending order of their reference indices. First, list X is checked, and then the images in list Y (which is 1-X) are checked.

[0186] i. Select list X such that list X is a list containing juxtaposed images for TMVP.

[0187] ii. Alternatively, X is simply set to 1 or 0.

[0188] c. The images to be examined are those identified by the motion vectors of spatially adjacent items, which have a given order.

[0189] 6. The PU segmentation currently applied to by ATMVP can be 2N×2N, N×N, 2N×N, N×2N, or asymmetric motion segmentation (AMP) partitioning, such as 2N×N / 2.

[0190] a. Alternatively, ATMVP can also be supported if other partition sizes are allowed, and such sizes can include, for example, 64×8.

[0191] b. Alternatively, the pattern may be applicable only to certain partitions, for example, 2N×2N.

[0192] 7. ATMVP candidates are marked as different types of merge candidates.

[0193] 8. When identifying vectors from adjacent items (as time vectors in the first level), multiple adjacent positions can be checked sequentially, such as those used in the construction of the merged candidate list. For each of the adjacent items, the motion vectors corresponding to either reference image list 0 (list 0) or reference image list 1 (list 1) can be checked sequentially. When two motion vectors are available, the motion vectors in list X can be checked first, followed by list Y (where Y equals 1-X), such that list X is a list containing juxtaposed images for the TMVP. In the ATMVP, the time vector is added as an offset to any center position of the subPU, where the components of the time vector may need to be shifted to an integer number. Such shifted center positions are used to identify the smallest unit that can be assigned to the motion vector, for example, a 4×4 size covering the current center position.

[0194] a. Alternatively, the motion vectors corresponding to List 1 can be checked before those corresponding to List 0.

[0195] b. Alternatively, the motion vectors corresponding to List 1 can be checked before those corresponding to List 0.

[0196] c. Alternatively, sequentially examine all motion vectors in all spatially adjacent items that correspond to list X, followed by the motion vectors corresponding to list Y (where Y equals 1-X). Here, list "X" can be a list indicating the location of the juxtaposed images, or simply set to 0 or 1.

[0197] d. The order of spatially adjacent items can be the same as the order used in the HEVC merge mode.

[0198] 9. When in the first level of recognition, the time vector does not contain information from the recognition reference image, such as... Figure 9 The motion source image shown can be easily set as a fixed image, such as a juxtaposed image used for TMVP.

[0199] a. In this case, the vector can be identified solely from the motion vector pointing to such a fixed image.

[0200] b. In this case, the vector can be identified only from motion vectors that point to any image but are further scaled towards a fixed image.

[0201] 10. In the first level of recognition, the vector is composed of recognition reference images, such as... Figure 9 The motion source image shown can be used to apply one or more of the following additional checks to the candidate motion vectors.

[0202] a. If a motion vector is associated with an intra-frame decoded image or slice, then such a motion vector is considered unusable and cannot be converted into a vector.

[0203] b. If the motion vector is identified within a block in the associated image (by, for example, by adding the current center coordinates to the motion vector), then such a motion vector is considered unavailable and cannot be converted to a vector.

[0204] 11. When in the first-level recognition vector, the vector components can be set to (half the width of the current PU, half the height of the current PU) to identify the bottom right pixel position in the motion source image. Here (x, y) indicates the horizontal and vertical components of a motion vector.

[0205] a. Alternatively, the vector components can be set as (sum(current PU half-width, M), sum(current PU half-height, n)), where the sum function (a, b) returns the sum of a and b. In one instance, when motion information is stored in 4×4 cells, both M and N are set to 2. In another instance, when motion information is stored in 8×8 cells, both M and N are set to 4.

[0206] 12. When ATMVP is applicable, the sub-block / sub-PU size is signaled in the parameter set (e.g., a sequence parameter set of an image parameter set). The size ranges from the minimum PU size to the CTU size. The size may also be predefined or signaled. The size may be, for example, as small as 4×4. Alternatively, the sub-block / sub-PU size may be derived based on the size of the PU or CU. For example, the sub-block / sub-PU may be set to equal the maximum (4×4, (CU width) >> M). The value of M may be predefined in the bitstream or signaled.

[0207] 13. Because ATMVP can be considered a new merge candidate, the maximum number of merge candidates can be increased by 1. For example, compared to HEVC, which occupies 5 candidates in the merge candidate list after streamlining, the maximum number of merge candidates can be increased to 6.

[0208] a. Alternatively, the ATMVP can be streamlined or unified with the regular TMVP candidates, so that the maximum number of merged candidates remains unchanged.

[0209] b. Alternatively, when ATMVP identifies an option as available, spatially adjacent candidates are excluded from the merge candidate list, for example, the last spatially adjacent candidate in the extraction order is not included.

[0210] 14. When deriving a time vector by considering multiple spatially adjacent motion vectors, motion vector similarity can be calculated based on the adjacent motion vectors of the current PU and the adjacent motion vectors identified by setting a specific time vector equal to the motion vector. The one that produces the highest motion similarity can be selected as the final time vector.

[0211] a. In an alternative approach, for each motion vector from neighboring positions N, the motion vector identifies a block (of the same size as the current PU) in the motion source image, where its neighboring positions N contain a set of motion information. This set of motion vectors is then compared with the set of motion information in the neighboring positions N of the current block.

[0212] b. In another alternative, for each motion vector from adjacent positions N, the motion vector identifies a block in the motion source image where its adjacent positions contain multiple sets of motion information. These multiple sets of motion vectors are compared with multiple sets of motion information from adjacent positions of the current PU in the same relative position. Motion information similarity is calculated. For example, the current PU has the following sets of motion information from A1, B1, A0, and B0, labeled MIA1, MIB1, MIA0, and MIB0. For the time vector TV, it identifies a block corresponding to the PU in the motion source image. Such blocks have motion information from the same relative positions A1, B1, A0, and B0 and are labeled TMIA1, TMIB1, TMIA0, and TMIB0. The motion similarity determined by TV is calculated as MStv, where the similarity between two sets of motion information is constrained using MVSim.

[0213] c. In both of the above scenarios, motion similarity MVSim can be used, where the two input parameters are two sets of motion information, each set containing at most two motion vectors and two reference indices. Thus, each pair of motion vectors in list X is associated with a reference image, the current image, and the motion source image in a different list X for different images. For each of the two motion vectors MVXN and TMVXN (where X equals 0 or 1), the motion vector difference MVDXN can be calculated as MVXN - TMVXN. Then, the difference MVSimX is calculated, for example, as abs(〖MVDX〗_N[0]) + abs(〖MVDX〗_N[1]), or (〖MVDX〗_N[0] * 〖MVDX〗_N[0] + 〖MVDX〗_N[1] * 〖MVDX〗_N[1]). If the two sets of motion information contain available motion vectors, then MVSim is set to equal MVSim0 + MVSim1.

[0214] i. In order to uniformly calculate the motion difference, both motion vectors need to be scaled proportionally toward the same fixed image, which may be, for example, the first reference image RefPicListX[0] of the current image list X.

[0215] ii. If the availability of motion vectors in list X from the first set differs from the availability of motion vectors in list X from the second set—that is, one reference index is -1 while the other is not -1—then the two sets of motion information are considered dissimilar in direction X. If the two sets are dissimilar in their settings, the final MVSim function may return a large value T, which may, for example, be considered infinity.

[0216] iii. Alternatively, for a pair of motion information sets, if one is predicted from list X (X equals 0 or 1) instead of list Y (Y equals 1-X) and the other has the same state, a weighted average between 1 and 2 can be used (e.g., MVSim equals MVSimX*1.5). When one set is predicted only from list X and the other only from list Y, MVSim is set to a large value T.

[0217] iv. Alternatively, for any set of motion information, two motion vectors will be generated as long as one motion vector is available. If only one motion vector is available (corresponding to list X), the motion vector is scaled to form a motion vector corresponding to another list Y.

[0218] d. Alternatively, the motion vector can be measured based on the difference between the neighboring pixels of the current PU and the neighboring pixels of a block identified by the motion vector (of the same size as the current PU). The motion vector that produces the smallest difference can be selected as the final time vector.

[0219] 15. When deriving the time vector of the current block, motion vectors and / or time vectors from neighboring blocks decoded by ATMVP may have a higher priority than motion vectors from other neighboring blocks.

[0220] a. In one instance, only the time vectors of adjacent blocks are checked first, and the first available time vector can be set as the time vector of the current block. Only if such a time vector does not exist is the normal motion vector further checked. In this case, the time vector of the ATMVP decoded block needs to be stored.

[0221] b. In another instance, only motion vectors from adjacent blocks decoded by ATMVP are examined first, and the first available one can be set as the time vector of the current block. Ordinary motion vectors are then examined only if such a time vector does not exist.

[0222] c. In another instance, only motion vectors from adjacent blocks decoded by ATMVP are checked first, and the first available one can be set as the time vector for the current block. If such motion vectors are not available, the checking of time vectors continues in a similar manner to item number 15a.

[0223] d. In another instance, the time vectors from adjacent blocks are checked first, and the first available time vector can be set as the time vector for the current block. If such a motion vector is not available, the checking of time vectors continues in a manner similar to item number 15b.

[0224] e. In another instance, the time and motion vectors of adjacent blocks decoded by ATMVP are first examined, and the first available one can be set as the time vector of the current block. Only if such time and motion vectors do not exist is the ordinary motion vector further examined.

[0225] 16. When considering multiple spatially adjacent motion vectors to derive a temporal vector, the motion vectors can be selected such that they minimize the distortion calculated from the pixel domain. For example, template matching can be used to derive a temporal vector such that the one that produces the minimum matching cost is selected as the final temporal vector.

[0226] 17. The derivation of the set of motion information from the corresponding block (in the motion source image) can be accomplished in such a way that when the motion vector is available in the corresponding block for any list X (indicating that the motion vector will be MVX), the motion vector is considered available for list X for the current sub-PU of the ATMVP candidate (by scaling MVX). If the motion vector is not available in the corresponding block for any list X, then the motion vector is considered unavailable for list X.

[0227] a. Alternatively, when a motion vector in the corresponding block is not available for list X but is available for list 1-X (indicated by Y and the motion vector is indicated as MVY), the motion vector is still considered available for list X (by scaling MVY toward the target reference image in list X).

[0228] b. Alternatively or additionally, when the motion vectors in the corresponding blocks of list X and list Y (equal to 1-X) are both available, it is not necessary to use the motion vectors from list X and list Y to directly scale and generate the two motion vectors of the current subPU by scaling.

[0229] i. In one instance, when compiling ATMVP candidates, a low-latency check, similar to that performed in TMVP, can be applied to each sub-PU. If, for each picture in the reference picture list of the current slice (indicated by refPic), the Picture Order Count (POC) value of refPic is less than the POC of the current slice, the current slice is considered to have a low-latency mode. In this low-latency mode, motion vectors from lists X and Y are scaled to produce motion vectors for the current sub-PUs in lists X and Y respectively. When not in a low-latency mode, only one motion vector MVZ from MVX or MVY is selected and scaled to produce two motion vectors for the current sub-PU. Similar to TMVP, in this case, Z is set to equal collocated_from_l0_flag, meaning it depends on whether the juxtaposed picture in the TMVP is in list X or list Y. Alternatively, Z is set as follows: if the motion source picture is identified as being from list X, then Z is set to X. Alternatively, Z is set to 0 when the motion source image belongs to two reference image lists, and RefPicList0[idx0] is the motion source image that first exists in list 0 and RefPicList(1)[idx1] is the motion source image that first exists in list 1, and is set to 1 if idx0 is less than or equal to idx1.

[0230] 18. The motion source image can be signaled, for example, generated by the video encoder 20 in the encoded bitstream. Specifically, for a B-slice, a signal is used to indicate whether the motion source image comes from list 0 or list 1. Alternatively, a signal can be used to indicate the reference index of list 0 or list 1 for the current image to identify the motion source image.

[0231] When identifying time vectors, if a vector points to an intra-decoded block in the associated motion source image, the vector is considered unusable (therefore other vectors can be considered).

[0232] Figure 11 This is a conceptual diagram illustrating sub-PU motion prediction from a reference image. In this example, the current image 380 contains the current PU 384 (e.g., PU). In this example, motion vector 392 identifies PU 386 of the reference image 382 relative to PU 384. PU 386 is segmented into sub-PUs 388A-388D, each sub-PU having a corresponding motion vector 390A-390D. Therefore, although the current PU 384 is not actually divided into separate sub-PUs, in this example, motion information from sub-PUs 388A-388D can be used to predict the current PU 384. Specifically, the video decoder can decode the sub-PUs of the current PU 384 using the corresponding motion vectors 390A-390D. However, the video decoder does not need to decode the syntax elements indicating that the current PU 384 is split into sub-PUs. In this way, multiple motion vectors 390A-390D inherited from the corresponding sub-PUs 388A-388D can be used to efficiently predict the current PU 384 without the signaling overhead of syntax elements used to split the current PU 384 into multiple sub-PUs.

[0233] Figure 12 This is a concept diagram showing relevant images from ATMVP (similar to TMVP). Specifically, Figure 12 The diagram illustrates a current image 404, a motion source image 406, and reference images 400 and 402. More specifically, the current image 404 contains the current block 408. A temporal motion vector 412 identifies a corresponding block 410 of the motion source image 406 relative to the current block 408. The corresponding block 410, in turn, contains a motion vector 414 that references the reference image 402 and acts as a high-level temporal motion vector predictor for at least a portion of the current block 408 (e.g., a sub-PU of the current block 408). That is, motion vector 414 can be added as a candidate motion vector predictor for the current block 408. If selected, the corresponding motion vector (i.e., motion vector 416) of the reference image 400 can be used to predict at least a portion of the current block 408.

[0234] Sub-PU related techniques for HEVC are also described in U.S. Patent Application No. 15 / 176,790, filed July 9, 2016, the entire contents of which are incorporated herein by reference. To enhance performance using sub-PU motion prediction, spatial-temporal motion information from neighboring sub-PUs (ATMVP_EXT) is utilized. In this example, the motion vector of each sub-PU is derived from information about neighboring blocks in the 3D domain. This means that neighboring blocks can be spatial neighbors in the current image or temporal neighbors in previously decoded images. Figure 13 A flowchart illustrating the process of deriving the Space-Time Motion Vector Predictor (STMVP) is shown. In addition to those described below, the methods described above for ATMVP (e.g., project numbers #1, #2, #3, #4, #6, #7, #12, #13) can be directly extended to STMVP.

[0235] like Figure 13 As shown, video encoder 20 and / or video decoder 30 can be configured to obtain available motion fields (430) from spatially or temporally neighboring blocks of the current sub-PU. In this context, a motion field is a set of optimal motion vectors selected as spatially / temporally neighboring blocks. For example, a block to the left or above the current block has already been decoded, and optimal motion vectors are available before decoding the current sub-PU. The available motion information from neighboring blocks is identical in video encoder 20 and video decoder 30. The motion information contains one or two three-dimensional vectors (MVx, Mvy, temporal direction): one vector for unidirectional prediction and two vectors for bidirectional prediction. Video encoder 20 and / or video decoder 30 can then derive motion information from the obtained neighboring motion fields (432). Video encoder 20 and / or video decoder 30 then determine whether all sub-PUs have been processed (434). If not, video encoder 20 and / or video decoder 30 move to the next sub-PU. If so, the video encoder 20 and / or video decoder 30 can determine the availability of the spatial-temporal sub-PU motion predictor (436). If available, the video encoder 20 and / or video decoder 30 insert the spatial-temporal sub-PU motion predictor into the merge list.

[0236] In the following description, the term "block" refers to a block unit used to store prediction-related information such as inter-frame or intra-frame prediction, intra-frame prediction mode, and motion information. This prediction information is stored and can be used to decode future blocks, for example, to predict prediction mode information for future blocks. In AVC and HEVC, the size of such a block is 4×4. Note that in the following description, we use "PU" to indicate inter-frame decoded block units and sub-PUs to indicate units from which motion information is derived from adjacent blocks. Any combination of the following techniques can be applied.

[0237] In one instance, video encoder 20 and / or video decoder 30 can be configured to obtain motion information from neighboring blocks. Sub-PUs and neighboring blocks can have different sizes. Consider a PU with multiple sub-PUs. The size of a sub-PU is typically equal to or greater than the size of a neighboring block. In one instance, such as... Figure 14 As shown in Figure 14, hash blocks represent neighboring blocks (a, b, ... i) outside the current PU, and the remaining non-hash blocks (A, B, ... P) represent sub-PUs within the current PU. The size of a sub-PU is the same as its neighboring blocks, as shown in Figure 14. In one instance, the size of the sub-PU is 4×4, but sub-PUs of different sizes can be used. Figure 15 This illustrates another instance where the sub-PU is larger than the adjacent block. In other instances, the sub-PU can be non-square, such as rectangular or triangular. In some instances, the size of the sub-PU can be indicated by a signal in the slice header.

[0238] In other instances, the process described in item #12 of the above discussion regarding ATMPV can be extended to STMVP. For example, when ATMVP is applicable, the sub-block / sub-PU size is signaled in a parameter set (e.g., a sequence parameter set of a picture parameter set). The size ranges from the minimum PU size to the CTU size. The size can also be predefined or signaled. The size can be, for example, as small as 4×4. Alternatively, the sub-block / sub-PU size can be derived based on the size of the PU or CU. For example, the sub-block / sub-PU can be set to equal the maximum (4×4, (width of CU) >> M). The value of M can be predefined in the bitstream or signaled.

[0239] Different check orders for sub-PUs can be used in STMVP. Figure 14 In the examples described below, it is assumed that the raster scan order (A, B, C, D, E...) is applied to the sub-PU for its motion prediction derivation. However, other scan orders can also be applied, and it should be noted that the techniques described herein are not limited to raster scan orders.

[0240] In STMVP, neighboring blocks can be categorized into two distinct types: spatial and temporal. Spatially neighboring blocks are already decoded blocks or scanned subPUs that are adjacent to the current subPU in the current picture or slice. Temporally neighboring blocks are blocks that are adjacent to juxtaposed blocks of the current subPU in previously decoded pictures. In one instance, temporally neighboring blocks are obtained using all reference pictures associated with the current PU. In another instance, STMVP is derived using a subset of the reference pictures. For example, only the first entry in each reference picture list is used.

[0241] Follow this definition, refer to Figure 14 For a sub-PU(A), all adjacent blocks (a, b, ... i) and their juxtaposed blocks in the previously decoded image are considered as available spatial and temporal adjacent blocks. Blocks B, C, D, E...P are not spatially available according to the raster scan order. However, all sub-PUs (from A to P) are temporally available adjacent blocks of PU(A) because their motion information can be found in their juxtaposed blocks in the previously decoded image. Taking sub-PU(G) as another example, its available spatial adjacent blocks include those from a, b... to i, and also those from A to F. In some instances, certain restrictions may be applied to spatial adjacent blocks; for example, spatial adjacent blocks (i.e., from a, b... to i) will be in the same LCU / slice / tile.

[0242] Video encoder 20 and / or video decoder 30 select a subset of all available neighboring blocks to derive motion information or motion fields for each sub-PU. The subset used for deriving each PU can be predefined. In other instances, the subset used for deriving can be signaled as a high-level syntax in the slice header, picture parameter set (PPS), and / or sequence parameter set (SPS). To optimize decoding performance, the subset for each sub-PU can be different. In practice, for simplicity, a fixed pattern of subset positions is preferred. For example, each sub-PU can use its directly above spatial neighbor, its directly left spatial neighbor, and its directly right temporal neighbor as a subset. Figure 14 As shown, when considering a sub-PU(J), the upper block (F) and the left block (I) are spatially available neighboring blocks, and the lower right block (O) is a temporally available neighboring block. Through this subset, sub-PUs within the current PU are processed sequentially due to handling dependencies.

[0243] To allow parallel processing of each subPU within the current PU, different subsets of adjacent blocks can be defined and used. In one instance, the subset contains only spatially adjacent blocks that do not belong to the current PU, such as blocks a, b, ... i. In this case, parallel processing is possible. In another instance, for a given subPU, if its spatially adjacent blocks are within the current PU, then the juxtaposed blocks of those spatially adjacent blocks can be placed in a subset and used to derive the motion information of the current subPU. For example, when considering subPU(j), the temporally juxtaposed blocks of the upper block (F), left block (I), and lower right block (O) are selected as subsets to derive the motion of subPU(J). In this case, the subset of subPU(J) contains three temporally adjacent blocks. In another instance, partial parallel processing can be enabled, where a PU is divided into several regions and each region (covering several subPUs) can be processed independently.

[0244] Sometimes, adjacent blocks are intra-decoded, where it is desirable to have rules for determining which blocks have replacement motion information to achieve better motion prediction and coding efficiency. For example, considering sub-PU(A), there might be a case where blocks b, c, and f are intra-decoded, and a, d, e, g, h, and i are inter-decoded.

[0245] For spatially adjacent items, a predefined order can be used to fill in the motion information of the intra-frame decoded block and the first found inter-frame decoded block. For example, the search order for the above adjacent items can be set to start from the directly above adjacent item and move right to the rightmost adjacent item, i.e., the order of b, c, d, and e. The search order for left adjacent items can be set to start from the directly left adjacent item and move down to the bottommost adjacent item. In this example, the order is f, g, h, and then i. If no inter-frame decoded block is found through the search process, then the above or left spatial adjacent items are considered unavailable.

[0246] For time-adjacent items, the same rules as specified in the TMVP export can be used. However, it should be noted that other rules can also be used, such as those based on direction of motion, temporal distance (searching across different reference images), and spatial location.

[0247] The video encoder 20 and / or video decoder 30 can then derive motion information for a given sub-PU. As part of this process, the video encoder 20 and / or video decoder 30 can perform target reference image determination and motion vector scaling. For adjacent blocks, motion vector scaling can be applied to the motion vectors associated with the adjacent blocks based on each list of reference images, so that the motion vectors of all adjacent blocks are mapped to the same reference image in each list. An example may involve two steps: first, determining the source motion vectors we use for scaling; second, determining the target reference image to which the source motion vectors are projected.

[0248] Several methods can be used for the first step.

[0249] (a) For each reference list, motion vector scaling is independent of motion vectors in the other reference list. For motion information of a given block, if there are no motion vectors in the reference list (e.g., unidirectional prediction mode instead of bidirectional prediction mode), motion vector scaling is not performed on the list.

[0250] (b) Motion vector scaling is independent of motion vectors in another reference list. For motion information of a given block, if no motion vector is available in any reference list, the motion vector can be scaled based on motion vectors in another reference list.

[0251] (c) The two motion vectors are scaled from a predefined list of references (such as TMVP mentioned above).

[0252] As an example, method (a) is used to scale the motion vectors of spatially adjacent blocks, and method (c) is used to scale the motion vectors of temporally adjacent blocks.

[0253] For the second step, the video encoder 20 and / or video decoder 30 may select a target reference image based on motion information (e.g., reference images) of adjacent blocks in the available space, according to specific rules. One example of such a rule is the majority rule, which selects a reference image shared by a majority of blocks. In this case, no signaling is required from the encoder to the decoder for the target reference image, as the same rules can be used to infer the same information on the decoder side. Alternatively, such a reference image may be explicitly specified in the slice header or signaled to the decoder in some other way. The target reference image is determined as the first reference image (refidx = 0) in each reference list.

[0254] Video encoder 20 and / or video decoder 30 can be configured to derive motion information for a given sub-PU. After retrieving motion information from neighboring blocks and performing motion scaling processing (if necessary) as described in the previous section, the motion information for the current sub-PU is exported. Assume there are N available neighboring blocks, each with motion information for a given sub-PU.

[0255] First, determine the prediction direction (InterDir). An example method is as follows:

[0256] a. Initialize InterDir to zero, then loop through the motion information of N available adjacent blocks;

[0257] b. If there is at least one motion vector in list 0, then InterDir = (InterDir bitwise OR 1).

[0258] c. If there is at least one motion vector in List 1, then InterDir = (InterDir bitwiseOR2).

[0259] Here, "bitwiseOR" indicates a bitwise OR operation. The value of InterDir is defined as: 0 (no inter-frame prediction), 1 (inter-frame prediction based on list 0), 2 (inter-frame prediction based on list 1), and 3 (inter-frame prediction based on both list 0 and list 1).

[0260] In another instance, similar to the determination of the target reference image used for the aforementioned motion vector scaling, most of the rules can be used to determine the value of InterDir for a given subPU based on the motion information of all available neighboring blocks.

[0261] After determining the InterDir, motion vectors can be exported. For each reference list based on the exported InterDir, there can be M motion vectors (M<=N) that are available when scaled to the target reference image using motion vectors, as described above. The motion vectors of the reference list can be exported as follows:

[0262]

[0263] Where w i and w j It is the weighting factor for the corresponding horizontal and vertical motion components, and O i and O j It depends on the offset value of the weighting factor.

[0264] Weighting factors can be determined based on various factors. In one instance, the same rule can be applied to all sub-PUs within a PU. The rule can be defined as follows. For example, the weighting factor can be determined based on the positional distance between the current sub-PU and its corresponding neighboring block. In another instance, the weighting factor can also be determined based on the point of reference (POC) distance between the target reference image before scaling and the reference image associated with the motion vectors of the corresponding neighboring block. In yet another instance, the weighting factor can be determined based on motion vector difference or consistency. For simplicity, all weighting factors can also be set to 1.

[0265] In another instance, different rules can be applied to sub-PUs within a PU. For example, the rules described above can be applied. Furthermore, for the sub-PU located in the first row / first column, the weighting factor of motion vectors derived from temporally adjacent blocks is set to 0, while for the remaining blocks, the weighting factor of motion vectors derived from spatially adjacent blocks is set to 0.

[0266] It should be noted that in practice, the above equations can be implemented as is or simplified for ease of implementation. For example, to avoid division or floating-point operations, fixed-point operations can be used to roughly estimate the above equations. One example is that to avoid division by 3, multiplication by 43 / 128 can be chosen instead to replace the division operation with multiplication and bit offset. Those variations in the implementation should be considered in the same spirit as the technology of this disclosure. Alternatively, nonlinear operations can be applied to derive motion vectors, such as median filters.

[0267] The video encoder 20 and / or video decoder 30 can also be configured to perform availability checks during the STMVP candidate list construction process. It is proposed that even if the motion vector predictor for each sub-PU is available, the STMVP mode can be reset to be unavailable for a given PU. For example, once the motion vector predictor for each sub-PU is derived for a given PU, some availability checks are performed to determine whether the STMVP mode should be made available for that PU. This operation is used to eliminate cases where it is highly unlikely that the STMVP mode will ultimately be selected for a given PU. When the STMVP mode is unavailable, the mode signaling does not include the STMVP. In the case where the STMVP mode is implemented by inserting SMTVPs into the merge list, when the STMVP mode is determined to be unavailable, the merge list does not include this STMVP candidate. Therefore, signaling overhead can be reduced.

[0268] Consider dividing a PU into M sub-PUs. In one instance, if N1 (N1 <= M) of the M sub-PUs have the same motion vector predictor (i.e., the same motion vector and the same reference image index), then STMVP is available only if N1 is less than a threshold or the predictor differs from other motion vector predictors in the merge list (with a smaller merge index). In another instance, if N2 (N2 <= M) sub-PUs in STMVP mode share the same motion vector predictor as their corresponding sub-PUs in ATMVP mode, then STMVP is available only if N2 is less than another threshold. In one instance of this disclosure, the two thresholds N1 and N2 are set to equal M.

[0269] If STMVP is available, the video encoder 20 and / or video decoder 30 insert STMPV candidates into the merge list. The process described in item #1 of the ATMVP above can be extended, and STMVP candidates can be inserted before or after the ATMVP candidates. In one instance, the STMVP candidate is inserted immediately after the ATMVP candidate in the merge list.

[0270] U.S. Patent Application No. 15 / 431,321, filed February 13, 2017, describes a Proof-of-Concept (POC)-based MV simplification technique, the entire contents of which are incorporated herein by reference. To maximize the efficiency of MV prediction, the uniqueness of available MVs can be checked. Otherwise, redundant MVs will lead to inefficient resource utilization, such as wasting budget or resources of the target device. Therefore, eliminating redundancy in MV candidates (so-called simplification) may be an important step in making MVs as unique and diverse as possible to provide more meaningful MV candidates in MV prediction.

[0271] This disclosure describes a proof-of-concept (POC)-based simplification method that has three main advantages: (1) higher accuracy, (2) simplicity, and (3) universality. The proposed technique achieves higher simplification accuracy because it can detect redundant MVs that existing simplification methods have not captured. Furthermore, it is simple because it requires no additional complexity. Finally, the POC-based simplification is universal because it can be applied to a variety of situations, such as spatial MVs for ATMVP / merge candidates, sub-PU (ATMVP and STMVP) MVs, TMVPs, combined MVs, and even zero MVs.

[0272]

[0273] Table 1. Possible MV pairs based on POC (Proof of Concept)

[0274] Table 1 summarizes the types of MV pairs that can be simplified using the POC-based approach. In category C1, MVs from the regular PU (not sub-PUs) are compared. Comparisons can be made between two unidirectional MVs (e.g., motion vectors for unidirectional prediction) or two bidirectional MVs (e.g., motion vectors for bidirectional prediction). From categories C2 to C4, sub-PU MVs are included in the comparison. In C2, POC-based simplification is used to determine if all sub-PU MVs within the PU are identical. This can be handled by applying the same technique from C1 to MVs from sub-PUs. The case where all sub-PU MVs are equal falls into C3, where MVs represent all MVs from sub-PUs, and therefore the same comparison as in C1 applies. However, if all sub-PU MVs of the candidates are not equal, and another candidate C4 with sub-PU MVs exists, then POC-based simplification is applied to every pair of MVs from sub-PUs located in the same position within the PU. C5 and C6 are both related to the construction of bidirectional MVs by combining two unidirectional MVs: one from L0 and the other from L1. If two one-way MVs are identical (e.g., identical MVs from the same reference image), then bidirectional MV construction is unnecessary because the resulting bidirectional MV will be identical to the one-way MV. Therefore, POC-based simplification can help save resources by more accurately detecting identical MVs, especially when L0 and L1 have the same reference image.

[0275] For a given list of candidates, two factors that determine the efficiency of merging candidate lists are (1) the order of the candidate lists (e.g., how to assign the order of candidates in the list) and (2) streamlining (e.g., eliminating redundancy among those candidates). Generally, it is preferable to place the most likely candidate first in the candidate list, as the index of the first candidate can be notified with fewer signals. Additionally, having more varied candidates in the list (e.g., less redundancy) increases the likelihood of more accurate motion vectors between candidates in the list.

[0276] This disclosure includes techniques for determining a set of mergeable candidates from a larger group of possible candidates. Additionally, this disclosure describes adaptive selection, sorting, and simplification techniques for merging candidates to achieve greater efficiency in the list of motion vector candidates. For adaptive sorting, the proposed techniques utilize additional MV information to assign higher priority to candidates that are more likely to have more accurate motion information (e.g., forming smaller indices in the list). For adaptive simplification, motion vector difference (MVD) can be used to determine whether two MVs are the same (or very close) by comparing the MVD with an adaptive threshold.

[0277] Due to the flexibility of the proposed techniques, the techniques disclosed herein can be applied to most existing state-of-the-art codecs, such as H.264, HEVC, or H.266, and can be easily extended to different partitioning frameworks, such as the QTBT structure described above. Furthermore, different combinations of the proposed techniques can be combined to form desired solutions for specific applications. That is, the following techniques can be applied independently or in any non-mutually exclusive combination.

[0278] In addition to the merge index in HEVC or H.266 reference software, the techniques described below can be performed without additional signaling. That is, in some instances, video encoder 20 and video decoder 30 can be configured to perform the following techniques based on a predetermined set of rules without using explicit signaling. Video encoder 20 can be configured to signal the merge index of the current block, and video decoder 30 can be configured to perform the same procedure as video encoder 20 to derive merge candidates. Therefore, using the received merge index, video decoder 30 can be configured to determine the same MV information without any mismatches.

[0279] Figure 16 This shows the set of neighboring blocks of the current block 450. For example... Figure 16As shown, the shaded neighboring blocks a, e, f, j, and k are the same neighboring blocks used as space merging candidates in HEVC. This disclosure proposes using motion information from additional neighboring blocks decoded before the current block 450. Such additional neighboring blocks may include neighboring blocks b, c, d, g, h, and i. By deriving the final motion vector candidate list from more neighboring blocks, the possibility of finding more accurate motion vectors within the motion vector candidate list is increased.

[0280] exist Figure 16 In this example, the current block 450 is 16×16, and each of the adjacent blocks is a 4×4 block. However, it should be noted that adjacent blocks can have different sizes based on the size of the current block. Typically, the video encoder 20 and the video decoder 30 can be configured to construct a motion vector candidate list for the current block 450, wherein the motion vector candidate list contains motion vector information from several adjacent blocks relative to the current block, wherein the number of adjacent blocks is greater than 5.

[0281] In another embodiment of this disclosure, video encoder 20 and video decoder 30 may be configured to construct a motion vector candidate list using a motion vector histogram-based sorting of motion information from neighboring blocks. Based on the assumption that motion is spatially homogeneous (e.g., likely identical or nearly identical at a given spatial location in an image), the primary motion information from neighboring blocks is more likely to be the motion information selected for the current block. Therefore, video encoder 20 and video decoder 30 may be configured to derive a motion vector histogram from the motion vector distribution of neighboring blocks. As described above, the motion vector information comprises a three-dimensional vector (MVx, MVy, direction), where MVx is the horizontal component of the motion vector, MVy is the vertical component of the motion vector, and the direction refers to either a past (reference list L0) or future (reference list L1) predicted direction. Reference Figure 13 The video encoder 20 and the video decoder 30 can determine that a particular motion vector has the same frequency for each of the adjacent blocks ak.

[0282] Video encoder 20 and video decoder 30 can use histogram information in several different ways. In one instance, video encoder 20 and video decoder 30 can use histogram information to determine which motion vectors, and therefore which adjacent blocks, are available as spatial merge candidates in the candidate list. In another instance, video encoder 20 and video decoder 30 can use histograms to determine the order in which certain spatial merge candidates are added to the list.

[0283] Typically, the video encoder 20 and video decoder 30 can be configured to derive motion vector histograms from adjacent pixels or blocks. As described above, Figure 16An example of 4×4 adjacent blocks (ak) to be used in the motion vector histogram of the current 16×16 block 450 is shown. The highlighted blocks (a, e, f, j, and k) are the locations of spatial merging candidates in HEVC.

[0284] In some instances, the video encoder 20 and the video decoder 30 derive motion vector histograms from the motion vector distribution of neighboring blocks of a specific size. Figure 16 This illustrates which neighboring blocks (ak) will be used to construct the MV histogram. The cell size of the neighboring blocks can be a specific size, such as 4×4, or some predefined minimum size for motion compensation. If a block does not have associated motion information (e.g., an intra-predicted block), it is ignored or can be filled with motion information from other neighboring blocks. For example, if the neighboring block h is an intra-predicted block, then the video encoder 20 and video decoder 30 can simply not use the neighboring block. In other instances, if the neighboring block h is an intra-predicted block, then the video encoder 20 and video decoder 30 can use motion information from the blocks to the left of the neighboring block h.

[0285] like Figure 16 As shown in the example, to construct the MV histogram of the current 16×16 block, the video encoder 20 and the video decoder 30 can examine 11 neighboring blocks (from block a to block k) with a size of 4×4. It should be noted that neighboring blocks (including the top row / left column) can be predefined as shown in Figure 16, or depending on the size and / or shape of the current block.

[0286] In another instance, the histogram can be reconstructed using weights proportional to the size of neighboring blocks. For example, the number of pixels (or units, i.e., 4x4 blocks) belonging to neighboring blocks can be used as the weights of the histogram. That is, motion vectors from larger blocks (specifically, blocks containing more pixels) have higher weights proportional to the number of pixels within those blocks. In another instance, the histogram weights can be determined by a combination of the two factors mentioned above: the number of pixels (or units) within neighboring blocks and the number of pixels (or units) adjacent to the current block.

[0287] Note that both video encoder 20 and video decoder 30 should follow the same rules to construct their histograms to avoid mismatches. Given the same histograms in video encoder 20 and video decoder 30, all of the following adaptive schemes for merging candidates will produce an equivalent merge list.

[0288] After determining the motion vector histogram, the video encoder 20 and video decoder 30 can then use the histogram to determine the order of spatial merging candidates in the motion vector candidate list. In some instances, the constructed histogram can be used to determine the order of spatial merging candidates for a given (fixed) N.f The order of space merging candidates, where N f This is the number of fixed space candidates. As an example, let N be a fixed number. f Spatial candidates can be adjacent blocks a, e, f, j, and k, as used in HEVC. Of course, any subset of the total number of candidates can be used. For example, see reference... Figure 16 Any fixed subset of adjacent blocks ak can be used as a spatial merging candidate.

[0289] Depending on the frequency of each motion vector from the available adjacent blocks, the most frequent motion vector from the histogram is first inserted into the merge list, and the least frequent motion vector from the histogram is the last spatial merge candidate to be inserted into the list. For example, Figure 16 Five spatial merge candidates (a, e, f, j, k) used in HEVC are shown. Both the video encoder 20 and the video decoder 30 can be configured to adaptively determine the order from the MV histogram, rather than following a fixed order of those candidates (the order of jefka in HEVC). In another instance, instead of checking each adjacent block (e.g., a 4×4 spatial merge candidate), the spatial merge candidates are reordered based on the size of the prediction block (e.g., the PU in HEVC), which contains the blocks used to derive the spatial merge candidates.

[0290] consider Figure 17 Examples. For example... Figure 17 As shown, three adjacent blocks (e, g, and h) have motion vector 0 (MV0), four distinct adjacent blocks (a, b, c, and d) have motion vector 1 (MV1), one adjacent block (f) has motion vector 2 (MV2), two distinct adjacent blocks (i and j) have motion vector 3 (MV3), and one adjacent block (k) has motion vector 4 (MV4). Therefore, the video encoder 20 and video decoder 30 will be configured to sort the list of motion vector candidates using fixed candidates a, e, f, j, and k, as follows: MV1 - candidate a (index 0), MV0 - candidate e (index 0), MV3 - candidate j (index 0), MV2 - candidate f (index 0), MV4 - candidate k (index 0). The example in Figure 17 assumes all adjacent blocks have the same weight. In some instances, if two or more fixed candidates have the same associated motion vector with the same number of occurrences in the histogram (e.g., ...), ... Figure 17 MV2 and MV4 in the code can then be used in a predetermined order for those inspection candidates. Figure 17 In this example, candidate f is placed before candidate k in the list. Of course, any predetermined order can be used.

[0291] In another embodiment of this disclosure, after determining the motion vector histogram, the video encoder 20 and video decoder 30 can then use the histogram to determine which neighboring blocks are used as spatial merge candidates in the motion vector candidate list, regardless of the order. That is, the video encoder 20 and video decoder 30 can determine which of all possible neighboring blocks is used as a spatial merge candidate in the list, rather than using a fixed N. f A number of space merging candidates. In this example, refer to... Figure 16 It is advisable to include all adjacent blocks ak in the motion vector candidate list as spatial merging candidates.

[0292] The video encoder 20 and video decoder 30 can use a motion vector histogram to determine which of the total number of available adjacent blocks will constitute a predetermined number (N) in the motion vector candidate list. h The candidates are as follows. As described above, instead of simply changing the order of a given list of candidates, the order of two locations (e.g., actual adjacent blocks) and the determined spatial merging candidates can be adaptively derived from the distribution of adjacent motion vectors in the determined histogram. For example, if N h =2, then the two most frequent motion vectors from adjacent blocks are placed in the order of frequency in the merge list. If more than one adjacent block is associated with the most frequent motion vector in the histogram, the video encoder 20 and video decoder 30 can use predetermined rules to determine which adjacent blocks to place in the candidate list. However, it should be noted that regardless of the rule used, the adjacent blocks associated with the most frequent motion vector in the histogram will be added to the motion vector candidate list. Based on this example, refer to Figure 17 Add the neighboring blocks associated with MV0 and the neighboring blocks associated with MV1 to the motion vector candidate list.

[0293] In some instances, video encoder 20 and video decoder 30 can be configured to use the two techniques described above for merge list construction using a determined histogram. That is, video encoder 20 and video decoder 30 can use histograms to sort a fixed set of candidates and add multiple non-fixed candidates (N) based on the histogram. h As mentioned above, N f The position of each space merge candidate is fixed in all blocks, for example, Figure 16 Blocks a, e, f, j, and k in the dataset. Additionally, N from adjacent blocks... h The most frequently occurring motion information is added to the list as spatial merging candidates, and then the selection is determined based on the frequency of the associated motion vectors appearing in the determined histogram (N). f +N hThe order of the 10 candidates.

[0294] In another instance, N f The position of each space merge candidate is fixed in all blocks, for example, Figure 16 The video encoder 20 and video decoder 30 use the determined histogram to determine the order of fixed candidates, which includes blocks a, e, f, j, and k. Additionally, N from adjacent blocks... h The most frequently occurring motion information is added to the list, but the other N h A candidate is inserted into a predetermined position (e.g., from...). Figure 16 (Before or after the motion vector of block e in the middle).

[0295] In another instance, this disclosure describes an adaptive ordering of PU merging candidates, such as the ATMVP and ATMVP_EXT candidates described above. In one instance of JEM2.0 software, ATMVP and ATMVP_EXT are always located between candidate k and candidate a (e.g., as shown in the example). Figure 16 (As shown). The video encoder 20 and video decoder 30 can be configured to adaptively place the ATMVP / ATMVP_EXT candidate based on conditions associated with other available merge candidates, ATMVP / ATMVP_EXT, or a combination thereof, instead of placing ATMVP / ATMVP_EXT in a fixed position in the merge list.

[0296] In some instances, the motion vector difference (MVD) between two spatial merging candidates can be used to determine the position of the ATMVP / ATMVP_EXT candidate. The video encoder 20 and video decoder 30 can be configured to calculate the MVD as a function of the motion vectors. In one instance, the sum of the absolute differences between two MVs is: MVD = abs(MV x [1]-MV x [0])+abs(MV y [1]-MV y [0]). In another instance, the function is defined as MVD = (MV x [1]-MV x [0])*(MV x [1]-MV x [0])+(MV y [1]-MV y [0])*(MV y [1]-MV y [0]). It should be noted that the function used to calculate MVD can vary based on the precision of the motion vector, such as integer, half, quarter, eighth, or sixteenth pixel precision.

[0297] For example, different from the latest JEM software between candidate k and a in Figure 16 which always places ATMVP / ATMVP_EXT, video encoder 20 and video decoder 30 can be configured to position the ATMVP / ATMVP_EXT candidate before candidate k, depending on the MVD (MVD jk ) between candidate j and candidate k. If the MVD jk is less than threshold TH1 or greater than another threshold TH2, that is, MVD jk < TH1 or MVD jk > TH2, then the ATMVP ATMVP_EXT is positioned before candidate k. For example, by calculating TH1 using the minimum MVD among all or (some) spatially adjacent motion vectors and calculating TH2 using the maximum MVD, an adaptive threshold can be used, where the calculation does not include the same MV pair and a pair (MVj and MVk). Since video encoder 20 and video decoder 30 both have the same adjacent motion vectors, the calculation will produce the same TH1 and TH2. Alternatively, both TH1 and TH2 can be determined experimentally. For example, TH1 = 2 and TH2 = 8 in one-sixteenth pixel MV accuracy. Otherwise, candidate k is before ATMVP / ATMVP_EXT in the list. Similarly, video encoder 20 and video decoder 30 can be configured to determine the order of the ATMVP / ATMVP_EXT candidate and candidate f by checking the MVD (MVD ef ) between candidate e and candidate f. If one or both of the candidates used for MVD calculation, that is, candidate k or candidate a in the above example, are not available in the merged list, the ATMVP / ATMVP_EXT candidate can be placed in the motion vector candidate list in the default order.

[0298] In another example, video encoder 20 and video decoder 30 can be configured to analyze the characteristics of ATMVP / ATMVP_EXT candidates (e.g., the variance of sub-block MVs or the spatial distribution of sub-block MVs) to determine which candidates will be placed in the motion vector candidate list. If the variance is in the range [TH1, TH2], a higher priority is assigned, i.e., a smaller index in the list. The range [TH1, TH2] can be determined by the average variance of the sub-block MVs of the previously decoded block that selected ATMVP or ATMVP_EXT as the best merging candidate: TH1 = C1 * Var1 and TH2 = C2 * Var2, where Var1 and Var2 are calculated and stored based on the previously decoded block. The coefficients C1 and C2 can be fixed as constants or depend on the size and / or shape of the current block. The range can depend on the size and / or shape of the current block. For larger blocks, both TH1 and TH2 increase, and the range widens. The range can depend on the motion vector precision.

[0299] In another example, video encoder 20 and video decoder 30 can be configured to analyze the state of spatial merging candidates and ATMVP / ATMVP_EXT candidates to determine the order among those candidates. For example, the average motion vector or the most frequent sub-block motion vector from the ATMVP or ATMVP_EXT candidate is considered the representative MV of the sub-block. The representative motion vector can be used to calculate the MVD for the spatial candidates, for example, Figure 16 In block f. If MVD is greater than TH1 but less than TH2, the video encoder 20 and video decoder 30 can be configured to place the ATMVP / ATMVP_EXT candidate before the spatial candidate.

[0300] In HEVC, two available bidirectional merging candidates (i.e., C1 and C2) are used to derive combined motion vector (combi-MV) candidates, which contain two motion vectors for two prediction directions: reference list L0 and reference list L1. It is assumed that both C1 and C2 have bidirectional MV: the MV of candidate C1... L0C1 and MV L1C1 MV of candidate C2 L0C2 and MV L1C2 The video encoder 20 and video decoder 30 can be configured to derive a new combi-mv(MV0, MV1) by obtaining L0 MV from C1 and L1 MV from C2: (MV0, MV1) = (MV0, MV1) L0C1 MV L1C2 Similarly, video encoder 20 and video decoder 30 can be configured to derive another combi-mv by taking the remaining MV: (MV0', MV1') = (MV0', MV1')L0C2 ,MV L1C1 ).

[0301] In some instances, the maximum number of combine-mv candidates is fixed. In HEVC, if the number of available combine candidates is less than the maximum number of combine candidates (e.g., 5 in HEVC), and there is more than one bidirectional combine candidate in the list, then a maximum of 12 combine-mv candidates can be considered combine candidates. In HEVC extensions, more combine candidates have been added, such as ATMVP and ATMVP_EXT candidates, so it is possible to increase the maximum number of combine-mv candidates from 12 to a larger number (e.g., 30).

[0302] In another embodiment of this disclosure, the video encoder 20 and video decoder 30 may be configured to consider more combi-mvs as needed (e.g., if the maximum number of merging candidates has not yet been reached). For example, if the similarity between available merging candidates is higher than a certain threshold, the combi-mv candidates will also be similar to the existing candidates, thus limiting the maximum number of combi-mvs. Similarity can be measured by sum of absolute differences (SAD), SATD, average luminance or chrominance values, pixel variance, and / or MV trajectory.

[0303] If more combi-MV options are considered, adaptive sorting can be employed to maximize the benefits of alternative combi-MV candidates. Given the order of combi-MV candidates, the following techniques rearrange the candidates for specific criteria. Candidates that do not meet the criteria are followed in the default order.

[0304] In some instances, video encoder 20 and video decoder 30 can be configured to reorder combi-mv candidates based on the similarity between the exported combi-mv and existing mv from available candidates. Assuming both C1 and C2 have bidirectional mv, the mv... C1 =(MV) L0C1 ,MV L1C1 And MV C2 =(MV) L0C2 ,MV L1C2 It can also export two combi-mv files as an MV. combi-1 =(MV L0C1 ,MV L1C2 And MV combi-2 =(MV) L0C2 ,MV L1C1 If MV L0C1 and MV L0C2 (and / or MV) L1C2 and MVL1C1 ) If the same picture is referenced, calculate the MV L0C1 and the MV L0C2 (and / or the MV L1C2 and the MV L1C1 ) Calculate the MVD between them. Then, the video encoder 20 and the video decoder 30 can be configured to add the derived combi-mv to the motion vector candidate list after pruning when one of the following conditions is met: (1) If the MVD is between two thresholds, TH1 < MVD < TH2, or (2) if the MV L0C1 and the MV L0C2 reference different pictures. Otherwise, leave the combi-mv. In one example for a quarter-pixel motion vector precision, when both the width and height of the current block are less than 8, TH1 = 2 and TH2 = 8. If the width and height of the current block are greater than 8 and less than 32, then TH1 = 8 and TH2 = 32. If the width and height are greater than 32, then TH1 = 16 and TH2 = 64. Once all the combi-mv candidates that meet those conditions are first added to the merge list, the video encoder 20 and the video decoder 30 can be configured to add the remaining combi-mv to the list after pruning. The thresholds TH1 and TH2 can be adaptively selected based on the size or shape of the current block, for example, max(width, height).

[0305] In another example, the video encoder 20 and the video decoder 30 can be configured to sort the combi-mv candidates based on the above MVD. For simplicity, set the MVD of the combi-mv candidates where the MV L0C1 and the MV L0C2 (or the MV L1C1 and the MV L1C2 ) reference different pictures to 0. If the combi-mv candidates have equal MVD values, they follow the default order. Once sorted, the video encoder 20 and the video decoder 30 can be configured to add the candidates to the motion vector candidate list after pruning.

[0306] In addition to the POC-based pruning, as described above, the video encoder 20 and the video decoder 30 can be configured to further prune the merge candidates using an adaptive criterion determined by additional information other than the motion vector itself (e.g., the size and / or shape of the current block, the type of the target merge candidate, and / or the position of the merge candidate in the case of a spatial candidate).

[0307] In some instances, the video encoder 20 and video decoder 30 can be configured to treat a pair of MVs with MVDs less than an adaptive threshold as identical, thus streamlining and further diversifying the list of motion vector candidates. The threshold can be adaptively selected based on the size and / or shape of the current block.

[0308] In some instances, such as the MVD-based simplification method with the aforementioned adaptive threshold, simplification can be applied to all types of merging candidates: spatial, temporal, sub-block, or combi-MV candidates. In yet another instance, different criteria can be considered for different types of candidates. As an example of spatial candidates, the distance between the locations of those derived candidates can be used as a metric to determine the adaptive threshold. For example, if two motion vectors are derived from adjacent blocks (e.g., blocks e and f in Figure 16), then by comparing the distance from a distant block (e.g., ...) Figure 16 The threshold values ​​of the motion vectors derived from blocks f and k are small, and the motion vectors are simplified by the threshold values.

[0309] In some instances, for bidirectional merging candidates, the similarity between two unidirectional prediction blocks (one from the L0 direction and the other from the L1 direction) can indicate the reliability of the merging candidate. Based on this observation, video encoder 20 and video decoder 30 can be configured to use measurements to distinguish bidirectional merging candidates by using the similarity of two unidirectional prediction blocks, and accordingly reorder the bidirectional merging candidates. For example, video encoder 20 and video decoder 30 can be configured to determine similarity using sum of absolute differences (SAD), SSE, SATD, average luminance or chrominance values, pixel variance, and / or MV trajectory. More sophisticated metrics can provide higher accuracy in measuring prediction performance. The determination of the metric can depend on the requirements of the target application.

[0310] If SAD is used, then for two given bidirectional merging candidates C1 and C2, two SADs are calculated for each bidirectional candidate between the L0 and L1 directions: SAD C1 and SAD C2 The video encoder 20 and video decoder 30 can be configured to include videos with a smaller final SAD (i.e., SAD) in the merge list. C1 or SAD C2 The candidate is placed before the other.

[0311] In some instances, all the techniques proposed above can be combined to construct a list of motion vector candidates. In yet another instance, a set or subset of the proposed techniques can be merged.

[0312] Figure 18 This is a flowchart illustrating an example encoding method of this disclosure. Figure 18Implementations may be performed by one or more hardware units of the video encoder 20, including a motion estimation unit 42 and a motion compensation unit 44.

[0313] In one embodiment of this disclosure, video encoder 20 may be configured to receive a current video data block (500). Video encoder 20 may derive a histogram (502) of motion vector information of neighboring blocks relative to the current block. In one embodiment of this disclosure, the number of neighboring blocks considered for a motion vector candidate list is based on the size of the current block, and the number of neighboring blocks is greater than 5. In this context, the term "consider" may include video encoder 20 analyzing neighboring blocks, determining whether neighboring blocks have associated motion information, and, if neighboring blocks have associated motion information, using said motion information to construct a motion vector candidate list. As described above, motion information may be directly added to the motion vector candidate list, or it may be used to construct a histogram that can be used to determine the order and / or position of neighboring blocks as spatial merging candidates in the motion vector candidate list. Video encoder 20 may be further configured to construct a motion vector candidate list of merging candidates for the current block based on motion information from a number of neighboring blocks relative to the current video data block. In some embodiments, the motion information considered is a derived histogram (504). Then, the video encoder 20 can determine the current motion vector from the motion vector candidate list (506) and encode the current video data block using the current motion vector (508).

[0314] In another embodiment of this disclosure, the video encoder 20 may be configured to sort spatially merged candidates of a predetermined fixed subset in a list of motion vector candidates based on an exported histogram.

[0315] In another embodiment of this disclosure, the video encoder 20 may be configured to determine a fixed number of spatial merging candidates from the total number of adjacent blocks based on the derived histogram to add to the motion vector candidate list.

[0316] In another embodiment of this disclosure, the video encoder 20 may be configured to determine a fixed number of spatial merging candidates from a total number of adjacent blocks based on a derived histogram to add to a motion vector candidate list, and to sort a predetermined fixed subset of spatial merging candidates and the determined fixed number of spatial merging candidates in the motion vector candidate list based on the derived histogram.

[0317] In another embodiment of this disclosure, the video encoder 20 may be configured to sort spatial merging candidates of a predetermined fixed subset in a motion vector candidate list based on a derived histogram, determine a fixed number of spatial merging candidates from a total number of adjacent blocks based on the derived histogram to add to the motion vector candidate list, and insert the determined fixed number of spatial merging candidates at predetermined positions in the motion vector candidate list.

[0318] In another embodiment of this disclosure, the video encoder 20 may be configured to add ATMVP candidates to a motion vector candidate list based on motion vector functions of one or more Advanced Temporal Motion Vector Prediction (ATMVP) candidates. In yet another embodiment of this disclosure, the video encoder 20 may be configured to determine the position in the motion vector candidate list for adding an ATMVP candidate based on motion vector functions of one or more ATMVP candidates.

[0319] In another embodiment of this disclosure, the video encoder 20 may be configured to determine a combined motion vector candidate by combining motion vector information from two bidirectional motion vector candidates, and add the combined motion vector candidate to a motion vector candidate list.

[0320] In another embodiment of this disclosure, the video encoder 20 may be configured to determine the position in the motion vector candidate list for adding a combined motion vector candidate based on a motion vector function of one or more combined motion vector candidates.

[0321] In another embodiment of this disclosure, the video encoder 20 may be configured to reduce the motion vector candidate list based on motion vector difference information of the motion vector candidates in the motion vector candidate list.

[0322] In another embodiment of this disclosure, the video encoder 20 may be configured to sort bidirectional candidates in a motion vector candidate list based on the motion vector difference information of the bidirectional candidates.

[0323] Figure 19 This is a flowchart illustrating an example decoding method of this disclosure. Figure 19 The technology can be implemented by one or more hardware units of the video decoder 30, including the motion compensation unit 72.

[0324] In one embodiment of this disclosure, video decoder 30 may be configured to receive a current video data block encoded using a merging mode (550). Video decoder 30 may derive a histogram (552) of motion vector information of neighboring blocks relative to the current block. In one embodiment of this disclosure, the number of neighboring blocks considered for a motion vector candidate list is based on the size of the current block, and the number of neighboring blocks is greater than 5. In this context, the term "consider" may include video decoder 30 analyzing neighboring blocks, determining whether neighboring blocks have associated motion information, and, if neighboring blocks have associated motion information, using said motion information to construct a motion vector candidate list. As described above, motion information may be directly added to the motion vector candidate list, or it may be used to construct a histogram that can be used to determine the order and / or position of neighboring blocks as spatial merging candidates in the motion vector candidate list. Video decoder 30 may be further configured to construct a motion vector candidate list of merging candidates for the current block based on motion information from a number of neighboring blocks relative to the current video data block. In some embodiments, the motion information considered is a derived histogram (554). Then, the video decoder 30 can determine the current motion vector from the motion vector candidate list (556) and use the current motion vector to decode the current video data block (558).

[0325] In another embodiment of this disclosure, the video decoder 30 may be configured to sort spatially merged candidates of a predetermined fixed subset in a list of motion vector candidates based on an exported histogram.

[0326] In another embodiment of this disclosure, the video decoder 30 may be configured to determine a fixed number of spatial merging candidates from the total number of adjacent blocks based on the derived histogram to add to the motion vector candidate list.

[0327] In another embodiment of this disclosure, the video decoder 30 may be configured to determine a fixed number of spatial merging candidates from a total number of adjacent blocks based on a derived histogram to add to a motion vector candidate list, and to sort a predetermined fixed subset of spatial merging candidates and the determined fixed number of spatial merging candidates in the motion vector candidate list based on the derived histogram.

[0328] In another embodiment of this disclosure, the video decoder 30 may be configured to sort spatial merging candidates of a predetermined fixed subset in a motion vector candidate list based on a derived histogram, determine a fixed number of spatial merging candidates from a total number of adjacent blocks based on the derived histogram to add to the motion vector candidate list, and insert the determined fixed number of spatial merging candidates at predetermined positions in the motion vector candidate list.

[0329] In another embodiment of this disclosure, video decoder 30 may be configured to add ATMVP candidates to a motion vector candidate list based on motion vector functions of one or more Advanced Temporal Motion Vector Prediction (ATMVP) candidates. In another embodiment of this disclosure, video encoder 20 may be configured to determine the position in the motion vector candidate list for adding ATMVP candidates based on motion vector functions of one or more ATMVP candidates.

[0330] In another embodiment of this disclosure, the video decoder 30 may be configured to determine a combined motion vector candidate by combining motion vector information from two bidirectional motion vector candidates, and add the combined motion vector candidate to a motion vector candidate list.

[0331] In another embodiment of this disclosure, the video decoder 30 may be configured to determine the position in the motion vector candidate list for adding a combined motion vector candidate based on a motion vector function of one or more combined motion vector candidates.

[0332] In another embodiment of this disclosure, the video decoder 30 may be configured to reduce the motion vector candidate list based on motion vector difference information of the motion vector candidates in the motion vector candidate list.

[0333] In another embodiment of this disclosure, the video decoder 30 may be configured to sort bidirectional candidates in a motion vector candidate list based on the motion vector difference information of the bidirectional candidates.

[0334] For example, the combination of the proposed techniques showed a 0.4% BD rate improvement in the random access configuration on the JEM2.0 software, as shown in the table below. The gains in the following examples come from the combination of tools: (1) histogram-based spatial merge candidate sorting; (2) MVD-based combined merge candidate sorting and simplification; (3) simplification of combined zero-mv candidates for ATMVP; and (4) increasing the number of merge candidates and combined merge candidates.

[0335]

[0336] It should be recognized that, depending on the instance, certain actions or events of any of the techniques described herein may be performed in a different order, and the actions or events may be added, combined, or omitted entirely (e.g., practicing the techniques does not necessarily require all the described actions or events). Furthermore, in some instances, actions or events may be performed simultaneously rather than sequentially, for example, through multithreaded processing, interrupt handling, or multiple processors.

[0337] In one or more instances, the described functionality may be implemented using hardware, software, firmware, or any combination thereof. If implemented in software, the functionality may be stored or transmitted as one or more instructions or code on a computer-readable medium and executed by a hardware-based processing unit. The computer-readable medium may comprise a computer-readable storage medium, which corresponds to a tangible medium such as a data storage medium or a communication medium, which (e.g.) contains any medium that facilitates the transfer of a computer program from one place to another according to a communication protocol. In this way, a computer-readable medium may generally correspond to (1) a non-transitory tangible computer-readable storage medium, or (2) a communication medium, such as a signal or carrier wave. The data storage medium may be any available medium accessible by one or more computers or one or more processors to retrieve instructions, code, and / or data structures for implementing the techniques described in this disclosure. A computer program product may comprise a computer-readable medium.

[0338] By way of example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage devices, magnetic disk storage devices or other magnetic storage devices, flash memory, or any other media that can be used to store desired program code in the form of instructions or data structures and that is accessible to a computer. Furthermore, any connection is appropriately referred to as computer-readable media. For example, coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of media if instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave. However, it should be understood that the computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but are specifically addressed to non-transient tangible storage media. As used herein, disks and optical discs include compact optical discs (CDs), laser optical discs, optical discs, digital versatile optical discs (DVDs), floppy disks, and Blu-ray discs, wherein disks typically reproduce data magnetically, while optical discs reproduce data optically using lasers. The combination of the above items should also be included in the scope of computer-readable media.

[0339] Instructions can be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Therefore, the term "processor" as used herein can refer to any of the above-described structures or any other structures suitable for implementing the techniques described herein. Additionally, in some aspects, the functionality described herein may be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated into combined codecs. Furthermore, the techniques can be fully implemented within one or more circuit or logic elements.

[0340] The techniques disclosed herein can be implemented in a wide variety of devices or apparatuses, including wireless handheld devices, integrated circuits (ICs), or IC sets (e.g., chipsets). The various components, modules, or units described in this disclosure are intended to emphasize functional aspects of a device configured to perform the disclosed techniques, but do not necessarily require implementation by different hardware units. Specifically, as described above, the various units may be combined with suitable software and / or firmware within a codec hardware unit, or provided by a collection of interoperable hardware units comprising one or more processors as described above.

[0341] Various examples have been described. These and other examples are within the scope of the appended claims.

Claims

1. A method for decoding video data, the method comprising: Receive the current block of video data encoded in merge mode; Based on motion information from multiple neighboring blocks relative to the current block, a motion vector candidate list for merging candidates of the current block of video data is constructed, wherein the multiple neighboring blocks considered for the motion vector candidate list are predefined and include all neighboring blocks adjacent to the top row of the current block or adjacent to the left column of the current block, wherein each neighboring block has a size of 4x4, and wherein the number of neighboring blocks among the multiple neighboring blocks is greater than 5; A histogram of motion vector information is derived from the motion vector distribution of the plurality of adjacent blocks, wherein the motion vector distribution indicates that a particular motion vector has the same frequency for each of the plurality of adjacent blocks; The order of spatial merging candidates in the motion vector candidate list is determined based on the frequency of the associated motion vectors in the histogram, wherein if two or more spatial merging candidates have associated motion vectors with the same frequency in the histogram, a predetermined order is used. Determine the current motion vector from the list of motion vector candidates; and The current block of video data is decoded using the current motion vector.

2. The method according to claim 1, further comprising: The ATMVP candidate is added to the motion vector candidate list based on the motion vector function of one or more Advanced Time Motion Vector Prediction (ATMVP) candidates.

3. The method according to claim 1, further comprising: Based on the motion vector function of one or more ATMVP candidates, determine the position in the motion vector candidate list for adding the ATMVP candidate.

4. The method according to claim 1, further comprising: The combined motion vector candidate is determined by combining motion vector information from two bidirectional motion vector candidates. as well as Add the combined motion vector candidates to the motion vector candidate list.

5. The method according to claim 4, further comprising: Based on the motion vector function of one or more combinations of motion vector candidates, determine the position in the motion vector candidate list for adding the combination of motion vector candidates.

6. The method according to claim 1, further comprising: The motion vector candidate list is simplified based on the motion vector difference information of the motion vector candidates in the motion vector candidate list.

7. An apparatus configured to decode video data, the apparatus comprising: A memory configured to store current video data blocks; as well as One or more processors configured to: Receive the current block of video data encoded in merge mode; Based on motion information from multiple neighboring blocks relative to the current block, a motion vector candidate list for merging candidates of the current block of video data is constructed, wherein the multiple neighboring blocks considered for the motion vector candidate list are predefined and include all neighboring blocks adjacent to the top row of the current block or adjacent to the left column of the current block, wherein each neighboring block has a size of 4x4, and wherein the number of neighboring blocks among the multiple neighboring blocks is greater than 5; A histogram of motion vector information is derived from the motion vector distribution of the plurality of adjacent blocks, wherein the motion vector distribution indicates that a particular motion vector has the same frequency for each of the plurality of adjacent blocks; The order of spatial merging candidates in the motion vector candidate list is determined based on the frequency of the associated motion vectors in the histogram, wherein if two or more spatial merging candidates have associated motion vectors with the same frequency in the histogram, a predetermined order is used. Determine the current motion vector from the list of motion vector candidates; and The current block of video data is decoded using the current motion vector.

8. The device according to claim 7, wherein, The one or more processors are further configured to add ATMVP candidates to the motion vector candidate list based on motion vector functions of one or more Advanced Time Motion Vector Prediction (ATMVP) candidates.

9. The device according to claim 7, wherein, The one or more processors are further configured to: determine, based on the motion vector function of one or more ATMVP candidates, the position in the motion vector candidate list for adding the ATMVP candidate.

10. The device according to claim 7, wherein, The one or more processors are further configured to: The combined motion vector candidate is determined by combining motion vector information from two bidirectional motion vector candidates; and Add the combined motion vector candidates to the motion vector candidate list.

11. The device according to claim 10, wherein, The one or more processors are further configured to: determine, based on a motion vector function of one or more combined motion vector candidates, the position in the motion vector candidate list for adding the combined motion vector candidate.

12. The device according to claim 7, wherein, The one or more processors are further configured to: reduce the motion vector candidate list based on the motion vector difference information of the motion vector candidates in the motion vector candidate list.

13. An apparatus configured to encode video data, the apparatus comprising: A memory configured to store the current block of video data; as well as One or more processors configured to: The current block that receives video data; Based on motion information from multiple neighboring blocks relative to the current block, a motion vector candidate list for merging candidates of the current block of video data is constructed, wherein the multiple neighboring blocks considered for the motion vector candidate list are predefined and include all neighboring blocks adjacent to the top row of the current block or adjacent to the left column of the current block, wherein each neighboring block has a size of 4x4, and wherein the number of neighboring blocks among the multiple neighboring blocks is greater than 5; A histogram of motion vector information is derived from the motion vector distribution of the plurality of adjacent blocks, wherein the motion vector distribution indicates that a particular motion vector has the same frequency for each of the plurality of adjacent blocks; The order of spatial merging candidates in the motion vector candidate list is determined based on the frequency of the associated motion vectors in the histogram, wherein if two or more spatial merging candidates have associated motion vectors with the same frequency in the histogram, a predetermined order is used. Determine the current motion vector from the list of motion vector candidates; and The current block of video data is encoded using the current motion vector.

14. A non-transitory computer-readable storage medium storing instructions, which, when executed, cause one or more processors configured to decode video data to perform the following operations: Receive the current block of video data encoded in merge mode; Based on motion information from multiple neighboring blocks relative to the current block, a candidate list of motion vectors for merging the current block of video data is constructed, wherein... The plurality of adjacent blocks considered for the motion vector candidate list are predefined and include all adjacent blocks that are adjacent to the top row of the current block or the left column of the current block, wherein each adjacent block has a size of 4x4 and wherein the number of adjacent blocks in the plurality of adjacent blocks is greater than 5; A histogram of motion vector information is derived from the motion vector distribution of the plurality of adjacent blocks, wherein the motion vector distribution indicates that a particular motion vector has the same frequency for each of the plurality of adjacent blocks; The order of spatial merging candidates in the motion vector candidate list is determined based on the frequency of the associated motion vectors in the histogram, wherein if two or more spatial merging candidates have associated motion vectors with the same frequency in the histogram, a predetermined order is used. Determine the current motion vector from the list of motion vector candidates; and The current block of video data is decoded using the current motion vector.

Images