Block-level collocated motion field projection for video coding

Abstract

Systems and techniques for processing video data are provided. For example, a device can obtain one or more first sets of collocated motion vector data and one or more second sets of collocated motion vector data associated with respective first and second blocks of video data included in a current frame of video data. The apparatus can project the one or more first sets of collocated motion vector data into a first projected motion field associated with a first buffer and the one or more second sets of collocated motion vector data into the first projected motion field associated with the first buffer. Based on the one or more first sets of projected collocated motion vector data and the one or more second sets, the device can decode the first block of video data based on the first projected motion field associated with the first buffer.

Description

Technical Field

[0001] The present disclosure generally relates to video encoding and decoding. For example, aspects of the present disclosure include improving video coding techniques related to collocated motion field projection. Background Art

[0002] Digital video capabilities can be incorporated into a variety of devices, including digital televisions, digital live broadcast systems, wireless broadcast systems, personal digital assistants (PDAs), laptop or desktop computers, tablet computers, e-book readers, digital cameras, digital recording devices, digital media players, video game devices, video game consoles, cellular or satellite wireless phones (so-called "smart phones"), video teleconferencing devices, video streaming devices, etc. Such devices allow video data to be processed and output for consumption. Digital video data includes a large amount of data to meet the needs of consumers and video providers. For example, consumers of video data expect the highest quality video, with high fidelity, high resolution, high frame rate, etc. As a result, the large amount of video data required to meet these needs burdens the communication networks and devices that process and store video data.

[0003] Digital video devices can implement video decoding techniques to compress video data. Video decoding is performed according to one or more video decoding standards or formats. For example, video decoding standards or formats include universal video coding (VVC), high-efficiency video coding (HEVC), advanced video coding (AVC), MPEG-2 Part 2 decoding (MPEG stands for Moving Picture Experts Group), etc., and proprietary video codecs / formats, such as AOMedia Video 1 (AV1) developed by the Open Media Alliance. Video decoding generally utilizes prediction methods (e.g., inter-frame prediction, intra-frame prediction, etc.) that utilize redundancy present in video images or sequences. The goal of video decoding techniques is to compress video data into a form that uses a lower bit rate while avoiding or minimizing the degradation of video quality. As evolving video services become available, decoding techniques with better decoding efficiency are needed. Summary of the invention

[0004] In some examples, systems and techniques for block-level collocated motion field projection are described. For example, the systems and techniques may read motion vectors associated with one or more collocated reference frames and project them into a block-level motion field. The one or more collocated reference frames may be associated with a current (e.g., currently decoded) frame of video data. Each corresponding block-level motion field may be associated with a block included in the current frame of video data. The block-level motion field and the block included in the current frame of video data may have the same relative position relative to the collocated reference frame and the current block of video data, respectively. The projected block-level motion field may be stored in one or more buffers. The motion field associated with the current block of video data may be used to decode the current block of video data concurrently with determining a motion field for one or more subsequent blocks of video data.

[0005] According to at least one illustrative example, a device for processing video data is provided, the device comprising at least one memory (e.g., the at least one memory is configured to store data, such as virtual content data, one or more images, etc.) and at least one processor coupled to the at least one memory (e.g., the at least one processor is implemented in a circuit). The at least one processor is configured and can: obtain one or more first sets of collocated motion vector data associated with a first block of video data included in a current frame of the video data; project each of the one or more first sets of collocated motion vector data into a first projected motion field associated with a first buffer; obtain one or more second sets of collocated motion vector data associated with a second block of video data included in the current frame of the video data; project each of the one or more second sets of collocated motion vector data into the first projected motion field associated with the first buffer; and decode the first block of video data based on the first projected motion field associated with the first buffer based on projecting each of the one or more first sets of collocated motion vector data and each of the one or more second sets of collocated motion vector data into the first projected motion field associated with the first buffer.

[0006] In another example, a method for processing video data is provided, the method comprising: obtaining one or more first sets of collocated motion vector data associated with a first block of video data included in a current frame of the video data; projecting each of the one or more first sets of collocated motion vector data into a first projected motion field associated with a first buffer; obtaining one or more second sets of collocated motion vector data associated with a second block of video data included in the current frame of the video data; projecting each of the one or more second sets of collocated motion vector data into the first projected motion field associated with the first buffer; and decoding the first block of video data based on the first projected motion field associated with the first buffer based on projecting each of the one or more first sets of collocated motion vector data and each of the one or more second sets of collocated motion vector data into the first projected motion field associated with the first buffer.

[0007] In another example, a non-transitory computer-readable medium having instructions stored thereon is provided, which instructions, when executed by one or more processors, cause the one or more processors to: obtain one or more first sets of collocated motion vector data associated with a first block of video data included in a current frame of video data; project each of the one or more first sets of collocated motion vector data into a first projected motion field associated with a first buffer; obtain one or more second sets of collocated motion vector data associated with a second block of video data included in the current frame of video data; project each of the one or more second sets of collocated motion vector data into the first projected motion field associated with the first buffer; and decode the first block of video data based on the first projected motion field associated with the first buffer based on projecting each of the one or more first sets of collocated motion vector data and each of the one or more second sets of collocated motion vector data into the first projected motion field associated with the first buffer.

[0008] In another example, an apparatus for processing video data is provided, the apparatus comprising: a component for obtaining one or more first sets of collocated motion vector data associated with a first block of video data included in a current frame of the video data; a component for projecting each of the one or more first sets of collocated motion vector data into a first projected motion field associated with a first buffer; a component for obtaining one or more second sets of collocated motion vector data associated with a second block of video data included in the current frame of the video data; a component for projecting each of the one or more second sets of collocated motion vector data into the first projected motion field associated with the first buffer; and a component for decoding the first block of video data based on the first projected motion field associated with the first buffer based on projecting each of the one or more first sets of collocated motion vector data and each of the one or more second sets of collocated motion vector data into the first projected motion field associated with the first buffer.

[0009] In some aspects, one or more of the devices described herein are, are part of, and / or include a mobile device or wireless communication device (e.g., a mobile phone or other mobile device), an extended reality (XR) device or system (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a wearable device (e.g., a network-connected watch or other wearable device), a camera, a personal computer, a laptop computer, a vehicle or a computing device or component of a vehicle, a server computer or server device, another device, or a combination thereof. In some aspects, the device includes one or more cameras for capturing one or more images. In some aspects, the device also includes a display for displaying one or more images, notifications, and / or other displayable data. In some aspects, the above-mentioned device may include one or more sensors (e.g., one or more inertial measurement units (IMUs), such as one or more gyroscopes, one or more gyrometers, one or more accelerometers, any combination thereof, and / or other sensors).

[0010] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter. The subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all drawings, and each claim.

[0011] The foregoing and other features and aspects will become more apparent upon reference to the following description, claims and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Illustrative aspects of the present application are described in detail below with reference to the following drawings:

[0013] Figure 1 is a block diagram illustrating examples of encoding devices and decoding devices according to some examples of the present disclosure;

[0014] Figure 2A is a conceptual diagram illustrating example spatial neighboring motion vector candidates for merge mode according to some examples of the present disclosure;

[0015] Figure 2B is a conceptual diagram illustrating example spatial neighboring motion vector candidates for an Advanced Motion Vector Prediction (AMVP) mode according to some examples of the present disclosure;

[0016] Figure 3A is a conceptual diagram illustrating example temporal motion vector predictor (TMVP) candidates according to some examples of the present disclosure;

[0017] Figure 3B is a conceptual diagram illustrating an example of motion vector scaling according to some examples of the present disclosure;

[0018] Figure 4A is a conceptual diagram illustrating an example of neighboring samples of a current coding unit for estimating motion compensation parameters of the current coding unit according to some examples of the present disclosure;

[0019] Figure 4B is a conceptual diagram illustrating an example of neighboring samples of a reference block for estimating motion compensation parameters of a current coding unit according to some examples of the present disclosure;

[0020] Figure 5 illustrating aspects of motion vector scaling for temporal merging candidates used in processing blocks according to some examples of the present disclosure;

[0021] Fig. 6A illustrates an example of collocated motion vector projection according to some examples of the present disclosure;

[0022] Figure 6B illustrates examples of available types of collocated reference frames according to some examples of the present disclosure;

[0023] FIG. 7A to FIG. 7D An example illustrating juxtaposed motion vector projections and overwriting of existing motion vector field projections according to some examples of the present disclosure;

[0024] Fig. 8A illustrates an example of a frame level raster scan order that may be used to perform collocated motion vector projection according to some examples of the present disclosure;

[0025] Figure 8BAn example diagram illustrating a system for performing collocated motion vector projection based on frame-level raster scanning according to some examples of the present disclosure;

[0026] Fig. 9A illustrates an example of a block level raster scan order that may be used to perform collocated motion vector projection according to some examples of the present disclosure;

[0027] Fig. 9B An example diagram illustrating a system for performing collocated motion vector projection based on block level processing according to some examples of the present disclosure;

[0028] Fig.10 is a diagram illustrating an example of block-based collocated motion vector projection using a sliding motion vector buffer according to some examples of the present disclosure;

[0029] Fig.11 is a diagram illustrating an example of block-based collocated motion vector projection using a circular motion vector buffer according to some examples of the present disclosure;

[0030] Fig.12 is a diagram illustrating an example of block-based collocated motion vector projection using a ping-pong motion vector buffer according to some examples of the present disclosure;

[0031] Fig.13 is a diagram illustrating an example of block-based collocated motion vector projection using a sliding motion vector buffer in which projection is performed in reverse N order and decoding is performed in Z order according to some examples of the present disclosure;

[0032] Fig.14 is a diagram illustrating an example of row-based indexing for overwriting conflicts associated with block-based collocated motion vector projections according to some examples of the present disclosure;

[0033] Fig.15 is a flow chart illustrating an example process for video coding using block-based collocated motion vector projection according to some examples of the present disclosure;

[0034] Fig.16 is a block diagram illustrating an example video encoding device according to some examples of the present disclosure; and

[0035] Fig.17 is a block diagram illustrating an example video decoding device according to some examples of the present disclosure. DETAILED DESCRIPTION

[0036] Some aspects and aspects of the present disclosure are provided below. Some aspects of these aspects and aspects can be applied independently, and some of them can be applied in combination, which is obvious to those skilled in the art. In the following description, specific details are set forth for explanation purposes to provide a thorough understanding of various aspects of the application. However, it is obvious that various aspects can be implemented without these specific details. Each drawing and description are not intended to be restrictive.

[0037] The following description provides only exemplary aspects and is not intended to limit the scope, applicability or configuration of the present disclosure. On the contrary, the following description of the exemplary aspects will provide a description that can be used to implement the exemplary aspects to those skilled in the art. It should be understood that various changes can be made to the function and arrangement of elements without departing from the essence and scope of the present application as set forth in the appended claims.

[0038] A video decoding device (e.g., an encoding device, a decoding device, or a combined encoding-decoding device) implements video compression techniques to efficiently decode (e.g., encode and / or decode) video data. Video compression techniques may include applying different prediction modes, including spatial prediction (e.g., intra-frame prediction or intra-prediction), temporal prediction (e.g., inter-frame prediction or inter-prediction), inter-layer prediction (across different layers of video data), and / or other prediction techniques for reducing or removing redundancy inherent in video sequences. The video encoder may partition each picture of the original video sequence into rectangular areas, which are referred to as video blocks or decoding units (described in more detail below). These video blocks may be encoded using a specific prediction mode.

[0039] Video blocks may be divided into one or more groups of smaller blocks in one or more ways. Blocks may include coding tree blocks, prediction blocks, transform blocks, or other suitable blocks. Unless otherwise specified, references to "blocks" may generally refer to such video blocks (e.g., coding tree blocks, coding blocks, prediction blocks, transform blocks, or other suitable blocks or sub-blocks, as will be understood by those of ordinary skill in the art). In addition, each of these blocks may also be referred to interchangeably herein as a "unit" (e.g., coding tree unit (CTU), decoding unit, prediction unit (PU), transform unit (TU), etc.). In some cases, a unit may indicate a decoding logic unit encoded in a bitstream, and a block may indicate a portion of a process in a video frame buffer for which a process is directed.

[0040] For inter-prediction mode, the video encoder can search for a block similar to the block being encoded in a frame (or picture) located at another time position (which is called a reference frame or reference picture). The video encoder can limit the search to a certain spatial displacement from the block to be encoded. A two-dimensional (2D) motion vector including a horizontal displacement component and a vertical displacement component can be used to locate the best match. For intra-prediction mode, the video encoder can use spatial prediction techniques to form a prediction block based on data from previously encoded neighboring blocks within the same picture.

[0041] In some cases, a collocated motion vector projection may be performed based on some (or all) of the temporal motion vector set determined for the starting frame relative to the current frame and its reference frame projection. In some cases, a temporal motion vector determined for a given frame with index x may be used to generate one or motion vector projections that may be located within a given frame x and / or may be located outside a given frame x. In some examples, if the position of the motion vector projection is not included in one of the following, the motion vector projection may be invalidated (e.g., rejected or not selected): a given frame from which the motion vector projection originates, a horizontally adjacent frame on the left side of the given frame, and a horizontally adjacent frame on the right side of the given frame. In some examples, collocated motion vector projection may be performed for up to three collocated reference frames (referred to as REF0, REF1, and REF2). Based on having the same projection end position, the motion vector associated with REF0 may be projected first and may be overwritten by a later projected motion vector associated with REF1 and / or REF2.

[0042] In some examples, collocated motion vector projection may be performed using a frame-level raster scan order. For example, for a given frame of video data, all REF0 motion vectors may be projected in the raster scan order of the frame, followed by projecting all REF1 motion vectors and REF2 motion vectors (e.g., sequentially and each in the raster scan order of the frame). Performing collocated motion vector projection by processing three collocated reference frames in a frame-level raster scan order may be associated with four read operations and four write operations for collocated data. For example, the four read operations may include three read operations for reading REF0, REF1, and REF2 motion vector data (e.g., before performing a corresponding motion vector projection for each reference frame) and a fourth read operation for reading the final contents of the projected motion vector buffer. The four write operations may include three write operations for writing the REF0, REF1, and REF2 motion vector projections to the projected motion vector buffer and a fourth write operation for writing a final motion vector prediction for the current frame to a memory (e.g., based on a combined motion vector projection field stored in the projected motion vector buffer).

[0043] There is a need to reduce the number of clock cycles used to perform collocated motion vector projection for one or more (e.g., up to three) collocated reference frames. There is a need to reduce the read / write bandwidth associated with decoding frames of video data using collocated motion vector projection. There is also a need to reduce the size (e.g., storage capacity) of one or more buffers associated with storing projected motion vectors determined when performing collocated motion vector projection.

[0044] Described herein are systems, apparatus, processes (also referred to as methods), and computer-readable media (collectively referred to herein as "systems and techniques") for performing collocated motion vector projection using block-level processing. In some aspects, the systems and techniques described herein may be used to perform on-the-fly (e.g., real-time or near real-time) collocated motion vector projection according to a block-based processing order. In some examples, the systems and techniques may be used to read and project reference frame motion vectors based on block-level scanning. In some aspects, a current decoded frame of video data may be associated with three collocated reference frames (e.g., REF0, REF1, REF2). The associated motion vector data included in the three collocated reference frames may be read and projected at a 64×64 block level. For example, the current decoded frame of video data may be divided into a plurality of 64×64 sized blocks, and motion vectors may be obtained from the three collocated reference frames (e.g., REF0, REF1, REF2) using the same 64×64 block size. In some cases, the associated motion vector data included in the collocated reference frames may be read and projected at a 128×128 block level. In some examples, systems and techniques may avoid writing projected motion vectors to DDR or other memory based on using the block-based processing order described herein for performing collocated motion vector projection.

[0045] In some aspects, systems and techniques may include one or more MV projection buffers implemented in hardware, wherein the size of the one or more MV projection buffers is set to three times the size of the block level (e.g., for a 64×64 block level, the one or more MV projection buffers may be sized to store three 64×64 blocks; for a 128×128 block level, the one or more MV projection buffers may be sized to store three 128×128 blocks).

[0046] In some examples, a given block of video data may be decoded in parallel with a collocated motion vector projection operation performed for one or more additional and / or subsequent blocks of video data. For example, the additional and / or subsequent blocks of video data may be horizontally adjacent to the right of a currently decoded block of video data (e.g., where both blocks are included in the same frame of video data). In some cases, the given block of video data may be decoded based on a determination that MV projection has been completed for the given block, where decoding is performed for subsequent blocks of video data independent of the MV projection operation.

[0047] The techniques described herein may be implemented using one or more decoding devices (including one or more encoding devices, decoding devices, or combined encoding-decoding devices). The decoding device may be implemented by one or more of a player device (such as a mobile device), an extended reality (XR) device, a vehicle or a computing system of a vehicle, a server device or system (e.g., a distributed server system including multiple servers, a single server device or system, etc.), or other devices or systems.

[0048] The systems and techniques described herein may be applied to any existing video codec, any video codec being developed, and / or any future video coding standard, including but not limited to High Efficiency Video Coding (HEVC), Advanced Video Coding (AVC), Versatile Video Coding (VVC), VP9, ​​AOMedia Video 1 (AV1) format / codec, and / or other video coding standards that are existing, developing, or to be developed. The systems and techniques described herein may improve the operation of communication systems and devices in a system by improving the performance of video data delivery by a device with improved compression and associated improved video quality based on improved motion vector selection from adaptive bilateral matching as described herein.

[0049] Various aspects of the systems and techniques will be described with respect to the accompanying figures.

[0050] Figure 1 1 is a block diagram illustrating an example of a system 100 including an encoding device 104 and a decoding device 112. The encoding device 104 may be part of a source device, and the decoding device 112 may be part of a receiving device. The source device and / or the receiving device may include an electronic device, such as a mobile or fixed telephone handset (e.g., a smart phone, a cellular phone, etc.), a desktop computer, a laptop or notebook computer, a tablet computer, a set-top box, a television, a camera, a display device, a digital media player, a video game console, a video streaming device, an Internet Protocol (IP) camera, or any other suitable electronic device. In some examples, the source device and the receiving device may include one or more wireless transceivers for wireless communication. The decoding techniques described herein are applicable to video decoding in various multimedia applications, including streaming video transmission (e.g., via the Internet), television broadcasting or transmission, encoding digital video for storage on a data storage medium, decoding digital video stored on a data storage medium, or other applications. As used herein, the term decoding may refer to encoding and / or decoding. In some examples, system 100 may support one-way or two-way video transmission to support applications such as video conferencing, video streaming, video playback, video broadcasting, gaming, and / or video telephony.

[0051] The encoding device 104 (or encoder) can be used to encode video data using a video coding standard, format, codec or protocol to generate a coded video bitstream. Examples of video coding standards and formats / codecs include ITU-T H.261, ISO / IEC MPEG-1 Visual, ITU-T H.262 or ISO / IEC MPEG-2 Visual, ITU-TH.263, ISO / IEC MPEG-4 Visual, ITU-T H.264 (also known as ISO / IEC MPEG-4 AVC) (including its scalable video coding (SVC) and multi-view video coding (MVC) extensions), high efficiency video coding (HEVC) or ITU-T H.265 and universal video coding (VVC) or ITU-T H.266. There are various extensions to HEVC to handle multi-layer video coding, including range and screen content coding extensions, 3D video coding (3D-HEVC) and multi-view extensions (MV-HEVC) and scalable extensions (SHVC). HEVC and its extensions have been developed by the Joint Collaborative Team on Video Coding (JCT-VC) and the Joint Collaborative Team on 3D Video Coding Extensions (JCT-3V) of the ITU-T Video Coding Experts Group (VCEG) and the ISO / IEC Motion Picture Experts Group (MPEG). VP9, ​​AOMedia Video 1 (AV1) developed by the Alliance for Open Media (AOMedia) of Open Media, and Essential Video Coding (EVC) are other video coding standards to which the techniques described herein may be applied.

[0052] The techniques described herein may be applied to any of the existing video codecs (e.g., High Efficiency Video Coding (HEVC), Advanced Video Coding (AVC), or other suitable existing video codecs), and / or may be efficient coding tools for any video coding standard being developed and / or future video coding standards (e.g., VVC and / or other video coding standards under development or to be developed). For example, the examples described herein may be performed using video codecs such as VVC, HEVC, AVC, and / or their extensions. However, the techniques and systems described herein may also be applicable to other coding standards, codecs, or formats, such as MPEG, JPEG (or other coding standards for still images), VP9, ​​AV1, their extensions, or other suitable coding standards that are already available or not yet available or not yet developed. For example, in some examples, the encoding device 104 and / or the decoding device 112 may operate according to a proprietary video codec / format (such as AV1, an extension of AVI, and / or a successor version of AV1 (e.g., AV2)) or other proprietary formats or industry standards. Thus, although the techniques and systems described herein may be described with reference to a particular video coding standard, one of ordinary skill in the art will appreciate that the description should not be construed as applying only to that particular standard.

[0053] refer to Figure 1 , video source 102 may provide video data to encoding device 104. Video source 102 may be part of a source device, or may be part of a device other than a source device. Video source 102 may include a video capture device (e.g., a video camera, a camera phone, a video phone, etc.), a video archive containing stored video, a video server or content provider that provides video data, a video feed interface that receives video from a video server or content provider, a computer graphics system for generating computer graphics video data, a combination of such sources, or any other suitable video source.

[0054] The video data from the video source 102 may include one or more input pictures or frames. A picture or frame is a static image, which in some cases is part of a video. In some examples, the data from the video source 102 may be a static image that is not part of a video. In HEVC, VVC and other video coding specifications, a video sequence may include a series of pictures. A picture may include three sample arrays, which are represented as SL, SCb and SCr. SL is a two-dimensional array of luma samples, SCb is a two-dimensional array of Cb chroma samples, and SCr is a two-dimensional array of Cr chroma samples. Chroma samples may also be referred to as "chroma" samples in this article. A pixel may refer to all three components (luminance samples and chroma samples) of a given position in an array of pictures. In other instances, a picture may be monochrome and may include only a luma sample array, in which case the terms pixel and sample may be used interchangeably. With respect to the example techniques described herein that refer to individual samples for illustrative purposes, the same techniques may be applied to pixels (e.g., all three sample components of a given position in an array of pictures). With respect to the example techniques described herein that refer to pixels (eg, all three sample components of a given position in an array of a picture) for illustrative purposes, the same techniques may be applied to individual samples.

[0055] The encoder engine 106 (or encoder) of the encoding device 104 encodes the video data to generate a coded video bitstream. In some examples, the coded video bitstream (or "video bitstream" or "bitstream") is a series of one or more decoded video sequences. The decoded video sequence (CVS) includes a series of access units (AUs) starting from an AU having a random access point picture in the base layer and having certain attributes, until the next AU having a random access point picture in the base layer and having certain attributes and not including the next AU. For example, certain attributes of the random access point picture that starts the CVS may include a RASL marker equal to 1 (e.g., NoRaslOutputFlag). Otherwise, the random access point picture (with a RASL marker equal to 0) does not start the CVS. The access unit (AU) includes one or more decoded pictures and control information corresponding to the decoded pictures sharing the same output time. The decoded slices of the picture are encapsulated as data units at the bitstream level, which are called network abstraction layer (NAL) units. For example, an HEVC video bitstream may include one or more CVSs, which include NAL units. Each NAL unit in the NAL unit has a NAL unit header. In one example, the header is one byte for H.264 / AVC (except for multi-layer extensions) and two bytes for HEVC. The syntax elements in the NAL unit header use designated bits and are therefore visible to all kinds of systems and transport layers (such as transport streams, real-time transport (RTP) protocols, file formats, etc.).

[0056] There are two types of NAL units in the HEVC standard, including video coding layer (VCL) NAL units and non-VCL NAL units. The VCL NAL unit includes decoded picture data that forms a decoded video bitstream. For example, a bit sequence that forms a decoded video bitstream exists in a VCL NAL unit. The VCL NAL unit may include a slice or slice segment (described below) of decoded picture data, and the non-VCL NAL unit includes control information related to one or more decoded pictures. In some cases, the NAL unit may be referred to as a packet. The HEVC AU includes a VCL NAL unit containing decoded picture data and a non-VCL NAL unit corresponding to the decoded picture data (if any). In addition to other information, the non-VCL NAL unit may also include a parameter set with advanced information related to the encoded video bitstream. For example, a parameter set may include a video parameter set (VPS), a sequence parameter set (SPS), and a picture parameter set (PPS). In some cases, each slice or other portion of the bitstream may reference a single valid PPS, SPS, and / or VPS to allow the decoding device 112 to access information that can be used to decode the slice or other portion of the bitstream.

[0057] A NAL unit may include a sequence of bits that form a decoded representation of video data (e.g., a coded video bitstream, a CVS of a bitstream, etc.), such as a decoded representation of a picture in a video. The encoder engine 106 generates a decoded representation of a picture by partitioning each picture into a plurality of slices. A slice is independent of other slices so that information in the slice can be decoded without relying on data from other slices within the same picture. A slice includes one or more slice segments, including an independent slice segment and, if present, one or more dependent slice segments that depend on a previous slice segment.

[0058] In HEVC, the slice is then partitioned into coding tree blocks (CTBs) of luma samples and chroma samples. The CTB of luma samples and one or more CTBs of chroma samples together with the syntax for the samples are called coding tree units (CTUs). CTUs may also be called "tree blocks" or "largest coding units" (LCUs). CTUs are the basic processing units for HEVC encoding. CTUs can be split into multiple coding units (CUs) of different sizes. A CU contains a luma sample array and a chroma sample array called a coding block (CB).

[0059] Luma CB and chroma CB can be further split into prediction blocks (PB). PB is a sample block of luma component or chroma component, which uses the same motion parameters for inter prediction or intra block copy (IBC) prediction (when available or enabled for use). Luma PB and one or more chroma PBs together with associated syntax form a prediction unit (PU). For inter prediction, a set of motion parameters (e.g., one or more motion vectors, reference indexes, etc.) are signaled in the bitstream for each PU, as well as inter prediction for luma PB and one or more chroma PBs. Motion parameters may also be referred to as motion information. CB may also be divided into one or more transform blocks (TB). TB represents a square block of samples of a color component, to which a residual transform (e.g., in some cases, the same two-dimensional transform) is applied to decode the prediction residual signal. Transform unit (TU) represents a TB of luma samples and chroma samples and corresponding syntax elements. Transform decoding is described in more detail below.

[0060] The size of the CU corresponds to the size of the decoding mode and may be a square shape. For example, the size of the CU may be 8×8 samples, 16×16 samples, 32×32 samples, 64×64 samples, or any other appropriate size up to the size of the corresponding CTU. The phrase "N×N" is used herein to refer to the pixel dimensions of the video block in terms of the vertical and horizontal dimensions (e.g., 8 pixels×8 pixels). The pixels in the block may be arranged in rows and columns. In some specific implementations, the block may not have the same number of pixels in the horizontal direction as in the vertical direction. The syntax data associated with the CU may describe, for example, the partitioning of the CU into one or more PUs. The partitioning mode may vary depending on whether the CU is encoded in an intra-prediction mode or an inter-prediction mode. The PU may be partitioned into a non-square shape. The syntax data associated with the CU may also describe, for example, the partitioning of the CU into one or more TUs according to the CTU. The TU may be square or non-square in shape.

[0061] According to the HEVC standard, a transform unit (TU) may be used to perform the transform. The TU may vary for different CUs. The size of the TU may be set based on the size of the PU within a given CU. The TU may have the same size as the PU or be smaller than the PU. In some examples, a quadtree structure called a residual quadtree (RQT) may be used to subdivide the residual samples corresponding to the CU into smaller units. The leaf nodes of the RQT may correspond to the TU. The pixel difference values ​​associated with the TU may be transformed to generate transform coefficients. The transform coefficients may then be quantized by the encoder engine 106.

[0062] Once the picture of the video data is partitioned into CUs, the encoder engine 106 predicts each PU using a prediction mode. The prediction unit or prediction block is then subtracted from the original video data to obtain a residual (described below). For each CU, the prediction mode can be signaled in the bitstream using syntax data. The prediction mode may include intra-frame prediction (or intra-picture prediction) or inter-frame prediction (or inter-picture prediction). Intra-frame prediction exploits the correlation between spatially adjacent samples within a picture. For example, using intra-frame prediction, each PU is predicted from adjacent image data in the same picture using, for example, DC prediction to find an average value for the PU, using plane prediction to fit the plane surface to the PU, using directional prediction to infer from adjacent data, or using any other suitable prediction type. Inter-frame prediction uses temporal correlation between pictures to derive motion compensated prediction for blocks of image samples. For example, using inter-frame prediction, each PU is predicted from image data in one or more reference pictures (before or after the current picture in output order) using motion compensated prediction. For example, a decision can be made at the CU level whether to use inter-picture prediction or intra-picture prediction to decode a picture area.

[0063] The encoder engine 106 and the decoder engine 116 (described in more detail below) may be configured to operate according to VVC. According to VVC, a video decoder (such as the encoder engine 106 and / or the decoder engine 116) partitions a picture into a plurality of coding tree units (CTUs) (wherein the CTB of a luminance sample and one or more CTBs of a chrominance sample together with the syntax for the sample are referred to as CTUs). The video decoder may partition the CTU according to a tree structure such as a quadtree-binary tree (QTBT) structure or a multi-type tree (MTT) structure. The QTBT structure removes the concept of multiple partition types, such as the separation between CU, PU, ​​and TU of HEVC. The QTBT structure includes two levels, including a first level partitioned according to quadtree partitioning, and a second level partitioned according to binary tree partitioning. The root node of the QTBT structure corresponds to the CTU. The leaf nodes of the binary tree correspond to the coding units (CUs).

[0064] In the MTT segmentation structure, blocks can be segmented using quadtree segmentation, binary tree segmentation, and one or more types of ternary tree segmentation. Ternary tree segmentation is a segmentation in which a block is split into three sub-blocks. In some examples, ternary tree segmentation divides a block into three sub-blocks without dividing the original block through the center. The segmentation types in MTT (e.g., quadtree, binary tree, and ternary tree) can be symmetric or asymmetric.

[0065] When operating according to the AV1 codec, the encoding device 104 and the decoding device 112 may be configured to decode the video data in blocks. In AV1, the largest decoding block that can be processed is called a super block. In AV1, a super block may be 128×128 luma samples or 64×64 luma samples. However, in subsequent video coding formats (e.g., AV2), super blocks may be defined by different (e.g., larger) luma sample sizes. In some examples, the super block is the top layer of the block quadtree. The encoding device 104 may further divide the super block into smaller decoding blocks. The encoding device 104 may divide the super block and other decoding blocks into smaller blocks using square or non-square partitioning. Non-square blocks may include N / 2×N blocks, N×N / 2 blocks, N / 4×N blocks, and N×N / 4 blocks. The encoding device 104 and the decoding device 112 may perform separate prediction and transform processing on each decoding block in the decoding block.

[0066] AV1 also defines tiles of video data. A tile is a rectangular array of super blocks that can be decoded independently of other tiles. That is, the encoding device 104 and the decoding device 112 can encode and decode coded blocks within a tile, respectively, without using video data from other tiles. However, the encoding device 104 and the decoding device 112 can perform filtering across tile boundaries. The size of the tiles can be uniform or non-uniform. Tile-based decoding can enable parallel processing and / or multi-threading specifically implemented by the encoder and decoder.

[0067] In some examples, the video coder may use a single QTBT or MTT structure to represent each of the luma and chroma components, while in other examples, the video coder may use two or more QTBT or MTT structures, such as one QTBT or MTT structure for the luma component and another QTBT or MTT structure for the two chroma components (or two QTBT and / or MTT structures for corresponding chroma components).

[0068] The video coder may be configured to use quadtree partitioning, QTBT partitioning, MTT partitioning, superblock partitioning, or other partitioning structures.

[0069] In some examples, a slice type is assigned to one or more slices of a picture. Slice types include intra-coded slices (I slices), inter-coded P slices, and inter-coded B slices. An I slice (intra-coded frame, independently decodable) is a slice of a picture that is coded only by intra prediction, and is therefore independently decodable because an I slice only requires data within the frame to predict any prediction unit or prediction block of the slice. A P slice (unidirectionally predicted frame) is a slice of a picture that can be coded with intra prediction and with unidirectional inter prediction. Each prediction unit or prediction block within a P slice is coded with intra prediction or inter prediction. When inter prediction is applied, the prediction unit or prediction block is predicted by only one reference picture, and therefore the reference sample comes from only one reference region of one frame. A B slice (bidirectionally predicted frame) is a slice of a picture that can be coded with intra prediction and inter prediction (e.g., either bidirectional prediction or unidirectional prediction). A prediction unit or prediction block of a B slice may be bidirectionally predicted from two reference pictures, where each picture contributes to one reference region and the sample sets of the two reference regions are weighted (e.g., with equal weights or with different weights) to produce a prediction signal for the bidirectionally predicted block. As described above, slices of a picture are coded independently. In some cases, a picture may be coded as only one slice.

[0070] As described above, intra-picture prediction of a picture exploits the correlation between spatially adjacent samples within the picture. There are multiple intra-prediction modes (also referred to as "intra-modes"). In some examples, intra-prediction of a luminance block includes 35 modes, including a planar mode, a DC mode, and 33 angular modes (e.g., a diagonal intra-prediction mode and an angular mode adjacent to the diagonal intra-prediction mode). The 35 modes of intra-prediction can be indexed. In other examples, more intra-modes can be defined, including prediction angles that may not yet be represented by the 33 angular modes. In other examples, the prediction angles associated with the angular modes may be different from those used in HEVC.

[0071] Inter-picture prediction uses temporal correlations between pictures in order to derive motion compensated predictions for blocks of image samples. Using a translational motion model, the position of a block in a previously decoded picture (reference picture) is indicated by a motion vector (Δx, Δy), where Δx specifies the horizontal displacement of the reference block relative to the position of the current block, and Δy specifies the vertical displacement of the reference block relative to the position of the current block. In some cases, the motion vector (Δx, Δy) may have integer sample accuracy (also referred to as integer accuracy), in which case the motion vector points to an integer pixel grid (or integer pixel sampling grid) of a reference frame. In some cases, the motion vector (Δx, Δy) may have fractional sample accuracy (also referred to as fractional pixel accuracy or non-integer accuracy) to more accurately capture the movement of underlying objects without being limited to an integer pixel grid of a reference frame.

[0072] The accuracy of the motion vector may be represented by the quantization level of the motion vector. For example, the quantization level may be integer accuracy (e.g., 1 pixel) or fractional pixel accuracy (e.g., 1 / 4 pixels, 1 / 2 pixel or other sub-pixel value). When the corresponding motion vector has fractional sample accuracy, interpolation is applied to the reference picture to derive the prediction signal. For example, the samples available at the integer positions can be filtered (e.g., using one or more interpolation filters) to estimate the values ​​at the fractional positions. The previously decoded reference picture is indicated by the reference index (refIdx) of the reference picture list. The motion vector and the reference index may be referred to as motion parameters. Two types of inter-picture prediction can be performed, including unidirectional prediction and bidirectional prediction.

[0073] In the case of inter-frame prediction using bidirectional prediction (also known as bidirectional inter-frame prediction), two motion parameter sets (Δx0, y0, refIdx0 and Δx1, y1, refIdx1) are used to generate two motion compensated predictions (from the same reference picture or possibly from different reference pictures). For example, in the case of bidirectional prediction, each prediction block uses two motion compensated prediction signals and generates B prediction units. The two motion compensated predictions are then combined to obtain the final motion compensated prediction. For example, the two motion compensated predictions can be combined by averaging. In another example, weighted prediction can be used, in which case different weights can be applied to each motion compensated prediction. The reference pictures that can be used in bidirectional prediction are stored in two separate lists, represented as list 0 and list 1. The motion parameters can be derived using a motion estimation process at the encoder.

[0074] In the case of inter prediction using unidirectional prediction (also referred to as unidirectional inter prediction), a motion compensation prediction is generated from a reference picture using a set of motion parameters (Δx0, y0, refIdx0). For example, in the case of unidirectional prediction, each prediction block uses at most one motion compensation prediction signal and generates P prediction units.

[0075] The PU may include data related to the prediction process (e.g., motion parameters or other suitable data). For example, when the PU is encoded using intra prediction, the PU may include data describing the intra prediction mode for the PU. As another example, when the PU is encoded using inter prediction, the PU may include data defining a motion vector for the PU. The data defining the motion vector for the PU may describe, for example, a horizontal component of the motion vector (Δx), a vertical component of the motion vector (Δy), a resolution for the motion vector (e.g., integer precision, quarter-pixel precision, or eighth-pixel precision), a reference picture to which the motion vector points, a reference index, a reference picture list for the motion vector (e.g., list 0, list 1, or list C), or any combination thereof.

[0076] AV1 includes two general techniques for encoding and decoding coded blocks of video data. The two general techniques are intra-frame prediction (e.g., intra-frame prediction or spatial prediction) and inter-frame prediction (e.g., inter-frame prediction or temporal prediction). In the context of AV1, when an intra-frame prediction mode is used to predict a block of a current frame of video data, the encoding device 104 and the decoding device 112 do not use video data from other frames of the video data. For most intra-frame prediction modes, the video encoding device 104 encodes the block of the current frame based on the difference between the sample values ​​in the current block and the prediction values ​​generated from the reference samples in the same frame. The video encoding device 104 determines the prediction values ​​generated from the reference samples based on the intra-frame prediction mode.

[0077] After performing prediction using intra prediction and / or inter prediction, the encoding device 104 may perform transformation and quantization. For example, after prediction, the encoder engine 106 may calculate a residual value corresponding to the PU. The residual value may include a pixel difference value between a current pixel block (PU) being decoded and a prediction block (e.g., a predicted version of the current block) used to predict the current block. For example, after generating a prediction block (e.g., issuing an inter prediction or an intra prediction), the encoder engine 106 may generate a residual block by subtracting the prediction block generated by the prediction unit from the current block. The residual block includes a set of pixel difference values ​​that quantizes the difference between the pixel values ​​of the current block and the pixel values ​​of the prediction block. In some examples, the residual block may be represented by a two-dimensional block format (e.g., a two-dimensional matrix or array of pixel values). In such examples, the residual block is a two-dimensional representation of pixel values.

[0078] Any residual data that may remain after performing the prediction is transformed using a block transform, which may be based on a discrete cosine transform, a discrete sine transform, an integer transform, a wavelet transform, other suitable transform functions, or any combination thereof. In some cases, one or more block transforms (e.g., of size 32×32, 16×16, 8×8, 4×4, or other suitable sizes) may be applied to the residual data in each CU. In some aspects, TUs may be used for the transform and quantization process implemented by the encoder engine 106. A given CU with one or more PUs may also include one or more TUs. As described in further detail below, the residual values ​​may be transformed into transform coefficients using a block transform, and then quantized and scanned using the TU to produce serialized transform coefficients for entropy coding.

[0079] In some aspects, after intra-prediction decoding or inter-prediction decoding using the PU of the CU, the encoder engine 106 may calculate residual data for the TU of the CU. The PU may include pixel data in the spatial domain (or pixel domain). The TU may include coefficients in the transform domain after applying the block transform. As previously noted, the residual data may correspond to a pixel difference between a pixel of an uncoded picture and a prediction value corresponding to the PU. The encoder engine 106 may form a TU including residual data for the CU, and may then transform the TU to generate transform coefficients for the CU.

[0080] The encoder engine 106 may perform quantization of the transform coefficients. Quantization provides further compression by quantizing the transform coefficients to reduce the amount of data used to represent the coefficients. For example, quantization may reduce the bit depth associated with some or all of the coefficients. In one example, a coefficient having an n-bit value may be rounded down to an m-bit value during quantization, where n is greater than m.

[0081] Once quantization is performed, the decoded video bitstream includes the quantized transform coefficients, prediction information (e.g., prediction mode, motion vectors, block vectors, etc.), segmentation information, and any other suitable data (such as other syntax data). The different elements of the decoded video bitstream can then be entropy encoded by the encoder engine 106. In some examples, the encoder engine 106 can scan the quantized transform coefficients using a predefined scan order to produce a serialized vector that can be entropy encoded. In some examples, the encoder engine 106 can perform adaptive scanning. After scanning the quantized transform coefficients to form a vector (e.g., a one-dimensional vector), the encoder engine 106 can entropy encode the vector. For example, the encoder engine 106 can use context adaptive variable length coding, context adaptive binary arithmetic coding, grammar-based context adaptive binary arithmetic coding, probability interval segmentation entropy coding, or another suitable entropy coding technique.

[0082] The output 110 of the encoding device 104 can transmit the NAL units constituting the encoded video bitstream data to the decoding device 112 of the receiving device on the communication link 120. The input 114 of the decoding device 112 can receive the NAL unit. The communication link 120 may include a channel provided by a wireless network, a wired network, or a combination of a wired network and a wireless network. The wireless network may include any wireless interface or a combination of wireless interfaces, and may include any suitable wireless network (e.g., the Internet or other wide area network, a packet-based network, WiFiTM, radio frequency (RF), ultra-wideband (UWB), WiFi direct connection, cellular, long-term evolution (LTE), WiMaxTM, etc.). The wired network may include any wired interface (e.g., optical fiber, Ethernet, power line Ethernet, Ethernet based on coaxial cable, digital signal line (DSL), etc.). The wired network and / or wireless network may be implemented using various equipment (such as base stations, routers, access points, bridges, gateways, switches, etc.). The encoded video bitstream data may be modulated according to a communication standard (such as a wireless communication protocol) and sent to a receiving device.

[0083] In some examples, the encoding device 104 may store the encoded video bitstream data in a storage device 108. The output end 110 may retrieve the encoded video bitstream data from the encoder engine 106 or from the storage device 108. The storage device 108 may include any of a variety of distributed or locally accessed data storage media. For example, the storage device 108 may include a hard drive, a storage disk, a flash memory, a volatile or non-volatile memory, or any other suitable digital storage medium for storing the encoded video data. The storage device 108 may also include a decoded picture buffer (DPB) for storing reference pictures for use in inter-frame prediction. In a further example, the storage device 108 may correspond to a file server or another intermediate storage device that can store the encoded video generated by the source device. In such a case, a receiving device including a decoding device 112 may access the stored video data from the storage device via streaming or downloading. The file server may be any type of server capable of storing the encoded video data and sending the encoded video data to the receiving device. Example file servers include web servers (e.g., for websites), FTP servers, network attached storage (NAS) devices, or local disk drives. The receiving device can access the encoded video data through any standard data connection (including an Internet connection). This may include a wireless channel (e.g., a Wi-Fi connection), a wired connection (e.g., DSL, cable modem, etc.), or a combination of the two suitable for accessing the encoded video data stored on a file server. The transmission of the encoded video data from the storage device 108 may be streaming transmission, download transmission, or a combination thereof.

[0084] The input end 114 of the decoding device 112 receives the encoded video bitstream data, and the video bitstream data can be provided to the decoder engine 116, or provided to the storage device 118 for later use by the decoder engine 116. For example, the storage device 118 may include a decoded picture buffer (DPB) for storing reference pictures used in inter-frame prediction. The receiving device including the decoding device 112 can receive the encoded video data to be decoded via the storage device 108. The encoded video data can be modulated according to a communication standard (such as a wireless communication protocol) and sent to the receiving device. The communication medium for sending the encoded video data may include any wireless or wired communication medium, such as a radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form a part of a packet-based network such as a local area network, a wide area network, or a global network (such as the Internet). The communication medium may include a router, a switch, a base station, or any other equipment that may be useful for facilitating communication from a source device to a receiving device.

[0085] The decoder engine 116 may decode the coded video bitstream data by entropy decoding (e.g., using an entropy decoder) and extracting elements of one or more decoded video sequences that constitute the coded video data. The decoder engine 116 may then rescale the coded video bitstream data and perform an inverse transform on it. The residual data is then passed to the prediction stage of the decoder engine 116. The decoder engine 116 then predicts a block of pixels (e.g., a PU). In some examples, the prediction is added to the output of the inverse transform (residual data).

[0086] The video decoding device 112 may output the decoded video to a video destination device 122, which may include a display or other output device for displaying the decoded video data to a consumer of the content. In some aspects, the video destination device 122 may be part of a receiving device that includes the decoding device 112. In some aspects, the video destination device 122 may be part of a separate device distinct from the receiving device.

[0087] In some aspects, the video encoding device 104 and / or the video decoding device 112 may be integrated with an audio encoding device and an audio decoding device, respectively. The video encoding device 104 and / or the video decoding device 112 may also include other hardware or software necessary to implement the decoding techniques described above, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic components, software, hardware, firmware, or any combination thereof. The video encoding device 104 and the video decoding device 112 may be integrated as part of a combined encoder / decoder (codec) in the corresponding device.

[0088] exist Figure 1 The example system shown in is an illustrative example that can be used in this article. The technology for processing video data using the technology described herein can be performed by any digital video encoding and / or decoding device. Although generally speaking, the technology of the present disclosure is performed by a video encoding device or a video decoding device, these technologies can also be performed by a combined video encoder-decoder (commonly referred to as a "codec"). In addition, the technology of the present disclosure can also be performed by a video preprocessor. The source device and the receiving device are merely examples of such decoding devices, wherein the source device generates decoded video data for sending to the receiving device. In some examples, the source device and the receiving device can operate in a substantially symmetrical manner so that each of these devices includes a video encoding and decoding component. Therefore, the example system can support one-way or two-way video transmission between video devices, for example, for video streaming, video playback, video broadcasting, or video telephony.

[0089] The present disclosure may generally involve "signaling" certain information (such as syntax elements). The term "signaling" may generally refer to the communication of values ​​of syntax elements and / or other data used to decode encoded video data. For example, video encoding device 104 may signal values ​​of syntax elements in a bitstream. Generally, signaling refers to generating values ​​in a bitstream. As noted above, video source 102 may transmit the bitstream to video destination device 122 in substantially real time or in non-real time (such as may occur when storing syntax elements to storage device 108 for later retrieval by video destination device 122).

[0090] The video bitstream may also include a supplemental enhancement information (SEI) message. For example, a SEI NAL unit may be part of the video bitstream. In some cases, the SEI message may contain information that is not required for the decoding process. For example, the information in the SEI message may not be necessary for the decoder to decode the video picture of the bitstream, but the decoder may use the information to improve the display or processing of the picture (e.g., the decoded output). The information in the SEI message may be embedded metadata. In an illustrative example, the information in the SEI message may be used by the decoder-side entity to improve the visibility of the content. In some instances, certain application standards may mandate the presence of such SEI messages in the bitstream, so that quality improvements can be brought to all devices that comply with the application standards (e.g., carrying a frame packing SEI message for a frame-compatible planar stereoscopic 3DTV video format, where an SEI message is carried for each frame of the video; processing recovery point SEI messages; using a pull-scan rectangle SEI message in DVB; and many other examples).

[0091] As described above, for each block, a motion information set (also referred to herein as motion parameters) may be available. The motion information set contains motion information for forward and backward prediction directions. The forward and backward prediction directions are the two prediction directions of the bidirectional prediction mode, in which case the terms "forward" and "reverse" do not necessarily have a geometric meaning. Instead, "forward" and "reverse" correspond to reference picture list 0 (RefPicList0 or L0) and reference picture list 1 (RefPicList1 or L1) of the current picture. In some examples, when only one reference picture list is available for a picture or slice, only RefPicList0 is available and the motion information for each block of the slice is always forward.

[0092] In some cases, a motion vector is used together with its reference index in a decoding process (e.g., motion compensation). Such a motion vector with an associated reference index is represented as a single prediction set of motion information. For each prediction direction, the motion information may include a reference index and a motion vector. In some cases, for simplicity, it may be assumed that the motion vector itself is referenced in a manner that has an associated reference index. The reference index is used to identify a reference picture in the current reference picture list (RefPicList0 or RefPicList1). The motion vector has a horizontal component and a vertical component that provide an offset from a coordinate position in the current picture to a coordinate in the reference picture identified by the reference index. For example, a reference index may indicate a specific reference picture that should be used for a block in the current picture, and a motion vector may indicate the position of the best matching block (the block that best matches the current block) in the reference picture.

[0093] In H.264 / AVC, each inter-frame macroblock (MB) can be partitioned in four different ways, including: one 16×16MB partition; two 16×8MB partitions; two 8×16MB partitions; and four 8×8MB partitions. Different MB partitions in one MB may have different reference index values ​​(RefPicList0 or RefPicList1) for each direction. In some cases, when the MB is not partitioned into four 8×8MB partitions, it may have only one motion vector for each MB partition in each direction. In some cases, when the MB is partitioned into four 8×8MB partitions, each 8×8MB partition may be further partitioned into sub-blocks, in which case each sub-block may have a different motion vector in each direction. In some examples, there are four different ways to obtain sub-blocks from an 8×8MB partition, including: one 8×8 sub-block; two 8×4 sub-blocks; two 4×8 sub-blocks; and four 4×4 sub-blocks. Each sub-block may have a different motion vector in each direction. Therefore, the motion vector exists at a level equal to or higher than the sub-block.

[0094] In AVC, temporal direct mode can be enabled at the MB level or MB partition level for skip and / or direct mode in B slices. For each MB partition, the motion vector is derived using the motion vector of the block in RefPicList1[0] co-located with the current MB partition. Each motion vector in the co-located block is scaled based on the POC distance.

[0095] Spatial direct mode can also be implemented in AVC. For example, in AVC, direct mode can also predict motion information from spatial neighbors.

[0096] As mentioned above, in HEVC, the largest coding unit in a slice is called a coding tree block (CTB). The CTB contains a quadtree whose nodes are coding units. In the HEVC Main Profile, the size of the CTB can range from 16×16 to 64×64. In some cases, an 8×8 CTB size can be supported. A coding unit (CU) can have the same size as a CTB and be as small as 8×8. In some cases, each coding unit is coded in one mode. When a CU is inter-coded, the CU can be further partitioned into 2 or 4 prediction units (PUs) or can become only one PU when no further partitioning is applied. When two PUs are present in one CU, they can be a rectangle of half the size or have the shape of the CU. 1 / 4 or 3 / 4 size. When a CU is inter-coded, there is one motion information set for each PU. In addition, each PU is coded with a unique inter prediction mode to derive a motion information set.

[0097] For example, for motion prediction in HEVC, there are two inter prediction modes, including merge mode and advanced motion vector prediction (AMVP) mode for prediction units (PUs). Special cases considered as merges are skipped. In AMVP mode or merge mode, a motion vector (MV) candidate list is maintained for multiple motion vector predictors. The motion vector of the current PU and the reference index in merge mode are generated by taking a candidate from the MV candidate list. In some examples, one or more scaling window offsets may be included in the MV candidate list together with the stored motion vector.

[0098] In an example where an MV candidate list is used for motion prediction of a block, the MV candidate list may be constructed separately by an encoding device and a decoding device. For example, the MV candidate list may be generated by an encoding device when encoding a block, and may be generated by a decoding device when decoding a block. Information related to motion information candidates in the MV candidate list (e.g., information related to one or more motion vectors, information related to one or more LIC flags in the MV candidate list that may be stored in some cases, and / or other information) may be signaled between the encoding device and the decoding device. For example, in merge mode, an index value to the stored motion information candidate may be signaled from the encoding device to the decoding device (e.g., in a syntax structure such as a picture parameter set (PPS), a sequence parameter set (SPS), a video parameter set (VPS), a slice header, a supplemental enhancement information (SEI) message transmitted in a video bitstream or separately from a video bitstream, and / or other signaling). The decoding device may construct an MV candidate list and use the signaled reference or index to obtain one or more motion information candidates from the constructed MV candidate list for motion compensated prediction. For example, the decoding device 112 may construct an MV candidate list and use the motion vector (and in some cases, the LIC flag) from the indexed position for motion prediction of the block. In the case of AMVP mode, in addition to the reference or index, a difference or residual value may be signaled as a delta. For example, for AMVP mode, the decoding device may construct one or more MV candidate lists and apply the delta value to one or more motion information candidates obtained using the signaled index value in performing motion compensated prediction of the block.

[0099] In some examples, the MV candidate list contains up to five candidates for merge mode and two candidates for AMVP mode. In other examples, different numbers of candidates may be included in the MV candidate list for merge mode and / or AMVP mode. The merge candidate may contain a motion information set. For example, the motion information set may include motion vectors corresponding to both the reference picture lists (list 0 and list 1) and the reference index. If the merge candidate is identified by the merge index, the reference picture used for the prediction of the current block and the associated motion vector are determined. However, in the AMVP mode for each potential prediction direction from list 0 or list 1, the reference index needs to be explicitly signaled to the MV candidate list together with the MVP index, because the AMVP candidate only contains motion vectors. In AMVP mode, the predicted motion vector can be further refined.

[0100] As can be seen above, the merge candidate corresponds to the full set of motion information, while the AMVP candidate contains only one motion vector for a specific prediction direction and reference index.Candidates for both modes are similarly derived from the same spatial and temporal neighboring blocks.

[0101] In some examples, merge mode allows an inter-predicted PU to inherit one or more identical motion vectors, prediction directions, and one or more reference picture indexes from an inter-predicted PU including a motion data location selected from a set of spatially adjacent motion data locations and one of two temporally co-located motion data locations. For AMVP mode, one or more motion vectors of a PU may be predictively coded relative to one or more motion vector predictors (MVPs) from an AMVP candidate list constructed by an encoder and / or decoder. In some instances, for unidirectional inter prediction of a PU, the encoder and / or decoder may generate a single AMVP candidate list. In some instances, for bidirectional prediction of a PU, the encoder and / or decoder may generate two AMVP candidate lists, one using motion data from spatially and temporally adjacent PUs in a forward prediction direction, and one using motion data from spatially and temporally adjacent PUs in a backward prediction direction.

[0102] Candidates for both modes can be derived from spatial and / or temporal neighboring blocks. Figure 2A and Figure 2B Included is a conceptual diagram illustrating spatial neighbor candidates. Figure 2A Spatial neighboring motion vector (MV) candidates for merge mode are illustrated. Figure 2B Spatial neighboring motion vector (MV) candidates for AMVP mode are illustrated. Spatial MV candidates are derived from neighboring blocks for a specific PU (PU0), but the method of generating candidates from blocks is different for merge mode and AMVP mode.

[0103] In merge mode, the encoder may form a merge candidate list by considering merge candidates from various motion data locations. Figure 2A As shown in Figure 2A The spatially adjacent motion data positions indicated by numbers 0 to 4 in the merge candidate list can be used to derive up to five spatial MV candidates. The MV candidates can be sorted in the merge candidate list in the order indicated by numbers 0 to 4. For example, the positions and order may include: left position (0), top position (1), top right position (2), bottom left position (3), and top left position (4). Figure 2A , block 200 includes PU0 202 and PU1 204. In some examples, when the video coder is to code the motion information of PU0 202 using merge mode, the video coder may add the motion information from spatial neighboring block 210, spatial neighboring block 212, spatial neighboring block 214, spatial neighboring block 216, and spatial neighboring block 218 to the candidate list in the order described above.

[0104] exist Figure 2BIn the AVMP mode shown, adjacent blocks are divided into two groups: the left group including blocks 0 and 1; and the upper group including blocks 2, 3, and 4. Figure 2B , blocks 0, 1, 2, 3, and 4 are labeled as blocks 230, 232, 234, 236, and 238, respectively. Here, block 220 includes PU0 222 and PU1 224, and blocks 230, 232, 234, 236, and 238 represent spatial neighbors of PU0 222. For each group, potential candidates among neighboring blocks that refer to the same reference picture as indicated by the signaled reference index have the highest priority to be selected to form the final candidate for the group. All neighboring blocks may not contain motion vectors pointing to the same reference picture. Therefore, if such a candidate cannot be found, the first available candidate will be scaled to form the final candidate, so that the temporal distance difference can be compensated.

[0105] Figure 3A and Figure 3B Included is a conceptual diagram illustrating temporal motion vector prediction. Figure 3A An example CU 300 including PU0 302 and PU1 304 is illustrated. PU0 302 includes a center block 310 for PU0 302 and a bottom block 306 to the right of PU0 302. Figure 3A Also shown is an external block 308 whose motion information may be predicted from the motion information of PU0 302, as discussed below. Figure 3B A current picture 342 is illustrated that includes a current block 326 for which motion information is to be predicted. Figure 3B Also illustrated are a collocated picture 330 (including a collocated block 324 to a current block 326) to a current picture 342, a current reference picture 340, and a collocated reference picture 332. The collocated block 324 is predicted using a collocated motion vector 320 of a temporal motion vector predictor (TMVP) candidate 322 used as motion information of the current block 326.

[0106] The video coder may add a temporal motion vector predictor (TMVP) candidate (e.g., TMVP candidate 322) to the MV candidate list after any spatial motion vector candidate, if enabled and available. The motion vector derivation process for the TMVP candidate is the same for both merge mode and AMVP mode. However, in some examples, the target reference index of the TMVP candidate in merge mode is always set to zero.

[0107] The main block location for TMVP candidate derivation is the right bottom block 306 outside the collocated PU 304, as shown in FIG. Figure 3A , to compensate for the deviation of the upper and left blocks used to generate spatial neighboring candidates. However, if block 306 is located outside the current CTB (or LCU) row (e.g., Figure 3A308 in PU 306) or if motion information for block 306 is not available, the block is replaced with the center block 310 of PU 302.

[0108] refer to Figure 3B , the motion vector of the TMVP candidate 322 may be derived from the collocated block 324 of the collocated picture 330, indicated in the slice level. Similar to the temporal direct mode in AVC, the motion vector of the TMVP candidate may be subjected to motion vector scaling, which is performed to compensate for the distance difference between the current picture 342 and the current reference picture 340, and the collocated picture 330 and the collocated reference picture 332. That is, the motion vector 320 may be scaled based on the distance difference between the current picture (e.g., current picture 342) and the current reference picture (e.g., current reference picture 340), and the collocated picture (e.g., collocated picture 330) and the collocated reference picture (e.g., collocated reference picture 332) to generate the TMVP candidate 322.

[0109] Other aspects of motion prediction are covered in the HEVC standard and / or other standards, formats or codecs. For example, several other aspects of merge mode and AMVP mode are covered. One aspect includes motion vector scaling. With respect to motion vector scaling, it may be assumed that the value of a motion vector is proportional to the distance of the pictures in presentation time. A motion vector associates two pictures, a reference picture and a picture containing the motion vector (i.e., a containing picture). When one motion vector is used to predict another motion vector, the distance between the containing picture and the reference picture is calculated based on a picture order count (POC) value.

[0110] For a motion vector to be predicted, its associated containing picture and reference picture may both be different. Therefore, a new distance is calculated (based on POC). In addition, the motion vector can be scaled based on these two POC distances. For spatial neighboring candidates, the containing pictures of the two motion vectors are the same, while the reference pictures are different. In HEVC, motion vector scaling is applied to both TMVP and AMVP for spatial and temporal neighboring candidates.

[0111] Another aspect of motion prediction includes artificial motion vector candidate generation. For example, if the motion vector candidate list is incomplete, artificial motion vector candidates are generated and inserted at the end of the list until all candidates are obtained. In merge mode, there are two types of artificial MV candidates: combined candidates derived only for B slices; and zero candidates used only for AMVP if the first type does not provide enough artificial candidates. For each pair of candidates that are already in the candidate list and have the necessary motion information, a bidirectional combined motion vector candidate is derived by combining the motion vector of the first candidate of the picture in reference list 0 and the motion vector of the second candidate of the picture in reference list 1.

[0112] In some implementations, a pruning process may be performed when a new candidate is added or inserted into the MV candidate list. For example, in some cases, MV candidates from different blocks may include the same information. In this case, storing duplicate motion information of multiple MV candidates in the MV candidate list may result in redundancy and reduced efficiency of the MV candidate list. In some examples, the pruning process may eliminate or minimize redundancy in the MV candidate list. For example, the pruning process may include comparing potential MV candidates to be added to the MV candidate list with MV candidates already stored in the MV candidate list. In an illustrative example, the horizontal displacement (ΔX) and vertical displacement (Δy) of the stored motion vector (indicating the position of the reference block relative to the position of the current block) may be compared with the horizontal displacement (Δx) and vertical displacement (Δy) of the motion vector of the potential candidate. If the comparison reveals that the motion vector of the potential candidate does not match any of the one or more stored motion vectors, the potential candidate is not considered a candidate to be pruned and may be added to the MV candidate list. If a match is found based on the comparison, the potential MV candidate is not added to the MV candidate list, thereby avoiding insertion of the same candidate. In some cases, to reduce complexity, only a limited number of comparisons are performed during the pruning process instead of comparing each potential MV candidate with all existing candidates.

[0113] In some decoding schemes such as HEVC, weighted prediction (WP) is supported, in which case a scaling factor (denoted by a), a shift number (denoted by s), and an offset (denoted by b) are used in motion compensation. Assuming that the pixel value in the position (x, y) of the reference picture is p(x, y), p'(x, y) = ((a*p(x, y) + (1 << (s-1))) >> s) + b is used as the prediction value in motion compensation instead of p(x, y).

[0114] When WP is enabled, for each reference picture of the current slice, a flag is signaled to indicate whether WP is applicable to the reference picture. If WP is applied to a reference picture, the WP parameter set (i.e., a, s, and b) is transmitted to the decoder and used for motion compensation from the reference picture. In some examples, in order to flexibly turn on / off WP for luma and chroma components, the WP flag and WP parameters are signaled for luma and chroma components, respectively. In WP, one same WP parameter set is used for all pixels in one reference picture.

[0115] Figure 4A4 is a diagram illustrating an example of neighboring reconstructed samples of a current block 402 and neighboring samples of a reference block 404 for unidirectional inter prediction. A motion vector MV 410 may be coded for the current block 402, where the MV 410 may include a reference index to a reference picture list and / or other motion information for identifying the reference block 404. For example, the MV may include a horizontal component and a vertical component that provide an offset from a coordinate position in the current picture to a coordinate in the reference picture identified by the reference index. Figure 4B 424 and the second reference block 426. In this case, two motion vectors MV0 and MV1 may be coded for the current block 422 to identify the first reference block 424 and the second reference block 426, respectively.

[0116] As previously explained, OBMC is an example motion compensation technique that can be implemented for motion compensation. OBMC can increase prediction accuracy and avoid blocking artifacts. In OBMC, the prediction can be a weighted sum of multiple predictions or include a weighted sum of multiple predictions. In some cases, blocks can be larger in each dimension and can overlap with adjacent blocks in a quadrant manner. Therefore, each pixel can belong to multiple blocks. For example, in some exemplary cases, each pixel can belong to 4 blocks. In this scheme, OBMC can implement four predictions for each pixel that total to a weighted average.

[0117] In some cases, OBMC can be turned on and off using a specific syntax at the CU level. In some examples, there are two directional modes (e.g., top, left, right, bottom, or bottom) in OBMC, including a CU boundary OBMC mode and a sub-block boundary OBMC mode. When the CU boundary OBMC mode is used, the original prediction block of the current CU MV and another prediction block (e.g., "OBMC block") using an adjacent CU MV are mixed. In some examples, the left top sub-block in the CU (e.g., the first or leftmost sub-block on the first / top row of the CU) has a top OBMC block and a left OBMC block, and the other topmost sub-blocks (e.g., other sub-blocks on the first / top row of the CU) may have only a top OBMC block. Other leftmost sub-blocks (e.g., sub-blocks on the first column of the CU on the left side of the CU) may have only a left OBMC block.

[0118] When sub-CU decoding tools (e.g., affine motion compensated prediction, advanced temporal motion vector prediction (ATMVP), etc.) that allow different MVs based on sub-blocks are enabled in the current CU, the sub-block boundary OBMC mode may be enabled. In the sub-block boundary OBMC mode, a separate OBMC block using the MV of a connected adjacent sub-block may be mixed with an original prediction block using the MV of the current sub-block. In some examples, in the sub-block boundary OBMC mode, a separate OBMC block using the MV of a connected adjacent sub-block may be mixed in parallel with an original prediction block using the MV of the current sub-block, as further described herein. In other examples, in the sub-block boundary mode, a separate OBMC block using the MV of a connected adjacent sub-block may be mixed sequentially with an original prediction block using the MV of the current sub-block. In some cases, the CU boundary OBMC mode may be executed before the sub-block boundary OBMC mode, and the predefined mixing order of the sub-block boundary OBMC mode may include top, left, bottom, and right.

[0119] The prediction of the MV based on the neighboring sub-blocks N (eg, the sub-blocks above the current sub-block, the left side of the current sub-block, the bottom side of the current sub-block, and the right side of the current sub-block) can be expressed as P N , and the prediction based on the MV of the current sub-block can be expressed as P C When sub-block N contains the same motion information as the current sub-block, the original prediction block may not be mixed with the prediction block based on the MV of sub-block N. In some cases, P N The 4 rows / columns of samples in P C The same samples were mixed.

[0120] Figure 5 Aspects of motion vector scaling 500 for a temporal merge candidate for use in a processing block are illustrated according to some examples of the present disclosure. In some examples, a temporal merge candidate derivation may be performed in which a merge candidate (e.g., a temporal merge candidate) is added to a merge candidate list. The temporal merge candidate derivation may be performed based on the scaled motion vector. The scaled motion vector may be derived based on a collocated CU included in a collocated reference picture. A reference picture list to be used for deriving the collocated CU may be explicitly signaled in a slice header.

[0121] For example, Figure 5 Depicted are a current picture 510 and a collocated picture 530 that may be associated with a current reference picture 515 and a collocated reference picture 535, respectively. Figure 5 Also depicted are current CU 512 (eg, associated with current picture 510) and collocated CU (eg, associated with collocated picture 530). Figure 5 , deriving or obtaining a scaled motion vector for temporal merge candidate derivation is exemplified in Figure 5Dashed line 511 is depicted, which is scaled using picture order count (POC) distances tb and td from the motion vector of collocated CU 532. In some examples, tb is the POC difference between current reference picture 515 and current picture 510, and td is the POC difference between collocated reference picture 535 and collocated picture 530. The reference picture index of the temporal merging candidate may be set equal to zero.

[0122] Fig. 6A Aspects of collocated motion vector projections 610 and 620 used in processing blocks are illustrated according to some examples of the present disclosure. Figure 6B Aspects of various reference frames that may be used as temporally collocated reference frames for collocated motion vector projection are illustrated.

[0123] The AV1 codec supports up to seven reference frames for inter-frame prediction. Figure 6B As illustrated, AV1 may support a set of reference frames 630 that includes some (or all) of the LAST, LAST2, LAST3, GOLDEN, BWDREF, ALTREF2, and ALTREF reference frames. The set of seven reference frames 630 may be used to perform inter prediction. In some cases, each reference frame in the set of reference frames 630 may be associated with one or more motion vectors. In some aspects, up to three reference frames (e.g., Figure 6B In one illustrative example, up to three reference frames that can be used as temporally collocated reference frames may be selected from a set of five reference frames. The five reference frames from which up to three temporally collocated reference frames are selected may include Figure 6B In some aspects, selection process 632 may be used to select up to three temporally collocated reference frames, wherein the selected temporally collocated reference frames are associated with a set of temporal motion vectors 642. As previously described, motion vectors of collocated reference frames (e.g., temporal motion vector 642) may be used to perform collocated motion vector projection.

[0124] Fig. 6AAn example of collocated motion vector projection 610, 620 is illustrated. In some cases, collocated motion vector projection can be performed based on projecting some (or all) of the temporal motion vectors 642 relative to the current frame and its reference frames. For example, one or more current frame offsets cur_offsets can be used to project one or more temporal motion vectors 642 associated with a start frame 624 (e.g., a reference frame for the current frame 622) relative to the current frame 622. One or more reference frame offsets ref_offset[rf] can be used to additionally project one or more temporal motion vectors 642 associated with the start frame 624 relative to a reference frame 626 (e.g., which is a reference frame for the start frame 624).

[0125] The example of collocated motion vector projection 610 presents a simplified representation of collocated motion vector projection 620 (e.g., current frame 622 may correspond to or be the same as block 612, start frame 624 may correspond to or be the same as block 614, and reference frame 626 may correspond to or be the same as block 616). In one illustrative example, blocks 612, 614, and 616 are 64×64 in size. In some aspects, a projection of a given temporal motion vector in temporal motion vector 642 may be valid when the projection is within the block to which the projection is associated, or when the projection is within one of the horizontally adjacent blocks. For example, when temporal motion vector 642 is projected based on block 614, the given projection may be valid if it is within block 614 (e.g., the block to which the projection is associated) or if it is within block 612 or block 616 (e.g., two horizontally adjacent blocks relative to block 614).

[0126] In some examples, collocated motion vector projection may be performed for up to three collocated reference frames (referred to as REF0, REF1, and REF2). The motion vector associated with REF0 (e.g., obtained from or included in the set of temporal motion vectors 642) may be projected first. In an illustrative example, for a 64×64 block size, the motion vector associated with REF0 may be projected in an 8×8 raster scan order for the entire frame. The valid projection of the REF0 MV may be located within the 64×64 REF0 block or within one of the 64×64 blocks horizontally adjacent to the REF0 block. A projection located outside one of the three 64×64 blocks may be considered invalid. After determining the valid projection of the REF0 MV, the motion vectors associated with REF1 and REF2 may be projected using the same process as described above. In some aspects, the projection of MVs (e.g., the set of temporal motion vectors 642) associated with REF0, REF1, and REF2 may be performed sequentially such that if both projections go to the same location within one of three adjacent 64×64 blocks, the MV projection associated with REF1 will overwrite the MV projection associated with REF0, and if both projections go to the same location, the MV projection with REF2 will overwrite the MV projection associated with REF1. In some cases, sequentially projecting REF0, REF1, and REF2 may be used to provide an MV projection associated with REF2 having the greatest relative priority and an MV projection associated with REF0 having the lowest relative priority, with a subsequent projection overwriting any MV projection already present at the same location.

[0127] FIG. 7A to FIG. 7D An example of collocated motion vector projection performed based on two collocated reference frames (eg, LAST_FRAME and ALT_FRAME) is illustrated. Fig. 7A An example of motion vector references included in LAST_FRAME is illustrated (e.g., before performing motion vector projection). The arrows originating from each of the motion vector references may represent a scaled motion vector that may be used to perform collocated motion vector projection for LAST_FRAME. The end of each arrow represents the block location to which the corresponding motion vector reference within LAST_FRAME will be projected (e.g., based on the scaled motion vector determined for the corresponding motion vector reference within LAST_FRAME).

[0128] Figure 7B Example based on Fig. 7AThe motion vector field generated for LAST_FRAME is projected using the scaled motion vector depicted in FIG. Block positions associated with no motion vector projection (e.g., empty block positions) may be set to zero or may be considered invalid. In some aspects, the projection may be performed by performing concatenated motion vector projection for LAST_FRAME in raster scan order (e.g., as Fig. 7A exemplified) to generate Figure 7B 704. Based on the scaled motion vector associated with MV ref_frame 704, MV ref_frame 704 is also projected to the same position 730, which is now occupied by the projected MV ref_frame 702. Because MV ref_frame 704 is projected at a later point in the raster scan order than MV ref_frame 702, conflicts between multiple MV ref_frames being projected to the same location 730 can be resolved by allowing later projections of MV ref_frame 704 to overwrite earlier projections of MV ref_frame 702. Figure 7B As illustrated, in the final motion vector field projection determined for LAST_FRAME, position 730 is occupied by the projected MV ref_frame 704 .

[0129] Figure 7C A second example of motion vector references and scaled motion vectors associated with a second collocated reference frame ALT_FRAME is illustrated. This can be compared to the projection of LAST_FRAME described above (e.g., Fig. 7A In the same or similar manner as described in the example (exemplified in FIG. 1 ), the motion vector reference included in ALT_FRAME is projected based on the associated scaled motion vector to determine the motion vector field projection for LAST_FRAME (e.g., as Figure 7B exemplified).

[0130] Can Figure 7CThe motion vector references depicted in (e.g., MV ref_frames included in ALT_FRAME) are projected onto the motion vector field projection previously determined for LAST_FRAME, where the motion vector projection associated with ALT_FRAME overwrites any existing projection associated with LAST_FRAME that is located at the same position within the motion vector field projection. Fig.7D The resulting motion vector field projection determined for LAST_FRAME and ALT_FRAME is illustrated in FIG.

[0131] In some examples, (e.g., Figure 7C ) MV ref_frame 714 included in the ALT_FRAME depicted in FIG. 7 is projected to position 730. As previously mentioned, the MV ref_frames included in the ALT_FRAME may be projected to the MV ref_frames determined for the LAST_FRAME and at Figure 7B In the LAST_FRAME motion vector field projection, position 730 is occupied by the LAST_FRAME MV ref_frame 704.

[0132] When ALT_FRAME MV ref_frame 730 is projected to the same position 730 (e.g., during collocated motion vector projection for ALT_FRAME), ALT_FRAME MV ref_frame 730 may overwrite LAST_FRAME MVref_frame 704. In one illustrative example, motion vector projection for ALT_FRAME may be performed in raster scan order (e.g., as described above). Based on the raster scan order, ALT_FRAME MV ref_frame 716 is also subsequently projected to the same position 730 now occupied by ALT_FRAME MV ref_frame 714. The later projected ALT_FRAME MV ref_frame 716 may overwrite the earlier projected ALT_FRAME MV ref_frame 716.

[0133] like Fig.7D As illustrated, the resulting motion vector field projection determined for LAST_FRAME and ALT_FRAME includes only the later projected ALT_FRAME MV ref_frame 716 at position 730. In some aspects, when a third collocated reference frame is used to perform collocated motion vector projection, the same method as described above with respect to FIG. 7A to FIG. 7D The third collocated reference frame is projected onto Fig.7DThe illustrated LAST_FRAME+ALT_FRAME motion vector field projections, where position conflicts (eg, MV projections to already occupied positions within the motion vector field projections) are resolved by overwriting earlier projections with later projections.

[0134] Fig. 8A and Figure 8B Various aspects of collocated motion vector projection based on frame level processing are illustrated. For example, Fig. 8A An example of a frame-level raster scan order 810 that may be used to perform collocated motion vector projection for a current decoded frame 802 is illustrated. The first set of MVs 812 associated with a first collocated reference frame (e.g., Fig. 8A ), a second set of MVs 814 associated with a second collocated reference frame (e.g., in Fig. 8A Frame1 MV') and a third set of MVs 816 associated with a third collocated reference frame (eg, Fig. 8A The collocated motion vector projection is performed by using the Frame2 MV (indicated as 'Frame2 MV' in the example).

[0135] In one illustrative example, the first collocated reference frame, the second collocated reference frame, and the third collocated reference frame may be associated with or obtained based on the current decoded frame 802. The first set of MVs 812 may be read from memory and retrieved using Fig. 8A 812 to perform collocated motion vector projection for the current decoded frame 802. For example, the same motion vector projection as described above may be performed for each MV in the first set of MVs 812 projected in the frame level raster scan order depicted in FIG. 6A to 7D In some aspects, the projected motion vectors (e.g., motion vector field projections) determined based on projecting the first set of MVs 812 in frame-level raster scan order may be written or otherwise stored in a buffer or memory associated with a decoding device used to decode the current frame 802.

[0136] For example, Figure 8BAn example diagram illustrating a system 820 implementing collocated motion vector projection with a frame level raster scan order 810. REF0 frame MV 812 is read from a DDR (e.g., memory) 840 and used to generate REF0 frame MV projections 852. In some aspects, REF0 frame MV 812 and the first set of MVs 812 may be the same. REF0 MV 812 may be read from the memory 840 and sequentially projected using the frame level raster scan order 810. As each REF0 MV 812 is projected, the resulting motion vector projection may be written to a projected MV buffer 825 included in the memory 840.

[0137] After all of the REF0 MVs 812 have been projected and written to the projected MV buffer 825, the example system 820 may then repeat the same process for the REF1 MV 814. The REF1 MV 814 may be used to determine the REF1 frame MV projections 854, again using the frame level raster scan order 810. Each entry of the REF1 frame MV projections 854 may be written to the projected MV buffer 825, wherein the REF1 frame MV projection entry overwrites any REF0 MV projection entry already stored in the projected MV buffer 825 at the same location (e.g., by overwriting the REF1 frame MV projection entry as described above with respect to the REF1 frame MV projection). FIG. 7A to FIG. 7D After all REF1 MVs 814 have been projected and written to the projected MV buffer 825 , the example system 820 may then repeat the same process again for REF2 MVs 816 .

[0138] In some cases, the example system 820 performs three consecutive raster scans, each over the entire frame 802, by using a frame-level raster scan order 810 to perform each of three consecutive motion vector projections (e.g., for REF0 MV 812, REF1 MV 814, and REF2 MV 816). The projected MV buffer 825 must additionally be sized to store MV projection entries over the entire frame 802 because the frame 802 cannot be decoded until the motion vector projection has been completed for all three collocated reference frames (e.g., REF0, REF1, REF2).

[0139] In some aspects, sequentially performing motion vector projection for each of the collocated reference frames associated with the currently decoded frame 802 may consume a significant number of computational cycles (e.g., clock cycles of one or more processors, etc.) when beginning to decode the frame 802. For example, a significant number of clock cycles may be consumed when the example system 802 waits for frame-level raster scan order motion vector projection to be sequentially completed for three collocated reference frames REF0, REF1, and REF2. In some cases, the sequential process of projecting MVs associated with REF0, REF1, and REF2 is itself sequential with the decoding of the frame 802, which may introduce further inefficiencies and / or delays into the MV projection-based decoding of the frame 802.

[0140] like Figure 8B As illustrated, performing collocated motion vector projection by processing three collocated reference frames in frame-level raster scan order may be associated with four read operations and four write operations for collocated data. For example, the four read operations may include three read operations for reading REF0, REF1, and REF2 MV data (e.g., 812, 814, and 816, respectively) before performing corresponding MV projection for each reference frame and a fourth read operation for reading the final contents of the projected MV buffer 825 to perform MV prediction at the MV prediction hardware 850. The four write operations may include three write operations for writing the REF0, REF1, and REF2 motion vector projections to the projected MV buffer 825 and a fourth write operation for writing the final MV prediction for the current frame 802 from the MV prediction hardware 850 to the memory 840.

[0141] There is a need to reduce the number of clock cycles used to perform collocated motion vector projection for one or more (e.g., up to three) collocated reference frames. There is a need to reduce the read / write bandwidth associated with decoding frames of video data using collocated motion vector projection. There is also a need to reduce the size (e.g., storage capacity) of one or more buffers associated with storing projected motion vectors determined when performing collocated motion vector projection.

[0142] The systems and techniques described herein may be used to perform on-the-fly collocated motion vector projection according to a block-based processing order. As will be described in more depth below, the systems and techniques may be used to read and project reference frame motion vectors at a block level. In one illustrative example, in a scenario where a current decoded frame of video data is associated with three collocated reference frames (e.g., REF0, REF1, REF2), the associated motion vector data included in the three collocated reference frames may be read and projected at a 64×64 block level. In some aspects, the associated motion vector data included in the collocated reference frames may be read and projected at a 128×128 block level. In some aspects, the systems and techniques may avoid writing the projected motion vectors to a DDR or other memory (e.g., as in a 4K block) based on using the block-based processing order described herein for performing collocated motion vector projection. Figure 8B In one illustrative example, systems and techniques may include one or more MV projection buffers implemented in hardware, wherein the one or more MV projection buffers are sized to three times the size of the block level (e.g., for a 64×64 block level, the one or more MV projection buffers may be sized to store three 64×64 blocks; for a 128×128 block level, the one or more MV projection buffers may be sized to store three 128×128 blocks).

[0143] Fig. 9A 9 is a diagram illustrating an example of a block-based (e.g., block-level) process 910 that may be used to more efficiently perform collocated motion vector projection. For example, a current decoded frame of video data may be divided into a plurality of blocks, and collocated motion vector projection may be performed for each block of the current decoded frame of video data. Fig. 9A In the example of block-based processing, three blocks of video data (e.g., 902a, 902b, and 902c) are illustrated. In some cases, a greater or lesser number of blocks may be utilized per decoded frame of video data. Blocks 902a-902c are illustrated as having a size of 64×64, but other block sizes (e.g., such as 128×128) may also be utilized.

[0144] In one illustrative example, the MV information block may be obtained from one or more collocated reference frames using the same block level (e.g., block size) associated with the video data blocks 902a-902c. The collocated reference frame may be associated with the currently decoded frame of video data and may include the information described above with respect to Figure 6B One or more of the described example collocated reference frames. In one illustrative example, the one or more collocated reference frames may include a first collocated reference frame REF0, a second collocated reference frame REF1, and a third collocated reference frame REF2.

[0145] The MV information associated with the collocated reference frame may be read and projected at the same block level (e.g., block size) associated with the video data block included in the currently decoded frame. For example, if the size of the video data blocks 902a-902c is 64×64, the first block 902a of video data may be decoded based on reading the 64×64 MV information block from the collocated reference frame. The 64×64 MV information block obtained for the collocated reference frame may be projected into a single 64×64 motion vector field projection generated for the first block 902a of video data, and the first block 902a of video data may be decoded based on determining that the projection of the associated 64×64 motion vector field projection has been completed.

[0146] In one illustrative example, a given block of video data (e.g., first block 902a) may be decoded in parallel with a collocated motion vector projection operation performed on one or more additional and / or subsequent blocks of video data. For example, when the collocated motion vector projection operation has been completed for the first block 902a and the second block 902b, the first block 902a may be decoded (e.g., the first block 902a may be decoded in parallel with the collocated motion vector projection operation performed on the third block 902c). When the collocated motion vector projection operation has been completed for the second block 902b and the third block 902c, the second block 902b may be decoded (e.g., the second block 902b may be decoded in parallel with the collocated motion vector projection operation performed on the fourth block). For example, a given block of video data may be decoded based on a determination that MV projection has been completed for the given block and adjacent or neighboring blocks of the given block, wherein decoding is performed for subsequent non-adjacent blocks of video data independently of the MV projection operation.

[0147] like Fig. 9A As illustrated, the systems and techniques may perform block-based collocated MV projection for each block (e.g., 902a-902c) based on the raster scan order applied within each block. For example, to decode the 64×64 block 902a of video data, a 64×64 block of MVs associated with each collocated reference frame (e.g., REF0, REF1, REF2) may be obtained and the MVs may be projected in the order of the raster scan applied within each block. Fig. 9A. The blocks of REF0 MV 912 may be projected in a raster scan order to generate REF0 block projections 952, which may be written to or stored in one or more buffers of MV projection buffer 970. After the blocks of REF0 MV 912 have been projected into MV projection buffer 970 (e.g., as REF1 block projections 952), the blocks of REF1 MV 914 may be projected in a raster scan order to generate REF1 block projections 954, which may also be written to or stored in one or more buffers of MV projection buffer 970.

[0148] By writing (e.g., projecting) REF1 block projection 954 into a buffer (e.g., included in MV projection buffer 970) that already includes REF0 block projection 952, systems and techniques can generate a combined MV field projection for REF1 and REF0, such as Figures 10 to 13 In a combined MV field projection for REF1 and REF0, a REF1 MV projection having the same position as an existing REF0 MV projection will overwrite the existing REF0 MV projection (e.g., as previously described with respect to FIG. 7A to FIG. 7D described).

[0149] After the block of REF1 MV 914 has been projected into MV projection buffer 970 (e.g., as REF1 block projection 954), the block of REF2 MV 916 may be projected in raster scan order to generate REF2 block projection 956. REF2 block projection 956 may also be written to or stored in one or more buffers of MV projection buffer 970, overwriting any REF0 and / or REF1 MV projections present at the same location as the REF2 MV projection.

[0150] In some aspects, one or more MV buffers included in the MV projection buffer 970 may be implemented as a sliding buffer, where the current decoded block of video data is decoded from a first buffer of the sliding MV buffer 970. After decoding the current block (e.g., the first block 902a of video data) from the first buffer of the sliding MV buffer 970, the contents of the first buffer may be invalidated (e.g., deleted, cleared, zeroed, etc.). The next block of video data (e.g., the second block 902b of video data) may be decoded based on sliding the sliding MV buffer 970 by a step size. For example, after invalidating the contents of the first buffer, the sliding MV buffer 970 may slide to the right by a step size. In some aspects, the old second buffer becomes the new first buffer from which the next block of video data (e.g., the second block 902b of video data) will be decoded; the old third buffer becomes the new second buffer; and the old first buffer (e.g., now empty after invalidating its contents) becomes the new third buffer.

[0151] In one illustrative example, a first buffer of sliding MV buffer 970 may be used to obtain an MV field projection for decoding a current decoded block of video data based on determining that the first buffer includes an MV projection associated with a current block of video data and includes MV projections for two horizontally neighboring blocks. For example, when the current decoded block of video data is block 902a, the first buffer in sliding MV buffer 970 may be used to decode block 902a when the first buffer includes an MV projection associated with block 902a (e.g., an MV projection associated with the current block of video data) and includes an MV projection associated with block 902b (e.g., an MV projection associated with a horizontally neighboring block to the right).

[0152] When the current decoded block of video data is block 902b, the first buffer in the sliding MV buffer 970 may be used to decode block 902b when the first buffer includes an MV projection associated with block 902b (e.g., an MV projection associated with the current block of video data), includes an MV projection associated with block 902a (e.g., an MV projection associated with a horizontally neighboring block on the left), and includes an MV projection associated with block 902c (e.g., an MV projection associated with a horizontally neighboring block on the right).

[0153] In some aspects, systems and techniques may be used to perform just-in-time collocated MV projection based on determining that a first buffer of a sliding MV buffer 970 includes MV field projections of a current block and two horizontally neighboring blocks. In such examples, the MV field projections stored in the first buffer of the sliding MV buffer 970 may be provided as input to an MV prediction unit 960, which generates a final MV and writes the final MV to a memory 940. Based on the final MV determined using the MV prediction unit 960, the current block of video data may be decoded just-in-time (e.g., before MV projection has been completed for any of the collocated reference frames and / or the currently decoded frame of the video data). As will be described below with respect to Fig.10 As described in greater depth, a current block of video data may be decoded in parallel with performing MV projection for a subsequent (eg, not yet decoded) block of video data.

[0154] For example, Fig.10 1002a), block B (e.g., block 1002b), block C (e.g., block 1002c), and block D (e.g., block 1002d). In some examples, the block level (e.g., block size) may be 64×64. The sliding MV buffer 1070 includes a first buffer 1071, a second buffer 1072, and a third buffer 1073. In an illustrative example, the size of each buffer 1071 to 1073 included in the sliding MV buffer 1070 may be set to store a 64×64 block of data. As will be described below, the size of each buffer 1071 to 1073 may be set to store a 64×64 block of MV projection.

[0155] The multiple blocks of video data included in the frame of video data 1002 may be scanned in raster order or processed in various other scan orders across the frame of video data 1002. Fig.10 As illustrated in the example of , collocated MV projection may be performed for frames of video data 1002 using a projection and decoding order of block A, block B, block C, block D, etc. A currently decoded frame of video data 1002 may be associated with one or more collocated reference frames (not shown), as has been previously described. In some examples, a currently decoded frame of video data 1002 may be associated with up to three collocated reference frames.

[0156] Fig.10The illustrated block-based collocated MV projection 1000 may apply the same 64×64 block-level scan to obtain a block of MVs from each of the three collocated reference frames associated with the current decoded frame of the video data 1002. For example, a 64×64 block of MVs included within block A of each of the three collocated reference frames (e.g., REF0, REF1, and REF2) may be obtained and projected into the sliding MV buffer 1070. In one illustrative example, three 64×64 blocks A MVs may be projected into a first buffer 1071 and a second buffer 1072 of the sliding MV buffer 1070. Because the three buffers 1071 to 1073 of the sliding MV buffer 1070 each store one 64×64 block of video data, in some aspects, each buffer 1071 to 1073 may store a different MV projection field.

[0157] For example, the block A MVs obtained from REF0 may be projected in the raster scan order of block A, where each given block A REF0 MV is projected into either the first buffer (e.g., corresponding to the MV projection field determined for decoding the block A video data 1002a) or the second buffer (e.g., corresponding to the MV projection field determined for decoding the block B video data 1002b), but not both. Fig. 6A and Figure 6B As described, the valid MV projection for a given block may be located in the same block or in one of the two horizontally adjacent blocks, wherein the invalid MV projection is located outside of these three blocks. Since block A is the first block for which MV projection is performed, the horizontally adjacent block on the left does not exist, and therefore the only valid MV projection location for block A will be the first buffer 1071 or will be the second buffer 1072.

[0158] After the REF0 block A MV is projected into the first buffer 1071 and the second buffer 1072, the REF1 block AMV is obtained and also projected into the first buffer 1071 and the second buffer 1072. The REF1 MV projection projected to a location where a REF0 MV projection already exists will overwrite the existing REF0 MV projection (e.g., as previously described with respect to FIG. 7A to FIG. 7D Subsequently, after the REF1 block A MV projection has been completed, the REF2 block A MV is obtained and projected again into the first buffer 1071 and the second buffer 1072, again overwriting any existing REF0 or REF1 projection already present at the same position as the REF2 MV projection.

[0159] When the REF2 block A MV projection has been completed, the first buffer 1071 includes MV projections from REF0 block A, REF1 block A, and REF2 block A that are projected to locations within block A. The second buffer 1072 includes MV projections from REF0 block A, REF1 block A, and REF2 block A that are projected to locations within horizontally adjacent blocks to the right of block A (e.g., projected into block B). The third buffer 1073 remains empty or otherwise does not include block A projections.

[0160] The contents of the first buffer 1071 (e.g., the MV projection field of the block A collocated reference frame MV into block A) are not yet available for decoding the block A video data 1002a. In order to decode the block A video data 1002a, the combined MV projection fields of the block A collocated reference frame and the block B collocated reference frame into block A are needed.

[0161] After the MV projection for block A is completed, the MV included in block B of each of the three collocated reference frames (e.g., REF0, REF1, and REF2) may be obtained and projected into buffers 1071, 1072, and 1073 in the same or similar manner as described above with respect to the MV projection for block A. When the MV projection for block B is performed, the REF0, REF1, and REF2 MVs projected from block B to block A (e.g., a horizontally adjacent block on the left) are written to the first buffer 1071. The REF0, REF1, and REF2 MVs projected from block B to another position within block B are written to the second buffer 1072. The REF0, REF1, and REF2 MVs projected from block B to block C (e.g., a horizontally adjacent block on the right) are written to the third buffer 1073.

[0162] The first buffer 1071 now includes the MV projection fields of the three block A reference frames MV projected into block A and the three block B reference frames MV projected into block A (e.g., the first buffer 1071 includes the MV projection fields of the block A and block B reference frames MV into block A). The MV projection fields of the first buffer 1071 can be used to decode the block A video data 1002a.

[0163] After decoding the block A video data 1002a, the MV projection fields of block A (e.g., the projections of the block A reference frame MV and the block B reference frame MV into block A) may be invalidated, cleared, zeroed, etc. The sliding MV buffer 1070 may then slide to the right by one step based on reassigning the old second buffer 1072 as the new first buffer 1071. The old third buffer 1073 may be reassigned as the new second buffer 1072, and the old first buffer 1071 (e.g., now empty) may be reassigned as the new third buffer 1073. In an illustrative example, the contents of the buffers included in the sliding MV buffer 1070 are not modified during the buffer reassignment. For example, before sliding the sliding MV buffer 1070, the old second buffer 1072 includes the projection of the block A reference frame MV into block B and the projection of the block B reference frame MV into block B. After sliding the sliding MV buffer 1070 and reassigning the old second buffer 1072 as the new first buffer 1071 , the new first buffer 1071 still includes the same projection of block A reference frame MV into block B and the projection of block B reference frame MV into block B.

[0164] As described above, after sliding the sliding MV buffer 1070 to the right by one step, the MV included in the block C of each of the three collocated reference frames (e.g., REF0, REF1, and REF2) may be obtained and projected into the buffers 1071, 1072, and 1073. In one illustrative example, because the sliding MV buffer 1070 slides to the right by one step before projecting the block C MV, the block C MV is projected into the newly reassigned first buffer 1071 (e.g., the old second buffer 1072 from the previous step), the newly reassigned second buffer 1072 (e.g., the old third buffer 1073 from the previous step), and the newly reassigned third buffer 1073 (e.g., the old first buffer 1071 cleared / invalidated in the previous step).

[0165] The new first buffer 1071 now contains the MV projection fields of the three block A reference frames projected into block B, the three block B reference frames projected into block B, and the three block C reference frames projected into block B (e.g., the new first buffer 1071 includes the combined MV projection fields of the three reference frames MV of block A, the three reference frames MV of block B, and the three reference frames MV of block C, all of which are projected into block B). The combined MV projection fields of the new first buffer 1071 can be used to decode the block B video data 1002b.

[0166] After decoding the block B video data 1002b, the MV projection field of the block B stored in the new first buffer 1071 may be invalidated, cleared, zeroed, etc., and the sliding MV buffer 1070 may then slide to the right by one step. After the new first buffer 1071 clears the block B MV projection field (e.g., after decoding the block B video data 1002b), the buffer 1071 may be reassigned as a new third buffer 1073 for decoding the block C video data 1002c. The second buffer 1072 may be reassigned as a new first buffer 1071 for decoding the block C video data 1002c, and the third buffer 1073 may be reassigned as a new second buffer 1072 for decoding the block C video data 1002c. The process may then be repeated for block C video data 1002c, block D video data 1002d, and so on until MV projection has been performed on all blocks included in the plurality of blocks used to partition frame 1002 (e.g., the process described above for decoding blocks A, B, and C using the sliding MV buffer 1070 may be repeated for all remaining 64×64 blocks included in the currently decoded frame of video data 1002).

[0167] In one illustrative example, the row index of a projected MV determined for a block (e.g., 64×64) of collocated reference frame MV data may be used to modify an overwrite of an existing projected MV entry stored in a sliding MV buffer (e.g., sliding MV buffer 1070). For example, the row index of a projected MV associated with the same position as an existing projected MV may be compared to the row index of the existing projected MV to determine whether an overwrite should be performed. In some aspects, the row index-based comparison of the projected MVs may be used to obtain a final MV projection field that is generated using the block-level MV projection previously described, but with the overwrite of the existing MV projection implemented as if a frame-level raster scan order MV projection were performed.

[0168] For example, Fig.14 An example diagram 1400 illustrates a process that may be used to modify an MV projection override associated with a block-level MV projection to be the same as an MV projection override associated with a frame-level raster scan order MV projection.

[0169] In the existing frame level raster scan order MV projection (e.g., such as described above with respect to Fig. 8A and Figure 8BIn the frame-level raster scan order MV projection described above, the projection of an MV with a higher row index may always overwrite a previously projected MV with a lower row index. For example, in the frame-level raster scan order method, an MV with a higher row index (e.g., closer to the bottom of the current decoded frame) is processed after all MVs with lower row indices (e.g., closer to the top of the current decoded frame). Therefore, in the frame-level raster scan order method, an MV with a higher row index may always overwrite a previously projected MV with a lower row index.

[0170] However, when a block-level MV projection method is implemented, an MV associated with a second block (e.g., horizontally adjacent to the right side of the first block) will overwrite a previously projected MV located in the first block, even if the projected MV from the second block has a lower row index than the previously projected MV located in the first block. Fig.14 Illustrated are a first 64x64 block 1410 and a horizontally adjacent second 64x64 block 1420. An associated first 64x64 projection space 1415 and second 64x64 projection space 1425 are also depicted.

[0171] The MVs of the first projection block 1410 are projected first, for example, using a raster scan order within the first projection block 1410. As illustrated, MVs A, B, and C can each be projected into the first projection space 1415 without overwrite conflicts. MV D is projected into the same position within the first projection space 1415 as the previously projected MV B, which results in an overwrite conflict that can be evaluated according to the systems and techniques described herein.

[0172] Based on the row index of MV D (eg, row index=4) being greater than the row index of MV B (eg, row index=3), the projection of MV D may overwrite the existing projection of MV B in the first projection space 1415 .

[0173] MV E can then be projected from the first projection block 1410 to the second projection space 1425 without an overwrite conflict. The projection of MV F from the second projection block 1420 to the first projection space 1415 results in an overwrite conflict because MV F is projected to the same location as an already existing projection of MV C (e.g., the same location as the first projection space 1415). In this example, the row index of MV F (e.g., row index=1) is less than the row index of MV C (e.g., row index=4), and the overwrite conflict can be resolved by maintaining the existing projection of MV C (e.g., by preventing MV F from overwriting the existing projection of MV C).

[0174] Since MV G is projected to the same location as the existing projection of MV A, another overwrite conflict is caused by the projection of MV G from the second projection block 1620 to the first projection space 1615. Here, the row index of MV G (e.g., row index=4) is greater than the row index of MV A (e.g., row index=0), and the overwrite conflict is resolved by allowing the projection of MV G to overwrite the existing projection of MV A in the first projection space 1415.

[0175] In some aspects, the systems and techniques described herein can implement the block-based MV projection buffer as a ring buffer (e.g., rather than a Fig.10 The illustrated sliding MV buffer 1070). For example, Fig.11 1 is a diagram illustrating an example of block-based collocated MV projection 1100 using a circular MV buffer 1170. In this example, the three reference frame MV projections associated with block A (e.g., REF0 block A, REF1 block A, and REF2 block A) can be the same as described above with respect to Fig.10 The three reference frames MV associated with block B are projected into the first buffer 1171 and the second buffer 1172 in the same manner as described above. Fig.10 The same manner as described is projected into the first buffer 1171, the second buffer 1172 and the third buffer 1173 respectively.

[0176] After the MV projection for blocks A and B has been completed, the first buffer 1171a may be used to decode the block A video data 1102a, as also described above with respect to Fig.10 However, after decoding the block A video data 1102a and invalidating the contents of the first buffer 1171a, the now empty first buffer 1171a is not reassigned as a new third buffer (e.g., the MV buffer 1170 does not slide like the sliding MV buffer 1070). Instead, the MV buffer 1170 may be implemented as a ring buffer.

[0177] As illustrated, three reference frame MV projections associated with block C (e.g., REF0 block C, REF1 block C, and REF2 block C) may be projected into the second buffer 1172, the third buffer 1173, and the first buffer 1171, respectively, in that order. Block B video data 1102b may then be decoded from the second buffer 1172, and the second buffer 1172 may be cleared after successful decoding. Three reference frame MV projections associated with block D may then be projected into the third buffer 1173, the first buffer 1171, and the second buffer 1172, respectively, in that order. Block C video data 1102c may then be decoded from the third buffer 1173, and the third buffer 1173 may be cleared after successful decoding.

[0178] Subsequently, three reference frame MV projections associated with the additional block E (e.g., horizontally adjacent to the right side of block D, not shown) may be projected into the first buffer 1171, the second buffer 1172, and the third buffer 1173, respectively, in that order. Block D video data 1102d may then be decoded from the first buffer 1171, and the first buffer 1171 may be cleared after successful decoding. The above process may then be repeated for the remaining blocks included in the current decoded frame 1102. In some aspects, the above description of Fig.14 The row index-based overwrite conflict control described above still works with Fig.11 The illustrated block-level concatenated MV projections are used together with a ring buffer implementation.

[0179] In another aspect, systems and techniques can implement a block-based MV projection buffer using a ping-pong memory buffer. For example, Fig.12 is a diagram illustrating an example of block-based collocated MV projection 1200 using a ping-pong MV buffer 1270. In this example, three reference frame MV projections associated with block A (e.g., REF0 block A, REF1 block A, and REF2 block A) may each be projected into a first ping buffer and a second ping buffer (ping_1 and ping_2, respectively), and may additionally each be projected into a first pong buffer and a second pong buffer (pong_1 and pong_2, respectively).

[0180] The three reference frames MV projections associated with block B may be projected into the first, second, and third ping buffers (e.g., ping_1, ping_2, and ping_3, respectively), and may additionally be projected into the first, second, and third pong buffers (e.g., pong_1, pong_2, and pong_3, respectively). Block A video data 1202a may be decoded from the ping_1 buffer.

[0181] Subsequently, three reference frame MV projections associated with block C may be projected into the first, second, and third ping buffers (e.g., ping_1, ping_2, and ping_3, respectively), and may additionally be projected into the first, second, and third pong buffers (e.g., pong_1, pong_2, and pong_3, respectively). Block B video data 1202 b may be decoded from the pong_2 buffer. After decoding block B video data 1202 b, block A projections may be invalidated (e.g., cleared) from all buffers of MV buffer 1270 (e.g., the block A projections may be invalidated from all ping buffers and all pong buffers).

[0182] Then, the three reference frame MV projections associated with block D may be projected into the first, second, and third ping buffers (e.g., ping_1, ping_2, and ping_3, respectively), respectively, and may additionally be projected into the first, second, and third pong buffers (e.g., pong_1, pong_2, and pong_3, respectively). Block C video data 1202c may be decoded from the ping_3 buffer. After decoding block C video data 1202c, block B projections may be invalidated (e.g., cleared) from all buffers of the MV buffer 1270 (e.g., the block B projections may be invalidated from all ping buffers and all pong buffers).

[0183] Then, three reference frame MV projections associated with the additional block E (e.g., horizontally adjacent to the right side of block D, not shown) may be projected into the first, second, and third ping buffers (e.g., ping_1, ping_2, and ping_3, respectively), respectively, and may additionally be projected into the first, second, and third pong buffers (e.g., pong_1, pong_2, and pong_3, respectively). Block D video data 1202d may be decoded from the pong_1 buffer. After decoding the block D video data 1202d, the block D projection may be invalidated (e.g., cleared) from all buffers of the MV buffer 1270 (e.g., the block D projection may be invalidated from all ping buffers and all pong buffers).

[0184] The above process may then be repeated for the remaining blocks included in the current decoded frame 1202. Fig.14 The row index-based overwrite conflict control described above still works with Fig.12 The illustrated block-level concatenated MV projection is used together with the ping-pong MV buffer implementation.

[0185] In another aspect, systems and techniques can implement the block-based MV projection described herein using a block-level scan size of 128×128 (e.g., as opposed to the 64×64 block size described above). Fig.13 As illustrated in the example diagram 1300 of , the frame of the currently decoded video data 1302 may be divided into a plurality of 128×128 blocks. Each 128×128 block of video data may include four 64×64 sub-blocks. For example, frame A video data 1302a may include sub-blocks A1, A2, A3, and A4, each of which is 64×64 in size.

[0186] The three reference frame MV projections associated with the frame A video data 1302a may be read in 128×128 size. The 64×64A1 and A2 portions of the frame A MV may be projected into buffers Up_1, Low_1, respectively, and may be projected into buffers Up_2, Low_2, respectively. The 64×64A3 and A4 portions of the frame A MV may be projected into buffers Up_1, Low_1, respectively; may be projected into buffers Up_2, Low_2, respectively; and may be projected into buffers Up_3, Low_3, respectively.

[0187] The three reference frame MV projections associated with the frame B video data 1302b may be read at 128x128 size. The 64x64 B1 and B2 portions of the frame B MV may be projected into buffers Up_2, Low_2, respectively; may be projected into buffers Up_3, Low_3, respectively; and may be projected into buffers Up_4, Low_4, respectively.

[0188] The block A video data 1302a may then be decoded using the first buffer and the second buffer (e.g., Up_1, Up_2, Low_1, Low_2). In some aspects, the projection of the 64×64 portion of the 128×128 block of video data (e.g., frame A video data 1302a, frame B video data 1302b, etc.) may be performed in reverse N order, and the decoding of the 64×64 portion of the 128×128 block of video data may be performed in Z order. In some cases, performing reverse N order concatenated motion vector projections (e.g., projection of a 64×64 portion or projection of two 64×128 portions split from a 128×128 block of video data) may be associated with a 50% reduction in DDR or other memory bandwidth at vertical tile boundaries. For example, for a 128×128 block of video data, at a vertical tile boundary, the left line buffer read may be 64. In some examples, the A1 through A4 64x64 sub-blocks of block A video data 1302a may be decoded in Z order, starting with A1 , followed by A3 , A2 , and A4 .

[0189] After decoding the four 64x64 sub-blocks of the 128x128 block A video data 1302a, buffers 1 and 2 may be invalidated, cleared, zeroed, etc. For example, buffers Up_1, Up_2, Low_1, and Low_2 may be invalidated. Subsequently, the 64x64 sub-blocks B3, B4 and C1, C3 may be projected into the high and low buffers of buffers 1 to 4, with the old buffers 3 and 4 being reassigned as the new buffers 1 and 2. The old buffers 1 and 2 (now cleared) may be assigned as the new buffers 3 and 4, thereby re-assigning the old buffers 1 and 2 to the new buffers 3 and 4 in the same manner as described above with respect to Fig.10 The sliding buffer can be realized in the same or similar manner as described.

[0190] Fig.15 1 is a flow chart illustrating an example of a process 1500 for encoding or decoding (decoding) video data in accordance with various aspects described herein. At block 1502, process 1500 includes obtaining one or more first sets of collocated motion vector data associated with a first block of video data included in a current frame of video data. For example, the first block of video data may be associated with Figure 4B The current block 422 is shown as follows. Fig.10 One or more of the illustrated blocks 1002a to 1002d, Fig.11 One or more of the illustrated blocks 1102a to 1102d, Fig.12 One or more of the illustrated blocks 1202a-1202d and / or Fig.13 One or more of the illustrated blocks 1302a-1302d may be the same or similar. In some cases, the current frame of video data may be the same as Figure 5 The current picture 510 shown in the example, Fig. 6A The current frame 622 is shown as an example. Fig.10 The illustrated frame 1002, Fig.11 The illustrated frame 1102, Fig.12 The illustrated frame 1202 and / or Fig.13 The illustrated frames 1302 are the same or similar.

[0191] In some examples, the one or more first sets of collocated motion vector data include three first sets of collocated motion vector data obtained from respective three reference frames associated with the current frame of video data. For example, the one or more first sets of collocated motion vector data may include Fig. 8A The illustrated Frame0 MV 812, Frame1 MV 814 and Frame2 MV 816 and / or Fig. 9A One or more of the illustrated Frame0 MV 912, Frame1 MV 914, and Frame2 MV 916. In some examples, the Figure 6BThree corresponding reference frames selected from the various types of reference frames 630 illustrated obtain three first sets of collocated motion vector data (for example, corresponding three reference frames can be obtained for each of Frame0 MV 812 / 912, corresponding three reference frames can be obtained for each of Frame1 MV 814 / 914, and corresponding three reference frames can be obtained for each of Frame2 MV 816 / 916).

[0192] In some cases, the positions of the three first sets of collocated motion vector data within the corresponding three reference frames are the same. The positions of the three first sets of collocated motion vector data within the corresponding three reference frames may be the same as the positions of the first block of video data within the current frame of video data. In some examples, the first block of video data may be a 64×64 block of video data, a 64×128 block of video data, a 128×64 block of video data, a 128×128 block of video data, etc. In some cases, one or more first sets of collocated motion vector data (e.g., three first sets of collocated motion vector data) may have the same size as the first block of video data.

[0193] At block 1504, process 1500 includes projecting each of the one or more first sets of collocated motion vector data into a first projected motion field associated with the first buffer. For example, each of the one or more first sets of collocated motion vector data may be projected into a first projected motion field associated with the first buffer. Fig. 9B The first projected motion field associated with the motion vector prediction buffer 970 is shown. One or more first sets of motion vector data are concatenated therein and Fig. 9A and Fig. 9B In some cases where the first projected motion field is the same or similar to the illustrated Ref-0 frame MV 912, the first projected motion field may be Fig. 9B The illustrated Ref0 block projection 952 is the same or similar.

[0194] wherein one or more first sets of motion vector data are concatenated with Fig.10 In some examples associated with the first block given by the illustrated block 1002a, the first projected motion field may be the same as or similar to the projection 1071. Fig.11 In some cases where the first projected motion field is associated with the first block given by the illustrated block 1102a, the first projected motion field may be the same or similar to the projection 1171. Fig.12 In some examples associated with the first block given in the illustrated block 1202a, the first projected motion field may be the same as or similar to one or more of the projections 1270; etc.

[0195] In some examples, projecting each of the one or more first sets of collocated motion vector data includes splitting each of the one or more first sets of collocated motion vector data into a first 64×128 set of collocated motion vector data and a second 64×128 set of collocated motion vector data (e.g., when the first block size and the collocated motion vector data size are 128×128). In some cases, the first 64×128 set of collocated motion vector data and the second 64×128 set of collocated motion vector data may be non-overlapping. In some aspects, the first 64×128 set of collocated motion vector data and the second 64×128 set of collocated motion vector data may be projected in reverse N order, for example, as Fig.13 exemplified.

[0196] At block 1506, process 1500 includes obtaining one or more second sets of collocated motion vector data associated with a second block of video data included in the current frame of video data. For example, the second block of video data may be associated with Fig.10 One or more of the illustrated blocks 1002a to 1002d, Fig.11 One or more of the illustrated blocks 1102a to 1102d, Fig.12 One or more of the illustrated blocks 1202a-1202d and / or Fig.13 One or more of the illustrated blocks 1302a-1302d are the same or similar.

[0197] In some cases, the second block of video data may be adjacent to the first block of video data. For example, the first block of video data may be Fig.10 The illustrated block is block 1002a, while the second block of video data may be block 1002b; the first block of video data may be Fig.11 The illustrated block 1102a, while the second block of video data may be block 1102b; etc. In some examples, the first block of video data may be Fig.10 The illustrated block is block 1002b, while the second block of video data may be block 1002c; the first block of video data may be Fig.11 Block 1102b is illustrated, while the second block of video data may be block 1102c; and so on.

[0198] In some examples, the one or more second sets of collocated motion vector data include three second sets of collocated motion vector data obtained from the corresponding three reference frames associated with the current frame of video data. For example, the three second sets of collocated motion vector data and the three first sets of collocated motion vector data may be obtained from the same three corresponding reference frames (e.g., the same corresponding three reference frames described above with respect to block 1502 of process 1500).

[0199] In some cases, the positions of the three second sets of collocated motion vector data within the corresponding three reference frames are the same. The positions of the three second sets of collocated motion vector data within the corresponding three reference frames may be the same as the positions of the second block of video data within the current frame of video data. In some examples, the second block of video data may be a 64×64 block of video data, a 64×128 block of video data, a 128×64 block of video data, a 128×128 block of video data, etc. In some cases, one or more second sets of collocated motion vector data (e.g., three second sets of collocated motion vector data) may have the same size as the second block of video data.

[0200] In some examples, the first block of video data and the second block of video data may have the same size. The one or more first sets of collocated motion vector data may have the same size as the one or more second sets of collocated motion vector data (e.g., both may additionally or alternatively be the same size as the first block of video data and the second block of video data).

[0201] At block 1508, process 1500 includes projecting each of the one or more second sets of collocated motion vector data into the first projected motion field associated with the first buffer. For example, each of the one or more second sets of collocated motion vector data may be projected into the first projected motion field associated with the first buffer. Fig. 9B In some aspects, each of the one or more second sets of collocated motion vector data may be projected into the same first projected motion field to which each of the one or more first sets of collocated motion vector data is projected (e.g., as described above with respect to block 1504 of process 1500). Fig. 9A and Fig. 9B In some cases where the Ref-0 frame MV 912 is the same or similar to the illustrated one or more second sets of concatenated motion vector data may be the same as Fig. 9A and Fig. 9B The illustrated Ref-1 frame MV914 is the same or similar.

[0202] wherein one or more first sets of motion vector data are concatenated with Fig.10 In some examples associated with the first block given by the illustrated block 1002a, one or more second sets of concatenated motion vector data may be associated with the first block given by Fig.10As previously described, the first projected motion field may be the same or similar to projection 1071. At block 1504, the collocated MV associated with block 1002a may be projected into first projected motion field 1071 (e.g., to produce Fig.10 At block 1508, the collocated MV associated with block 1002b may be projected on top of the existing projection of the first projected motion field 1071 (eg, generating Fig.10 'Proj A and B' illustrated).

[0203] At block 1510, process 1500 includes decoding the first block of video data based on projecting each of the one or more first sets of collocated motion vector data and each of the one or more second sets of collocated motion vector data into the first projected motion field associated with the first buffer. For example, the collocated MV data associated with block 1002a may be projected into buffer 1071 and the collocated MV data associated with block 1002b may be projected into buffer 1071. Fig.10 The illustrated block 1002a is decoded wherein a projection of collocated MV data associated with block 1002b overwrites a conflicting projection of collocated MV data associated with block 1002a (eg, overwrites a projection at the same location within the first projected motion field).

[0204] In some examples, process 1500 may also include projecting each of the one or more first sets of collocated motion vector data into a second projected motion field associated with a second buffer, wherein the second projected motion field is associated with decoding the second block of video data. For example, collocated MV data associated with block 1002a may be projected into a second projected motion field associated with second buffer 1072, such as Fig.10 The juxtaposed MV data associated with block 1102a may be projected into a second projection motion field associated with a second buffer 1172, such as Fig.11 exemplified; etc.

[0205] In some cases, process 1500 may also include projecting each of the one or more second sets of collocated motion vector data into the second projected motion field associated with the second buffer; and projecting each of the one or more second sets of collocated motion vector data into a third projected motion field associated with a third buffer, wherein the third projected motion field is associated with decoding a third block of video data, the third block of video data being positioned adjacent to the second block of video data. For example, collocated MV data associated with block 1002 b may be projected into the second projected motion field associated with second buffer 1072 and may be projected into the third projected motion field associated with third buffer 1073, as shown in FIG. Fig.10 1102b associated with the block MV data may be projected into the second projection motion field associated with the second buffer 1172, and may be projected into the third projection motion field associated with the third buffer 1173, such as Fig.11 In the above example, the third block of video data may be positioned adjacent to the second block of video data (eg, the third block of video data may be positioned adjacent to the second block of video data). Fig.10 The illustrated block 1002c; Fig.11 The illustrated block 1102c; etc. are the same or similar).

[0206] In some examples, process 1500 may include obtaining one or more third sets of collocated motion vector data associated with the third block of video data included in the current frame of video data; and projecting each of the one or more third sets of collocated motion vector data into the second projected motion field associated with the second buffer. Fig.10 The juxtaposed MV data associated with the illustrated third block 1002c is projected into a second projected motion field associated with the second buffer 1072; Fig.11 The second block of video data may be decoded based on the second projected motion field associated with the second buffer after the collocated MVs associated with the first block, the second block, and the third block have been projected into the second projected motion field associated with the second buffer. For example, after the collocated MVs associated with the first block 1002a, the second block 1002b, and the third block 1002c have each been projected into the second projected motion field associated with the second buffer 1072, the collocated MVs associated with the first block 1002a, the second block 1002b, and the third block 1002c may be decoded based on the second projected motion field associated with the second buffer. Fig.10After the collocated MVs associated with the first block 1102a, the second block 1102b, and the third block 1102c have each been projected into the second projected motion field associated with the second buffer 1172, the second block 1102b may be decoded; Fig.11 The illustrated second block 1102b is decoded; etc.

[0207] In some examples, process 1500 may also include projecting each of the one or more third sets of concatenated motion vector data into the third projected motion field associated with the third buffer; and deleting the first projected motion field from the first buffer based on successfully decoding the first block of video data. Fig.10 The collocated MV data associated with the illustrated third block 1002c may be projected into a third projected motion field associated with the third buffer 1073, and the first projected motion field stored in the first buffer 1071 may be cleared or deleted based on successfully decoding the first block 1002a (e.g., based on an earlier projection of the first block 1002a and the second block 1002b into the first buffer 1071, as described above). In some examples, the process 1500 may also include projecting each of the one or more third sets of collocated motion vector data into a fourth projected motion field, where the fourth projected motion field is written to the first buffer. For example, upon deleting Fig.10 After the contents of the illustrated first buffer 1071, the first buffer 1071 can be used to store the fourth projected motion field. In some examples, the process 1500 can also include deleting the second projected motion field from the second buffer based on successfully decoding the second block of video data. In some cases, the first buffer, the second buffer, and the third buffer can be included in a sliding buffer. In some examples, the first buffer, the second buffer, and the third buffer can be included in a ring buffer or a ping-pong memory.

[0208] In some implementations, the processes (or methods) described herein (including process 1500) may be performed by a computing device or apparatus (such as Figure 1 For example, these processes may be performed by Figure 1 and Fig.16 The encoding device 104 shown in the figure is provided by another video source side device or a video sending device. Figure 1 and Fig.17The decoding device 112 shown and / or performed by another client-side device (such as a player device, a display, or any other client-side device). In some cases, the computing device or apparatus may include one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, and / or other components configured to perform the steps of process 1500.

[0209] In some examples, the computing device may include a mobile device, a desktop computer, a server computer and / or server system, or other types of computing devices. The components of the computing device (e.g., one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, and / or other components) may be implemented in circuits. For example, the components may include electronic circuits or other electronic hardware and / or may be implemented using electronic circuits or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, graphics processing units (GPUs), digital signal processors (DSPs), central processing units (CPUs), and / or other suitable electronic circuits) and / or may include computer software, firmware, or any combination thereof for performing the various operations described herein and / or may be implemented using computer software, firmware, or any combination thereof for performing the various operations described herein.

[0210] In some examples, the computing device or apparatus may include a camera configured to capture video data (e.g., a video sequence) including video frames. In some examples, the camera or other capture device that captures the video data is separate from the computing device, in which case the computing device receives or obtains the captured video data. The computing device may also include a network interface configured to communicate the video data. The network interface may be configured to communicate data based on an Internet Protocol (IP) or other types of data. In some examples, the computing device or apparatus may include a display for displaying output video content (such as samples of pictures of a video bitstream).

[0211] The processes may be described with respect to logical flow diagrams whose operations represent sequences of operations that can be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, each operation represents a computer-executable instruction stored on one or more computer-readable storage media that, when executed by one or more processors, performs the described operations. Generally speaking, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform specific functions or implement specific data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be combined in any order and / or in parallel to implement the processes.

[0212] Additionally, these processes may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented by hardware or a combination thereof as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed together on one or more processors. As noted above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program including a plurality of instructions that can be executed by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0213] The decoding techniques discussed herein can be implemented in an example video encoding and decoding system (e.g., system 100). In some examples, a system includes a source device that provides encoded video data that will be decoded later by a destination device. Specifically, the source device provides video data to the destination device via a computer-readable medium. The source device and the destination device may include any of a variety of devices, including desktop computers, notebook (i.e., laptop) computers, tablet computers, set-top boxes, telephone handsets (such as so-called "smart" phones, so-called "smart" tablets), televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, etc. In some cases, the source device and the destination device may be equipped for wireless communication.

[0214] The destination device may receive the encoded video data to be decoded via a computer-readable medium. The computer-readable medium may include any type of medium or device capable of moving the encoded video data from the source device to the destination device. In one example, the computer-readable medium may include a communication medium that enables the source device to directly send the encoded video data to the destination device in real time. The encoded video data may be modulated according to a communication standard (such as a wireless communication protocol) and sent to the destination device. The communication medium may include any wireless or wired communication medium, such as a radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form a part of a packet-based network such as a local area network, a wide area network, or a global network (such as the Internet). The communication medium may include a router, a switch, a base station, or any other equipment that may be useful for facilitating communication from the source device to the destination device.

[0215] In some examples, the encoded data may be output from the output interface to a storage device. Similarly, the encoded data may be accessed from the storage device via an input interface. The storage device may include any data storage medium in a variety of distributed or locally accessed data storage media, such as a hard drive, a Blu-ray disc, a DVD, a CD-ROM, a flash memory, a volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded video data. In another example, the storage device may correspond to a file server or another intermediate storage device that can store the encoded video generated by the source device. The destination device may access the stored video data from the storage device via streaming or downloading. The file server may be any type of server capable of storing the encoded video data and sending the encoded video data to the destination device. Example file servers include web servers (e.g., for websites), FTP servers, network attached storage (NAS) devices, or local disk drives. The destination device may access the encoded video data via any standard data connection (including an Internet connection). The data connection may include a wireless channel (e.g., a Wi-Fi connection), a wired connection (e.g., a DSL, a cable modem, etc.) suitable for accessing the encoded video data stored on the file server, or a combination of the two. The transmission of the encoded video data from the storage device may be streaming transmission, download transmission, or a combination thereof.

[0216] The techniques of the present disclosure are not necessarily limited to wireless applications or settings. These techniques can be applied to video decoding to support any of a variety of multimedia applications, such as over-the-air television broadcasts, cable television transmissions, satellite television transmissions, Internet streaming video transmissions (such as Dynamic Adaptive Streaming over HTTP (DASH)), digital video encoded on a data storage medium, decoding of digital video stored on a data storage medium, or other applications. In some examples, the system can be configured to support one-way or two-way video transmission to support applications such as video streaming, video playback, video broadcasting, and / or video telephony.

[0217] In one example, the source device includes a video source, a video encoder, and an output interface. The destination device may include an input interface, a video decoder, and a display device. The video encoder of the source device may be configured to apply the technology disclosed herein. In other examples, the source device and the destination device may include other components or arrangements. For example, the source device may receive video data from an external video source such as an external camera. Similarly, the destination device may be docked with an external display device rather than including an integrated display device.

[0218] The above example system is just an example. The technology for parallel processing of video data can be performed by any digital video encoding and / or decoding device. Although generally speaking, the technology of the present disclosure is performed by a video encoding device, these technologies can also be performed by a video encoder / decoder (commonly referred to as a "codec"). In addition, the technology of the present disclosure can also be performed by a video preprocessor. The source device and the destination device are just examples of such decoding devices, wherein the source device generates decoded video data to send to the destination device. In some examples, the source device and the destination device can operate in a substantially symmetrical manner so that each of these devices includes a video encoding and decoding component. Therefore, the example system can support one-way or two-way video transmission between video devices, for example, for video streaming, video playback, video broadcasting, or video telephony.

[0219] The video source may include a video capture device, such as a video camera, a video archive including previously captured video, and / or a video feed interface for receiving video from a video content provider. As another alternative, the video source may generate computer graphics-based data as the source video, or generate a combination of real-time video, archived video, and computer-generated video. In some cases, if the video source is a video camera, the source device and the destination device may form a so-called camera phone or video phone. However, as mentioned above, the technology described in this disclosure is generally applicable to video decoding, and may be applicable to wireless and / or wired applications. In each case, the captured, pre-captured or computer-generated video may be encoded by a video encoder. The encoded video information may then be output to a computer-readable medium by an output interface.

[0220] As noted, computer-readable media may include transient media (such as wireless broadcasts or wired network transmissions) or storage media (i.e., non-transitory storage media) such as hard disks, flash drives, compact discs, digital video discs, Blu-ray discs, or other computer-readable media. In some examples, a network server (not shown) may receive encoded video data from a source device and provide the encoded video data to a destination device, for example, via a network transmission. Similarly, a computing device of a media production facility (such as a disc stamping facility) may receive encoded video data from a source device and produce a disc containing the encoded video data. Thus, in various examples, a computer-readable medium may be understood to include one or more computer-readable media in various forms.

[0221] An input interface of a destination device receives information from a computer-readable medium. The information of the computer-readable medium may include syntax information defined by a video encoder, which is also used by a video decoder, including syntax elements that describe characteristics and / or processing of blocks and other coded units (e.g., groups of pictures (GOPs)). A display device displays the decoded video data to a user and may include any of a variety of display devices, such as a cathode ray tube (CRT), a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, or another type of display device. Various aspects of the present application have been described.

[0222] The details of the encoding device 104 and the decoding device 112 are respectively described in Fig.16 and Fig.17 Shown in. Fig.16 1 is a block diagram illustrating an example encoding device 104 that can implement one or more of the techniques described in the present disclosure. For example, the encoding device 104 can generate the syntax structures described herein (e.g., syntax structures of VPS, SPS, PPS or other syntax elements). The encoding device 104 can perform intra-frame prediction decoding and inter-frame prediction decoding of video blocks within a video slice. As previously described, intra-frame decoding relies at least in part on spatial prediction to reduce or remove spatial redundancy within a given video frame or picture. Inter-frame decoding relies at least in part on temporal prediction to reduce or remove temporal redundancy within adjacent or surrounding frames of a video sequence. Intra-frame mode (I mode) may refer to any of several spatial-based compression modes. Inter-frame mode (such as unidirectional prediction (P mode) or bidirectional prediction (B mode)) may refer to any of several temporal-based compression modes.

[0223] The encoding device 104 includes a segmentation unit 35, a prediction processing unit 41, a filter unit 63, a picture memory 64, a summer 50, a transform processing unit 52, a quantization unit 54, and an entropy coding unit 56. The prediction processing unit 41 includes a motion estimation unit 42, a motion compensation unit 44, and an intra-frame prediction processing unit 46. For video block reconstruction, the encoding device 104 also includes an inverse quantization unit 58, an inverse transform processing unit 60, and a summer 62. The filter unit 63 is intended to represent one or more loop filters, such as a deblocking filter, an adaptive loop filter (ALF), and a sample adaptive offset (SAO) filter. Although the filter unit 63 is not limited to a single-loop filter, the filter unit 63 is not limited to a single-loop filter, and the filter unit 63 is not limited to a single-loop filter. Fig.16 6 is shown as an in-loop filter, but in other configurations, filter unit 63 may be implemented as a post-loop filter. Post-processing device 57 may perform additional processing on the encoded video data generated by encoding device 104. In some examples, the techniques of this disclosure may be implemented by encoding device 104. However, in other examples, one or more of the techniques of this disclosure may be implemented by post-processing device 57.

[0224] like Fig.16 As shown, the encoding device 104 receives video data, and the segmentation unit 35 segments the data into video blocks. The segmentation may also include segmentation into slices, slice segments, tiles or other larger units and video block segmentation (e.g., according to the quadtree structure of LCU and CU). The encoding device 104 generally illustrates the components for encoding the video blocks within the video slice to be encoded. The slice may be divided into multiple video blocks (and may be divided into sets of video blocks called tiles). The prediction processing unit 41 may select one of multiple possible decoding modes for the current video block based on the error result (e.g., decoding rate and distortion level, etc.), such as one of one or more inter-frame prediction decoding modes in multiple intra-frame prediction decoding modes. The prediction processing unit 41 may provide the resulting intra-frame decoded block or inter-frame decoded block to the summer 50 to generate residual block data and to the summer 62 to reconstruct the encoded block for use as a reference picture.

[0225] Intra-prediction processing unit 46 within prediction processing unit 41 may perform intra-predictive coding of the current video block relative to one or more neighboring blocks in the same frame or slice as the current block to be coded to provide spatial compression. Motion estimation unit 42 and motion compensation unit 44 within prediction processing unit 41 may perform inter-predictive coding of the current video block relative to one or more predictive blocks in one or more reference pictures to provide temporal compression.

[0226] Motion estimation unit 42 may be configured to determine an inter-prediction mode for a video slice according to a predetermined pattern for a video sequence. The predetermined pattern may designate a video slice in a sequence as a P slice, a B slice, or a GPB slice. Motion estimation unit 42 and motion compensation unit 44 may be highly integrated but are illustrated separately for conceptual purposes. Motion estimation performed by motion estimation unit 42 is the process of generating motion vectors, which estimate the motion of a video block. For example, a motion vector may indicate the displacement of a prediction unit (PU) of a video block within a current video frame or picture relative to a predictive block within a reference picture.

[0227] A predictive block is a block that is found to closely match a PU of a video block to be coded in terms of pixel difference, which may be determined by a sum of absolute difference (SAD), a sum of squared difference (SSD), or other difference metric. In some examples, encoding device 104 may calculate values ​​for sub-integer pixel positions of a reference picture stored in picture memory 64. For example, encoding device 104 may interpolate values ​​for quarter-pixel positions, eighth-pixel positions, or other fractional pixel positions of a reference picture. Thus, motion estimation unit 42 may perform a motion search relative to full pixel positions and fractional pixel positions and output a motion vector with fractional pixel precision.

[0228] Motion estimation unit 42 calculates a motion vector for a PU by comparing the position of a PU of a video block in an inter-coded slice to the position of a predictive block of a reference picture. The reference picture may be selected from a first reference picture list (List 0) or a second reference picture list (List 1), each of which identifies one or more reference pictures stored in picture memory 64. Motion estimation unit 42 communicates the calculated motion vector to entropy encoding unit 56 and motion compensation unit 44.

[0229] Motion compensation performed by motion compensation unit 44 may involve extracting or generating a predictive block based on a motion vector determined by motion estimation, possibly performing interpolation to sub-pixel precision. Upon receiving the motion vector of the PU of the current video block, motion compensation unit 44 may locate the predictive block pointed to by the motion vector in the reference picture list. The encoding device 104 forms a residual video block by subtracting the pixel values ​​of the predictive block from the pixel values ​​of the current video block being decoded, thereby forming a pixel difference value. The pixel difference value forms the residual data of the block and may include both a luminance difference component and a chrominance difference component. Summer 50 represents one or more components that perform a subtraction operation. Motion compensation unit 44 may also generate syntax elements associated with video blocks and video slices for use by the decoding device 112 when decoding video blocks of the video slices.

[0230] Intra-prediction processing unit 46 may perform intra-prediction on the current block as an alternative to inter-prediction performed by motion estimation unit 42 and motion compensation unit 44, as described above. Specifically, intra-prediction processing unit 46 may determine an intra-prediction mode for encoding the current block. In some examples, intra-prediction processing unit 46 may encode the current block using various intra-prediction modes, for example, during a separate encoding process, and intra-prediction processing unit 46 may select an appropriate intra-prediction mode from the tested modes to use. For example, intra-prediction processing unit 46 may calculate rate-distortion values ​​using rate-distortion analysis for various tested intra-prediction modes, and may select an intra-prediction mode with the best rate-distortion characteristics among the tested modes. Rate-distortion analysis typically determines the amount of distortion (or error) between the encoded block and the original unencoded block that is encoded to produce the encoded block, as well as the bit rate (i.e., the number of bits) used to produce the encoded block. Intra-prediction processing unit 46 may calculate ratios from the distortions and rates for the various encoded blocks to determine which intra-prediction mode exhibits the best rate-distortion value for the block.

[0231] In any case, after selecting an intra-prediction mode for a block, intra-prediction processing unit 46 may provide information indicating the selected intra-prediction mode for the block to entropy encoding unit 56. Entropy encoding unit 56 may encode the information indicating the selected intra-prediction mode. Encoding device 104 may include in the transmitted bitstream configuration data a definition of coding contexts for various blocks and an indication of the most probable intra-prediction mode, an intra-prediction mode index table, and a modified intra-prediction mode index table for each of the contexts. The bitstream configuration data may include a plurality of intra-prediction mode index tables and a plurality of modified intra-prediction mode index tables (also referred to as codeword mapping tables).

[0232] After the prediction processing unit 41 generates a predictive block for the current video block via inter-frame prediction or intra-frame prediction, the encoding device 104 forms a residual video block by subtracting the predictive block from the current video block. The residual video data in the residual block may be included in one or more TUs and applied to the transform processing unit 52. The transform processing unit 52 transforms the residual video data into residual transform coefficients using a transform (such as a discrete cosine transform (DCT) or a conceptually similar transform). The transform processing unit 52 may convert the residual video data from a pixel domain to a transform domain, such as a frequency domain.

[0233] Transform processing unit 52 may pass the resulting transform coefficients to quantization unit 54. Quantization unit 54 quantizes the transform coefficients to further reduce the bit rate. The quantization process may reduce the bit depth associated with some or all of the coefficients. The degree of quantization may be modified by adjusting a quantization parameter. In some examples, quantization unit 54 may then perform a scan of the matrix including the quantized transform coefficients. Alternatively, entropy encoding unit 56 may perform the scan.

[0234] After quantization, entropy encoding unit 56 entropy encodes the quantized transform coefficients. For example, entropy encoding unit 56 may perform context adaptive variable length coding (CAVLC), context adaptive binary arithmetic coding (CABAC), syntax-based context adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or another entropy encoding technique. After entropy encoding by entropy encoding unit 56, the encoded bitstream may be sent to decoding device 112, or archived for later sending or retrieval by decoding device 112. Entropy encoding unit 56 may also entropy encode motion vectors and other syntax elements for the current video slice being coded.

[0235] Inverse quantization unit 58 and inverse transform processing unit 60 apply inverse quantization and inverse transform, respectively, to reconstruct the residual block in the pixel domain for later use as a reference block of a reference picture. Motion compensation unit 44 may calculate a reference block by adding the residual block to a predictive block of one of the reference pictures in a reference picture list. Motion compensation unit 44 may also apply one or more interpolation filters to the reconstructed residual block to calculate sub-integer pixel values ​​for use in motion estimation. Summer 62 adds the reconstructed residual block to the motion compensated prediction block produced by motion compensation unit 44 to produce a reference block for storage in picture memory 64. The reference block may be used by motion estimation unit 42 and motion compensation unit 44 as a reference block for inter-prediction of a block in a subsequent video frame or picture.

[0236] In this way, Fig.16 The encoding device 104 represents an apparatus configured to perform any of the techniques described herein (including the above-mentioned Fig.15 In some cases, some of the techniques of this disclosure may also be implemented by post-processing device 57.

[0237] Fig.17 1 is a block diagram illustrating an example decoding device 112. The decoding device 112 includes an entropy decoding unit 80, a prediction processing unit 81, an inverse quantization unit 86, an inverse transform processing unit 88, a summer 90, a filter unit 91, and a picture memory 92. The prediction processing unit 81 includes a motion compensation unit 82 and an intra-frame prediction processing unit 84. In some examples, the decoding device 112 may perform substantially the same Fig.16 The encoding pass described by the encoding device 104 is a decoding pass that is the inverse of the encoding pass.

[0238] During the decoding process, the decoding device 112 receives an encoded video bitstream representing video blocks and associated syntax elements of an encoded video slice transmitted by the encoding device 104. In some aspects, the decoding device 112 may receive the encoded video bitstream from the encoding device 104. In some aspects, the decoding device 112 may receive the encoded video bitstream from a network entity 79, such as a server, a media-aware network element (MANE), a video editor / splicer, or other such device configured to implement one or more of the above-described techniques. The network entity 79 may or may not include the encoding device 104. Some of the techniques described in the present disclosure may be implemented by the network entity 79 before the network entity 79 sends the encoded video bitstream to the decoding device 112. In some video decoding systems, the network entity 79 and the decoding device 112 may be parts of separate devices, while in other instances, the functionality described with respect to the network entity 79 may be performed by the same device including the decoding device 112.

[0239] The entropy decoding unit 80 of the decoding device 112 entropy decodes the bitstream to generate quantized coefficients, motion vectors and other syntax elements. The entropy decoding unit 80 forwards the motion vectors and other syntax elements to the prediction processing unit 81. The decoding device 112 may receive syntax elements at the video slice level and / or the video block level. The entropy decoding unit 80 may process and parse both fixed length syntax elements and variable length syntax elements in one or more parameter sets (such as VPS, SPS, and PPS).

[0240] When the video slice is decoded as an intra-coded (I) slice, the intra-prediction processing unit 84 of the prediction processing unit 81 may generate prediction data for the video block of the current video slice based on the signaled intra-prediction mode and data from a previously decoded block of the current frame or picture. When the video frame is decoded as an inter-coded (i.e., B, P, or GPB) slice, the motion compensation unit 82 of the prediction processing unit 81 generates a predictive block for the video block of the current video slice based on the motion vector and other syntax elements received from the entropy decoding unit 80. The predictive block may be generated from one of the reference pictures in the reference picture list. The decoding device 112 may construct the reference frame lists (list 0 and list 1) using a default construction technique based on the reference pictures stored in the picture memory 92.

[0241] Motion compensation unit 82 determines prediction information for a video block of the current video slice by parsing the motion vectors and other syntax elements, and uses the prediction information to generate a predictive block for the current video block being decoded. For example, motion compensation unit 82 may use one or more syntax elements in a parameter set to determine a prediction mode (e.g., intra prediction or inter prediction) for coding a video block of a video slice, an inter-prediction slice type (e.g., a B slice, a P slice, or a GPB slice), construction information for one or more reference picture lists for the slice, motion vectors for each inter-coded video block of the slice, an inter-prediction state for each inter-coded video block of the slice, and other information for decoding a video block in the current video slice.

[0242] The motion compensation unit 82 may also perform interpolation based on interpolation filters. The motion compensation unit 82 may calculate interpolated values ​​for sub-integer pixels of the reference blocks using interpolation filters as used by the encoding device 104 during encoding of the video blocks. In this case, the motion compensation unit 82 may determine the interpolation filters used by the encoding device 104 from the received syntax elements and may use the interpolation filters to produce the predictive blocks.

[0243] Inverse quantization unit 86 inverse quantizes or dequantizes the quantized transform coefficients provided in the bitstream and decoded by entropy decoding unit 80. The inverse quantization process may include determining the degree of quantization using quantization parameters calculated by encoding device 104 for each video block in the video slice, and likewise determining the degree of inverse quantization that should be applied. Inverse transform processing unit 88 applies an inverse transform (e.g., an inverse DCT or other suitable inverse transform), an inverse integer transform, or a conceptually similar inverse transform process to the transform coefficients to produce residual blocks in the pixel domain.

[0244] After motion compensation unit 82 generates a predictive block for the current video block based on the motion vector and other syntax elements, decoding device 112 forms a decoded video block by adding the residual block from inverse transform processing unit 88 to the corresponding predictive block generated by motion compensation unit 82. Summer 90 represents one or more components that perform this summation operation. If desired, a loop filter (in the decoding loop or after the decoding loop) may also be used to smooth pixel transitions or otherwise improve video quality. Filter unit 91 is intended to represent one or more loop filters, such as a deblocking filter, an adaptive loop filter (ALF), and a sample adaptive offset (SAO) filter. Although filter unit 91 is Fig.17 91 is shown as an in-loop filter, but in other configurations, filter unit 91 may be implemented as a post-loop filter. The decoded video blocks in a given frame or picture are then stored in picture memory 92, which stores reference pictures for subsequent motion compensation. Picture memory 92 also stores the decoded video for later viewing on a display device such as Figure 1 The video is presented on the video destination device 122 shown.

[0245] In this way, Fig.17 The decoding device 112 represents a device configured to perform any of the techniques described herein (including the above-mentioned Fig.15 An example of a video decoder that does not include the process described herein.

[0246] As used herein, the term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or carrying instructions and / or data. Computer-readable media may include non-transient media in which data may be stored and does not include carrier waves and / or transient electronic signals that are propagated wirelessly or over a wired connection. Examples of non-transient media may include, but are not limited to, disks or tapes, optical storage media (such as compact disks (CDs) or digital versatile disks (DVDs)), flash memory, memory, or memory devices. Computer-readable media may store thereon code and / or machine-executable instructions that may represent procedures, functions, subroutines, programs, routines, subroutines, modules, software packages, categories, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, independent variables, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or sent via any suitable means, including memory sharing, message passing, token passing, network sending, etc.

[0247] In some examples, computer-readable storage devices, media, and memories may include cables or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer-readable storage media expressly exclude media such as energy, carrier signals, electromagnetic waves, and signals themselves.

[0248] Specific details are provided in the above description to provide a detailed understanding of the aspects and examples provided herein. However, it will be understood by those skilled in the art that examples can be implemented without these specific details. For clarity, in some instances, the present technology may be presented as including separate functional blocks, including functional blocks comprising devices, device components, steps or routines in methods embodied in software or a combination of hardware and software. Additional components other than those components shown in the accompanying drawings and / or described herein may be used. For example, circuits, systems, networks, processes and other components may be shown as components in block diagram form so as not to obscure aspects of the present disclosure with unnecessary details. In other instances, known circuits, processes, algorithms, structures and techniques may be shown without unnecessary details to avoid obscuring aspects of the present disclosure.

[0249] The above may describe various examples as processes or methods, which are depicted as flow charts, flow diagrams, data flow diagrams, structure diagrams, or block diagrams. Although a flow chart may describe an operation as a sequential process, many of the operations may be performed in parallel or concurrently. In addition, the order of the operations may be rearranged. The process is terminated when the operations of the process are completed, but the process may have additional steps not included in the accompanying drawings. A process may correspond to a method, function, procedure, subroutine, subprogram, etc. When a process corresponds to a function, the termination of the process may correspond to the function returning to the calling function or main function.

[0250] The processes and methods according to the above examples can be implemented using stored computer executable instructions or otherwise obtained from computer readable media. Such instructions may include, for example, instructions and data that configure a general-purpose computer, a special-purpose computer, or a processing device to perform a certain function or group of functions. Portions of the computer resources used may be accessed via a network. Computer executable instructions may be, for example, binary, intermediate format instructions, such as assembly language, firmware, source code, etc. Examples of computer readable media that can be used to store instructions, information used, and / or information created during the methods according to the described examples include disks or optical disks, flash memory, USB devices with non-volatile memory, networked storage devices, etc.

[0251] Devices implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) for performing the necessary tasks may be stored in a computer-readable or machine-readable medium. The processor may perform the necessary tasks. Typical examples of form factors include laptop computers, smart phones, mobile phones, tablet devices or other small form factor personal computers, personal digital assistants, rack-mounted devices, stand-alone devices, etc. The functionality described herein may also be embodied in peripheral devices or add-in cards. By way of additional examples, such functionality may also be implemented on circuit boards among different chips or different processes executed on a single device.

[0252] The instructions, the media for conveying such instructions, the computing resources for executing them, and other structures for supporting such computing resources are example means for providing the functionality described in this disclosure.

[0253] In the foregoing description, various aspects of the present application are described with reference to the specific examples of the present application, but those skilled in the art will recognize that the present application is not limited thereto. Therefore, although the illustrative examples of the present application have been described in detail herein, it should be understood that the inventive concept can be implemented and adopted in various other ways, and the appended claims are intended to be interpreted as including these variations, unless limited by the prior art. The various features and aspects of the above-mentioned applications can be used individually or in combination. In addition, without departing from the broader spirit and scope of this specification, various aspects of the present disclosure can be utilized in any number of environments and applications beyond those described herein. Therefore, the description and the accompanying drawings should be considered as illustrative rather than restrictive. For the purpose of illustration, each method is described in a specific order. It should be understood that in an alternative example, the method can be performed in an order different from the described order.

[0254] It should be understood by those of ordinary skill in the art that the less than ("<") and greater than (">") symbols or terms used herein may be replaced by less than or equal to ("≤") and greater than or equal to ("≥") symbols, respectively, without departing from the scope of the present specification.

[0255] Where a component is described as being “configured to” perform certain operations, such configuration may be achieved, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., a microprocessor or other suitable electronic circuits) to perform the operations, or any combination thereof.

[0256] The phrase "coupled to" means that any component is directly or indirectly physically connected to another component, and / or any component is directly or indirectly in communication with another component (e.g., connected to another component via a wired or wireless connection and / or other suitable communication interface).

[0257] Claim language or other language in this disclosure that states "at least one of" a set and / or "one or more of" a set indicates that one member of the set or multiple members of the set (in any combination) satisfies the claim. For example, claim language reciting "at least one of A and B" means A, B, or A and B. In another example, claim language reciting "at least one of A, B, and C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language "at least one of" a set and / or "one or more of" a set does not limit the set to the items listed in the set. For example, claim language reciting "at least one of A and B" may mean A, B, or A and B, and may additionally include items not listed in the set of A and B.

[0258] Various illustrative logic blocks, modules, circuits, and algorithmic steps described in conjunction with the examples disclosed herein may be implemented as electronic hardware, computer software, firmware, or a combination thereof. In order to clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been generally described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints proposed for the entire system. The technician may implement the described functionality in different ways for each specific application, but such specific implementation decisions should not be interpreted as departing from the scope of the present application.

[0259] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as general-purpose computers, wireless communication devices, or integrated circuit devices with multiple uses, including applications in wireless communication devices, mobile phones, and other devices. Any features described as modules or components may be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, these techniques may be implemented at least in part by a computer-readable data storage medium including a program code, the program code including instructions that execute one or more of the above methods when executed. A computer-readable data storage medium may form part of a computer program product, which may include packaging materials. A computer-readable medium may include a memory or data storage medium, such as a random access memory (RAM) (such as a synchronous dynamic random access memory (SDRAM)), a read-only memory (ROM), a non-volatile random access memory (NVRAM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic or optical data storage medium, and the like. Additionally or alternatively, the technology may be implemented at least in part by a computer-readable communication medium that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer, such as a propagated signal or wave.

[0260] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such processors may be configured to perform any of the techniques described in the present disclosure. A general-purpose processor may be a microprocessor; however, in an alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Therefore, the term "processor" as used herein may refer to any of the aforementioned structures, any combination of the aforementioned structures, or any other structure or device suitable for implementing the techniques described herein. In addition, in some aspects, the functionality described herein may be provided in a dedicated software module or hardware module configured for encoding and decoding, or incorporated into a combined video encoder-decoder (CODEC).

[0261] Illustrative aspects of the present disclosure include:

[0262] Aspect 1: A device for processing video data, the device comprising: at least one memory; and at least one processor, the at least one processor being coupled to the at least one memory, the at least one processor being configured to: obtain one or more first sets of collocated motion vector data associated with a first block of video data included in a current frame of the video data; project each of the one or more first sets of collocated motion vector data into a first projected motion field associated with a first buffer; obtain one or more second sets of collocated motion vector data associated with a second block of video data included in the current frame of the video data; project each of the one or more second sets of collocated motion vector data into the first projected motion field associated with the first buffer; and decode the first block of video data based on the first projected motion field associated with the first buffer based on projecting each of the one or more first sets of collocated motion vector data and each of the one or more second sets of collocated motion vector data into the first projected motion field associated with the first buffer.

[0263] Aspect 2: An apparatus according to Aspect 1, wherein the at least one processor is further configured to: project each of the one or more first sets of collocated motion vector data into a second projected motion field associated with a second buffer, wherein the second projected motion field is associated with decoding the second block of video data.

[0264] Aspect 3: The apparatus of aspect 2, wherein the second block of video data is adjacent to the first block of video data.

[0265] Aspect 4: An apparatus according to any one of Aspects 2 to 3, wherein the at least one processor is further configured to: project each of the one or more second sets of collocated motion vector data into the second projected motion field associated with the second buffer; and project each of the one or more second sets of collocated motion vector data into a third projected motion field associated with a third buffer, wherein the third projected motion field is associated with decoding a third block of video data, the third block of video data being positioned adjacent to the second block of video data.

[0266] Aspect 5: An apparatus according to Aspect 4, wherein the at least one processor is further configured to: obtain one or more third sets of collocated motion vector data associated with the third block of video data included in the current frame of video data; project each of the one or more third sets of collocated motion vector data into the second projected motion field associated with the second buffer; and decode the second block of video data based on the second projected motion field associated with the second buffer.

[0267] Aspect 6: An apparatus according to Aspect 5, wherein the at least one processor is further configured to: project each of the one or more third sets of collocated motion vector data into the third projected motion field associated with the third buffer; delete the first projected motion field from the first buffer based on successful decoding of the first block of video data; and project each of the one or more third sets of collocated motion vector data into a fourth projected motion field, wherein the fourth projected motion field is written to the first buffer.

[0268] Aspect 7: The apparatus according to any one of aspects 5 to 6, wherein the at least one processor is further configured to: delete the second projected motion field from the second buffer based on successfully decoding the second block of video data.

[0269] Aspect 8: The apparatus according to any one of aspects 6 to 7, wherein the first buffer, the second buffer and the third buffer are included in a sliding buffer.

[0270] Aspect 9: The apparatus according to any one of aspects 6 to 8, wherein the first buffer, the second buffer and the third buffer are included in a ring buffer or a ping-pong memory.

[0271] Aspect 10: An apparatus according to any one of Aspects 1 to 9, wherein: the one or more first sets of collocated motion vector data include three first sets of collocated motion vector data obtained from corresponding three reference frames associated with the current frame of video data; and the one or more second sets of collocated motion vector data include three second sets of collocated motion vector data obtained from the corresponding three reference frames associated with the current frame of video data.

[0272] Aspect 11: An apparatus according to Aspect 10, wherein: the positions of the three first sets of juxtaposed motion vector data within the corresponding three reference frames are the same; and the positions of the three first sets of juxtaposed motion vector data within the corresponding three reference frames are the same as the position of the first block of video data within the current frame of video data.

[0273] Aspect 12: An apparatus according to any one of Aspects 10 to 11, wherein: the positions of the three second sets of juxtaposed motion vector data within the corresponding three reference frames are the same; and the positions of the three second sets of juxtaposed motion vector data within the corresponding three reference frames are the same as the position of the second block of video data within the current frame of video data.

[0274] Aspect 13: An apparatus according to any one of Aspects 1 to 12, wherein: the first block of video data is a 128×128 block of video data; and the one or more first sets of collocated motion vector data have the same size as the first block of video data.

[0275] Aspect 14: An apparatus according to Aspect 13, wherein, in order to project each of the one or more first sets of collocated motion vector data, the at least one processor is configured to: split each of the one or more first sets of collocated motion vector data into a first 64×128 set of collocated motion vector data and a second 64×128 set of collocated motion vector data; and project the first 64×128 set and the second 64×128 set in reverse N order.

[0276] Aspect 15: The apparatus of aspect 14, wherein the first 64x128 set and the second 64x128 set are non-overlapping.

[0277] Aspect 16: A method for processing video data, the method comprising: obtaining one or more first sets of collocated motion vector data associated with a first block of video data included in a current frame of the video data; projecting each of the one or more first sets of collocated motion vector data into a first projected motion field associated with a first buffer; obtaining one or more second sets of collocated motion vector data associated with a second block of video data included in the current frame of the video data; projecting each of the one or more second sets of collocated motion vector data into the first projected motion field associated with the first buffer; and decoding the first block of video data based on the first projected motion field associated with the first buffer based on projecting each of the one or more first sets of collocated motion vector data and each of the one or more second sets of collocated motion vector data into the first projected motion field associated with the first buffer.

[0278] Aspect 17: According to the method according to Aspect 16, the method also includes: projecting each of the one or more first sets of juxtaposed motion vector data into a second projected motion field associated with a second buffer, wherein the second projected motion field is associated with decoding the second block of video data.

[0279] Aspect 18: The method of aspect 17, wherein the second block of video data is adjacent to the first block of video data.

[0280] Aspect 19: According to the method described in any one of Aspects 17 to 18, the method further includes: projecting each of the one or more second sets of juxtaposed motion vector data into the second projected motion field associated with the second buffer; and projecting each of the one or more second sets of juxtaposed motion vector data into a third projected motion field associated with a third buffer, wherein the third projected motion field is associated with decoding a third block of video data, and the third block of video data is positioned adjacent to the second block of video data.

[0281] Aspect 20: According to the method described in Aspect 19, the method also includes: obtaining one or more third sets of collocated motion vector data associated with the third block of video data included in the current frame of video data; projecting each third set of the one or more third sets of collocated motion vector data into the second projected motion field associated with the second buffer; and decoding the second block of video data based on the second projected motion field associated with the second buffer.

[0282] Aspect 21: According to the method of Aspect 20, the method also includes: projecting each of the one or more third sets of collocated motion vector data into the third projected motion field associated with the third buffer; deleting the first projected motion field from the first buffer based on successful decoding of the first block of video data; and projecting each of the one or more third sets of collocated motion vector data into a fourth projected motion field, wherein the fourth projected motion field is written to the first buffer.

[0283] Aspect 22: The method according to any one of aspects 20 to 21, further comprising: deleting the second projected motion field from the second buffer based on successfully decoding the second block of video data.

[0284] Aspect 23: The method according to any one of aspects 21 to 22, wherein the first buffer, the second buffer and the third buffer are included in a sliding buffer.

[0285] Aspect 24: The method according to any one of Aspects 21 to 23, wherein the first buffer, the second buffer and the third buffer are included in a ring buffer or a ping-pong memory.

[0286] Aspect 25: A method according to any one of Aspects 16 to 24, wherein: the one or more first sets of collocated motion vector data include three first sets of collocated motion vector data obtained from corresponding three reference frames associated with the current frame of video data; and the one or more second sets of collocated motion vector data include three second sets of collocated motion vector data obtained from the corresponding three reference frames associated with the current frame of video data.

[0287] Aspect 26: A method according to Aspect 25, wherein: the positions of the three first sets of juxtaposed motion vector data within the corresponding three reference frames are the same; and the positions of the three first sets of juxtaposed motion vector data within the corresponding three reference frames are the same as the positions of the first block of video data within the current frame of video data.

[0288] Aspect 27: A method according to any one of Aspects 25 to 26, wherein: the positions of the three second sets of juxtaposed motion vector data within the corresponding three reference frames are the same; and the positions of the three second sets of juxtaposed motion vector data within the corresponding three reference frames are the same as the position of the second block of video data within the current frame of video data.

[0289] Aspect 28: A method according to any one of Aspects 16 to 27, wherein: the first block of video data is a 128×128 block of video data; and the one or more first sets of collocated motion vector data have the same size as the first block of video data.

[0290] Aspect 29: A method according to any one of Aspects 16 to 28, wherein projecting each of the one or more first sets of collocated motion vector data includes: splitting each of the one or more first sets of collocated motion vector data into a first 64×128 set of collocated motion vector data and a second 64×128 set of collocated motion vector data; and projecting the first 64×128 set and the second 64×128 set in reverse N order.

[0291] Aspect 30: The method of aspect 29, wherein the first 64x128 set and the second 64x128 set are non-overlapping.

[0292] Aspect 31: A non-transitory computer-readable storage medium having instructions stored thereon, the instructions, when executed by one or more processors, causing the one or more processors to perform any of the operations described in aspects 1 to 30.

[0293] Aspect 32: An apparatus comprising means for performing any of the operations described in aspects 1 to 30.

[0294] Aspect 33: A method of performing any of the operations described in aspects 1 to 15.

Claims

1. A device for processing video data, the device comprising: at least one memory; and at least one processor, the at least one processor coupled to the at least one memory, the at least one processor configured to: obtaining one or more first sets of collocated motion vector data associated with a first block of video data included in a current frame of video data; projecting each of the one or more first sets of concatenated motion vector data into a first projected motion field associated with the first buffer; obtaining one or more second sets of collocated motion vector data associated with a second block of video data included in the current frame of video data; projecting each of the one or more second sets of concatenated motion vector data into the first projected motion field associated with the first buffer; as well as Based on projecting each of the one or more first sets of collocated motion vector data and each of the one or more second sets of collocated motion vector data into the first projected motion field associated with the first buffer, the first block of video data is decoded based on the first projected motion field associated with the first buffer.

2. The apparatus of claim 1 , wherein the at least one processor is further configured to: Each of the one or more first sets of concatenated motion vector data is projected into a second projected motion field associated with a second buffer, wherein the second projected motion field is associated with decoding the second block of video data.

3. The apparatus of claim 2, wherein the second block of video data is adjacent to the first block of video data.

4. The apparatus of claim 2, wherein the at least one processor is further configured to: projecting each of the one or more second sets of concatenated motion vector data into the second projected motion field associated with the second buffer; and Each of the one or more second sets of collocated motion vector data is projected into a third projected motion field associated with a third buffer, wherein the third projected motion field is associated with decoding a third block of video data, the third block of video data being positioned adjacent to the second block of video data.

5. The apparatus of claim 4, wherein the at least one processor is further configured to: obtaining one or more third sets of collocated motion vector data associated with the third block of video data included in the current frame of video data; projecting each of the one or more third sets of concatenated motion vector data into the second projected motion field associated with the second buffer; and The second block of video data is decoded based on the second projected motion field associated with the second buffer.

6. The apparatus of claim 5, wherein the at least one processor is further configured to: projecting each of the one or more third sets of concatenated motion vector data into the third projected motion field associated with the third buffer; deleting the first projected motion field from the first buffer based on successfully decoding the first block of video data; as well as Each of the one or more third sets of concatenated motion vector data is projected into a fourth projected motion field, wherein the fourth projected motion field is written to the first buffer.

7. The apparatus of claim 5, wherein the at least one processor is further configured to: The second projected motion field is deleted from the second buffer based on successfully decoding the second block of video data. 8 . The apparatus of claim 6 , wherein the first buffer, the second buffer, and the third buffer are included in a sliding buffer. 9 . The apparatus of claim 6 , wherein the first buffer, the second buffer, and the third buffer are included in a ring buffer or a ping-pong memory.

10. The device according to claim 1, wherein: The one or more first sets of collocated motion vector data include three first sets of collocated motion vector data obtained from respective three reference frames associated with the current frame of video data; and The one or more second sets of collocated motion vector data include three second sets of collocated motion vector data obtained from the respective three reference frames associated with the current frame of video data.

11. The device according to claim 10, wherein: The positions of the three first sets of concatenated motion vector data within the respective three reference frames are identical; and The positions of the three first sets of concatenated motion vector data within the respective three reference frames are the same as the positions of the first block of video data within the current frame of video data.

12. The device according to claim 10, wherein: The positions of the three second sets of concatenated motion vector data within the corresponding three reference frames are identical; and The positions of the three second sets of concatenated motion vector data within the respective three reference frames are the same as the positions of the second block of video data within the current frame of video data.

13. The device according to claim 1, wherein: The first block of video data is a 128x128 block of video data; and The one or more first sets of collocated motion vector data have the same size as the first block of video data.

14. The apparatus of claim 13, wherein to project each of the one or more first sets of concatenated motion vector data, the at least one processor is configured to: splitting each of the one or more first sets of collocated motion vector data into a first 64x128 set of collocated motion vector data and a second 64x128 set of collocated motion vector data; and The first 64x128 set and the second 64x128 set are projected in reverse N order.

15. The apparatus of claim 14, wherein the first 64x128 set and the second 64x128 set are non-overlapping.

16. A method for processing video data, the method comprising: obtaining one or more first sets of collocated motion vector data associated with a first block of video data included in a current frame of video data; projecting each of the one or more first sets of concatenated motion vector data into a first projected motion field associated with the first buffer; obtaining one or more second sets of collocated motion vector data associated with a second block of video data included in the current frame of video data; projecting each of the one or more second sets of concatenated motion vector data into the first projected motion field associated with the first buffer; as well as Based on projecting each of the one or more first sets of collocated motion vector data and each of the one or more second sets of collocated motion vector data into the first projected motion field associated with the first buffer, the first block of video data is decoded based on the first projected motion field associated with the first buffer.

17. The method according to claim 16, further comprising: Each of the one or more first sets of concatenated motion vector data is projected into a second projected motion field associated with a second buffer, wherein the second projected motion field is associated with decoding the second block of video data.

18. The method of claim 17, wherein the second block of video data is adjacent to the first block of video data.

19. The method according to claim 17, further comprising: projecting each of the one or more second sets of concatenated motion vector data into the second projected motion field associated with the second buffer; as well as Each of the one or more second sets of collocated motion vector data is projected into a third projected motion field associated with a third buffer, wherein the third projected motion field is associated with decoding a third block of video data, the third block of video data being positioned adjacent to the second block of video data.

20. The method according to claim 19, further comprising: obtaining one or more third sets of collocated motion vector data associated with the third block of video data included in the current frame of video data; projecting each of the one or more third sets of concatenated motion vector data into the second projected motion field associated with the second buffer; as well as The second block of video data is decoded based on the second projected motion field associated with the second buffer.

21. The method according to claim 20, further comprising: projecting each of the one or more third sets of concatenated motion vector data into the third projected motion field associated with the third buffer; deleting the first projected motion field from the first buffer based on successfully decoding the first block of video data; as well as Each of the one or more third sets of concatenated motion vector data is projected into a fourth projected motion field, wherein the fourth projected motion field is written to the first buffer.

22. The method according to claim 20, further comprising: The second projected motion field is deleted from the second buffer based on successfully decoding the second block of video data.

23. The method of claim 21, wherein the first buffer, the second buffer, and the third buffer are included in a sliding buffer.

24. The method of claim 21, wherein the first buffer, the second buffer, and the third buffer are included in a ring buffer or a ping-pong memory.

25. The method of claim 16, wherein: The one or more first sets of collocated motion vector data include three first sets of collocated motion vector data obtained from respective three reference frames associated with the current frame of video data; and The one or more second sets of collocated motion vector data include three second sets of collocated motion vector data obtained from the respective three reference frames associated with the current frame of video data.

26. The method of claim 25, wherein: The positions of the three first sets of concatenated motion vector data within the respective three reference frames are identical; and The positions of the three first sets of concatenated motion vector data within the respective three reference frames are the same as the positions of the first block of video data within the current frame of video data.

27. The method of claim 25, wherein: The positions of the three second sets of concatenated motion vector data within the corresponding three reference frames are identical; and The positions of the three second sets of concatenated motion vector data within the respective three reference frames are the same as the positions of the second block of video data within the current frame of video data.

28. The method of claim 16, wherein: The first block of video data is a 128x128 block of video data; and The one or more first sets of collocated motion vector data have the same size as the first block of video data.

29. The method of claim 16, wherein projecting each of the one or more first sets of concatenated motion vector data comprises: splitting each of the one or more first sets of collocated motion vector data into a first 64x128 set of collocated motion vector data and a second 64x128 set of collocated motion vector data; as well as The first 64x128 set and the second 64x128 set are projected in reverse N order.

30. The method of claim 29, wherein the first 64x128 set and the second 64x128 set are non-overlapping.

Images