Storing unaligned reference pixel tiles

Abstract

Systems and techniques are provided for caching unaligned pixel tiles. A method includes determining a first codec region including a first region of a frame; determining whether pixels of a first version of a pixel tile corresponding to a location within the frame are stored in a cache while coding the block from the second codec region; based on whether the pixel is stored in the cache, it is determined whether to read the first version of the pixel tile from the cache or retrieve a second version of the pixel tile from a memory device, the second version of the pixel tile includes pixels from the first codec region that are not in the first version of the pixel tile; and coding a block based on the first version of the pixel tile read from the cache or the second version of the pixel tile retrieved from the memory device.

Description

Technical Field

[0001] The present application relates generally to video encoding and decoding.For example, aspects of the present disclosure relate to systems and techniques for storing (eg, caching) unaligned reference pixel tiles. Background Art

[0002] Digital video capabilities can be incorporated into a variety of devices, including digital televisions, digital live broadcast systems, wireless broadcast systems, personal digital assistants (PDAs), laptop or desktop computers, tablet computers, e-book readers, digital cameras, digital recording devices, digital media players, video game devices, cellular or satellite wireless phones, mobile phones (e.g., so-called "smart phones"), video teleconferencing devices, video streaming devices, etc. Such devices allow video data to be processed and output for consumption. Digital video data includes a large amount of data to meet the needs of consumers and video providers. For example, consumers of video data expect the highest quality video, with high fidelity, high resolution, high frame rate, etc. The large amount of video data required to meet these needs has burdened the communication networks and devices that process and store video data.

[0003] Digital video devices may implement video coding techniques to compress video data. Video coding may be performed according to one or more video coding standards or formats. For example, video coding standards or formats include universal video coding (VVC), basic video coding (EVC), high-efficiency video coding (HEVC), VP8, VP9, ​​advanced video coding (AVC), MPEG-2 Part 2 coding (MPEG stands for Moving Picture Experts Group), etc., and proprietary video codecs / formats, such as AOMedia Video 1 (AV1) and SMPTE 421 (also known as VC-1) developed by the Open Media Alliance, etc. Video coding generally utilizes prediction methods (e.g., inter-frame prediction, intra-frame prediction, etc.) that utilize redundancy present in video images or sequences. The goal of video coding techniques is to compress video data into a form that uses a lower bit rate while avoiding or minimizing degradation of video quality. As evolving video services become available, coding techniques with better coding efficiency are needed. Summary of the invention

[0004] Systems, methods, and computer-readable media for caching misaligned reference pixel tiles are disclosed. According to at least one example, a method for encoding or decoding (decoding) video data is provided. The method may include: determining a first codec zone associated with a first block to be decoded, the first codec zone being one of a plurality of codec zones of a frame, wherein each of the plurality of codec zones of the frame is independently decodable; determining whether reference pixels of a first version of a pixel tile are stored in a cache while decoding one or more blocks from a second codec zone, wherein the cache is locally hosted on a chip of a hardware decoder, and wherein the pixel tile corresponds to a position within the frame; determining whether reference pixels of a first version of a pixel tile are stored in a cache while decoding the one or more blocks from the second codec zone ... Simultaneously determining whether the reference pixels of the first version of the pixel block are stored in the cache, determining whether to read the first version of the pixel block from the cache or to retrieve a second version of the pixel block from a memory device external to the chip of the hardware decoder, wherein the second version of the pixel block includes reference pixels from the first codec zone that are not in the first version of the pixel block; and decoding the first block associated with the first codec zone based at least in part on the first version of the pixel block read from the cache or the second version of the pixel block retrieved from the memory device.

[0005] In another example, a device for encoding or decoding (decoding) video data is provided, the device comprising at least one memory and at least one processor (e.g., configured in a circuit), the at least one processor coupled to the at least one memory. The at least one processor is configured to: determine a first codec zone associated with a first block to be decoded, the first codec zone being one of a plurality of codec zones of a frame, wherein each of the plurality of codec zones of the frame is independently decodable; determine whether reference pixels of a first version of a pixel map block are stored in a cache while decoding one or more blocks from a second codec zone, wherein the cache is locally hosted on a chip of a hardware decoder, and wherein the pixel map block corresponds to a position within the frame; determine whether reference pixels of a first version of a pixel map block are stored in a cache while decoding one or more blocks from the ... Determine whether the reference pixels of the first version of the pixel block are stored in the cache while decoding, determine whether to read the first version of the pixel block from the cache or retrieve a second version of the pixel block from a memory device external to the chip of the hardware decoder, wherein the second version of the pixel block includes reference pixels from the first codec area that are not in the first version of the pixel block; and decode the first block associated with the first codec area based at least in part on the first version of the pixel block read from the cache or the second version of the pixel block retrieved from the memory device.

[0006] In another example, a non-transitory computer-readable medium is provided, the non-transitory computer-readable medium having instructions stored thereon that, when executed by one or more processors, cause the one or more processors to: determine a first codec zone associated with a first block to be decoded, the first codec zone being one of a plurality of codec zones of a frame, wherein each of the plurality of codec zones of the frame is independently decodable; determine whether reference pixels for a first version of a pixel map block are stored in a cache while decoding one or more blocks from a second codec zone, wherein the cache is locally hosted on a chip of a hardware decoder, and wherein the pixel map block corresponds to the frame ; based on determining whether the reference pixels of the first version of the pixel block are stored in the cache while decoding the one or more blocks from the second codec zone, determine whether to read the first version of the pixel block from the cache or retrieve a second version of the pixel block from a memory device external to the chip of the hardware decoder, wherein the second version of the pixel block includes reference pixels from the first codec zone that are not in the first version of the pixel block; and decode the first block associated with the first codec zone based at least in part on the first version of the pixel block read from the cache or the second version of the pixel block retrieved from the memory device.

[0007] In another example, an apparatus for encoding or decoding (decoding) video data is provided. The apparatus includes: a component for determining a first codec zone associated with a first block to be decoded, the first codec zone being one of a plurality of codec zones of a frame, wherein each of the plurality of codec zones of the frame is independently decodable; a component for determining whether reference pixels of a first version of a pixel map block are stored in a cache while decoding one or more blocks from a second codec zone, wherein the cache is locally hosted on a chip of a hardware decoder, and wherein the pixel map block corresponds to a position within the frame; and a component for determining whether reference pixels of a first version of a pixel map block are stored in a cache while decoding the one or more blocks from the second codec zone. while determining whether the reference pixels of the first version of the pixel block are stored in the cache, is a component for determining whether to read the first version of the pixel block from the cache or to retrieve a second version of the pixel block from a memory device external to the chip of the hardware decoder, wherein the second version of the pixel block includes reference pixels from the first codec zone that are not in the first version of the pixel block; and a component for decoding the first block associated with the first codec zone based at least in part on the first version of the pixel block read from the cache or the second version of the pixel block retrieved from the memory device.

[0008] In some aspects, one or more of the devices described above are, may be part of, or may include a mobile device, a camera device, an encoder, a decoder, an Internet of Things (IoT) device, and / or an extended reality (XR) device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device). In some aspects, the device includes a camera device. In some examples, the device may include a vehicle, a mobile device (e.g., a mobile phone or so-called "smart phone" or other mobile device), a wearable device, a personal computer, a laptop computer, a tablet computer, a server computer, a robotic device or system, an aviation system, or other device, or a part of these devices. In some aspects, the device includes an image sensor (e.g., a camera) or multiple image sensors (e.g., multiple cameras) for capturing one or more images. In some aspects, the device includes one or more displays for displaying one or more images, notifications, and / or other displayable data. In some aspects, the device includes one or more speakers, one or more light-emitting devices, and / or one or more microphones. In some aspects, the device described above may include one or more sensors.

[0009] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter. The subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all drawings, and each claim.

[0010] The foregoing and other features and aspects will become more apparent upon reference to the following description, claims and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Illustrative aspects of the present application are described in detail below with reference to the following drawings:

[0012] Figure 1 is a block diagram illustrating examples of encoding devices and decoding devices according to some examples of the present disclosure;

[0013] Figure 2A is a conceptual diagram illustrating example spatial neighboring motion vector candidates for merge mode according to some examples of the present disclosure;

[0014] Figure 2B is a conceptual diagram illustrating example spatial neighboring motion vector candidates for an Advanced Motion Vector Prediction (AMVP) mode according to some examples of the present disclosure;

[0015] Figure 3A is a conceptual diagram illustrating example temporal motion vector predictor (TMVP) candidates according to some examples of the present disclosure;

[0016] Figure 3B is a conceptual diagram illustrating an example of motion vector scaling according to some examples of the present disclosure;

[0017] Figure 4A is a conceptual diagram illustrating an example of neighboring samples of a current coding unit for estimating motion compensation parameters of the current coding unit according to some examples of the present disclosure;

[0018] Figure 4B is a conceptual diagram illustrating an example of neighboring samples of a reference block for estimating motion compensation parameters of a current coding unit according to some examples of the present disclosure;

[0019] Figure 5 is a diagram showing an example data flow for processing video data according to some examples of the present disclosure;

[0020] Fig. 6A , Figure 6B and Figure 6C is a diagram showing an example data flow for processing video data according to some examples of the present disclosure;

[0021] Figure 7 is a flow chart illustrating an example process 700 for caching misaligned pixel tiles according to some examples of the present disclosure;

[0022] Figure 8 is a block diagram illustrating an example video encoding device according to some examples of the present disclosure; and

[0023] Fig. 9 is a block diagram illustrating an example video decoding device according to some examples of the present disclosure. DETAILED DESCRIPTION

[0024] Provided below are some aspects and examples of the present disclosure. As will be apparent to those skilled in the art, some of these aspects and examples can be applied independently, and some of them can be applied in combination. In the following description, specific details are set forth for explanation purposes to provide a thorough understanding of various aspects of the application. However, it is apparent that various aspects can be implemented without these specific details. Each drawing and description are not intended to be restrictive.

[0025] The following description provides examples only and is not intended to limit the scope, applicability or configuration of the present disclosure. On the contrary, the following description of the examples of the present disclosure will provide an enabling description for implementing the examples of the present disclosure to those skilled in the art. It should be understood that various changes may be made to the functions and arrangements of the elements without departing from the scope of the present application as set forth in the appended claims.

[0026] As previously explained, digital video devices may implement video coding techniques to compress video data. Video coding may be performed according to one or more video coding standards or formats. For example, video coding standards or formats include Versatile Video Coding (VVC), Basic Video Coding (EVC), High Efficiency Video Coding (HEVC), VP8, VP9, ​​Advanced Video Coding (AVC), MPEG-2 Part 2 Coding (MPEG stands for Moving Picture Experts Group), etc., and proprietary video codecs / formats, such as AOMedia Video 1 (AV1) and SMPTE 421 (also known as VC-1) developed by the Alliance for Open Media, etc. Video coding typically uses a prediction method that exploits the redundancy present in a video image or sequence. The goal of video coding techniques is to compress video data into a form that uses a lower bit rate while avoiding or minimizing the degradation of video quality.

[0027] Video compression techniques for video coding may include applying different prediction modes, including spatial prediction (e.g., intra-frame prediction or intra-prediction), temporal prediction (e.g., inter-frame prediction or inter-prediction), inter-layer prediction (across different layers of video data), and / or other prediction techniques for reducing or removing redundancy inherent in video sequences. The video encoder may partition each picture of the original video sequence into a plurality of rectangular regions, which are referred to as video blocks or coding units. These video blocks may be encoded using a specific prediction mode.

[0028] Motion compensation is commonly used in the decoding of video data for video compression. In some examples, motion compensation may include and / or implement algorithmic techniques for predicting frames in a video based on previous and / or future frames of the video by taking into account the motion of the camera and / or elements (e.g., objects, etc.) in the video. Motion compensation may describe a picture based on a transformation from a reference picture to a current picture. The reference picture may be a picture that is previous in time or even from the future. In some examples, motion compensation may improve compression efficiency by allowing images to be accurately synthesized from previously sent and / or stored images.

[0029] One example of a motion compensation technique includes block motion compensation (BMC), also known as motion compensated discrete cosine transform (MC DCT), in which a frame is partitioned into non-overlapping blocks of pixels and each block is predicted from one or more blocks in one or more reference frames. In BMC, the block is shifted to the location of the predicted block. This shift is represented by a motion vector (MV) or motion compensation vector. In order to exploit the redundancy between adjacent block vectors, BMC can be used to encode only the difference between the current motion vector and the previous motion vector in the video bitstream.

[0030] Generally speaking, in BMC, the current reconstructed block consists of a prediction block from the previous frame (e.g., referenced by a motion vector) and residual data sent in the bitstream for the current block. Another example of motion compensation technology includes overlapped block motion compensation (OBMC). OBMC can increase prediction accuracy and avoid block effects. In OBMC, the prediction can be or include a weighted sum of multiple predictions. In addition, the block can be larger in each dimension and can overlap with adjacent blocks in a quadrant manner. Therefore, each pixel can belong to multiple blocks. For example, in some exemplary cases, each pixel can belong to 4 blocks. In such a scheme, OBMC can achieve 4 predictions totaling a weighted average for each pixel.

[0031] In some video coding standards or schemes, in order to decode a frame (e.g., a current frame being decoded), a video engine pre-fetches a previously decoded frame (e.g., a previous frame) or block, and uses the previous frame or block as a reference frame or block for a new frame or block being decoded (e.g., a current frame). In some cases, the video engine may include a hardware decoder (e.g., an encoder, a decoder, or a combined encoder-decoder or codec) configured to encode and / or decode video data. When the video engine pre-fetches a previous frame or block, the video engine may cache pixels of the previous frame or block for faster access from a cache memory (e.g., relative to access from a storage device such as an external random access memory (RAM) device).

[0032] In some examples, the video engine may fetch pixels of one or more frames (e.g., the current frame being decoded, a previously decoded frame, etc.) in a pixel block, where each pixel block is referred to herein as a "pixel tile". The pixel tile may include a region of a predefined size, such as a region of 32x16 (meaning 32 pixels wide and 16 pixels high), 64x32, 128x64, 32x32, 64x64, 128x128, or other sizes. The video engine may retrieve the pixel tile from a memory (e.g., a random access memory (RAM) or other memory) and store the pixel tile in a reference cache for future use (e.g., future reads). In some cases, the memory may be stored externally from a chip (e.g., a hardware decoder) of the video engine. In some examples, the cache may be stored locally on a chip (e.g., a hardware decoder) of the video engine. The video engine may store the pixel tile in a reference cache together with a tag indicating the position of the pixel tile within a reference frame. For example, a video engine may store a pixel tile along with a tag indicating the X and Y position coordinates of the pixel tile in a reference frame.

[0033] When the video engine determines to read / retrieve a pixel tile from the reference cache, the video engine may provide the location of the pixel tile (e.g., X and Y location coordinates) to indicate which pixel tile the video engine wants to read / retrieve from the reference cache. For example, the video engine may read / retrieve a pixel tile from the reference cache based on the X and Y location coordinates of the pixel tile that the video engine is attempting to read / retrieve from the reference cache. In order to identify the requested pixel tile in the reference cache, the location of the pixel tile provided by the video engine (e.g., via a read or retrieval query) may be compared to the location identified in the tag of the pixel tile in the reference cache. When the location of the pixel tile provided by the video engine matches the location identified in the tag of the pixel tile in the reference cache, the pixel tile in the reference cache associated with the tag having the matching location may be read out and transmitted to the video engine for processing.

[0034] In some examples, the video engine may decode a frame one codec zone at a time. A codec zone refers to a pixel partition within a frame used when decoding (e.g., encoding and / or decoding) a frame. Each codec zone can be encoded or decoded independently of other codec zones. Illustrative examples of codec zones include codec tiles (or tiles), slices, or other partitions of a video frame (e.g., any independently encodable partition of a video frame). For example, when encoding a frame, the entire frame may not be encoded as a single structure due to, for example, the complexity of searching the entire frame in order to encode the frame. Instead, a frame may be divided into codec zones and encoded (or decoded) by the codec zones (e.g., each codec zone is decoded independently of each other). In some cases, a codec zone may be larger than a pixel tile (e.g., a cache pixel block of a frame). However, in other cases, a codec zone may be the same size as a pixel tile or any other size.

[0035] Intra-block copy (IBC) is a video coding mode in which a video engine may code (e.g., encode and / or decode) a current block of video data in a picture using one or more previously coded blocks of video data within the same frame or picture as the current block. For example, the video engine may use the frame being decoded as a reference frame for coding the frame. In some cases, one or more blocks of previously coded video data are directly above or in line horizontally with (e.g., to the left of) the current block of video data.

[0036] In some aspects, the IBC mode may use a reference cache, where the video engine may retrieve a pixel tile of a frame or picture from a memory (e.g., a random access memory (RAM) or other memory) and store the pixel tile in the reference cache for future use (e.g., future reading) to decode other pixel tiles of the frame or picture (or pixel tiles of other frames or pictures). As previously described, in some examples, the video engine may decode (e.g., encode or decode) each codec region of a frame independently, for example, one codec region at a time. However, in some cases, pixel tiles in a frame (which may have a predefined size as mentioned above) may be "misaligned" in the reference cache and / or relative to the codec regions used to segment, encode, and / or decode the frame. For example, in some cases, pixel data for misaligned pixel tiles in a frame is part of or included within multiple codec regions of the frame. In this case, when the video engine is decoding a first codec zone that includes a portion of a pixel block and the remainder of the pixel block is located in a second codec zone, since the codec zones are independently decodable (for example, the first codec zone is decoded before the second codec zone, resulting in pixels from the second codec zone not being decoded when the block of the first codec zone is decoded), the pixel block may include only pixels from the first codec zone (rather than pixels from the second codec zone) when it is added to the reference cache. This pixel block is referred to herein as a "partial pixel block" or "partial block". For example, the video engine may fetch a partial pixel block from memory and cache the partial pixel block in the reference cache, in which case the partial pixel block is partially filled (with pixels from the first codec zone instead of pixels from the second codec zone) when it is added to the reference cache. When the video engine is decoding (e.g., encoding or decoding) a block within the second codec zone, the video engine may obtain the partial pixel map block from the reference cache to perform decoding of the block of the second codec zone, where the block references a portion of the second codec zone that includes the remainder of the pixel map block (including pixels of the second codec zone that were not included in the partial pixel map block when the partial pixel map block was added to the reference cache). However, because the pixels from the second codec zone were not included in the partial pixel map block when the partial pixel map block was added to the reference cache, the partial pixel map block cannot be used to decode the block of the second codec zone.

[0037] In one illustrative example, a first set of pixels of a pixel tile may reside in a first codec zone, and a second set of pixels of the pixel tile may reside in a second codec zone adjacent to the first codec zone. In this case, when the video engine is decoding the first codec zone containing the first set of pixels of the pixel tile (e.g., decoding a block or other decoding structure / construction contained in the first codec zone), the pixel tile may be only partially populated with the first set of pixels of the pixel tile (e.g., from the first codec zone). The video engine may fetch a partial pixel tile (e.g., including the first set of pixels) and cache only the partial pixel tile in the reference cache. Thus, when the video engine decodes a block of the second codec zone with reference to pixels in the second set of pixels of the second codec zone, the video engine may obtain a partial pixel tile (e.g., including the first set of pixels), wherein the remaining portion of the stale or incomplete data of the partial pixel tile corresponds to the second set of pixels of the pixel tile contained in the second codec zone.

[0038] Described herein are systems, devices, methods, and computer-readable media (hereinafter collectively referred to as "systems and techniques") for caching unaligned reference pixel tiles in intra-frame block copies. In some examples, the systems and techniques described herein can ensure that whenever a new codec zone associated with invalid pixel data (e.g., stale or incomplete pixel data cached in a reference cache during decoding of a different codec zone) is decoded, a partial tile can be cached, read / retrieved, and "refreshed" (e.g., by retrieving all pixels of the pixel tile from the reference cache). In some examples, the systems and techniques described herein can sequentially assign numbers to each codec zone in a reference frame in a particular order (e.g., raster scan order). To read / retrieve pixel tile data, a video engine can generate a query that provides information (e.g., an indication such as a "tag") indicating the location of the requested pixel tile data (e.g., an address such as an X and Y coordinate), as previously explained. In addition to providing the location of the requested pixel tile data within the reference frame, the tag can also indicate the codec zone associated with the pixel tile data. Thus, if a pixel tile includes pixels from multiple codec zones, the tags may indicate the location within the reference frame of each portion of the pixel tile contained within the corresponding codec zone and the different codec zones. The use of such tags may allow the video engine to cache, read / retrieve, and refresh portions of the tile each time a new codec zone is decoded.

[0039] To illustrate, if a first set of pixels of a pixel tile reside in a first codec zone and a second set of pixels of the pixel tile reside in a second codec zone adjacent to the first codec zone, when the video engine determines to read / retrieve the first set of pixels residing in the first codec zone, the video engine may provide a query with a tag identifying the first codec zone in which the first set of pixels resides and the location (e.g., address) of the first set of pixels within a reference frame. Similarly, when the video engine determines to read / retrieve the second set of pixels residing in the second codec zone, the video engine may provide a query with a tag identifying the second codec zone in which the second set of pixels resides and the location (e.g., address) of the second set of pixels within a reference frame.

[0040] As indicated above, when the video engine determines to read / retrieve a pixel tile (or a portion of a pixel tile residing in a codec zone), the video engine may provide a tag in the read / retrieve query that identifies the codec zone information (e.g., corresponding to the codec zone being decoded when the pixel tile is stored in the reference cache) and the address / location of the pixel tile data being requested. For any pixel tile or portion of a pixel tile contained in a particular codec zone, the reference cache may store the codec zone information and the location (e.g., address and / or coordinates) of the pixel tile data in an associated tag. The codec zone information may include, for example, but not limited to, a codec zone number, a codec zone address, a codec zone identifier, a codec zone description, and / or any other codec zone information.

[0041] When the video engine checks the tag to determine whether the corresponding pixel tile data is included in the reference cache, the video engine may compare the codec region information in the tag and the address information in the tag (e.g., which identifies the location of the requested pixel tile data) with the codec region information and address information in each tag in the reference cache. If both the codec region information and address information in the tag match the codec region information and address information in the tag in the reference cache, the query returns a hit (e.g., a match). If the query returns a hit, the video engine may read / retrieve the requested pixel tile data from the reference cache. On the other hand, if the codec region information and / or address information in the tag do not match the codec region information and / or address information in the tag in the reference cache, the query returns a miss. If the query returns a miss, the video engine may read / retrieve the requested pixel tile data from the memory device and refresh a portion of the pixel tile by storing the retrieved pixel tile in the reference cache together with the codec region information and / or address information. The memory device is different / separate from the reference cache. For example, the memory device may be an external memory device and / or a random access memory (RAM) device (e.g., static RAM (SRAM), synchronous RAM (SDRAM), double data rate SDRAM (DDR SDRAM), etc.) that is not local to the video engine (e.g., a hardware decoder). In other words, the memory device is considered to be off-chip memory relative to the video engine.

[0042] As shown above, the systems and techniques described herein can ensure that partial tiles can be cached, read / retrieved, and refreshed whenever a new codec region is decoded.Additional details regarding the systems and techniques will be described with respect to the accompanying drawings.

[0043] Figure 11 is a block diagram showing an example of a system 100 including an encoding device 104 and a decoding device 112. The encoding device 104 may be part of a source device, and the decoding device 112 may be part of a receiving device. The source device and / or the receiving device may include an electronic device, such as a mobile or fixed telephone handset (e.g., a smart phone, a cellular phone, etc.), a desktop computer, a laptop or notebook computer, a tablet computer, a set-top box, a television, a camera, a display device, a digital media player, a video game console, a video streaming device, an Internet Protocol (IP) camera, or any other suitable electronic device. In some examples, the source device and the receiving device may include one or more wireless transceivers for wireless communication. The decoding techniques described herein are applicable to video decoding in various multimedia applications, including streaming video transmission (e.g., via the Internet), television broadcasting or transmission, encoding digital video for storage on a data storage medium, decoding digital video stored on a data storage medium, or other applications. As used herein, the term decoding may refer to encoding and / or decoding. In some examples, system 100 may support one-way or two-way video transmission to support applications such as video conferencing, video streaming, video playback, video broadcasting, gaming, and / or video telephony.

[0044] The encoding device 104 (or encoder) can be used to encode video data using a video coding standard, format, codec or protocol to generate a coded video bitstream. Examples of video coding standards and formats, codecs include ITU-T H.261, ISO / IEC MPEG-1 video, ITU-T H.262 or ISO / IEC MPEG-2 video, ITU-T H.263, ISO / IEC MPEG-4 video, ITU-T H.264 (also known as ISO / IEC MPEG-4 AVC) (including its scalable video coding (SVC) and multi-view video coding (MVC) extensions), high efficiency video coding (HEVC) or ITU-T H.265 and universal video coding (VVC) or ITU-TH.266. There are various extensions to HEVC to handle multi-layer video coding, including range and screen content coding extensions, 3D video coding (3D-HEVC) and multi-view extensions (MV-HEVC) and scalable extensions (SHVC). HEVC and its extensions have been developed by the Joint Collaborative Team on Video Coding (JCT-VC) and the Joint Collaborative Team on 3D Video Coding Extensions Development (JCT-3V) of the ITU-T Video Coding Experts Group (VCEG) and the ISO / IEC Motion Picture Experts Group (MPEG). VP9, ​​AOMedia Video 1 (AV1) developed by the Alliance for Open Media (AOMedia), and Essential Video Coding (EVC) are other video coding standards to which the techniques described herein may be applied.

[0045] The techniques described herein may be applied to any of the existing video codecs (e.g., High Efficiency Video Coding (HEVC), Advanced Video Coding (AVC), or other suitable existing video codecs), and / or may be effective coding tools for any video coding standards being developed and / or future video coding standards (e.g., VVC and / or other video coding standards under development or to be developed). For example, the examples described herein may be performed using video codecs such as VVC, HEVC, AVC, and / or their extensions. However, the techniques and systems described herein may also be applicable to other coding standards, codecs, or formats, such as MPEG, JPEG (or other coding standards for still images), VP9, ​​AV1, their extensions, or other suitable coding standards that are already available or not yet available or not yet developed. For example, in some examples, the encoding device 104 and / or the decoding device 112 may operate according to a proprietary video codec / format (such as AV1, an extension of AVI, and / or a successor version of AV1 (e.g., AV2)) or other proprietary formats or industry standards. Thus, although the techniques and systems described herein may be described with reference to a particular video coding standard, one of ordinary skill in the art will appreciate that the description should not be construed as applying only to that particular standard.

[0046] refer to Figure 1 , video source 102 may provide video data to encoding device 104. Video source 102 may be part of a source device, or may be part of a device other than a source device. Video source 102 may include a video capture device (e.g., a video camera, a camera phone, a video phone, etc.), a video archive including stored video, a video server or content provider providing video data, a video feed interface receiving video from a video server or content provider, a computer graphics system for generating computer graphics video data, a combination of such sources, or any other suitable video source.

[0047] The video data from the video source 102 may include one or more input pictures or frames. A picture or frame is a static image, which in some cases is part of a video. In some examples, the data from the video source 102 may be a static image that is not part of a video. In HEVC, VVC and other video coding specifications, a video sequence may include a series of pictures. A picture may include three sample arrays, which are represented as SL, SCb and SCr. SL is a two-dimensional array of luma samples, SCb is a two-dimensional array of Cb chroma samples, and SCr is a two-dimensional array of Cr chroma samples. Chroma samples may also be referred to as "chroma" samples in this article. A pixel may refer to all three components (luminance samples and chroma samples) at a given position in an array of a picture. In other cases, a picture may be monochrome and may include only a luma sample array, in which case the terms pixel and sample may be used interchangeably. With respect to the example techniques described herein that refer to individual samples for illustrative purposes, the same techniques may be applied to pixels (e.g., all three sample components at a given position in an array of a picture). With respect to the example techniques described herein that refer to pixels (eg, all three sample components of a given position in an array of a picture) for illustrative purposes, the same techniques may be applied to individual samples.

[0048] The encoder engine 106 (or encoder) of the encoding device 104 encodes the video data to generate an encoded video bitstream. In some examples, the encoded video bitstream (or "video bitstream" or "bitstream") is a series of one or more decoded video sequences. The decoded video sequence (CVS) includes a series of access units (AUs) starting from an AU having a random access point picture in the base layer and having certain attributes, until the next AU having a random access point picture in the base layer and having certain attributes and not including the next AU. For example, certain attributes of the random access point picture that starts the CVS may include a RASL flag equal to 1 (e.g., NoRaslOutputFlag). Otherwise, the random access point picture (with a RASL flag equal to 0) does not start the CVS. The access unit (AU) includes one or more decoded pictures and control information corresponding to the decoded pictures sharing the same output time. The decoded slices of the picture are encapsulated as data units at the bitstream level, which are called network abstraction layer (NAL) units. For example, a HEVC video bitstream may include one or more CVSs, which include NAL units. Each NAL unit in the NAL unit has a NAL unit header. In one example, the header is one byte for H.264 / AVC (except for multi-layer extensions) and two bytes for HEVC. The syntax elements in the NAL unit header use designated bits and are therefore visible to all kinds of systems and transport layers (such as transport streams, real-time transport (RTP) protocols, file formats, etc.).

[0049] There are two types of NAL units in the HEVC standard, including video coding layer (VCL) NAL units and non-VCL NAL units. The VCL NAL unit includes decoded picture data that forms a decoded video bitstream. For example, a bit sequence that forms a decoded video bitstream exists in a VCL NAL unit. The VCL NAL unit may include a slice or slice segment (described below) of the decoded picture data, and the non-VCL NAL unit includes control information related to one or more decoded pictures. In some cases, the NAL unit may be referred to as a packet. The HEVC AU includes a VCL NAL unit containing the decoded picture data and a non-VCL NAL unit corresponding to the decoded picture data (if any). In addition to other information, the non-VCL NAL unit may also include a parameter set with advanced information related to the encoded video bitstream. For example, a parameter set may include a video parameter set (VPS), a sequence parameter set (SPS), and a picture parameter set (PPS). In some cases, each slice or other portion of the bitstream may reference a single valid PPS, SPS, and / or VPS to allow the decoding device 112 to access information that can be used to decode the slice or other portion of the bitstream.

[0050] A NAL unit may include a sequence of bits that form a decoded representation of video data (e.g., a coded video bitstream, a CVS of a bitstream, etc.), such as a decoded representation of a picture in a video. The encoder engine 106 generates a decoded representation of a picture by partitioning each picture into a plurality of slices. A slice is independent of other slices so that information in the slice can be decoded without relying on data from other slices within the same picture. A slice includes one or more slice segments, including an independent slice segment and, if present, one or more dependent slice segments that depend on previous slice segments.

[0051] In HEVC, the slice is then partitioned into coding tree blocks (CTBs) of luma samples and chroma samples. The CTB of luma samples and one or more CTBs of chroma samples together with the syntax for the samples are called coding tree units (CTUs). CTUs may also be called "tree blocks" or "largest coding units" (LCUs). CTUs are the basic processing units for HEVC encoding. CTUs can be split into multiple coding units (CUs) of different sizes. A CU contains a luma sample array and a chroma sample array called a coding block (CB).

[0052] Luma CB and chroma CB can be further split into prediction blocks (PB). PB is a sample block of luma component or chroma component, which uses the same motion parameters for inter prediction or intra block copy (IBC) prediction (when available or enabled). Luma PB and one or more chroma PBs together with associated syntax form a prediction unit (PU). For inter prediction, a set of motion parameters (e.g., one or more motion vectors, reference indexes, etc.) are signaled in the bitstream for each PU, as well as inter prediction for luma PB and one or more chroma PBs. Motion parameters may also be referred to as motion information. CB may also be divided into one or more transform blocks (TB). TB represents a square block of samples of a color component, to which a residual transform (e.g., in some cases, the same two-dimensional transform) is applied to decode the prediction residual signal. Transform unit (TU) represents a TB of luma samples and chroma samples and corresponding syntax elements. Transform decoding is described in more detail below.

[0053] The size of the CU corresponds to the size of the decoding mode and can be a square shape. For example, the size of the CU can be 8x8 samples, 16x16 samples, 32x32 samples, 64x64 samples, or any other appropriate size up to the size of the corresponding CTU. The phrase "NxN" is used herein to refer to the pixel size of the video block in the vertical and horizontal dimensions (e.g., 8 pixels x8 pixels). The pixels in the block can be arranged in rows and columns. In some specific implementations, the block may not have the same number of pixels in the horizontal direction as in the vertical direction. The syntax data associated with the CU can describe, for example, the partitioning of the CU into one or more PUs. The partitioning mode may vary depending on whether the CU is encoded in an intra-frame prediction mode or an inter-frame prediction mode. The PU can be partitioned into a non-square shape. The syntax data associated with the CU can also, for example, describe the partitioning of the CU into one or more TUs according to the CTU. The TU can be square or non-square in shape.

[0054] According to the HEVC standard, a transform unit (TU) may be used to perform the transform. The TU may vary for different CUs. The size of the TU may be set based on the size of the PU within a given CU. The TU may have the same size as the PU or be smaller than the PU. In some examples, a quadtree structure called a residual quadtree (RQT) may be used to subdivide the residual samples corresponding to the CU into smaller units. The leaf nodes of the RQT may correspond to the TU. The pixel difference values ​​associated with the TU may be transformed to generate transform coefficients. The transform coefficients may then be quantized by the encoder engine 106.

[0055] Once the picture of the video data is partitioned into CUs, the encoder engine 106 predicts each PU using a prediction mode. The prediction unit or prediction block is then subtracted from the original video data to obtain a residual (described below). For each CU, the prediction mode can be signaled in the bitstream using syntax data. The prediction mode may include intra-frame prediction (or intra-picture prediction) or inter-frame prediction (or inter-picture prediction). Intra-frame prediction exploits the correlation between spatially adjacent samples within a picture. For example, using intra-frame prediction, each PU is predicted from adjacent image data in the same picture using, for example, DC prediction to find an average value for the PU, using plane prediction to fit the plane surface to the PU, using directional prediction to infer from adjacent data, or using any other suitable prediction type. Inter-frame prediction uses temporal correlation between pictures to derive motion compensated prediction for blocks of image samples. For example, using inter-frame prediction, each PU is predicted from image data in one or more reference pictures (before or after the current picture in output order) using motion compensated prediction. For example, a decision can be made at the CU level whether to use inter-picture prediction or intra-picture prediction to decode a picture area.

[0056] The encoder engine 106 and the decoder engine 116 (described in more detail below) may be configured to operate according to VVC. According to VVC, a video decoder (such as the encoder engine 106 and / or the decoder engine 116) partitions a picture into a plurality of coding tree units (CTUs) (wherein the CTB of a luminance sample and one or more CTBs of a chrominance sample together with the syntax for the sample are referred to as CTUs). The video decoder may partition the CTU according to a tree structure such as a quadtree-binary tree (QTBT) structure or a multi-type tree (MTT) structure. The QTBT structure removes the concept of multiple partition types, such as the separation between CU, PU, ​​and TU of HEVC. The QTBT structure includes two levels, including a first level partitioned according to quadtree partitioning, and a second level partitioned according to binary tree partitioning. The root node of the QTBT structure corresponds to the CTU. The leaf nodes of the binary tree correspond to the coding units (CUs).

[0057] In the MTT segmentation structure, blocks can be segmented using quadtree segmentation, binary tree segmentation, and one or more types of ternary tree segmentation. A ternary tree segmentation is a segmentation in which a block is split into three sub-blocks. In some examples, a ternary tree segmentation divides a block into three sub-blocks without dividing the original block through the center. The segmentation types in MTT (e.g., quadtree, binary tree, and ternary tree) can be symmetric or asymmetric.

[0058] When operating according to the AV1 codec, the encoding device 104 and the decoding device 112 may be configured to decode the video data in units of blocks. In AV1, the largest decoding block that can be processed is called a super block. In AV1, a super block may be 128x128 luma samples or 64x64 luma samples. However, in subsequent video coding formats (e.g., AV2), super blocks may be defined by different (e.g., larger) luma sample sizes. In some examples, the super block is the top level of the block quadtree. The encoding device 104 may further divide the super block into smaller decoding blocks. The encoding device 104 may divide the super block and other decoding blocks into smaller blocks using square or non-square division. Non-square blocks may include N / 2xN blocks, NxN / 2 blocks, N / 4xN blocks, and NxN / 4 blocks. The encoding device 104 and the decoding device 112 may perform separate prediction and transform processing on each decoding block.

[0059] AV1 also defines tiles of video data. A tile is a rectangular array of super blocks that can be decoded independently of other tiles. That is, the encoding device 104 and the decoding device 112 can encode and decode coded blocks within a tile, respectively, without using video data from other tiles. However, the encoding device 104 and the decoding device 112 can perform filtering across tile boundaries. The size of the tiles can be uniform or non-uniform. Tile-based decoding can enable parallel processing and / or multi-threading specifically implemented by the encoder and decoder.

[0060] In some examples, the video coder may use a single QTBT or MTT structure to represent each of the luma and chroma components, while in other examples, the video coder may use two or more QTBT or MTT structures, such as one QTBT or MTT structure for the luma component and another QTBT or MTT structure for two chroma components (or two QTBT and / or MTT structures for corresponding chroma components).

[0061] The video coder may be configured to use quadtree partitioning, QTBT partitioning, MTT partitioning, superblock partitioning, or other partitioning structures.

[0062] In some cases, the encoder engine 106 and the decoder engine 116 may be configured to operate according to another video coding standard such as, for example, HEVC. According to HEVC, a video decoder such as the encoder engine 106 and / or the decoder engine 116 divides each picture into one or more slices. Each slice may be independent of other slices so that the information carried in the slice may be decoded without relying on data from other slices within the same picture. A slice may include one or more slice segments, wherein the first slice segment of a slice is referred to as an independent slice segment and is independent of other slice segments. Subsequent slice segments, if any, are referred to as dependent slice segments because they depend on previous slice segments.

[0063] In some examples, a slice type is assigned to one or more slices of a picture. Slice types include intra-coded slices (I slices), inter-coded P slices, and inter-coded B slices. An I slice (intra-coded frame, independently decodable) is a slice of a picture that is coded only by intra prediction, and is therefore independently decodable because an I slice only requires data within the frame to predict any prediction unit or prediction block of the slice. A P slice (unidirectionally predicted frame) is a slice of a picture that can be coded with intra prediction and with unidirectional inter prediction. Each prediction unit or prediction block within a P slice is coded with intra prediction or inter prediction. When inter prediction is applied, the prediction unit or prediction block is predicted by only one reference picture, and therefore the reference sample comes from only one reference region of one frame. A B slice (bidirectionally predicted frame) is a slice of a picture that can be coded with intra prediction and inter prediction (e.g., either bidirectional prediction or unidirectional prediction). A prediction unit or prediction block of a B slice may be bidirectionally predicted from two reference pictures, where each picture contributes to one reference region and the sample sets of the two reference regions are weighted (e.g., with equal weights or with different weights) to produce a prediction signal for the bidirectionally predicted block. As described above, slices of a picture are coded independently. In some cases, a picture may be coded as only one slice.

[0064] Intra-picture prediction of a picture exploits the correlation between spatially adjacent samples within a picture. There are multiple intra-prediction modes (also referred to as "intra-modes"). In some examples, intra-prediction of a luma block includes 35 modes, including a planar mode, a DC mode, and 33 angular modes (e.g., a diagonal intra-prediction mode and an angular mode adjacent to the diagonal intra-prediction mode). The 35 modes of intra-prediction can be indexed. In other examples, more intra-modes can be defined, including prediction angles that may not yet be represented by the 33 angular modes. In other examples, the prediction angles associated with the angular modes may be different from those used in HEVC.

[0065] Inter-picture prediction uses temporal correlations between pictures in order to derive motion compensated predictions for blocks of image samples. Using a translational motion model, the position of a block in a previously decoded picture (reference picture) is indicated by a motion vector (Δx, Δy), where Δx specifies the horizontal displacement of the reference block relative to the position of the current block, and Δy specifies the vertical displacement of the reference block relative to the position of the current block. In some cases, the motion vector (Δx, Δy) may have integer sample accuracy (also called integer accuracy), in which case the motion vector points to an integer pixel grid (or integer pixel sampling grid) of a reference frame. In some cases, the motion vector (Δx, Δy) may have fractional sample accuracy (also called fractional pixel accuracy or non-integer accuracy) to more accurately capture the movement of underlying objects without being limited to an integer pixel grid of a reference frame.

[0066] The accuracy of the motion vector can be represented by the quantization level of the motion vector. For example, the quantization level can be integer accuracy (e.g., 1 pixel) or fractional pixel accuracy (e.g., 1 / 4 pixel, 1 / 2 pixel, or other sub-pixel value). When the corresponding motion vector has fractional sample accuracy, interpolation can be applied to the reference picture to derive the prediction signal. For example, the samples available at the integer position can be filtered (e.g., using one or more interpolation filters) to estimate the value at the fractional position. The previously decoded reference picture is indicated by the reference index (refIdx) of the reference picture list. The motion vector and the reference index can be referred to as motion parameters. Two types of inter-picture prediction can be performed, including unidirectional prediction and bidirectional prediction.

[0067] In the case of inter-frame prediction using bidirectional prediction (also called bidirectional inter-frame prediction), two sets of motion parameters (e.g., Δx0, y0, refIdx0 and Δx1, y1, refIdx1) can be used to generate two motion compensated predictions (from the same reference picture or possibly from different reference pictures). For example, in the case of bidirectional prediction, each prediction block uses two motion compensated prediction signals and generates B prediction units. The two motion compensated predictions are then combined to obtain the final motion compensated prediction. For example, the two motion compensated predictions can be combined by averaging. In another example, weighted prediction can be used, in which case different weights can be applied to each motion compensated prediction. The reference pictures that can be used in bidirectional prediction are stored in two separate lists, represented as list 0 and list 1, respectively. The motion parameters can be derived using a motion estimation process at the encoder.

[0068] In the case of inter prediction using unidirectional prediction (also called unidirectional inter prediction), a motion compensation prediction is generated from a reference picture using one motion parameter set (Δx0, y0, refIdx0). For example, in the case of unidirectional prediction, each prediction block uses at most one motion compensation prediction signal and generates P prediction units.

[0069] The PU may include data related to the prediction process (e.g., motion parameters or other suitable data). For example, when the PU is encoded using intra prediction, the PU may include data describing the intra prediction mode for the PU. As another example, when the PU is encoded using inter prediction, the PU may include data defining a motion vector for the PU. The data defining the motion vector for the PU may describe, for example, a horizontal component of the motion vector (Δx), a vertical component of the motion vector (Δy), a resolution for the motion vector (e.g., integer precision, quarter-pixel precision, or eighth-pixel precision), a reference picture to which the motion vector points, a reference index, a reference picture list for the motion vector (e.g., list 0, list 1, or list C), or any combination thereof.

[0070] AV1 includes two general techniques for encoding and decoding coded blocks of video data. The two general techniques are intra-frame prediction (e.g., intra-frame prediction or spatial prediction) and inter-frame prediction (e.g., inter-frame prediction or temporal prediction). In the content of AV1, when an intra-frame prediction mode is used to predict a block of a current frame of video data, the encoding device 104 and the decoding device 112 do not use video data from other frames of the video data. For most intra-frame prediction modes, the video encoding device 104 encodes a block of the current frame based on the difference between the sample value in the current block and the prediction value generated from the reference sample in the same frame. The video encoding device 104 determines the prediction value generated from the reference sample based on the intra-frame prediction mode.

[0071] After performing prediction using intra prediction and / or inter prediction, the encoding device 104 may perform transformation and quantization. For example, after prediction, the encoder engine 106 may calculate a residual value corresponding to the PU. The residual value may include a pixel difference value between a current pixel block (PU) being decoded and a prediction block (e.g., a predicted version of the current block) used to predict the current block. For example, after generating a prediction block (e.g., using inter prediction or intra prediction), the encoder engine 106 may generate a residual block by subtracting the prediction block generated by the prediction unit from the current block. The residual block includes a set of pixel difference values ​​that quantizes the difference between the pixel values ​​of the current block and the pixel values ​​of the prediction block. In some examples, the residual block may be represented by a two-dimensional block format (e.g., a two-dimensional matrix or array of pixel values). In such examples, the residual block is a two-dimensional representation of the pixel values.

[0072] Any residual data that may remain after prediction is performed is transformed using a block transform, which may be based on a discrete cosine transform, a discrete sine transform, an integer transform, a wavelet transform, other suitable transform functions, or any combination thereof. In some cases, one or more block transforms (e.g., of size 32×32, 16×16, 8×8, 4×4, or other suitable sizes) may be applied to the residual data in each CU. In some examples, TUs may be used for the transform and quantization process implemented by the encoder engine 106. A given CU with one or more PUs may also include one or more TUs. As described in further detail below, the residual values ​​may be transformed into transform coefficients using a block transform, and then quantized and scanned using the TUs to produce serialized transform coefficients for entropy coding.

[0073] In some examples, after intra-prediction or inter-prediction coding using a PU of a CU, the encoder engine 106 may calculate residual data for a TU of the CU. A PU may include pixel data in a spatial domain (or pixel domain). A TU may include coefficients in a transform domain after applying a block transform. As previously noted, the residual data may correspond to a pixel difference between a pixel of an uncoded picture and a prediction value corresponding to the PU. The encoder engine 106 may form a TU including the residual data for the CU, and may then transform the TU to generate transform coefficients for the CU.

[0074] The encoder engine 106 may perform quantization of the transform coefficients. Quantization provides further compression by quantizing the transform coefficients to reduce the amount of data used to represent the coefficients. For example, quantization may reduce the bit depth associated with some or all of the coefficients. In one example, a coefficient having an n-bit value may be rounded down to an m-bit value during quantization, where n is greater than m.

[0075] Once quantization is performed, the decoded video bitstream includes the quantized transform coefficients, prediction information (e.g., prediction mode, motion vectors, block vectors, etc.), segmentation information, and any other suitable data (such as other syntax data). The different elements of the decoded video bitstream can then be entropy encoded by the encoder engine 106. In some examples, the encoder engine 106 can scan the quantized transform coefficients using a predefined scan order to produce a serialized vector that can be entropy encoded. In some examples, the encoder engine 106 can perform adaptive scanning. After scanning the quantized transform coefficients to form a vector (e.g., a one-dimensional vector), the encoder engine 106 can entropy encode the vector. For example, the encoder engine 106 can use context adaptive variable length coding, context adaptive binary arithmetic coding, grammar-based context adaptive binary arithmetic coding, probability interval segmentation entropy coding, or another suitable entropy coding technique.

[0076] The output 110 of the encoding device 104 may transmit the NAL units constituting the encoded video bitstream data to a decoding device 112 of a receiving device over a communication link 120. The input 114 of the decoding device 112 may receive the NAL units. The communication link 120 may include a channel provided by a wireless network, a wired network, or a combination of a wired network and a wireless network. The wireless network may include any wireless interface or combination of wireless interfaces and may include any suitable wireless network (e.g., the Internet or other wide area network, a packet-based network, a WiFi network, or a wireless network). TM , Radio Frequency (RF), Ultra-Wideband (UWB), WiFi Direct, Cellular, Long Term Evolution (LTE), WiMax TM The wired network may include any wired interface (e.g., optical fiber, Ethernet, power line Ethernet, Ethernet based on coaxial cable, digital signal line (DSL), etc.). The wired network and / or wireless network may be implemented using various equipment (such as base stations, routers, access points, bridges, gateways, switches, etc.). The encoded video bitstream data may be modulated according to a communication standard (such as a wireless communication protocol) and sent to a receiving device.

[0077] In some examples, the encoding device 104 may store the encoded video bitstream data in a storage device 108. The output end 110 may retrieve the encoded video bitstream data from the encoder engine 106 or from the storage device 108. The storage device 108 may include any of a variety of distributed or locally accessed data storage media. For example, the storage device 108 may include a hard drive, a storage disk, a flash memory, a volatile or non-volatile memory, or any other suitable digital storage medium for storing the encoded video data. The storage device 108 may also include a decoded picture buffer (DPB) for storing reference pictures for use in inter-frame prediction. In a further example, the storage device 108 may correspond to a file server or another intermediate storage device that can store the encoded video generated by the source device. In such a case, a receiving device including a decoding device 112 can access the stored video data from the storage device via streaming or downloading. The file server can be any type of server capable of storing the encoded video data and sending the encoded video data to the receiving device. Example file servers include web servers (e.g., for websites), FTP servers, network attached storage (NAS) devices, or local disk drives. The receiving device can access the encoded video data through any standard data connection (including an Internet connection). The data connection may include a wireless channel (e.g., a Wi-Fi connection), a wired connection (e.g., DSL, cable modem, etc.) suitable for accessing the encoded video data stored on a file server, or a combination of the two. The transmission of the encoded video data from the storage device 108 can be streaming, download transmission, or a combination thereof.

[0078] The input end 114 of the decoding device 112 receives the encoded video bitstream data, and the video bitstream data can be provided to the decoder engine 116, or provided to the storage device 118 for later use by the decoder engine 116. For example, the storage device 118 may include a DPB for storing reference pictures used in inter-frame prediction. The receiving device including the decoding device 112 can receive the encoded video data to be decoded via the storage device 108. The encoded video data can be modulated according to a communication standard (such as a wireless communication protocol) and sent to the receiving device. The communication medium for sending the encoded video data may include any wireless or wired communication medium, such as a radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form a part of a packet-based network such as the following: a local area network, a wide area network, or a global network such as the Internet. The communication medium may include a router, a switch, a base station, or any other equipment that may be useful for facilitating communication from a source device to a receiving device.

[0079] The decoder engine 116 can decode the encoded video bitstream data by entropy decoding (e.g., using an entropy decoder) and extracting the elements of one or more encoded video sequences that constitute the encoded video data. The decoder engine 116 can then rescale the encoded video bitstream data and perform an inverse transform on it. The residual data is then passed to the prediction stage of the decoder engine 116. The decoder engine 116 then predicts a pixel block (e.g., a PU). In some examples, the prediction is added to the output of the inverse transform (residual data).

[0080] The video decoding device 112 may output the decoded video to a video destination device 122, which may include a display or other output device for displaying the decoded video data to a consumer of the content. In some aspects, the video destination device 122 may be part of a receiving device that includes the decoding device 112. In some aspects, the video destination device 122 may be part of a separate device distinct from the receiving device.

[0081] In some examples, the video encoding device 104 and / or the video decoding device 112 may be integrated with an audio encoding device and an audio decoding device, respectively. The video encoding device 104 and / or the video decoding device 112 may also include other hardware or software necessary to implement the decoding techniques described above, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic components, software, hardware, firmware, or any combination thereof. The video encoding device 104 and the video decoding device 112 may be integrated as part of a combined encoder / decoder (codec) in the respective devices.

[0082] exist Figure 1 The example system shown in is an illustrative example that can be used in this article. The technology for processing video data using the technology described herein can be performed by any digital video encoding and / or decoding device. Although generally speaking, the technology of the present disclosure is performed by a video encoding device or a video decoding device, these technologies can also be performed by a combined video encoder-decoder (commonly referred to as a "codec"). In addition, the technology of the present disclosure can also be performed by a video preprocessor. The source device and the receiving device are merely examples of such decoding devices, wherein the source device generates the decoded video data for sending to the receiving device. In some examples, the source device and the receiving device can operate in a substantially symmetrical manner so that each of these devices includes a video encoding and decoding component. Therefore, the example system can support one-way or two-way video transmission between video devices, for example, for video streaming, video playback, video broadcasting, or video telephony.

[0083] Extensions to the HEVC standard include a multi-view video coding extension (referred to as MV-HEVC) and a scalable video coding extension (referred to as SHVC). The MV-HEVC and SHVC extensions share the concept of layered coding, in which different layers are included in the encoded video bitstream. Each layer in the decoded video sequence is addressed by a unique layer identifier (ID). The layer ID may be present in the header of the NAL unit to identify the layer associated with the NAL unit. In MV-HEVC, different layers typically represent different views of the same scene in the video bitstream. In SHVC, different scalable layers are provided that represent the video bitstream at different spatial resolutions (or picture resolutions) or at different reconstruction fidelity. The scalable layer may include a base layer (where layer ID = 0) and one or more enhancement layers (where layer ID = 1, 2, ... n). The base layer may conform to the profile of the first version of HEVC and represent the lowest available layer in the bitstream. Compared to the base layer, the enhancement layer has an increased spatial resolution, temporal resolution or frame rate and / or reconstruction fidelity (or quality). Enhancement layers are organized hierarchically and may (or may not) depend on lower layers. In some examples, a single standard codec may be used to code the different layers (e.g., all layers are encoded using HEVC, SHVC, or other coding standards). In some examples, multiple standard codecs may be used to code the different layers. For example, the base layer may be coded using AVC, while one or more enhancement layers may be coded using SHVC and / or MV-HEVC extensions to the HEVC standard.

[0084] Typically, a layer includes a set of VCL NAL units and a corresponding set of non-VCL NAL units. NAL units are assigned specific layer ID values. Layers can be hierarchical in the sense that layers can depend on lower layers. A layer set refers to a set of self-contained layers represented in a bitstream, which means that layers in a layer set can depend on other layers in the layer set during decoding, but do not depend on any other layers to be decoded. Therefore, the layers in a layer set can form an independent bitstream that can represent video content. The set of layers in a layer set can be obtained from another bitstream through the operation of a sub-bitstream extraction process. A layer set may correspond to a set of layers to be decoded when a decoder wants to operate according to certain parameters.

[0085] As previously described, the HEVC bitstream includes a group of NAL units, including VCL NAL units and non-VCL NAL units. The VCL NAL unit includes decoded picture data that forms the decoded video bitstream. For example, the bit sequence that forms the decoded video bitstream is present in the VCL NAL unit. In addition to other information, the non-VCL NAL unit may also contain a parameter set with advanced information related to the encoded video bitstream. For example, the parameter set may include a video parameter set (VPS), a sequence parameter set (SPS), and a picture parameter set (PPS). Examples of the goals of the parameter set include bit rate efficiency, fault tolerance, and providing a system layer interface. Each slice references a single active PPS, SPS, and VPS to access information that the decoding device 112 can use to decode the slice. An identifier (ID) may be decoded for each parameter set, including a VPS ID, an SPS ID, and a PPS ID. The SPS includes an SPS ID and a VPS ID. The PPS includes a PPS ID and an SPS ID. Each slice header includes a PPS ID. Using the ID, a valid parameter set may be identified for a given slice.

[0086] The PPS includes information applied to all slices in a given picture, in which case all slices in the picture reference the same PPS. Slices in different pictures may also reference the same PPS. The SPS includes information applied to all pictures in the same decoded video sequence (CVS) or bitstream. As previously described, the decoded video sequence is a series of access units (AUs) starting from a random access point picture (e.g., an instantaneous decoding reference (IDR) picture or a broken link access (BLA) picture, or other appropriate random access point picture) in the base layer and having certain properties (described above) until the next AU (or the end of the bitstream) with a random access point picture in the base layer and having certain properties and not including the next AU. The information in the SPS may not change from picture to picture within the decoded video sequence. The pictures in the decoded video sequence may use the same SPS. The VPS includes information applied to all layers within the decoded video sequence or bitstream. The VPS includes a syntax structure with syntax elements applied to the entire decoded video sequence. In some examples, the VPS, SPS, or PPS may be sent in-band with the encoded bitstream. In some examples, the VPS, SPS, or PPS may be sent out-of-band in a separate transmission from the NAL units containing the coded video data.

[0087] The present disclosure may generally involve "signaling" certain information (such as syntax elements). The term "signaling" may generally refer to the communication of values ​​of syntax elements and / or other data used to decode encoded video data. For example, video encoding device 104 may signal the value of a syntax element in a bitstream. Generally speaking, signaling refers to generating a value in a bitstream. As noted above, video source 102 may transmit the bitstream to video destination device 122 in substantially real time or in non-real time (such as may occur when storing syntax elements to storage device 108 for later retrieval by video destination device 122).

[0088] The video bitstream may also include a supplemental enhancement information (SEI) message. For example, a SEI NAL unit may be part of a video bitstream. In some cases, an SEI message may contain information that is not required for the decoding process. For example, the information in the SEI message may not be necessary for a decoder to decode a video picture of the bitstream, but the decoder may use the information to improve the display or processing of the picture (e.g., the decoded output). The information in the SEI message may be embedded metadata. In an illustrative example, the information in the SEI message may be used by a decoder-side entity to improve the visibility of the content. In some instances, certain application standards may indicate the presence of such SEI messages in the bitstream, so that quality improvements can be brought to all devices that comply with the application standards (e.g., carrying a frame encapsulation SEI message for a frame-compatible planar stereoscopic 3DTV video format, where an SEI message is carried for each frame of the video; processing recovery point SEI messages; using a pull-scan rectangle SEI message in DVB; and many other examples).

[0089] As described above, for each block, a motion information set (also referred to herein as motion parameters) may be available. The motion information set contains motion information for forward and backward prediction directions. The forward and backward prediction directions are the two prediction directions of the bidirectional prediction mode, in which case the terms "forward" and "reverse" do not necessarily have a geometric meaning. Instead, "forward" and "reverse" correspond to reference picture list 0 (RefPicList0 or L0) and reference picture list 1 (RefPicList1 or L1) of the current picture. In some examples, when only one reference picture list is available for a picture or slice, only RefPicList0 is available and the motion information for each block of the slice is always forward.

[0090] In some cases, a motion vector is used together with its reference index in a decoding process (e.g., motion compensation). Such a motion vector with an associated reference index is represented as a single prediction set of motion information. For each prediction direction, the motion information may include a reference index and a motion vector. In some cases, for simplicity, the motion vector itself may be referenced by assuming that the motion vector has an associated reference index. The reference index is used to identify a reference picture in the current reference picture list (RefPicList0 or RefPicList1). The motion vector has a horizontal component and a vertical component that provide an offset from a coordinate position in the current picture to a coordinate in the reference picture identified by the reference index. For example, a reference index may indicate a specific reference picture that should be used for a block in the current picture, and a motion vector may indicate the position of the best matching block (the block that best matches the current block) in the reference picture.

[0091] The picture order count (POC) may be used in video coding standards to identify the display order of pictures. Although there are cases where two pictures within a decoded video sequence may have the same POC value, this situation does not usually occur within a decoded video sequence. When multiple decoded video sequences are present in the bitstream, pictures with the same POC value may be closer to each other in decoding order. The POC value of a picture may be used for reference picture list construction, derivation of reference picture sets such as in HEVC, and motion vector scaling.

[0092] In H.264 / AVC, each inter-frame macroblock (MB) can be partitioned in four different ways, including: one 16x16 MB partition; two 16x8 MB partitions; two 8x16 MB partitions; and four 8x8 MB partitions. Different MB partitions in one MB may have different reference index values ​​(RefPicList0 or RefPicList1) for each direction. In some cases, when the MB is not partitioned into four 8x8 MB partitions, the MB may have only one motion vector for each MB partition in each direction. In some cases, when the MB is partitioned into four 8x8 MB partitions, each 8x8 MB partition may be further partitioned into multiple sub-blocks, in which case each sub-block may have a different motion vector in each direction. In some examples, there are four different ways to obtain sub-blocks from an 8x8 MB partition, including: one 8x8 sub-block; two 8x4 sub-blocks; two 4x8 sub-blocks; and four 4x4 sub-blocks. Each sub-block may have a different motion vector in each direction. Therefore, the motion vector exists at a level equal to or higher than the sub-block.

[0093] In AVC, temporal direct mode can be enabled at the MB level or MB partition level for skip and / or direct mode in B slices. For each MB partition, a motion vector is derived using the motion vector of a block in RefPicList1[0] co-located with the current MB partition. Each motion vector in the co-located block is scaled based on the POC distance.

[0094] Spatial direct mode can also be implemented in AVC. For example, in AVC, direct mode can also predict motion information from spatial neighbors.

[0095] As mentioned above, in HEVC, the largest coding unit in a slice is called a coding tree block (CTB). The CTB contains a quadtree whose nodes are coding units. In the HEVC main profile, the size of the CTB can range from 16x16 to 64x64. In some cases, an 8x8 CTB size can be supported. A coding unit (CU) may have the same size as a CTB and be as small as 8x8. In some cases, each coding unit is coded in one mode. When a CU is inter-coded, the CU may be further partitioned into 2 or 4 prediction units (PUs) or become only one PU when further partitioning is not applied. When two PUs are present in one CU, they may be a rectangle of half the size or two rectangles of 1 / 4 or 3 / 4 the size of the CU.

[0096] When a CU is inter-coded, there is a motion information set for each PU. In addition, each PU is coded with a unique inter prediction mode to derive a motion information set.

[0097] For example, for motion prediction in HEVC, there are two inter prediction modes, including merge mode and advanced motion vector prediction (AMVP) mode for prediction units (PUs). Special cases considered as merges are skipped. In AMVP mode or merge mode, a motion vector (MV) candidate list is maintained for multiple motion vector predictors. The motion vector of the current PU and the reference index in merge mode are generated by taking a candidate from the MV candidate list. In some examples, one or more scaling window offsets may be included in the MV candidate list together with the stored motion vectors.

[0098] In an example where an MV candidate list is used for motion prediction of a block, the MV candidate list may be constructed separately by an encoding device and a decoding device. For example, the MV candidate list may be generated by an encoding device when encoding a block, and may be generated by a decoding device when decoding a block. Information related to motion information candidates in the MV candidate list (e.g., information related to one or more motion vectors, information related to one or more LIC flags in the MV candidate list that may be stored in some cases, and / or other information) may be signaled between the encoding device and the decoding device. For example, in merge mode, an index value to the stored motion information candidate may be signaled from the encoding device to the decoding device (e.g., in a syntax structure such as a picture parameter set (PPS), a sequence parameter set (SPS), a video parameter set (VPS), a slice header, a supplemental enhancement information (SEI) message transmitted in a video bitstream or separately from a video bitstream, and / or other signaling). The decoding device may construct an MV candidate list and use the signaled reference or index to obtain one or more motion information candidates from the constructed MV candidate list for motion compensated prediction. For example, the decoding device 112 may construct an MV candidate list and use the motion vector (and in some cases, the LIC flag) from the indexed position for motion prediction of the block. In the case of AMVP mode, in addition to the reference or index, a difference or residual value may also be signaled as a delta. For example, for AMVP mode, the decoding device may construct one or more MV candidate lists and apply the delta value to one or more motion information candidates obtained using the signaled index value in performing motion compensated prediction of the block.

[0099] In some examples, the MV candidate list contains up to five candidates for merge mode and only two candidates for AMVP mode. In other examples, different numbers of candidates may be included in the MV candidate list for merge mode and / or AMVP mode. The merge candidate may contain a motion information set. For example, the motion information set may include motion vectors corresponding to both the reference picture lists (list 0 and list 1) and the reference index. If the merge candidate is identified by the merge index, the reference picture used for the prediction of the current block and the associated motion vector are determined. However, in the AMVP mode for each potential prediction direction from list 0 or list 1, the reference index needs to be explicitly signaled to the MV candidate list together with the MVP index because the AMVP candidate only contains motion vectors. In AMVP mode, the predicted motion vector can be further refined.

[0100] As can be seen above, the merge candidate corresponds to the full set of motion information, while the AMVP candidate contains only one motion vector for a specific prediction direction and reference index.Candidates for both modes are similarly derived from the same spatial and temporal neighboring blocks.

[0101] In some examples, merge mode allows an inter-predicted PU to inherit one or more identical motion vectors, prediction directions, and reference picture indices from an inter-predicted PU that includes a motion data location selected from a set of spatially adjacent motion data locations and one of two temporally co-located motion data locations. For AMVP mode, one or more motion vectors of a PU may be predictively coded relative to one or more motion vector predictors (MVPs) from an AMVP candidate list constructed by an encoder and / or decoder. In some instances, for unidirectional inter prediction of a PU, the encoder and / or decoder may generate a single AMVP candidate list. In some instances, for bidirectional prediction of a PU, the encoder and / or decoder may generate two AMVP candidate lists, one using motion data from spatially and temporally adjacent PUs in a forward prediction direction, and one using motion data from spatially and temporally adjacent PUs in a backward prediction direction.

[0102] Candidates for both modes can be derived from spatial and / or temporal neighboring blocks. Figure 2A and Figure 2B Included is a conceptual diagram showing spatial neighbor candidates. Figure 2A Spatial neighboring motion vector (MV) candidates for merge mode are shown. Figure 2B Spatial neighboring motion vector (MV) candidates for AMVP mode are shown. Spatial MV candidates are derived from neighboring blocks for a specific PU (PU0), but the method of generating candidates from blocks is different for merge and AMVP modes.

[0103] In merge mode, the encoder may form a merge candidate list by considering merge candidates from various motion data locations. Figure 2A As shown in Figure 2A Up to five spatial MV candidates can be derived from the spatially adjacent motion data positions indicated by numbers 0 to 4 in the merge candidate list. The MV candidates can be sorted in the order indicated by numbers 0 to 4 in the merge candidate list. For example, the positions and order may include: left position (0), top position (1), top right position (2), bottom left position (3), and top left position (4). Figure 2A , block 200 includes PU0 202 and PU1 204. In some examples, when the video coder is to code the motion information of PU0 202 using merge mode, the video coder may add the motion information from spatial neighboring block 210, spatial neighboring block 212, spatial neighboring block 214, spatial neighboring block 216, and spatial neighboring block 218 to the candidate list in the order described above.

[0104] exist Figure 2BIn the AVMP mode shown, adjacent blocks are divided into two groups: the left group including blocks 0 and 1; and the upper group including blocks 2, 3, and 4. Figure 2B , blocks 0, 1, 2, 3, and 4 are labeled as blocks 230, 232, 234, 236, and 238, respectively. Here, block 220 includes PU0 222 and PU1 224, and blocks 230, 232, 234, 236, and 238 represent spatial neighbors of PU0 222. For each group, potential candidates among neighboring blocks that refer to the same reference picture as indicated by the signaled reference index have the highest priority to be selected to form the final candidate for the group. All neighboring blocks may not contain motion vectors pointing to the same reference picture. Therefore, if such a candidate cannot be found, the first available candidate will be scaled to form the final candidate, so that the temporal distance difference can be compensated.

[0105] Figure 3A and Figure 3B Included is a conceptual diagram showing temporal motion vector prediction. Figure 3A An example CU 300 is shown including PU0 302 and PU1 304. PU0 302 includes a center block 310 for PU0 302 and a bottom block 306 to the right of PU0 302. Figure 3A Also shown is an external block 308 whose motion information may be predicted from the motion information of PU0 302, as discussed below. Figure 3B A current picture 342 is shown including a current block 326 for which motion information is to be predicted. Figure 3B Also shown are collocated picture 330 to current picture 342 (including collocated block 324 to current block 326), current reference picture 340, and collocated reference picture 332. Collocated block 324 is predicted using collocated motion vector 320 of temporal motion vector predictor (TMVP) candidate 322 used as motion information of current block 326.

[0106] The video coder may add a temporal motion vector predictor (TMVP) candidate (e.g., TMVP candidate 322) to the MV candidate list after any spatial motion vector candidate, if enabled and available. The motion vector derivation process for the TMVP candidate is the same for both merge mode and AMVP mode. However, in some examples, the target reference index of the TMVP candidate in merge mode is always set to zero.

[0107] The main block location for TMVP candidate derivation is the right bottom block 306 outside the collocated PU 304, as shown in FIG. Figure 3A , to compensate for the deviation of the upper and left blocks used to generate spatial neighboring candidates. However, if block 306 is located outside the current CTB (or LCU) row (e.g., Figure 3A308 in PU 306) or if motion information for block 306 is not available, the block is replaced with the center block 310 of PU 302.

[0108] refer to Figure 3B , the motion vector of the TMVP candidate 322 may be derived from the collocated block 324 of the collocated picture 330, indicated in the slice level. Similar to the temporal direct mode in AVC, the motion vector of the TMVP candidate may be subjected to motion vector scaling, which is performed to compensate for the distance difference between the current picture 342 and the current reference picture 340, and the collocated picture 330 and the collocated reference picture 332. That is, the motion vector 320 may be scaled based on the distance difference between the current picture (e.g., current picture 342) and the current reference picture (e.g., current reference picture 340), and the collocated picture (e.g., collocated picture 330) and the collocated reference picture (e.g., collocated reference picture 332) to generate the TMVP candidate 322.

[0109] Other aspects of motion prediction are covered in the HEVC standard and / or other standards, formats or codecs. For example, several other aspects of merge mode and AMVP mode are covered. One aspect includes motion vector scaling. With respect to motion vector scaling, it may be assumed that the value of a motion vector is proportional to the distance of the pictures in presentation time. A motion vector associates two pictures, a reference picture and a picture containing the motion vector (i.e., a containing picture). When one motion vector is used to predict another motion vector, the distance between the containing picture and the reference picture is calculated based on a picture order count (POC) value.

[0110] For a motion vector to be predicted, its associated containing picture and reference picture may both be different. Therefore, a new distance is calculated (based on POC). In addition, the motion vector can be scaled based on these two POC distances. For spatial neighboring candidates, the containing pictures of the two motion vectors are the same, while the reference pictures are different. In HEVC, motion vector scaling is applied to both TMVP and AMVP for spatial and temporal neighboring candidates.

[0111] Another aspect of motion prediction includes artificial motion vector candidate generation. For example, if the motion vector candidate list is incomplete, artificial motion vector candidates are generated and inserted at the end of the list until all candidates are obtained. In merge mode, there are two types of artificial MV candidates: combined candidates derived only for B slices; and zero candidates used only for AMVP if the first type does not provide enough artificial candidates. For each pair of candidates that are already in the candidate list and have the necessary motion information, a bidirectional combined motion vector candidate is derived by combining the motion vector of the first candidate of the picture in reference list 0 and the motion vector of the second candidate of the picture in reference list 1.

[0112] In some implementations, a pruning process may be performed when a new candidate is added or inserted into the MV candidate list. For example, in some cases, MV candidates from different blocks may include the same information. In this case, storing duplicate motion information of multiple MV candidates in the MV candidate list may result in redundancy and reduced efficiency of the MV candidate list. In some examples, the pruning process may eliminate or minimize redundancy in the MV candidate list. For example, the pruning process may include comparing potential MV candidates to be added to the MV candidate list with MV candidates already stored in the MV candidate list. In an illustrative example, the horizontal displacement (Δx) and vertical displacement (Δy) of the stored motion vector (indicating the position of the reference block relative to the position of the current block) may be compared with the horizontal displacement (Δx) and vertical displacement (Δy) of the motion vector of the potential candidate. If the comparison reveals that the motion vector of the potential candidate does not match any of the one or more stored motion vectors, the potential candidate is not considered a candidate to be pruned and may be added to the MV candidate list. If a match is found based on the comparison, the potential MV candidate is not added to the MV candidate list, thereby avoiding insertion of the same candidate. In some cases, to reduce complexity, only a limited number of comparisons are performed during the pruning process instead of comparing each potential MV candidate with all existing candidates.

[0113] In some decoding schemes such as HEVC, weighted prediction (WP) is supported, in which case a scaling factor (denoted by a), a shift number (denoted by s), and an offset (denoted by b) are used in motion compensation. Assuming that the pixel value in the position (x, y) of the reference picture is p(x, y), p'(x, y) = ((a*p(x, y) + (1 << (s-1))) >> s) + b is used as the prediction value in motion compensation instead of p(x, y).

[0114] When WP is enabled, for each reference picture of the current slice, a flag is signaled to indicate whether WP is applicable to the reference picture. If WP is applied to a reference picture, the WP parameter set (i.e., a, s, and b) is transmitted to the decoder and used for motion compensation from the reference picture. In some examples, in order to flexibly turn on / off WP for luma and chroma components, the WP flag and WP parameters are signaled separately for luma and chroma components. In WP, one same WP parameter set is used for all pixels in one reference picture.

[0115] Figure 4A4 is a diagram showing an example of neighboring reconstructed samples of a current block 402 and neighboring samples of a reference block 404 for uni-directional inter prediction. A motion vector MV 410 may be coded for the current block 402, wherein the MV 410 may include a reference index to a reference picture list and / or other motion information for identifying the reference block 404. For example, the MV may include a horizontal component and a vertical component that provide an offset from a coordinate position in the current picture to a coordinate in the reference picture identified by the reference index. Figure 4B 424 and the second reference block 426. In this case, two motion vectors MV0 and MV1 may be coded for the current block 422 to identify the first reference block 424 and the second reference block 426, respectively.

[0116] As previously explained, OBMC is an example motion compensation technique that can be implemented for motion compensation. OBMC can increase prediction accuracy and avoid blocking artifacts. In OBMC, a prediction can be or include a weighted sum of multiple predictions. In some cases, blocks can be larger in each dimension and can overlap with adjacent blocks in a quadrant manner. Therefore, each pixel can belong to multiple blocks. For example, in some exemplary cases, each pixel can belong to 4 blocks. In this scheme, OBMC can implement four predictions for each pixel that total to a weighted average.

[0117] In some cases, OBMC can be turned on and off using a specific syntax at the CU level. In some examples, there are two directional modes (e.g., top, left, right, bottom, or bottom) in OBMC, including a CU boundary OBMC mode and a sub-block boundary OBMC mode. When the CU boundary OBMC mode is used, the original prediction block of the current CU MV and another prediction block (e.g., "OBMC block") using an adjacent CU MV are mixed. In some examples, the left top sub-block in the CU (e.g., the first or leftmost sub-block on the first / top row of the CU) has a top OBMC block and a left OBMC block, and the other topmost sub-blocks (e.g., other sub-blocks on the first / top row of the CU) may have only a top OBMC block. Other leftmost sub-blocks (e.g., sub-blocks on the first column of the CU on the left side of the CU) may have only a left OBMC block.

[0118] When sub-CU decoding tools (e.g., affine motion compensated prediction, advanced temporal motion vector prediction (ATMVP), etc.) that allow different MVs based on sub-blocks are enabled in the current CU, the sub-block boundary OBMC mode may be enabled. In the sub-block boundary OBMC mode, a separate OBMC block using the MV of a connected adjacent sub-block may be mixed with an original prediction block using the MV of the current sub-block. In some examples, in the sub-block boundary OBMC mode, a separate OBMC block using the MV of a connected adjacent sub-block may be mixed in parallel with an original prediction block using the MV of the current sub-block, as further described herein. In other examples, in the sub-block boundary mode, a separate OBMC block using the MV of a connected adjacent sub-block may be mixed sequentially with an original prediction block using the MV of the current sub-block. In some cases, the CU boundary OBMC mode may be executed before the sub-block boundary OBMC mode, and the predefined mixing order of the sub-block boundary OBMC mode may include top, left, bottom, and right.

[0119] The prediction of the MV based on the neighboring sub-blocks N (eg, the sub-blocks above the current sub-block, the left side of the current sub-block, the bottom side of the current sub-block, and the right side of the current sub-block) can be expressed as P N , and the prediction based on the MV of the current sub-block can be expressed as P C When sub-block N contains the same motion information as the current sub-block, the original prediction block may not be mixed with the prediction block based on the MV of sub-block N. In some cases, P N The 4 rows / columns of samples in P C The same samples were mixed.

[0120] Figure 5 is a diagram showing an example data flow 500 for processing video data. Figure 5 In an example of , the video engine 512 may pre-fetch a previous frame 504 to use as a reference for a frame 506 being decoded (e.g., a new frame, a current frame). In some examples, the previous frame 504 may include a previously decoded frame associated with the same sequence of video frames as the frame 506. In addition, the previous frame 504 and the frame 506 being decoded may be stored in the memory 502 for access on demand. The memory 502 may include, for example, a random access memory (RAM) device external to a chip or processor of a hardware decoder (e.g., the encoding device 104 and / or the decoding device 112).

[0121] In some examples, the video engine 512 may extract pixels of the previous frame 504 in blocks of pixels, referred to herein as pixel tiles. For example, the video engine 512 may fetch pixel tiles 508 and 510 of the previous frame 504 from the memory 502. The video engine 512 may cache the pixel tiles 508 and 510 in a reference cache 514. The reference cache 514 may provide the hardware decoder with faster access to the pixel tiles 508 and 510 than the memory 502. In some examples, the reference cache 514 may include cache memory hosted locally on a chip or processor of the hardware decoder. In other words, the reference cache 514 is considered to be on-chip memory of the hardware decoder.

[0122] The video engine 512 may cache the pixel tiles 508 and 510 in the reference cache 514 along with tags 520 and 522 associated with the pixel tiles 508 and 510, respectively. The tags 520 and 522 may include an indication of the location of the pixel tiles 508 and 510 within the previous frame 504 used as a reference frame for coding the frame 506. For example, the tag 520 may include an indication of the location of the pixel tile 508 within the previous frame 504, and the tag 522 may include an indication of the location of the pixel tile 510 within the previous frame 504.

[0123] In some cases, tags 520 and 522 may include addresses of pixel tiles 508 and 510, respectively, within previous frame 504. For example, tag 520 may include an indication of the coordinates of pixel tile 508 within previous frame 504, and tag 522 may include an indication of the coordinates of pixel tile 510 within previous frame 504.

[0124] When the video engine 512 needs pixel tiles 508 and 510 as reference pixels to decode the frame 506, the video engine 512 may transmit corresponding requests 516 for pixel tiles 508 and pixel tiles 510 to the reference cache 514. The corresponding requests 516 may include an indication of the location (e.g., address) of the pixel tiles being requested. For example, the request for pixel tile 508 may include the address (e.g., coordinates) of pixel tile 508 within the previous frame 504, and the request for pixel tile 510 may include the address (e.g., coordinates) of pixel tile 510 within the previous frame 504. When reference cache 514 receives a request (e.g., request 516) for a pixel tile (e.g., pixel tile 508 or pixel tile 510) in reference cache 514, video engine 512 and / or a processor associated with reference cache 514 may compare the position information in the request with the position information in the tags (e.g., tag 520 and tag 522) in reference cache 514. If the request results in a cache hit (e.g., a match between the position information in the request and the position information in the tags), video engine 512 may retrieve the pixel tile associated with the position information that resulted in the cache hit from the reference cache.

[0125] Once the video engine 512 retrieves pixels 518 of a pixel tile (e.g., pixel tile 508 or pixel tile 510) from the reference cache 514 that the video engine 512 will use as reference pixels for decoding the frame 506, the video engine 512 may use the decoded pixels 518 of the pixel tile to code the frame 506. For example, the video engine 512 may retrieve the pixels of the pixel tile 508 from the reference cache 514 and use the pixels of the pixel tile 508 to predict the pixels of the frame 506 being coded. Similarly, the video engine 512 may retrieve the pixels of the pixel tile 510 from the reference cache 514 and use the pixels of the pixel tile 510 to predict the pixels of the frame 506 being coded.

[0126] Fig. 6A 6 is a diagram showing an example data flow 600 for processing a frame 602 of video data (e.g., based on performing intra-frame block copying). For example, based on performing intra-frame block copying, the frame 602 being decoded is also used as a reference frame (e.g., blocks of the frame 602 can be used as reference blocks when decoding other blocks of the frame). In some cases, the frame 602 is decoded one codec region at a time. For example, the video engine 512 can first decode the codec region 604 of the frame 602, and then decode the codec region 606 of the frame 602.

[0127] As shown, pixel map 510 is entirely within codec zone 604. However, pixel map 508 spans codec zone 604 and codec zone 606. Specifically, a portion of pixel map 608, including a portion of pixel map 508, is within codec zone 604, and a portion of pixel map 610, including a portion of pixel map 508, is within codec zone 606. Therefore, when codec zone 604 is decoded (before codec zone 606 is decoded), pixel map 508 will be partially filled, and a portion of pixel map 610 of pixel map 508 will be lost and / or stale.

[0128] exist Fig. 6A In the example of , when codec region 604 is being decoded by video engine 512, video engine 512 may fetch partial pixel map 608 and cache partial pixel map 608 in reference cache 514. When codec region 606 is being decoded by video engine 512, video engine 512 may request pixels from partial pixel map 610 of pixel map 508 using the position information of pixel map 508. Since partial pixel map 610 has not yet been decoded and stored in reference cache 514, when video engine 512 requests pixels from partial pixel map 508 using the address of pixel map 610, the request will not find the pixels associated with partial pixel map 610 in reference cache 514, and the request will therefore be completed with stale, invalid, and / or incomplete data corresponding to partial pixel map 610.

[0129] Figure 6B 6 is a diagram illustrating another example data flow 640 for processing a frame 602 of video data (eg, based on performing intra-frame block copying). Figure 6B In the example of , each codec zone (e.g., codec zone 604 and codec zone 606) is assigned a number. In some examples, each codec zone is assigned a number sequentially in a raster scan order. The codec zone number associated with a partial pixel tile (e.g., partial pixel tile 608 and partial pixel tile 610) can be included in a tag (e.g., tag 644, 646, or 648) of the pixel tile together with the location information (e.g., address) of the pixel tile within frame 602.

[0130] For example, the tag 644 corresponding to the partial pixel map 608 may include codec region information that identifies the codec region number of the codec region 604 being decoded when the pixel map is stored in the reference cache 514 (and / or the codec region in which the partial pixel map 608 resides), and the location of the pixel map 508 including the partial pixel map 608. Similarly, the tag 646 corresponding to the partial pixel map 610 may include codec region information that identifies the codec region number of the codec region 606 being decoded when the pixel map is stored in the reference cache 514 (and / or the codec region in which the partial pixel map 610 resides), and the location of the pixel map 508 including the partial pixel map 610. Therefore, when the video engine 512 needs to retrieve the partial pixel map 608 from the reference cache 514, the video engine 512 can use the tag 644 to locate and retrieve the partial pixel map 608. When the video engine 512 needs to retrieve a partial pixel tile 610 from the reference cache 514, the video engine 512 can use the tag 646 to locate and retrieve the partial pixel tile 610. Thus, the tags 644 and 646 can allow the video engine 512 to retrieve any partial pixel tile that the video engine 512 needs, regardless of which codec region is currently being decoded or has been decoded, thereby avoiding the retrieval of stale, invalid, and / or incomplete pixel tile data, as previously described with respect to Fig. 6A Explained.

[0131] The tag 648 associated with the pixel map tile 510 may include codec region information that identifies the codec region number of the codec region 604 that was being encoded when the pixel map tile was stored in the reference cache 514 (and / or the codec region in which the pixel map tile 510 resides) and the position of the pixel map tile 510 within the frame 602. Thus, the video engine 512 may similarly use the tag 648 associated with the pixel map tile 510 to locate and retrieve the pixel map tile 510 from the reference cache 514.

[0132] When the video engine 512 uses a tag (e.g., tag 644, tag 646, or tag 648) to locate and retrieve pixel tile data from the reference cache 514, the codec zone number and pixel tile location information in the tag (e.g., tag 644, tag 644, or tag 648) may be used to locate and retrieve the entire pixel tile or a portion of the pixel tile. For example, if the pixel tile resides entirely within codec zone 604, such as pixel tile 510, then the tag associated with the pixel tile (e.g., tag 648) may be used to locate and retrieve the entire pixel tile from the reference cache 514 because the tag identifies the codec zone (e.g., codec zone 604) in which the pixel tile resides and the address of the pixel tile within frame 602.

[0133] On the other hand, if the pixel tile includes a portion (e.g., partial pixel tile 608) that resides in one codec zone and another portion (e.g., partial pixel tile 610) that resides in another codec zone, then the tag (e.g., tag 644 or tag 646) associated with the portion of the pixel tile (e.g., partial pixel tile 608 or partial pixel tile 610) retrieved by the video engine 512 can be used to locate and retrieve that particular portion of the entire pixel tile (e.g., without stale, invalid, and / or incomplete data) because the tag includes codec zone information identifying the codec zone (e.g., codec zone 604 or codec zone 606) corresponding to the codec zone being decoded (and / or the codec zone in which the portion of the pixel tile resides) when the pixel tile is stored in the reference cache 514. The tag may also include the address of the pixel tile within the frame 602. Thus, tag 644 may be used to locate and retrieve a portion of pixel tile 608 in its entirety from reference cache 514 when needed, and tag 646 may be used to locate and retrieve a portion of pixel tile 610 in its entirety from reference cache 514 when needed.

[0134] In some examples, when a tag is used to retrieve a pixel tile or a partial pixel tile, an access request (e.g., request 516) from the video engine 512 to the reference cache 514 may generate a cache hit when both the codec region number and the pixel tile location information in the access request match the codec region number and the pixel tile location information in the tag on the reference cache 514. If the access request generates a hit, the requested pixel (e.g., the requested pixel tile or the partial pixel tile) may be read from the reference cache 514, which is faster than reading the requested pixel from the memory 502. If the access request does not generate a hit, the requested pixel (e.g., the requested pixel tile or the partial pixel tile) may be read from the memory 502 because the failure to generate a hit indicates that the requested pixel is not available in the reference cache 514. Figure 6B The data flow 640 shown in FIG. 6 and the use of codec region numbers in the tags (eg, in tags 644 - 648 ) may ensure that partial pixel tiles may be cached in the reference cache 514 and refreshed whenever a new codec region is decoded.

[0135] In some cases, the video engine 512 may decode frames on a per-slice basis. For example, when implementing certain video coding standards and / or schemes, the video engine 512 may decode each frame on a per-slice basis rather than on a per-codec region basis.

[0136] Figure 6C6 is a diagram illustrating another example data flow 660 for processing a frame 602 of video data (eg, based on performing intra-frame block copying). Figure 6C In the example of , the video engine 512 decodes the frame 602 on a per-slice basis. For example, the frame 602 includes a slice 662 and a slice 664, and the video engine 512 first decodes the slice 662 of the frame 602, and then decodes the slice 664 of the frame 602. To create labels 666, 668, 670 associated with the pixel tile 510, the partial pixel tile 608, and the partial pixel tile 610, respectively, the video engine 512 may assign a number to each slice (e.g., slice 662 and slice 664). In some examples, each slice is assigned a number sequentially in a raster scan order.

[0137] The slice number associated with the partial pixel block (e.g., the partial pixel block 608 and the partial pixel block 610) may be included in the tag (e.g., tag 666, 668, or 670) of the pixel block, along with the location information (e.g., address) of the pixel block within the frame 602. For example, the tag 666 corresponding to the partial pixel block 608 may include the slice number of the slice 662 being decoded when the pixel block is stored in the reference cache 514 (and / or the codec region where the partial pixel block 608 resides), and the location of the pixel block 508 including the partial pixel block 608. Similarly, the tag 668 corresponding to the partial pixel block 610 may include the slice number of the slice 664 being decoded when the pixel block is stored in the reference cache 514 (and / or the codec region where the partial pixel block 610 resides), and the location of the pixel block 508 including the partial pixel block 610. Thus, when the video engine 512 needs to retrieve a partial pixel tile 608 from the reference cache 514, the video engine 512 may use the tag 666 to locate and retrieve the partial pixel tile 608. When the video engine 512 needs to retrieve a partial pixel tile 610 from the reference cache 514, the video engine 512 may use the tag 668 to locate and retrieve the partial pixel tile 610. Thus, the tags 666 and 668 may allow the video engine 512 to retrieve any partial pixel tile that the video engine 512 needs, regardless of what slice is currently being (or has been) decoded, which may prevent the video engine 512 from retrieving stale, invalid, and / or incomplete pixel tile data, as previously described with respect to Fig. 6A Explained.

[0138] The tag 670 associated with the pixel map 510 may include the slice number of the slice 662 that was being decoded when the pixel map was stored in the reference cache 514 (and / or the codec region in which the pixel map 510 resides), and the position of the pixel map 510 within the frame 602. Thus, the video engine 512 may similarly use the tag 670 associated with the pixel map 510 to locate and retrieve the pixel map 510 from the reference cache 514.

[0139] When the video engine 512 uses a tag (e.g., tag 666, tag 668, or tag 670) to locate and retrieve pixel tile data from the reference cache 514, the slice number and pixel tile location information in the tag (e.g., tag 666, tag 668, or tag 670) may be used to locate and retrieve the entire pixel tile or a portion of the pixel tile. For example, if the pixel tile resides entirely within slice 662, such as pixel tile 510, then the tag associated with the pixel tile (e.g., tag 670) may be used to locate and retrieve the entire pixel tile from the reference cache 514 because the tag identifies the slice (e.g., slice 662) in which the pixel tile resides and the address of the pixel tile within frame 602.

[0140] On the other hand, if the pixel tile includes a portion (e.g., partial pixel tile 608) that resides in one slice and another portion (e.g., partial pixel tile 610) that resides in another slice, then the tag (e.g., tag 666 or tag 668) associated with the portion of the pixel tile (e.g., partial pixel tile 608 or partial pixel tile 610) retrieved by the video engine 512 can be used to locate and retrieve that particular portion of the entire pixel tile (e.g., without stale, invalid, and / or incomplete data) because the tag identifies the slice (e.g., slice 662 or slice 664) in which the portion of the pixel tile resides and the address of the pixel tile within the frame 602. Thus, the video engine 512 can use tag 666 to locate and retrieve the partial pixel tile 608 in its entirety from the reference cache 514 when needed, and the video engine 512 can use tag 668 to locate and retrieve the partial pixel tile 610 in its entirety from the reference cache 514 when needed.

[0141] In some examples, when a tag is used to retrieve a pixel tile or a partial pixel tile, an access request (e.g., request 516) from the video engine 512 to the reference cache 514 may generate a cache hit when both the slice number and the pixel tile location information in the access request match the slice number and the pixel tile location information in the tag on the reference cache 514. If the access request generates a hit, the requested pixel (e.g., the requested pixel tile or the partial pixel tile) may be read from the reference cache 514, which is faster than reading the requested pixel from the memory 502. If the access request does not generate a hit, the requested pixel (e.g., the requested pixel tile or the partial pixel tile) may be read from the memory 502 because the failure to generate a hit indicates that the requested pixel is not available in the reference cache 514. Figure 6C The data flow 660 shown in FIG. 6 and the use of slice numbers in the tags (eg, in tags 666 - 670 ) may ensure that partial pixel tiles may be cached in the reference cache 514 and refreshed whenever a new slice is decoded.

[0142] Figure 7 7 is a flow chart illustrating an example process 700 for encoding or decoding (decoding) video data (e.g., based on cache-unaligned pixel tiles) according to aspects described herein. At block 702, the process may include determining a first codec region (e.g., codec region 604, codec region 606, slice 662, or slice 664) associated with a first block to be decoded. The first codec region may include one of a plurality of regions of a frame (e.g., frame 602). In some examples, each of the plurality of regions of the frame may be independently coded (e.g., independently coded from one another).

[0143] At block 704, process 700 may include determining whether reference pixels of a first version of a pixel map (e.g., pixel map 508, pixel map 510, partial pixel map 608, or partial pixel map 610) are stored in a cache (e.g., reference cache 514) while decoding one or more blocks from a second codec zone (e.g., codec zone 604, codec zone 606, slice 662, or slice 664). In some examples, the cache may be hosted locally on a chip (e.g., a processor, processing circuitry, etc.) of a hardware coder (e.g., encoding device 104 and / or decoding device 112). In some cases, the pixel map corresponds to a location within a frame (e.g., frame 602). In some cases, the first codec zone may include one of the first codec zone or the first slice, and the second codec zone may include one of the second codec zone or the second slice.

[0144] In some examples, determining whether reference pixels for a first version of a pixel map are stored in a cache while decoding one or more blocks from a second codec zone may include determining that the first version of the pixel map is associated with the first codec zone or the second codec zone. In some cases, a hardware decoder (e.g., encoding device 104 and / or decoding device 112) or a video engine of the hardware decoder (e.g., video engine 512) may determine whether reference pixels for a first version of a pixel map are stored in a cache while decoding one or more blocks from a second codec zone based on a tag (e.g., tag 644, tag 646, tag 648, tag 666, tag 668, or tag 670). In some examples, the tag may include a first indication of a position of the pixel map within a frame and a second indication associating the first version of the pixel map with one of the first codec zone or the second codec zone.

[0145] At block 706, process 700 may include determining whether to read the first version of the pixel map from the cache or retrieve a second version of the pixel map (e.g., pixel map 508, pixel map 510, partial pixel map 608, or partial pixel map 610) from a memory device external to a chip of the hardware decoder (e.g., memory 502) based on determining whether reference pixels for the first version of the pixel map are stored in the cache while decoding one or more blocks from the second codec region. In some examples, the second version of the pixel map may include reference pixels from the first codec region that are not in the first version of the pixel map.

[0146] In some examples, the memory device may include a random access memory (RAM) device, such as, for example, a double data rate (DDR) synchronous dynamic random access memory (SDRAM) or any other type of memory device. In addition, the RAM device may be a memory separate from the cache.

[0147] At block 708 , process 700 may include decoding a first block associated with the first codec region based at least in part on the first version of the pixel map read from the cache or the second version of the pixel map retrieved from the memory device.

[0148] In some aspects, process 700 may include determining that reference pixels for a first version of a pixel block are stored in a cache while decoding one or more blocks from a second codec zone; and determining to retrieve a second version of the pixel block from a memory device based on determining that reference pixels for the first version of the pixel block are stored in the cache while decoding one or more blocks from the second codec zone.

[0149] In some aspects, process 700 may include: determining that reference pixels for a first version of a pixel block are not stored in a cache while decoding one or more blocks from a second codec zone; and determining to read the first version of the pixel block from the cache based on determining that reference pixels for the first version of the pixel block are not stored in the cache while decoding one or more blocks from the second codec zone.

[0150] In some aspects, process 700 may include determining a location of the pixel tile within a frame. Process 700 may include determining that a first version of the pixel tile resides within one of a first codec region or a second codec region. Process 700 may also include generating a tag that includes a first indication of a location of the pixel tile within a frame and data indicating that the first version of the pixel tile resides within one of the first codec region or the second codec region. Process 700 may include storing the tag in a cache and associating it with the first version of the pixel tile.

[0151] In some aspects, process 700 may include generating a request (e.g., request 516) to access reference pixels of a first version of a pixel tile from a cache. Process 700 may include determining that both a corresponding position of a pixel tile identified in the request and a corresponding codec region identified in the request match a position of the pixel tile identified in a tag associated with the first version of the pixel tile and one of the first codec region or the second codec region identified in the tag. Process 700 may also include determining, based on the match, that reference pixels of the first version of the pixel tile are stored in the cache while decoding one or more blocks from the second codec region. In some examples, the request may identify a corresponding position of the pixel tile within a frame and a corresponding codec region being decoded when the first version of the pixel tile is stored in the cache (and / or a codec region in which the first version of the pixel tile resides).

[0152] In some aspects, process 700 may include generating a request to access reference pixels of a first version of a pixel tile from a cache. In some examples, the request identifies a corresponding position of the pixel tile within a frame and a corresponding codec zone that is being decoded when the first version of the pixel tile is stored in the cache (and / or a codec zone in which the first version of the pixel tile resides). In some aspects, process 700 may also include: determining that the corresponding position of the pixel tile identified in the request and / or the corresponding codec zone identified in the request do not match the position of the pixel tile identified in the tag associated with the first version of the pixel tile and the position of one of the first codec zone or the second codec zone identified in the tag; and based on determining that the corresponding position of the pixel tile identified in the request and at least one of the corresponding codec zones identified in the request do not match the position of the pixel tile identified in the tag associated with the first version of the pixel tile and the first codec zone or the second codec zone identified in the tag, determining that the reference pixels of the first version of the pixel tile are not stored in the cache while decoding one or more blocks from the second codec zone.

[0153] In some aspects, process 700 may include: determining that a first version of a pixel tile resides within a first codec zone and a second version of the pixel tile resides within a second codec zone; and based on determining that the first version of the pixel tile resides within the first codec zone and the second version of the pixel tile resides within the second codec zone, prohibiting use of a cache to store or access the first version of the pixel tile and the second version of the pixel tile.

[0154] In some cases, process 700 may be implemented by an encoder and / or a decoder.

[0155] In some implementations, the processes (or methods) described herein (including process 700) may be performed by a computing device or apparatus (such as Figure 1 For example, the system 100 shown in FIG. Figure 1 and Figure 8 The encoding device 104 shown in the figure is provided by another video source side device or a video sending device. Figure 1 and Fig. 9 The processes are performed by the decoding device 112 shown and / or by another client-side device (such as a player device, a display, or any other client-side device). In some cases, the computing device or apparatus may include one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, and / or other components configured to perform the steps of process 700.

[0156] In some examples, the computing device may include a mobile device, a desktop computer, a server computer and / or server system, or other types of computing devices. The components of the computing device (e.g., one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, and / or other components) may be implemented in circuits. For example, the components may include electronic circuits or other electronic hardware and / or may be implemented using electronic circuits or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, graphics processing units (GPUs), digital signal processors (DSPs), central processing units (CPUs), and / or other suitable electronic circuits) and / or may include computer software, firmware, or any combination thereof for performing the various operations described herein and / or may be implemented using computer software, firmware, or any combination thereof for performing the various operations described herein.

[0157] In some examples, the computing device or apparatus may include a camera configured to capture video data (e.g., a video sequence) including video frames. In some examples, the camera or other capture device that captures the video data is separate from the computing device, in which case the computing device receives or obtains the captured video data. The computing device may also include a network interface configured to communicate the video data. The network interface may be configured to communicate data based on an Internet Protocol (IP) or other types of data. In some examples, the computing device or apparatus may include a display for displaying output video content (such as samples of pictures of a video bitstream).

[0158] The processes may be described relative to logical flow diagrams whose operations represent sequences of operations that can be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, each operation represents a computer-executable instruction stored on one or more computer-readable storage media that, when executed by one or more processors, performs the described operations. Generally speaking, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform specific functions or implement specific data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be combined in any order and / or in parallel to implement the processes.

[0159] In addition, these processes can be performed under the control of one or more computer systems configured with executable instructions, and can be implemented by hardware as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed together on one or more processors, or a combination thereof. As noted above, the code can be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program including a plurality of instructions that can be executed by one or more processors. The computer-readable or machine-readable storage medium can be non-transitory.

[0160] The decoding techniques discussed herein can be implemented in an example video encoding and decoding system (e.g., system 100). In some examples, a system includes a source device that provides encoded video data that will be decoded by a target device later. Specifically, the source device provides video data to the target device via a computer-readable medium. The source device and the target device may include any of a variety of devices, including desktop computers, notebook (i.e., laptop) computers, tablet computers, set-top boxes, telephone handsets (such as so-called "smart" phones, so-called "smart" tablets), televisions, cameras, display devices, digital media players, video game consoles, video streaming devices, etc. In some cases, the source device and the target device may be equipped for wireless communication.

[0161] The target device may receive the encoded video data to be decoded via a computer-readable medium. The computer-readable medium may include any type of medium or device capable of moving the encoded video data from the source device to the target device. In one example, the computer-readable medium may include a communication medium that enables the source device to directly send the encoded video data to the target device in real time. The encoded video data may be modulated according to a communication standard (such as a wireless communication protocol) and sent to the target device. The communication medium may include any wireless or wired communication medium, such as a radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form a part of a packet-based network such as a local area network, a wide area network, or a global network such as the Internet. The communication medium may include a router, a switch, a base station, or any other equipment that may be useful for facilitating communication from the source device to the target device.

[0162] In some examples, the encoded data can be output from the output interface to a storage device. Similarly, the encoded data can be accessed from the storage device through the input interface. The storage device may include any data storage medium in a variety of distributed or locally accessed data storage media, such as a hard drive, a Blu-ray disc, a DVD, a CD-ROM, a flash memory, a volatile or non-volatile memory, or any other suitable digital storage medium for storing the encoded video data. In another example, the storage device may correspond to a file server or another intermediate storage device that can store the encoded video generated by the source device. The target device can access the stored video data from the storage device via streaming or downloading. The file server can be any type of server that can store the encoded video data and send the encoded video data to the target device. Example file servers include web servers (e.g., for websites), FTP servers, network attached storage (NAS) devices, or local disk drives. The target device can access the encoded video data through any standard data connection (including an Internet connection). The data connection may include a wireless channel (e.g., a Wi-Fi connection), a wired connection (e.g., a DSL, a cable modem, etc.) suitable for accessing the encoded video data stored on the file server, or a combination of the two. The transmission of the encoded video data from the storage device may be streaming, download transmission, or a combination thereof.

[0163] The techniques of the present disclosure are not necessarily limited to wireless applications or settings. These techniques can be applied to video decoding to support any of a variety of multimedia applications, such as over-the-air television broadcasts, cable television transmissions, satellite television transmissions, Internet streaming video transmissions (such as Dynamic Adaptive Streaming over HTTP (DASH)), digital video encoded on a data storage medium, decoding of digital video stored on a data storage medium, or other applications. In some examples, the system can be configured to support one-way or two-way video transmission to support applications such as video streaming, video playback, video broadcasting, and / or video telephony.

[0164] In one example, the source device includes a video source, a video encoder, and an output interface. The target device may include an input interface, a video decoder, and a display device. The video encoder of the source device may be configured to apply the technology disclosed herein. In other examples, the source device and the target device may include other components or arrangements. For example, the source device may receive video data from an external video source such as an external camera. Similarly, the target device may be docked with an external display device, rather than including an integrated display device.

[0165] The above example system is just an example. The technology for processing video data simultaneously can be performed by any digital video encoding and / or decoding device. Although generally speaking the technology of the present disclosure is performed by a video encoding device, these technologies can also be performed by a video encoder / decoder (commonly referred to as a "codec"). In addition, the technology of the present disclosure can also be performed by a video preprocessor. The source device and the target device are just examples of such decoding devices, wherein the source device generates the decoded video data to send to the target device. In some examples, the source device and the target device can operate in a substantially symmetrical manner so that each of these devices includes a video encoding and decoding component. Therefore, the example system can support one-way or two-way video transmission between video devices, for example, for video streaming, video playback, video broadcasting or video telephony.

[0166] The video source may include a video capture device, such as a video camera, a video archiving unit including previously captured video, and / or a video feed interface for receiving video from a video content provider. As another alternative, the video source may generate computer graphics-based data as the source video, or generate a combination of real-time video, archived video, and computer-generated video. In some cases, if the video source is a video camera, the source device and the target device may form a so-called camera phone or video phone. However, as mentioned above, the technology described in the present disclosure may be generally applicable to video decoding, and may be applicable to wireless and / or wired applications. In each case, the captured, pre-captured or computer-generated video may be encoded by a video encoder. The encoded video information may then be output to a computer-readable medium by an output interface.

[0167] As noted, computer-readable media may include transient media (such as wireless broadcasts or wired network transmissions) or storage media (i.e., non-transient storage media), such as hard disks, flash drives, compact discs, digital video discs, Blu-ray discs, or other computer-readable media. In some examples, a network server (not shown) may receive the encoded video data from a source device and provide the encoded video data to a target device, for example, via a network transmission. Similarly, a computing device of a media production facility (such as a disc stamping facility) may receive the encoded video data from a source device and produce a disc containing the encoded video data. Therefore, in various examples, computer-readable media may be understood to include one or more computer-readable media in various forms.

[0168] An input interface of a target device receives information from a computer-readable medium. The information of the computer-readable medium may include syntax information defined by a video encoder, which is also used by a video decoder, and the syntax information includes syntax elements describing characteristics and / or processing of blocks and other decoded units (e.g., groups of pictures (GOPs)). A display device displays the decoded video data to a user and may include any of a variety of display devices, such as a cathode ray tube (CRT), a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, or another type of display device. Various aspects of the present application have been described.

[0169] The details of the encoding device 104 and the decoding device 112 are respectively described in Figure 8 and Fig. 9 Shown in. Figure 8 1 is a block diagram illustrating an example encoding device 104 that may implement one or more of the techniques described in the present disclosure. For example, the encoding device 104 may generate the syntax structures described herein (e.g., syntax structures of a VPS, SPS, PPS, or other syntax elements). The encoding device 104 may perform intra-prediction and inter-prediction coding of video blocks within a video slice. As previously described, intra-coding relies at least in part on spatial prediction to reduce or remove spatial redundancy within a given video frame or picture. Inter-coding relies at least in part on temporal prediction to reduce or remove temporal redundancy within adjacent or surrounding frames of a video sequence. Intra-mode (I-mode) may refer to any of several spatial-based compression modes. Inter-mode (such as unidirectional prediction (P-mode) or bidirectional prediction (B-mode)) may refer to any of several temporal-based compression modes.

[0170] The encoding device 104 includes a segmentation unit 35, a prediction processing unit 41, a filter unit 63, a picture memory 64, a summer 50, a transform processing unit 52, a quantization unit 54, and an entropy coding unit 56. The prediction processing unit 41 includes a motion estimation unit 42, a motion compensation unit 44, and an intra-frame prediction processing unit 46. For video block reconstruction, the encoding device 104 also includes an inverse quantization unit 58, an inverse transform processing unit 60, and a summer 62. The filter unit 63 is intended to represent one or more loop filters, such as a deblocking filter, an adaptive loop filter (ALF), and a sample adaptive offset (SAO) filter. Although the filter unit 63 is not limited to a single-loop filter, the filter unit 63 is not limited to a single-loop filter, and the filter unit 63 is not limited to a single-loop filter. Figure 8 6 is shown as an in-loop filter, but in other configurations, filter unit 63 may be implemented as a post-loop filter. Post-processing device 57 may perform additional processing on the encoded video data generated by encoding device 104. In some examples, the techniques of this disclosure may be implemented by encoding device 104. However, in other cases, one or more of the techniques of this disclosure may be implemented by post-processing device 57.

[0171] like Figure 8 As shown, the encoding device 104 receives video data, and the segmentation unit 35 segments the data into video blocks. Segmentation may also include segmentation into slices, slice segments, tiles or other larger units and video block segmentation (e.g., according to a quadtree structure of LCUs and CUs). The encoding device 104 generally shows the components for encoding video blocks within a video slice to be encoded. A slice may be divided into multiple video blocks (and may be divided into sets of video blocks called tiles). The prediction processing unit 41 may select one of a plurality of possible decoding modes for the current video block based on error results (e.g., decoding rate and distortion level, etc.), such as one of one or more inter-frame prediction decoding modes in a plurality of intra-frame prediction decoding modes. The prediction processing unit 41 may provide the resulting intra-frame decoded or inter-frame decoded block to the summer 50 to generate residual block data and to the summer 62 to reconstruct the encoded block for use as a reference picture.

[0172] Intra-prediction processing unit 46 within prediction processing unit 41 may perform intra-prediction coding of the current video block relative to one or more neighboring blocks in the same frame or slice as the current block to be coded to provide spatial compression. Motion estimation unit 42 and motion compensation unit 44 within prediction processing unit 41 may perform inter-prediction coding of the current video block relative to one or more prediction blocks in one or more reference pictures to provide temporal compression.

[0173] Motion estimation unit 42 may be configured to determine an inter-prediction mode for a video slice according to a predetermined pattern for a video sequence. The predetermined pattern may designate a video slice in a sequence as a P slice, a B slice, or a GPB slice. Motion estimation unit 42 and motion compensation unit 44 may be highly integrated but are shown separately for conceptual purposes. Motion estimation performed by motion estimation unit 42 is the process of generating motion vectors, which estimate the motion of a video block. For example, a motion vector may indicate the displacement of a prediction unit (PU) of a video block within a current video frame or picture relative to a prediction block within a reference picture.

[0174] A prediction block is a block that is found to closely match a PU of a video block to be coded in terms of pixel difference, which may be determined by a sum of absolute difference (SAD), a sum of squared difference (SSD), or other difference metric. In some examples, encoding device 104 may calculate values ​​for sub-integer pixel positions of a reference picture stored in picture memory 64. For example, encoding device 104 may interpolate values ​​for quarter-pixel positions, eighth-pixel positions, or other fractional pixel positions of a reference picture. Thus, motion estimation unit 42 may perform a motion search relative to full pixel positions and fractional pixel positions and output a motion vector with fractional pixel precision.

[0175] Motion estimation unit 42 calculates a motion vector for a PU by comparing the position of a PU of a video block in an inter-coded slice to the position of a prediction block of a reference picture. The reference picture may be selected from a first reference picture list (List 0) or a second reference picture list (List 1), each of which identifies one or more reference pictures stored in picture memory 64. Motion estimation unit 42 communicates the calculated motion vector to entropy encoding unit 56 and motion compensation unit 44.

[0176] Motion compensation performed by motion compensation unit 44 may involve extracting or generating a prediction block based on a motion vector determined by motion estimation, thereby possibly performing interpolation to sub-pixel precision. Upon receiving the motion vector of the PU of the current video block, motion compensation unit 44 may locate the prediction block pointed to by the motion vector in the reference picture list. The encoding device 104 forms a residual video block by subtracting the pixel values ​​of the prediction block from the pixel values ​​of the current video block being decoded, thereby forming a pixel difference value. The pixel difference value forms the residual data of the block and may include both a luminance difference component and a chrominance difference component. Summer 50 represents one or more components that perform a subtraction operation. Motion compensation unit 44 may also generate syntax elements associated with video blocks and video slices for use by the decoding device 112 when decoding video blocks of the video slices.

[0177] Intra-prediction processing unit 46 may perform intra-prediction on the current block as an alternative to inter-prediction performed by motion estimation unit 42 and motion compensation unit 44, as described above. Specifically, intra-prediction processing unit 46 may determine an intra-prediction mode for encoding the current block. In some examples, intra-prediction processing unit 46 may encode the current block using various intra-prediction modes, for example, during a separate encoding process, and intra-prediction processing unit 46 may select an appropriate intra-prediction mode to use from the tested modes. For example, intra-prediction processing unit 46 may calculate rate-distortion values ​​using rate-distortion analysis for various tested intra-prediction modes, and may select an intra-prediction mode with the best rate-distortion characteristics among the tested modes. Rate-distortion analysis typically determines the amount of distortion (or error) between the encoded block and the original unencoded block that is encoded to produce the encoded block, as well as the bit rate (i.e., the number of bits) used to produce the encoded block. Intra-prediction processing unit 46 may calculate ratios from the distortions and rates for the various encoded blocks to determine which intra-prediction mode exhibits the best rate-distortion value for the block.

[0178] In any case, after selecting an intra-prediction mode for a block, intra-prediction processing unit 46 may provide information indicating the selected intra-prediction mode for the block to entropy encoding unit 56. Entropy encoding unit 56 may encode the information indicating the selected intra-prediction mode. Encoding device 104 may include in the transmitted bitstream configuration data a definition of coding contexts for various blocks and an indication of the most probable intra-prediction mode, an intra-prediction mode index table, and a modified intra-prediction mode index table for each of the contexts. The bitstream configuration data may include a plurality of intra-prediction mode index tables and a plurality of modified intra-prediction mode index tables (also referred to as codeword mapping tables).

[0179] After the prediction processing unit 41 generates a prediction block for the current video block via inter-frame prediction or intra-frame prediction, the encoding device 104 forms a residual video block by subtracting the prediction block from the current video block. The residual video data in the residual block may be included in one or more TUs and applied to the transform processing unit 52. The transform processing unit 52 transforms the residual video data into residual transform coefficients using a transform (such as a discrete cosine transform (DCT) or a conceptually similar transform). The transform processing unit 52 may convert the residual video data from a pixel domain to a transform domain, such as a frequency domain.

[0180] Transform processing unit 52 may pass the resulting transform coefficients to quantization unit 54. Quantization unit 54 quantizes the transform coefficients to further reduce the bit rate. The quantization process may reduce the bit depth associated with some or all of the coefficients. The degree of quantization may be modified by adjusting a quantization parameter. In some examples, quantization unit 54 may then perform a scan of the matrix including the quantized transform coefficients. Alternatively, entropy encoding unit 56 may perform the scan.

[0181] After quantization, entropy encoding unit 56 entropy encodes the quantized transform coefficients. For example, entropy encoding unit 56 may perform context adaptive variable length coding (CAVLC), context adaptive binary arithmetic coding (CABAC), syntax-based context adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or another entropy encoding technique. After entropy encoding by entropy encoding unit 56, the encoded bitstream may be sent to decoding device 112, or archived for later sending or retrieval by decoding device 112. Entropy encoding unit 56 may also entropy encode motion vectors and other syntax elements for the current video slice being coded.

[0182] Inverse quantization unit 58 and inverse transform processing unit 60 apply inverse quantization and inverse transform, respectively, to reconstruct the residual block in the pixel domain for later use as a reference block of a reference picture. Motion compensation unit 44 may calculate a reference block by adding the residual block to a prediction block of one of the reference pictures in a reference picture list. Motion compensation unit 44 may also apply one or more interpolation filters to the reconstructed residual block to calculate sub-integer pixel values ​​for use in motion estimation. Summer 62 adds the reconstructed residual block to the motion compensated prediction block produced by motion compensation unit 44 to produce a reference block for storage in picture memory 64. The reference block may be used by motion estimation unit 42 and motion compensation unit 44 as a reference block for inter-prediction of a block in a subsequent video frame or picture.

[0183] In this way, Figure 8 The encoding device 104 represents an apparatus configured to perform any of the techniques described herein (including the above-mentioned Figure 7 In some cases, some of the techniques of the present disclosure may also be implemented by post-processing device 57.

[0184] Fig. 9 1 is a block diagram illustrating an example decoding device 112. The decoding device 112 includes an entropy decoding unit 80, a prediction processing unit 81, an inverse quantization unit 86, an inverse transform processing unit 88, a summer 90, a filter unit 91, and a picture memory 92. The prediction processing unit 81 includes a motion compensation unit 82 and an intra-frame prediction processing unit 84. In some examples, the decoding device 112 may perform substantially the same Figure 8 The encoding pass described in the encoding device 104 is a decoding pass that is the inverse of the encoding pass.

[0185] During the decoding process, the decoding device 112 receives an encoded video bitstream representing video blocks and associated syntax elements of the encoded video slice transmitted by the encoding device 104. In some aspects, the decoding device 112 may receive the encoded video bitstream from the encoding device 104. In some aspects, the decoding device 112 may receive the encoded video bitstream from the encoding device 104. In some aspects, the decoding device 112 may receive the encoded video bitstream from a network entity 79 (such as a server, a media-aware network element (MANE), a video editor / splicer, or other such device configured to implement one or more of the above-mentioned techniques. The network entity 79 may or may not include the encoding device 104. Some of the techniques described in the present disclosure may be implemented by the network entity 79 before the network entity 79 sends the encoded video bitstream to the decoding device 112. In some video decoding systems, the network entity 79 and the decoding device 112 may be parts of separate devices, while in other examples, the functionality described with respect to the network entity 79 may be performed by the same device including the decoding device 112.

[0186] The entropy decoding unit 80 of the decoding device 112 entropy decodes the bitstream to generate quantized coefficients, motion vectors and other syntax elements. The entropy decoding unit 80 forwards the motion vectors and other syntax elements to the prediction processing unit 81. The decoding device 112 may receive syntax elements at the video slice level and / or the video block level. The entropy decoding unit 80 may process and parse both fixed length syntax elements and variable length syntax elements in one or more parameter sets (such as VPS, SPS, and PPS).

[0187] When the video slice is decoded as an intra-coded (I) slice, the intra-prediction processing unit 84 of the prediction processing unit 81 may generate prediction data for the video block of the current video slice based on the signaled intra-prediction mode and data from a previously decoded block of the current frame or picture. When the video frame is decoded as an inter-coded (i.e., B, P, or GPB) slice, the motion compensation unit 82 of the prediction processing unit 81 generates a prediction block for the video block of the current video slice based on the motion vector and other syntax elements received from the entropy decoding unit 80. The prediction block may be generated from one of the reference pictures in the reference picture list. The decoding device 112 may construct the reference frame lists (list 0 and list 1) using a default construction technique based on the reference pictures stored in the picture memory 92.

[0188] Motion compensation unit 82 determines prediction information for a video block of the current video slice by parsing the motion vectors and other syntax elements, and uses the prediction information to generate a prediction block for the current video block being decoded. For example, motion compensation unit 82 may use one or more syntax elements in a parameter set to determine a prediction mode (e.g., intra prediction or inter prediction) for coding a video block of a video slice, an inter-prediction slice type (e.g., a B slice, a P slice, or a GPB slice), construction information for one or more reference picture lists for the slice, a motion vector for each inter-coded video block of the slice, an inter-prediction state for each inter-coded video block of the slice, and other information for decoding a video block in the current video slice.

[0189] The motion compensation unit 82 may also perform interpolation based on interpolation filters. The motion compensation unit 82 may calculate interpolated values ​​for sub-integer pixels of the reference block using interpolation filters as used by the encoding device 104 during encoding of the video block. In this case, the motion compensation unit 82 may determine the interpolation filters used by the encoding device 104 from the received syntax elements and may use the interpolation filters to produce the prediction block.

[0190] Inverse quantization unit 86 inverse quantizes or dequantizes the quantized transform coefficients provided in the bitstream and decoded by entropy decoding unit 80. The inverse quantization process may include determining the degree of quantization using quantization parameters calculated by encoding device 104 for each video block in the video slice, and likewise determining the degree of inverse quantization that should be applied. Inverse transform processing unit 88 applies an inverse transform (e.g., an inverse DCT or other suitable inverse transform), an inverse integer transform, or a conceptually similar inverse transform process to the transform coefficients to produce residual blocks in the pixel domain.

[0191] After motion compensation unit 82 generates a prediction block for the current video block based on the motion vector and other syntax elements, decoding device 112 forms a decoded video block by adding the residual block from inverse transform processing unit 88 to the corresponding prediction block generated by motion compensation unit 82. Summer 90 represents one or more components that perform this summation operation. If desired, a loop filter (in the decoding loop or after the decoding loop) may also be used to smooth pixel transitions or otherwise improve video quality. Filter unit 91 is intended to represent one or more loop filters, such as a deblocking filter, an adaptive loop filter (ALF), and a sample adaptive offset (SAO) filter. Although filter unit 91 is Fig. 9 91 is shown as an in-loop filter, but in other configurations, filter unit 91 may be implemented as a post-loop filter. The decoded video blocks in a given frame or picture are then stored in picture memory 92, which stores reference pictures for subsequent motion compensation. Picture memory 92 also stores the decoded video for later viewing on a display device such as Figure 1 The video shown is presented on target device 122).

[0192] In this way, Fig. 9 The decoding device 112 represents a device configured to perform any of the techniques described herein (including the above-mentioned Figure 7 An example of a video decoder of the process described).

[0193] As used herein, the term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing or carrying instructions and / or data. Computer-readable media may include non-transient media in which data may be stored and does not include carrier waves and / or transient electronic signals propagated wirelessly or on a wired connection. Examples of non-transient media may include, but are not limited to, disks or tapes, optical storage media (such as compact disks (CDs) or digital versatile disks (DVDs)), flash memory, memory, or memory devices. Computer-readable media may store thereon code and / or machine-executable instructions, which may represent procedures, functions, subroutines, programs, routines, subroutines, modules, software packages, categories, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by transmitting and / or receiving information, data, independent variables, parameters, or memory contents. Information, independent variables, parameters, data, etc. may be transmitted, forwarded, or sent via any suitable means, including memory sharing, message passing, token passing, network transmission, etc.

[0194] In some examples, computer-readable storage devices, media, and memories may include cables or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer-readable storage media specifically exclude media such as power consumption, carrier signals, electromagnetic waves, and signals themselves.

[0195] Specific details are provided in the above description to provide a detailed understanding of the aspects and examples provided herein. However, it will be understood by those skilled in the art that examples can be implemented without these specific details. For clarity, in some cases, the present technology may be presented as including separate functional blocks, including functional blocks comprising devices, device components, steps or routines in methods embodied in software or a combination of hardware and software. Additional components other than those components shown in the accompanying drawings and / or described herein may be used. For example, circuits, systems, networks, processes and other components may be shown as components in block diagram form so as not to obscure aspects of the present disclosure with unnecessary details. In other examples, known circuits, processes, algorithms, structures and techniques may be shown without unnecessary details to avoid obscuring aspects of the present disclosure.

[0196] The above may describe various examples as processes or methods, which are depicted as flow charts, flow diagrams, data flow diagrams, structure diagrams, or block diagrams. Although a flow chart may describe an operation as a sequential process, many of the operations in the operation may be performed in parallel or concurrently. In addition, the order of the operations may be rearranged. The process is terminated when the operation of the process is completed, but the process may have additional steps not included in the accompanying drawings. A process may correspond to a method, function, procedure, subroutine, subprogram, etc. When a process corresponds to a function, the termination of the process may correspond to the function returning to the calling function or main function.

[0197] The processes and methods according to the above examples can be implemented using stored computer executable instructions or otherwise obtained from computer readable media. These instructions may include, for example, instructions and data that configure a general-purpose computer, a special-purpose computer, or a processing device to perform a certain function or function group. Parts of the computer resources used can be accessed through a network. Computer executable instructions can be, for example, binary, intermediate format instructions, such as assembly language, firmware, source code, etc. Examples of computer readable media that can be used to store instructions, information used, and / or information created during the methods according to the described examples include disks or optical disks, flash memory, USB devices with non-volatile memory, networked storage devices, etc.

[0198] The equipment implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description language, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) for performing necessary tasks may be stored in a computer-readable or machine-readable medium. The processor may perform the necessary tasks. Typical examples of form factors include laptop computers, smart phones, mobile phones, tablet devices, or other small form factor personal computers, personal digital assistants, rack-mounted devices, stand-alone devices, etc. The functionality described herein may also be embodied in peripheral devices or plug-in cards. By way of additional examples, such functionality may also be implemented on circuit boards among different chips or different processes executed on a single device.

[0199] Instructions, media for conveying such instructions, computing resources for executing them, and other structures for supporting such computing resources are example means for providing the functionality described in this disclosure.

[0200] In the foregoing description, various aspects of the present application are described with reference to the specific examples of the present application, but those skilled in the art will recognize that the present application is not limited thereto. Therefore, although the illustrative examples of the present application have been described in detail herein, it should be understood that the inventive concept can be implemented and adopted in various other ways, and the appended claims are intended to be interpreted as including these variations, unless limited by the prior art. The various features and aspects of the above-mentioned applications can be used individually or in combination. In addition, without departing from the broader spirit and scope of this specification, various aspects of the present disclosure can be utilized in any number of environments and applications beyond those described herein. Therefore, the description and the accompanying drawings should be considered as illustrative rather than restrictive. For the purpose of illustration, each method is described in a specific order. It should be understood that in an alternative example, the method can be performed in an order different from the described order.

[0201] It should be understood by those of ordinary skill in the art that the less than ("<") and greater than (">") symbols or terms used herein may be replaced by less than or equal to ("≤") and greater than or equal to ("≥") symbols, respectively, without departing from the scope of the present specification.

[0202] Where a component is described as being “configured to” perform certain operations, such configuration may be achieved, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., a microprocessor or other suitable electronic circuits) to perform the operations, or any combination thereof.

[0203] The phrase "coupled to" means that any component is directly or indirectly physically connected to another component, and / or any component is directly or indirectly in communication with another component (e.g., connected to another component via a wired or wireless connection and / or other suitable communication interface).

[0204] Claim language or other language in this disclosure that states "at least one of" a set and / or "one or more of" a set indicates that one member of the set or multiple members of the set (in any combination) satisfies the claim. For example, claim language reciting "at least one of A and B" means A, B, or A and B. In another example, claim language reciting "at least one of A, B, and C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language "at least one of" a set and / or "one or more of" a set does not limit the set to the items listed in the set. For example, claim language reciting "at least one of A and B" may mean A, B, or A and B, and may additionally include items not listed in the set of A and B.

[0205] Various illustrative logic blocks, modules, circuits, and algorithmic steps described in conjunction with the examples disclosed herein may be implemented as electronic hardware, computer software, firmware, or a combination thereof. In order to clearly illustrate this interchangeability of hardware and software, various illustrative components, frames, modules, circuits, and steps have been generally described in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints proposed for the entire system. The technician may implement the described functionality in different ways for each specific application, but such specific implementation decisions should not be interpreted as departing from the scope of the present application.

[0206] The technology described herein can also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such technology can be implemented in any of a variety of devices, such as general-purpose computers, wireless communication device mobile phones, or integrated circuit devices with multiple uses, including applications in wireless communication device mobile phones and other devices. Any features described as modules or components can be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, these technologies can be implemented at least in part by a computer-readable data storage medium including a program code, the program code including instructions, which execute one or more of the above methods when executed. Computer-readable data storage media can form a part of a computer program product, which can include packaging materials. Computer-readable media can include memory or data storage media, such as random access memory (RAM) (such as synchronous dynamic random access memory (SDRAM)), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, etc. Additionally or alternatively, the technology may be implemented at least in part by a computer-readable communication medium that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer, such as a propagated signal or wave.

[0207] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such processors may be configured to perform any of the techniques described in the present disclosure. A general-purpose processor may be a microprocessor; but in an alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Therefore, the term "processor" as used herein may refer to any of the aforementioned structures, any combination of the aforementioned structures, or any other structure or device suitable for implementing the techniques described herein. In addition, in some aspects, the functionality described herein may be provided in a dedicated software module or hardware module configured for encoding and decoding, or incorporated into a combined video encoder-decoder (CODEC).

[0208] Illustrative aspects of the present disclosure include:

[0209] Aspect 1. A device for encoding or decoding (decoding) video data, the device comprising: a memory; and one or more processors coupled to the memory, the one or more processors being configured to: determine a first codec zone associated with a first block to be decoded, the first codec zone being one of a plurality of codec zones of a frame, wherein each of the plurality of codec zones of the frame is independently decodable; determine whether reference pixels of a first version of a pixel map block are stored in a cache while decoding one or more blocks from a second codec zone, wherein the cache is locally hosted on a chip of a hardware decoder, and wherein the pixel map block corresponds to a position within the frame; based on Determine whether the reference pixels of the first version of the pixel block are stored in the cache while decoding the one or more blocks from the second codec zone, determine whether to read the first version of the pixel block from the cache or retrieve a second version of the pixel block from a memory device external to the chip of the hardware decoder, wherein the second version of the pixel block includes reference pixels from the first codec zone that are not in the first version of the pixel block; and decode the first block associated with the first codec zone based at least in part on the first version of the pixel block read from the cache or the second version of the pixel block retrieved from the memory device.

[0210] Aspect 2. An apparatus according to Aspect 1, wherein in order to determine whether the reference pixels of the first version of the pixel block are stored in the cache while decoding the one or more blocks from the second codec zone, the one or more processors are configured to determine whether the first version of the pixel block is associated with the first codec zone or the second codec zone.

[0211] Aspect 3. An apparatus according to any one of Aspects 1 to 2, wherein the one or more processors are configured to determine whether the reference pixels of the first version of the pixel tile are stored in the cache based on a tag while decoding the one or more blocks from the second codec zone.

[0212] Aspect 4. An apparatus according to Aspect 3, wherein the tag comprises a first indication of the position of the pixel block within the frame and a second indication associating the first version of the pixel block with one of the first codec zone or the second codec zone.

[0213] Aspect 5. An apparatus according to any one of Aspects 1 to 4, wherein the first codec region comprises one of a first codec tile or a first slice, and wherein the second codec region comprises one of a second codec tile or a second slice.

[0214] Aspect 6. An apparatus according to any one of Aspects 1 to 5, wherein the one or more processors are configured to: determine that the reference pixels of the first version of the pixel block are stored in the cache while decoding the one or more blocks from the second codec zone; and determine to retrieve the second version of the pixel block from the memory device based on determining that the reference pixels of the first version of the pixel block are stored in the cache while decoding the one or more blocks from the second codec zone.

[0215] Aspect 7. An apparatus according to Aspect 6, wherein the one or more processors are configured to: determine that the reference pixels of the first version of the pixel block are not stored in the cache while decoding the one or more blocks from the second codec zone; and determine to read the first version of the pixel block from the cache based on determining that the reference pixels of the first version of the pixel block are not stored in the cache while decoding the one or more blocks from the second codec zone.

[0216] Aspect 8. An apparatus according to any one of Aspects 1 to 7, wherein the one or more processors are configured to: determine the position of the pixel block within the frame; determine that the first version of the pixel block resides within one of the first codec zone or the second codec zone; generate a tag, the tag comprising a first indication of the position of the pixel block within the frame and data indicating that the first version of the pixel block resides within one of the first codec zone or the second codec zone; and store the tag in the cache and associate it with the first version of the pixel block.

[0217] Aspect 9. An apparatus according to Aspect 8, wherein the one or more processors are configured to: generate a request for accessing the reference pixels of the first version of the pixel block from the cache, wherein the request identifies a corresponding position of the pixel block within the frame and a corresponding codec zone that is decoded when the first version of the pixel block is stored in the cache; determine that both the corresponding position of the pixel block identified in the request and the corresponding codec zone identified in the request match the position of the pixel block identified in the tag associated with the first version of the pixel block and one of the first codec zone or the second codec zone identified in the tag; and based on the match, determine that the reference pixels of the first version of the pixel block are stored in the cache while decoding the one or more blocks from the second codec zone.

[0218] Aspect 10. An apparatus according to aspect 8, wherein the one or more processors are configured to: generate a request to access the reference pixels of the first version of the pixel tile from the cache, wherein the request identifies a corresponding position of the pixel tile within the frame and a corresponding codec region that is decoded when the first version of the pixel tile is stored in the cache; determine whether at least one of the corresponding position of the pixel tile identified in the request and the corresponding codec region identified in the request is consistent with the position of the pixel tile identified in the tag associated with the first version of the pixel tile and the corresponding codec region identified in the tag; based on determining that the corresponding position of the pixel block identified in the request and at least one of the corresponding codec zones identified in the request do not match the position of the pixel block identified in the tag associated with the first version of the pixel block and the first codec zone or the second codec zone identified in the tag, determining that the reference pixels of the first version of the pixel block are not stored in the cache while decoding the one or more blocks from the second codec zone.

[0219] Aspect 11. An apparatus according to any one of Aspects 1 to 10, wherein the one or more processors are configured to: determine that the first version of the pixel block resides within the first codec zone and the second version of the pixel block resides within the second codec zone; and based on determining that the first version of the pixel block resides within the first codec zone and the second version of the pixel block resides within the second codec zone, prohibit the use of the cache to store or access the first version of the pixel block and the second version of the pixel block.

[0220] Aspect 12. An apparatus according to any one of aspects 1 to 11, wherein the memory device comprises a random access memory (RAM) device, and wherein the RAM device is a memory separate from the cache.

[0221] Aspect 13. An apparatus according to any one of aspects 1 to 12, wherein the apparatus comprises a decoder.

[0222] Aspect 14. An apparatus according to any one of aspects 1 to 13, the apparatus further comprising a display configured to display one or more output pictures associated with the video data.

[0223] Aspect 15. An apparatus according to any one of aspects 1 to 14, wherein the apparatus comprises an encoder.

[0224] Aspect 16. The apparatus according to any one of aspects 1 to 15, further comprising a camera configured to capture a picture associated with the video data.

[0225] Aspect 17. An apparatus according to any one of aspects 1 to 16, wherein the apparatus is a mobile device.

[0226] Aspect 18. A method for encoding or decoding (decoding) video data, the method comprising: determining a first codec zone associated with a first block to be decoded, the first codec zone being one of a plurality of codec zones of a frame, wherein each of the plurality of codec zones of the frame is independently decodable; determining whether reference pixels of a first version of a pixel map block are stored in a cache while decoding one or more blocks from a second codec zone, wherein the cache is locally hosted on a chip of a hardware decoder, and wherein the pixel map block corresponds to a position within the frame; determining a reference pixel of a first version of a pixel map block based on the one or more blocks from the second codec zone being decoded. Determine whether the reference pixels of the first version of the pixel block are stored in the cache while decoding the block, determine whether to read the first version of the pixel block from the cache or retrieve a second version of the pixel block from a memory device external to the chip of the hardware decoder, wherein the second version of the pixel block includes reference pixels from the first codec zone that are not in the first version of the pixel block; and decode the first block associated with the first codec zone based at least in part on the first version of the pixel block read from the cache or the second version of the pixel block retrieved from the memory device.

[0227] Aspect 19. A method according to Aspect 18, wherein determining whether the reference pixels of the first version of the pixel block are stored in the cache while decoding the one or more blocks from the second codec zone includes: determining whether the first version of the pixel block is associated with the first codec zone or the second codec zone.

[0228] Aspect 20. A method according to any one of Aspects 18 to 19, the method also includes determining whether the reference pixels of the first version of the pixel tile are stored in the cache based on a tag while decoding the one or more blocks from the second codec zone.

[0229] Aspect 21. A method according to Aspect 20, wherein the label includes a first indication of the position of the pixel block within the frame and a second indication associating the first version of the pixel block with one of the first codec zone or the second codec zone.

[0230] Aspect 22. A method according to any one of Aspects 18 to 21, wherein the first codec zone comprises one of a first codec tile or a first slice, and wherein the second codec zone comprises one of a second codec tile or a second slice.

[0231] Aspect 23. A method according to any one of Aspects 18 to 22, the method further comprising: determining that the reference pixels of the first version of the pixel block are stored in the cache while decoding the one or more blocks from the second codec zone; and determining to retrieve the second version of the pixel block from the memory device based on determining that the reference pixels of the first version of the pixel block are stored in the cache while decoding the one or more blocks from the second codec zone.

[0232] Aspect 24. The method according to Aspect 23 further includes: determining that the reference pixels of the first version of the pixel block are not stored in the cache while decoding the one or more blocks from the second codec zone; and determining to read the first version of the pixel block from the cache based on determining that the reference pixels of the first version of the pixel block are not stored in the cache while decoding the one or more blocks from the second codec zone.

[0233] Aspect 25. A method according to any one of Aspects 18 to 24, the method further comprising: determining the position of the pixel block within the frame; determining that the first version of the pixel block resides within one of the first codec zone or the second codec zone; generating a tag, the tag comprising a first indication of the position of the pixel block within the frame and data indicating that the first version of the pixel block resides within one of the first codec zone or the second codec zone; and storing the tag in the cache and associating it with the first version of the pixel block.

[0234] Aspect 26. The method according to Aspect 25 further includes: generating a request for accessing the reference pixels of the first version of the pixel block from the cache, wherein the request identifies the corresponding position of the pixel block within the frame and the corresponding codec zone when the first version of the pixel block is stored in the cache; determining that both the corresponding position of the pixel block identified in the request and the corresponding codec zone identified in the request match the position of the pixel block identified in the tag associated with the first version of the pixel block and the first codec zone or the second codec zone identified in the tag; and based on the match, determining that the reference pixels of the first version of the pixel block are stored in the cache while decoding the one or more blocks from the second codec zone.

[0235] Aspect 27. A method according to aspect 25, the method further comprising: generating a request to access the reference pixels of the first version of the pixel block from the cache, wherein the request identifies a corresponding position of the pixel block within the frame and a corresponding codec zone when the first version of the pixel block is stored in the cache; determining whether at least one of the corresponding position of the pixel block identified in the request and the corresponding codec zone identified in the request is consistent with the position of the pixel block identified in the tag associated with the first version of the pixel block and the corresponding codec zone identified in the tag. the first codec zone or the second codec zone does not match; and based on determining that the corresponding position of the pixel block identified in the request and at least one of the corresponding codec zone identified in the request do not match the position of the pixel block identified in the tag associated with the first version of the pixel block and the first codec zone or the second codec zone identified in the tag, determining that the reference pixels of the first version of the pixel block are not stored in the cache while decoding the one or more blocks from the second codec zone.

[0236] Aspect 28. A method according to any one of Aspects 18 to 27, the method further comprising: determining that the first version of the pixel block resides in the first codec zone and the second version of the pixel block resides in the second codec zone; and based on determining that the first version of the pixel block resides in the first codec zone and the second version of the pixel block resides in the second codec zone, prohibiting the use of the cache to store or access the first version of the pixel block and the second version of the pixel block.

[0237] Aspect 29. The method according to any one of aspects 18 to 28, wherein the memory device comprises a random access memory (RAM) device, and wherein the RAM device is a memory separate from the cache.

[0238] Aspect 30. The method according to any one of aspects 18 to 29, wherein coding the first block associated with the first codec zone comprises decoding the first block.

[0239] Aspect 31. The method according to any one of aspects 18 to 30, wherein decoding the first block associated with the first codec zone includes encoding the first block.

[0240] Aspect 32. Provided is a non-transitory computer-readable medium having instructions stored thereon, which, when executed by one or more processors, cause the one or more processors to perform operations according to any one of Aspects 1 to 31.

[0241] Aspect 33. Provided is an apparatus for encoding or decoding (decoding) video data, the apparatus comprising one or more components for performing the operations according to any one of aspects 1 to 31.

Claims

1. A device for encoding or decoding video data, the device comprising: Memory; and one or more processors coupled to the memory, the one or more processors configured to: determining a first codec zone associated with a first block to be decoded, the first codec zone being one of a plurality of codec zones of a frame, wherein each of the plurality of codec zones of the frame is independently decodable; determining whether reference pixels for a first version of a pixel map block are stored in a cache while decoding one or more blocks from a second codec zone, wherein the cache is hosted locally on a chip of a hardware decoder, and wherein the pixel map block corresponds to a location within the frame; determining whether to read the first version of the pixel map block from the cache or to retrieve a second version of the pixel map block from a memory device external to the chip of the hardware decoder based on determining whether the reference pixels of the first version of the pixel map block are stored in the cache while decoding the one or more blocks from the second codec region, wherein the second version of the pixel map block includes reference pixels from the first codec region that are not in the first version of the pixel map block; as well as The first block associated with the first codec region is decoded based at least in part on the first version of the pixel map read from the cache or the second version of the pixel map retrieved from the memory device.

2. An apparatus according to claim 1, wherein in order to determine whether the reference pixels of the first version of the pixel map block are stored in the cache while decoding the one or more blocks from the second codec zone, the one or more processors are configured to determine whether the first version of the pixel map block is associated with the first codec zone or the second codec zone.

3. The apparatus of claim 1 , wherein the one or more processors are configured to determine whether the reference pixels of the first version of the pixel tile are stored in the cache while decoding the one or more blocks from the second codec zone based on a tag.

4. A device according to claim 3, wherein the tag includes a first indication of the position of the pixel tile within the frame and a second indication associating the first version of the pixel tile with one of the first codec region or the second codec region.

5. The device of claim 1, wherein the first codec region comprises one of a first codec tile or a first slice, and wherein the second codec region comprises one of a second codec tile or a second slice.

6. The apparatus of claim 1, wherein the one or more processors are configured to: determining that the reference pixels of the first version of the pixel map are stored in the cache while decoding the one or more blocks from the second codec zone; and Based on determining that the reference pixels of the first version of the pixel map block are stored in the cache while decoding the one or more blocks from the second codec zone, determining to retrieve the second version of the pixel map block from the memory device.

7. The apparatus of claim 6, wherein the one or more processors are configured to: determining, while decoding the one or more blocks from the second codec zone, that the reference pixels of the first version of the pixel map are not stored in the cache; and Based on determining that the reference pixels of the first version of the pixel map block are not stored in the cache while decoding the one or more blocks from the second codec zone, determining to read the first version of the pixel map block from the cache.

8. The apparatus of claim 1, wherein the one or more processors are configured to: determining the position of the pixel block within the frame; determining that the first version of the pixel block resides within one of the first codec region or the second codec region; generating a tag comprising a first indication of the location of the pixel tile within the frame and data indicating that the first version of the pixel tile resides within the one of the first codec region or the second codec region; as well as The tag is stored in the cache and associated with the first version of the pixel tile.

9. The apparatus of claim 8, wherein the one or more processors are configured to: generating a request to access the reference pixels of the first version of the pixel tile from the cache, wherein the request identifies a corresponding position of the pixel tile within the frame and a corresponding codec region when the first version of the pixel tile is stored in the cache; determining that both the corresponding position of the pixel tile identified in the request and the corresponding codec region identified in the request match the position of the pixel tile identified in the tag associated with the first version of the pixel tile and the one of the first codec region or the second codec region identified in the tag; as well as Based on the match, determining that the reference pixels of the first version of the pixel map are stored in the cache while decoding the one or more blocks from the second codec zone.

10. The apparatus of claim 8, wherein the one or more processors are configured to: generating a request to access the reference pixels of the first version of the pixel tile from the cache, wherein the request identifies a corresponding position of the pixel tile within the frame and a corresponding codec region when the first version of the pixel tile is stored in the cache; determining that at least one of the corresponding position of the pixel tile identified in the request and the corresponding codec region identified in the request does not match the position of the pixel tile identified in the tag associated with the first version of the pixel tile and the one of the first codec region or the second codec region identified in the tag; as well as Based on determining that at least one of the corresponding position of the pixel map identified in the request and the corresponding codec zone identified in the request does not match the position of the pixel map identified in the tag associated with the first version of the pixel map and the first codec zone or the second codec zone identified in the tag, determining that the reference pixels of the first version of the pixel map are not stored in the cache while decoding the one or more blocks from the second codec zone.

11. The apparatus of claim 1 , wherein the one or more processors are configured to: determining that the first version of the pixel tile resides within the first codec region and the second version of the pixel tile resides within the second codec region; and Based on determining that the first version of the pixel tile resides within the first codec zone and the second version of the pixel tile resides within the second codec zone, prohibiting use of the cache to store or access the first version of the pixel tile and the second version of the pixel tile.

12. The apparatus of claim 1, wherein the memory device comprises a random access memory (RAM) device, and wherein the RAM device is a separate memory from the cache.

13. The apparatus of claim 1, wherein the apparatus comprises a decoder.

14. The apparatus of claim 13, further comprising a display configured to display one or more output pictures associated with the video data.

15. The apparatus of claim 1, wherein the apparatus comprises an encoder.

16. The apparatus of claim 15, further comprising a camera configured to capture a picture associated with the video data.

17. The apparatus of claim 1, wherein the apparatus is a mobile device.

18. A method for encoding or decoding video data, the method comprising: determining a first codec zone associated with a first block to be decoded, the first codec zone being one of a plurality of codec zones of a frame, wherein each of the plurality of codec zones of the frame is independently decodable; determining whether reference pixels for a first version of a pixel map block are stored in a cache while decoding one or more blocks from a second codec zone, wherein the cache is hosted locally on a chip of a hardware decoder, and wherein the pixel map block corresponds to a location within the frame; determining whether to read the first version of the pixel map block from the cache or to retrieve a second version of the pixel map block from a memory device external to the chip of the hardware decoder based on determining whether the reference pixels of the first version of the pixel map block are stored in the cache while decoding the one or more blocks from the second codec region, wherein the second version of the pixel map block includes reference pixels from the first codec region that are not in the first version of the pixel map block; as well as The first block associated with the first codec region is decoded based at least in part on the first version of the pixel map read from the cache or the second version of the pixel map retrieved from the memory device.

19. The method of claim 18, wherein determining whether the reference pixels of the first version of the pixel map are stored in the cache while decoding the one or more blocks from the second codec zone comprises: A determination is made as to whether the first version of the pixel tile is associated with the first codec region or the second codec region.

20. The method of claim 18, further comprising determining whether the reference pixels of the first version of the pixel tile are stored in the cache while decoding the one or more blocks from the second codec zone based on a tag.

21. The method of claim 20, wherein the tag comprises a first indication of the position of the pixel tile within the frame and a second indication associating the first version of the pixel tile with one of the first codec region or the second codec region.

22. The method of claim 18, wherein the first codec region comprises one of a first codec tile or a first slice, and wherein the second codec region comprises one of a second codec tile or a second slice.

23. The method according to claim 18, further comprising: determining that the reference pixels of the first version of the pixel map are stored in the cache while decoding the one or more blocks from the second codec zone; as well as Based on determining that the reference pixels of the first version of the pixel map block are stored in the cache while decoding the one or more blocks from the second codec zone, determining to retrieve the second version of the pixel map block from the memory device.

24. The method according to claim 23, further comprising: determining, while decoding the one or more blocks from the second codec zone, that the reference pixels of the first version of the pixel map are not stored in the cache; as well as Based on determining that the reference pixels of the first version of the pixel map block are not stored in the cache while decoding the one or more blocks from the second codec zone, determining to read the first version of the pixel map block from the cache.

25. The method according to claim 18, further comprising: determining the position of the pixel block within the frame; determining that the first version of the pixel block resides within one of the first codec region or the second codec region; generating a tag comprising a first indication of the location of the pixel tile within the frame and data indicating that the first version of the pixel tile resides within the one of the first codec region or the second codec region; as well as The tag is stored in the cache and associated with the first version of the pixel tile.

26. The method according to claim 25, further comprising: generating a request to access the reference pixels of the first version of the pixel tile from the cache, wherein the request identifies a corresponding position of the pixel tile within the frame and a corresponding codec region when the first version of the pixel tile is stored in the cache; determining that both the corresponding position of the pixel tile identified in the request and the corresponding codec region identified in the request match the position of the pixel tile identified in the tag associated with the first version of the pixel tile and the one of the first codec region or the second codec region identified in the tag; as well as Based on the match, determining that the reference pixels of the first version of the pixel map are stored in the cache while decoding the one or more blocks from the second codec zone.

27. The method according to claim 25, further comprising: generating a request to access the reference pixels of the first version of the pixel tile from the cache, wherein the request identifies a corresponding position of the pixel tile within the frame and a corresponding codec region when the first version of the pixel tile is stored in the cache; determining that at least one of the corresponding position of the pixel tile identified in the request and the corresponding codec region identified in the request does not match the position of the pixel tile identified in the tag associated with the first version of the pixel tile and the one of the first codec region or the second codec region identified in the tag; as well as Based on determining that at least one of the corresponding position of the pixel map identified in the request and the corresponding codec zone identified in the request does not match the position of the pixel map identified in the tag associated with the first version of the pixel map and the first codec zone or the second codec zone identified in the tag, determining that the reference pixels of the first version of the pixel map are not stored in the cache while decoding the one or more blocks from the second codec zone.

28. The method of claim 18, wherein the memory device comprises a random access memory (RAM) device, and wherein the RAM device is a separate memory from the cache.

29. The method of claim 18, wherein coding the first block associated with the first codec region comprises decoding the first block.

30. The method of claim 18, wherein decoding the first block associated with the first codec zone comprises encoding the first block.

Images