Video decoding system and method with improved buffer storage and bandwidth efficiency

Abstract

Video decoding systems and techniques are described. A decoder reads first video data from a first block of a video frame. The decoder retrieves neighboring video data from the line buffer. The adjacent video data is from an adjacent block adjacent to a first block in the video frame. The decoder processes the first video data and the retrieved neighboring video data using a constrained directional enhancement filter (CDEF) to generate filtered first video data. The decoder amplifies the filtered first video data using an amplifier to generate amplified filtered first video data. For example, after generating the filtered first video data, the decoder amplifies the retrieved neighboring video data using an amplifier to generate amplified neighboring video data. The decoder processes the amplified filtered first video data and the amplified neighboring video data using a loop recovery (LR) filter to generate output video data.

Description

Technical Field

[0001] The present application is related to video decoding and decompression. More specifically, the present application relates to systems and methods for performing improved video decoding that reduces the storage space required for adjacent block buffers, reduces redundancy of amplifier components, and improves bandwidth and efficiency. Background Art

[0002] Digital video capabilities can be integrated into a wide range of devices, including digital televisions, digital direct broadcast systems, wireless broadcast systems, personal digital assistants (PDAs), portable or desktop computers, tablet computers, e-book readers, digital cameras, digital recording devices, digital media players, video game devices, cellular or satellite radio telephones, mobile phones (e.g., so-called "smart phones"), video teleconferencing devices, video streaming devices, etc. Such devices allow video data to be processed and output for consumption. Digital video data includes large amounts of data to meet the needs of consumers and video providers. For example, consumers of video data want videos with the highest quality, high fidelity, resolution, frame rate, etc. The large amounts of video data required to meet these needs burden the communication networks and devices that process and store the video data.

[0003] Digital video devices may implement video decoding techniques to compress video data. Video decoding may be performed according to one or more video decoders and / or decoding formats. For example, video decoders and / or decoding formats include Versatile Video Coding (VVC), Basic Video Coding (EVC), High Efficiency Video Coding (HEVC), VP8, VP9, ​​Advanced Video Coding (AVC), MPEG-2 Part 2 Coding (MPEG stands for Moving Picture Experts Group), etc., as well as proprietary video decoders / formats such as AOMedia Video 1 (AV1) developed by the Alliance for Open Media and SMPTE 421 (also known as VC-1). Video decoding generally uses a prediction method (e.g., inter-frame prediction, intra-frame prediction, etc.) that exploits the redundancy present in a video image or sequence. One goal of video decoding techniques is to compress video data into a form that uses a lower bit rate while avoiding or minimizing degradation of video quality. One goal of video decoding techniques is to reproduce the original video data as closely as possible from the compressed video data. As evolving video services become available, there is a need for coding and decoding techniques with improved coding and decoding efficiencies. Summary of the invention

[0004] Systems and techniques for video decoding are described. In some examples, a decoder reads first video data from a first block of a video frame. The decoder retrieves adjacent video data from a line buffer. The adjacent video data is from an adjacent block adjacent to the first block in the video frame. The decoder processes the first video data and the retrieved adjacent video data using a constrained directional enhancement filter (CDEF) to generate filtered first video data. The decoder amplifies the filtered first video data using an amplifier to generate amplified filtered first video data. For example, after generating the filtered first video data, the decoder amplifies the retrieved adjacent video data using an amplifier to generate amplified adjacent video data. The decoder processes the amplified filtered first video data and the amplified adjacent video data using a loop recovery (LR) filter to generate output video data.

[0005] In one example, a device for media processing is provided. The device includes a memory and one or more processors (e.g., implemented in a circuit) coupled to the memory. The one or more processors are configured to and can: read first video data from a first block of a video frame; retrieve adjacent video data from a line buffer, wherein the adjacent video data is from an adjacent block adjacent to the first block in the video frame; process the first video data and the adjacent video data using a constrained directional enhancement filter (CDEF) to generate filtered first video data, wherein the adjacent video data used by the CDEF is retrieved from the line buffer; amplify the filtered first video data using an amplifier to generate amplified filtered first video data; amplify the adjacent video data retrieved from the line buffer using an amplifier to generate amplified adjacent video data; and process the amplified filtered first video data and the amplified adjacent video data using a loop recovery (LR) filter to generate output video data.

[0006] In another example, an image processing method is provided. The method includes: reading first video data from a first block of a video frame; retrieving adjacent video data from a line buffer, wherein the adjacent video data is from an adjacent block adjacent to the first block in the video frame; processing the first video data and the adjacent video data using a constrained directional enhancement filter (CDEF) to generate filtered first video data, wherein the adjacent video data used by the CDEF is retrieved from the line buffer; amplifying the filtered first video data using an amplifier to generate amplified filtered first video data; amplifying the adjacent video data retrieved from the line buffer using an amplifier to generate amplified adjacent video data; and processing the amplified filtered first video data and the amplified adjacent video data using a loop recovery (LR) filter to generate output video data.

[0007] In another example, a non-transitory computer-readable medium is provided having instructions stored thereon that, when executed by one or more processors, cause the one or more processors to: read first video data from a first block of a video frame; retrieve adjacent video data from a line buffer, wherein the adjacent video data is from an adjacent block adjacent to the first block in the video frame; process the first video data and the adjacent video data using a constrained directional enhancement filter (CDEF) to generate filtered first video data, wherein the adjacent video data used by the CDEF is retrieved from the line buffer; amplify the filtered first video data using an amplifier to generate amplified filtered first video data; amplify the adjacent video data retrieved from the line buffer using an amplifier to generate amplified adjacent video data; and process the amplified filtered first video data and the amplified adjacent video data using a loop restoration (LR) filter to generate output video data.

[0008] In another example, a device for image processing is provided. The device includes: a component for reading first video data from a first block of a video frame; a component for retrieving adjacent video data from a line buffer, wherein the adjacent video data is from an adjacent block adjacent to the first block in the video frame; a component for processing the first video data and the adjacent video data using a constrained directional enhancement filter (CDEF) to generate filtered first video data, wherein the adjacent video data used by the CDEF is retrieved from the line buffer; a component for amplifying the filtered first video data using an amplifier to generate amplified filtered first video data; a component for amplifying the adjacent video data retrieved from the line buffer using an amplifier to generate amplified adjacent video data; and a component for processing the amplified filtered first video data and the amplified adjacent video data using a loop recovery (LR) filter to generate output video data.

[0009] In some aspects, after generating the filtered first video data, adjacent video data is upscaled.

[0010] In some aspects, one or more of the above methods, apparatus, and computer-readable media further include filtering the first video data using a deblocking (DB) filter to read the first video data, wherein the adjacent video data is filtered using the DB filter before retrieving the adjacent video data from the line buffer.

[0011] In some aspects, one or more of the above methods, apparatuses, and computer-readable media further include: reading adjacent video data from adjacent blocks of the video frame before retrieving the adjacent video data from the line buffer; and storing the adjacent video data in the line buffer before retrieving the adjacent video data from the line buffer. In some aspects, one or more of the above methods, apparatuses, and computer-readable media further include: filtering the first video data and the adjacent video data using a deblocking (DB) filter before storing the adjacent video data in the line buffer. In some aspects, the adjacent video data is unamplified when retrieved from the line buffer.

[0012] In some aspects, reading the first video data and retrieving the adjacent video data comprises reading the first video data and retrieving the adjacent video data in a vertical raster scan order. In some aspects, the adjacent video data retrieved from the line buffer comprises a portion of one or more pixel rows corresponding to one or more columns along the vertical raster scan order, wherein a width of the portion of the one or more pixel rows is less than an entire width of the first block.

[0013] In some aspects, one or more of the above methods, apparatuses, and computer-readable media further include: setting an indication for the first video data, wherein the indication is retained in the filtered first video data, the upscaled filtered first video data, and the output video data; and based on the presence of the indication in the output video data, identifying a first subset of the output video data corresponding to the first block and a second subset of the output video data corresponding to the adjacent block, wherein the first subset of the output video data corresponding to the first block is configured to be displayed as part of the decoded video. In some aspects, one or more of the above methods, apparatuses, and computer-readable media further include: setting an indication for the adjacent video data, wherein the indication is retained during processing using CDEF, in the upscaled adjacent video data, and in the output video data; and based on the presence of the indication in the output video data, identifying a first subset of the output video data corresponding to the first block and a second subset of the output video data corresponding to the adjacent block, wherein the first subset of the output video data corresponding to the first block is configured to be displayed as part of the decoded video.

[0014] In some aspects, one or more of the above-mentioned methods, devices and computer-readable media also include: copying at least one row of pixels in the upscaled adjacent video data to generate modified upscaled adjacent video data; and in response to generating the modified upscaled adjacent video data, processing the amplified and filtered first video data and the upscaled adjacent video data using an LR filter to generate output video data.

[0015] In some aspects, the neighboring block is located above the first block in the video frame.

[0016] In some aspects, the adjacent video data comprises 4×W pixels, where W is a tile width corresponding to at least one tile of the video frame. In some aspects, the memory bandwidth of the line buffer is 4×W×F pixels per second, where W is a tile width corresponding to at least one tile of the video frame, where F is a frame rate associated with a video, where the video comprises the video frame.

[0017] In some aspects, the CDEF, amplifier, and LR filters are associated with the AOMedia Video 1 (AV1) format.

[0018] In some aspects, one or more of the above methods, apparatuses, and computer-readable media further include: outputting the output video data for storage in at least one memory. In some aspects, one or more of the above methods, apparatuses, and computer-readable media further include: using a display to cause the output video data to be displayed. In some aspects, one or more of the above methods, apparatuses, and computer-readable media further include: using a communication interface to cause the output video data to be sent to a recipient device.

[0019] In some aspects, the first video data and the adjacent video data are encoded using a video encoder prior to reading the first video data and retrieving the adjacent video data.

[0020] In some aspects, the apparatus is part of and / or includes a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a head mounted display (HMD) device, a wireless communication device, a mobile device (e.g., a mobile phone and / or a mobile handset and / or a so-called "smart phone" or other mobile device), a camera, a personal computer, a laptop computer, a server computer, a vehicle or a computing device or component of a vehicle, another device, or a combination thereof. In some aspects, the apparatus includes a camera or multiple cameras for capturing one or more images. In some aspects, the apparatus further includes a display for displaying one or more images, notifications, and / or other displayable data. In some aspects, the apparatus may include one or more sensors (e.g., one or more inertial measurement units (IMUs), such as one or more gyroscopes, one or more gyrometers, one or more accelerometers, any combination thereof, and / or other sensors).

[0021] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter. The subject matter of the present invention should be understood by reference to appropriate portions of the entire specification of this patent, any or all drawings, and each claim.

[0022] The foregoing and other features and aspects will become more apparent upon reference to the following description, claims and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Illustrative aspects of the present application are described in detail below with reference to the following drawings:

[0024] Figure 1 is a block diagram illustrating an example of a system including an encoding device and a decoding device according to some examples;

[0025] Figure 2 is a block diagram illustrating a decoder system according to some examples, the decoder system including two amplifiers and storing deblocking (DB) filtered neighboring block data and amplified DB filtered neighboring block data in a neighboring block line buffer;

[0026] Figure 3 is a block diagram illustrating a decoder system according to some examples, the decoder system including an amplifier and storing deblocking (DB) filtered neighboring block data in a neighboring block line buffer;

[0027] Figure 4 is a conceptual diagram illustrating reading, processing, and / or outputting pixel data from a superblock and neighboring blocks according to a vertical raster scan order according to some examples;

[0028] Figure 5A is a conceptual diagram illustrating reading, processing, and / or outputting pixel data from a block according to a horizontal raster scan order according to some examples;

[0029] Figure 5B is a conceptual diagram illustrating reading, processing, and / or outputting pixel data from a block according to a vertical raster scan order according to some examples;

[0030] Figure 6 is a conceptual diagram illustrating pixel data for filtering a block using a loop restoration (LR) filter according to some examples;

[0031] Figure 7 is a conceptual diagram illustrating duplication of pixel rows from an adjacent block row buffer to generate additional pixel rows for use by a loop recovery (LR) filter according to some examples;

[0032] Figure 8 is a flowchart illustrating a decoder process according to some examples; and

[0033] Fig. 9 is a schematic diagram illustrating an example of a computing system for implementing certain aspects described herein. DETAILED DESCRIPTION

[0034] Certain aspects of the present disclosure are provided below. Some of these aspects may be applied independently, and some of them may be applied in combination, which will be apparent to those skilled in the art. In the following description, specific details are set forth for the purpose of explanation, so as to provide a thorough understanding of aspects of the present application. However, it is apparent that various aspects may be practiced without these specific details. The drawings and descriptions are not intended to be restrictive.

[0035] The following description provides only example aspects and is not intended to limit the scope, applicability or configuration of the present disclosure. On the contrary, the following description of the example aspects will provide an enabling description for implementing the example aspects for those skilled in the art. It should be understood that various changes may be made to the function and arrangement of the elements without departing from the spirit and scope of the present application as set forth in the appended claims.

[0036] A camera is a device that uses an image sensor to receive light and capture image frames (such as still images or video frames). The terms "image", "image frame" and "frame" are used interchangeably herein. A camera can be configured with various image capture and image processing settings. Different settings produce images of different appearances. Some camera settings are determined and applied before or during the capture of one or more image frames, such as ISO, exposure time, aperture size, f / stop, shutter speed, focal length, and gain. For example, settings or parameters can be applied to an image sensor to capture one or more image frames. Other camera settings can configure post-processing of one or more image frames, such as changes to contrast, brightness, saturation, sharpness, levels, curves, or colors. For example, settings or parameters can be applied to a processor (e.g., an image signal processor or ISP) to process one or more image frames captured by an image sensor.

[0037] Video decoding devices implement video compression techniques to efficiently encode and decode video data. Video compression techniques may include applying different prediction modes, including spatial prediction (e.g., prediction within a frame or intra-frame prediction), temporal prediction (e.g., prediction between frames or inter-frame prediction), inter-layer prediction (across different layers of video data), and / or other prediction techniques to reduce or remove redundancy inherent in video sequences. The video encoder may partition each picture of the original video sequence into rectangular areas called video blocks or decoding units (described in more detail below). These video blocks may be encoded using specific prediction modes.

[0038] The video blocks may be divided into one or more groups of smaller blocks in one or more ways. Blocks may include coding tree blocks, prediction blocks, transform blocks, or other suitable blocks. Unless otherwise specified, general references to "blocks" may refer to such video blocks (e.g., coding tree blocks, coding blocks, prediction blocks, transform blocks, or other suitable blocks or sub-blocks, as will be understood by a person of ordinary skill). In addition, each of these blocks may also be referred to interchangeably herein as a "unit" (e.g., coding tree unit (CTU), decoding unit, prediction unit (PU), transform unit (TU), etc.) In some cases, a unit may indicate a decoding logic unit encoded in a bitstream, and a block may indicate a portion of a video frame buffer for which a process is directed.

[0039] For inter prediction mode, the video encoder can search for a block similar to the block being encoded in a frame (or picture) located in another temporal location, which is called a reference frame or reference picture. The video encoder can limit the search to a certain spatial displacement from the block to be encoded. A two-dimensional (2D) motion vector including a horizontal displacement component and a vertical displacement component can be used to locate the best match. For intra prediction mode, the video encoder can use spatial prediction techniques to form a prediction block based on data from previously encoded neighboring blocks in the same picture.

[0040] The video encoder can determine a prediction error. For example, the prediction error can be determined as the difference between the pixel values ​​in the block being encoded and the predicted block. The prediction error can also be referred to as a residual. The video encoder can also apply a transform (e.g., a discrete cosine transform (DCT) or other suitable transform) to the prediction error to generate transform coefficients. After the transform, the video encoder can quantize the transform coefficients. The quantized transform coefficients and motion vectors can be represented using syntax elements and, together with the control information, form a decoded representation of the video sequence. In some cases, the video encoder can entropy decode the syntax elements to further reduce the number of bits required for their representation.

[0041] The AV1 video codec / format specifies a decoder that decodes video data by passing it through a deblocking (DB) filter, a constrained directional enhancement filter (CDEF), and a loop restoration (LR) filter in sequence. Both the DB filter and the CDEF filter process the video data at the original resolution of the video data received by the decoder. However, the LR filter is configured to filter the upscaled video data, so the video data is first upscaled before being passed to the LR filter.

[0042] When decoding video data corresponding to a particular block of video data, the decoder uses data from the block and data from at least one adjacent block as input. For example, the decoder can receive first video data from a first block of a video frame and receive adjacent video data from an adjacent block of the video frame that is adjacent to the first block. The decoder can process both the first video data and the adjacent video data using a DB filter.

[0043] The decoder under the AV1 decoder / format may store two instances of adjacent video data in a line buffer corresponding to adjacent blocks. The first instance of adjacent video data stored by the decoder in the line buffer is stored as processed by the DB filter without further change. The second instance of adjacent video data stored by the decoder in the line buffer is amplified by the first amplification module after being processed by the DB filter before being stored. In terms of storage space, storing two instances of adjacent video data in the line buffer may be inefficient. For example, storing two instances of adjacent video data in the line buffer may require the line buffer to be large enough to store two instances of adjacent video data. This storage of two instances of adjacent video data in the line buffer may also be inefficient in terms of time because it may require two write operations to the line buffer. This storage of two instances of adjacent video data in the line buffer may also be inefficient in terms of bandwidth because it may require the decoder to use a component with a higher read and / or write bandwidth to read and / or write the two instances in a timely manner, rather than using a component with a lower read and / or write bandwidth and may be more energy-efficient.

[0044] In some examples, in a decoder under the AV1 decoder / format, the CDEF filter processes the first video data by receiving as input the first video data from the DB filter and a first (unamplified) instance of adjacent video data from a line buffer. The example decoder passes the CDEF-filtered first video data to an adjacent upscaling module to generate upscaled CDEF-filtered first video data. The adjacent upscaling module passes the upscaled CDEF-filtered first video data to the LR filter. Including two separate upscaling modules (a first upscaling module that upscales adjacent instances of adjacent video data for storage in a line buffer and a second upscaling module that upscales the CDEF-filtered first video data) is inefficient in terms of using additional hardware space for redundant functions. In an example decoder under the AV1 decoder / format, the LR filter processes the first video data to generate output video data by receiving as input the upscaled CDEF-filtered first video data from the second upscaling module and a second (upscaled) instance of adjacent video data from a line buffer.

[0045] The systems and methods described herein describe a decoder. In the decoder, an instance of adjacent video data is stored in a line buffer after being processed by a DB filter. This improves efficiency over other decoders because the line buffer can be smaller, only one write operation is required to write the adjacent video data to the line buffer, and components with lower bandwidth (and therefore improved power efficiency) can be used. The adjacent video data is not amplified before being stored in the line buffer, further reducing the space required to store the adjacent video data in the line buffer.

[0046] The decoder includes an amplification module located between the CDEF filter and the LR filter. The decoder can apply a first indication to the first video data and a second indication to the adjacent video data. These indications can persist through DB filtering, CDEF filtering, amplification and / or LR filtering. In this way, once the CDEF filter generates the first video data filtered by CDEF based on the first video data and the adjacent video data from the line buffer, the CDEF filter can pass all of its data (the first video data and the adjacent video data filtered by CDEF) to the amplification module, which can amplify all of the video data while retaining these indications. The LR filter then receives all of the amplified video data, processes it and outputs it, wherein the indication still indicates which output data comes from the first block and which output data comes from the adjacent block. The decoder can use the output data corresponding to the first block as part of the decoded video.

[0047] In some examples, the adjacent blocks are located vertically above the first block. If the decoder processes the video data within the first block and / or the adjacent blocks in a conventional horizontal raster scan order (from left to right, from the top row to the bottom row), the decoder may end up having to store the entire (one or more) rows of the adjacent blocks in the row buffer. However, if the decoder processes the video data within the first block and / or the adjacent blocks in a vertical raster scan order (from top to bottom, from the left column to the right column), the decoder can store smaller portions of (one or more) rows of the adjacent blocks in the row buffer at any given time, such as one or more blocks at a time. This can further improve efficiency and reduce the space required for the row buffer.

[0048] In some examples, the CDEF filter uses two rows of video data from the adjacent block closest to the first block. In some examples, the LR filter uses three rows of video data from the adjacent block closest to the first block. In some examples, the decoder can store the two rows of video data from the adjacent block closest to the first block in the line buffer. The decoder can copy the video data from the adjacent row of the adjacent block to use as the third row of the LR filter. This can further improve efficiency and reduce the space required for the line buffer.

[0049] With these improvements, in some examples, the minimum size of a row buffer can be reduced by 66.67%, the read / write bandwidth of components of the row buffer can be reduced by 66.67%, and the amount of writes to the row buffer can be reduced from 2 to 1.

[0050] Video decoding systems and techniques are described. A decoder reads first video data from a first block of a video frame. The decoder retrieves adjacent video data from a line buffer. The adjacent video data is from an adjacent block adjacent to the first block in the video frame. The decoder processes the first video data and the retrieved adjacent video data using a constrained directional enhancement filter (CDEF) to generate filtered first video data. The decoder amplifies the filtered first video data using an amplifier to generate amplified filtered first video data. For example, after generating the filtered first video data, the decoder amplifies the retrieved adjacent video data using an amplifier to generate amplified adjacent video data. The decoder processes the amplified filtered first video data and the amplified adjacent video data using a loop recovery (LR) filter to generate output video data.

[0051] Various aspects of the application will be described with reference to the accompanying drawings. Figure 1 1 is a block diagram illustrating an example of a system 100 including an encoding device 104 and a decoding device 112. The encoding device 104 may be part of a source device, and the decoding device 112 may be part of a receiving device. The source device and / or the receiving device may include an electronic device such as a mobile or fixed telephone handset (e.g., a smart phone, a cellular phone, etc.), a desktop computer, a laptop or notebook computer, a tablet computer, a set-top box, a television, a camera, a display device, a digital media player, a video game console, a video streaming device, an Internet Protocol (IP) camera, or any other suitable electronic device. In some examples, the source device and the receiving device may include one or more wireless transceivers for wireless communication. The decoding techniques described herein may be applied to video decoding in various multimedia applications, including streaming video transmission (e.g., via the Internet), television broadcasting or transmission, encoding of digital video stored on a data storage medium, decoding of digital video stored on a data storage medium, or other applications. In some examples, the system 100 may support one-way or two-way video transmission to support applications such as video conferencing, video streaming, video playback, video broadcasting, gaming, and / or video telephony.

[0052] The encoding device 104 (or encoder) can be used to encode video data using a video coding standard or protocol to generate a coded video bitstream. Examples of video coding standards include ITU-T H.261, ISO / IEC MPEG-1 Visual, ITU-T H.262 or ISO / IEC MPEG-2 Visual, ITU-T H.263, ISO / IEC MPEG-4 Visual, ITU-T H.264 (also known as ISO / IEC MPEG-4 AVC), including its scalable video coding (SVC) and multi-view video coding (MVC) extensions, and high efficiency video coding (HEVC) or ITU-T H.265. There are various extensions to HEVC involving multi-layer video coding, including range and screen content coding extensions, 3D video coding (3D-HEVC) and multi-view extensions (MV-HEVC) and scalable extensions (SHVC). HEVC and its extensions have been developed by the Joint Video Coding Task Force (JCT-VC) of the ITU-T Video Coding Experts Group (VCEG) and the ISO / IEC Moving Picture Experts Group (MPEG), and the Joint Task Force on Development of 3D Video Coding Extensions (JCT-3V).

[0053] MPEG and ITU-T VCEG have also formed the Joint Video Exploration Team (JVET) to explore new coding tools for the next generation video coding standard - Versatile Video Coding (VVC). The reference software is called the VVC Test Model (VTM) (or JEM (Joint Exploration Model)). One goal of VVC is to provide significant improvements in the compression performance of the existing HEVC standard, helping to deploy higher quality video services and emerging applications (e.g., such as 360° immersive multimedia, high dynamic range (HDR) video, etc.). VP9, ​​Alliance for Open Media (AOMedia) Video 1 (AV1), and Essential Video Coding (EVC) are other video codecs and / or coding standards to which the techniques described in this article can be applied.

[0054] The techniques described herein may be applied to any existing video codec (e.g., High Efficiency Video Coding (HEVC), Advanced Video Coding (AVC), or other suitable existing video codecs), and / or may be effective coding tools for any video coding standard being developed and / or future video coding standards (such as VVC and / or other video coding standards being developed or to be developed). For example, the examples described herein may be performed using video codecs such as VVC, HEVC, AVC, and / or their extensions. However, the techniques and systems described herein may also be applicable to other coding standards, such as MPEG, JPEG (or other coding standards for still images), VP9, ​​AV1, their extensions, or other suitable codecs and / or coding formats that are already available or not yet available or developed. Therefore, although the techniques and systems described herein may be described with reference to a particular video coding standard, those of ordinary skill in the art will appreciate that the description should not be interpreted as being applicable only to that particular standard.

[0055] Many of the embodiments described herein provide examples using the JEM model, VVC, HEVC standards, and / or their extensions. However, the techniques and systems described herein may also be applicable to other coding standards, such as AVC, MPEG, JPEG (or other coding standards for still images), their extensions, or other suitable coding standards that are already available or not yet available or developed. Therefore, although the techniques and systems described herein may be described with reference to a specific video coding standard, a person of ordinary skill in the art will understand that the description should not be interpreted as being applicable only to that specific standard.

[0056] Reference Figure 1 , video source 102 may provide video data to encoding device 104. Video source 102 may be part of a source device, or may be part of a device other than a source device. Video source 102 may include a video capture device (e.g., a video camera, a camera phone, a video phone, etc.), a video archive containing stored video, a video server or content provider that provides video data, a video feed interface that receives video from a video server or content provider, a computer graphics system for generating computer graphics video data, a combination of these sources, or any other suitable video source.

[0057] The video data from video source 102 may include one or more input pictures or frames. A picture or frame is a still image and, in some cases, is part of a video. In some examples, the data from video source 102 may be a still image that is not part of a video. In HEVC, VVC, and other video coding specifications, a video sequence may include a series of pictures. A picture may include three sample arrays, denoted S L , S Cb and S Cr . SL is a two-dimensional array of brightness samples, S Cb is a two-dimensional array of Cb chromaticity samples, and S Cr is a two-dimensional array of Cr chrominance samples. Chrominance samples may also be referred to herein as "chrominance" samples. A pixel may refer to all three components (luminance and chrominance samples) at a given position in a picture array. In other cases, a picture may be monochrome and may include only an array of luminance samples, in which case the terms pixel and sample may be used interchangeably. With respect to the exemplary techniques described herein that reference a single sample for illustrative purposes, the same techniques may be applied to pixels (e.g., all three sample components at a given position in a picture array). With respect to the exemplary techniques described herein that reference pixels (e.g., all three sample components at a given position in a picture array) for illustrative purposes, the same techniques may be applied to a single sample.

[0058] There are two types of network abstraction layer (NAL) units in the HEVC standard, including video coding layer (VCL) NAL units and non-VCL NAL units. VCL NAL units include a slice or slice segment (described below) of decoded picture data, while non-VCL NAL units include control information related to one or more decoded pictures. In some cases, NAL units can be referred to as packets. HEVC AUs include VCL NAL units and non-VCL NAL units, VCL NAL units contain decoded picture data, and non-VCL NAL units (if any) correspond to the decoded picture data.

[0059] The NAL unit may contain a sequence of bits that form a coded representation of video data (e.g., a coded video bitstream, a CVS of a bitstream, etc.), such as a coded representation of a picture in a video. The encoder engine 106 generates a coded representation of a picture by partitioning each picture into a plurality of slices. A slice is independent of other slices so that information in a slice is decoded without relying on data from other slices in the same picture. A slice includes one or more slice segments, including independent slice segments and, if present, one or more dependent slice segments that depend on previous slice segments.

[0060] In HEVC, the slice is then partitioned into coding tree blocks (CTBs) of luma samples and chroma samples. The CTB of luma samples and one or more CTBs of chroma samples and the syntax of the samples are called coding tree units (CTUs). CTUs may also be called "tree blocks" or "largest coding units" (LCUs). CTUs are the basic processing unit for HEVC encoding. CTUs can be split into multiple coding units (CUs) of different sizes. A CU contains arrays of luma and chroma samples called coding blocks (CBs).

[0061] The luma and chroma CBs can be further split into prediction blocks (PBs). A PB is a block of samples of a luma component or a chroma component that uses the same motion parameters for inter-frame prediction or intra-frame block copy (IBC) prediction (when available or allowed). The luma PB and one or more chroma PBs together with the associated syntax form a prediction unit (PU). For inter-frame prediction, a set of motion parameters (e.g., one or more motion vectors, reference indices, etc.) is signaled in the bitstream for each PU, and the set of motion parameters is used for inter-frame prediction of the luma PB and one or more chroma PBs. Motion parameters may also be referred to as motion information. The CB may also be partitioned into one or more transform blocks (TBs). A TB represents a square block of samples of a color component to which a residual transform (e.g., the same two-dimensional transform in some cases) is applied to decode a prediction residual signal. A transform unit (TU) represents a TB of luma and chroma samples, and corresponding syntax elements. Transform decoding will be described in more detail below.

[0062] The size of the CU corresponds to the size of the decoding mode and may be in the shape of a square. For example, the size of the CU may be 8×8 samples, 16×16 samples, 32×32 samples, 64×64 samples, or any other suitable size, up to the size of the corresponding CTU. The phrase "N×N" is used herein to refer to the pixel dimensions (e.g., 8 pixels × 8 pixels) of the vertical and horizontal dimensions of the video block. The pixels in the block may be arranged in rows and columns. In some embodiments, the block may not have the same number of pixels in the horizontal direction as in the vertical direction. The syntax data associated with the CU may describe, for example, the partitioning of the CU into one or more PUs. The partitioning mode may differ between whether the CU is encoded in an intra-prediction mode or an inter-prediction mode. The PU may be partitioned into a non-square shape. The syntax data associated with the CU may also describe, for example, the partitioning of the CU into one or more TUs according to the CTU. The shape of the TU may be square or non-square.

[0063] According to the HEVC standard, a transform unit (TU) may be used to perform a transform. A TU may be different for different CUs. A TU may be sized based on the size of a PU within a given CU. A TU may be the same size as a PU or smaller. In some examples, a quadtree structure called a residual quadtree (RQT) may be used to further divide the residual samples corresponding to a CU into smaller units. The leaf nodes of the RQT may correspond to TUs. The pixel difference values ​​associated with the TU may be transformed to produce transform coefficients. The transform coefficients may then be quantized by the encoder engine 106.

[0064] Once the picture of the video data is partitioned into CUs, the encoder engine 106 predicts each PU using a prediction mode. The prediction unit or prediction block is then subtracted from the original video data to obtain a residual (described below). For each CU, the prediction mode can be signaled in the bitstream using syntax data. The prediction mode may include intra-frame prediction (or intra-picture prediction) or inter-frame prediction (or inter-picture prediction). Intra-frame prediction exploits the correlation between spatially adjacent samples in a picture. For example, using intra-frame prediction, each PU is predicted from adjacent image data in the same picture, such as using DC prediction to find the average value of the PU, using plane prediction to fit a plane surface to the PU, using directional prediction to extrapolate from adjacent data, or any other suitable type of prediction. Inter-frame prediction uses temporal correlation between pictures to derive motion compensated predictions for blocks of image samples. For example, using inter-frame prediction, each PU is predicted from image data in one or more reference pictures (before or after the current picture in output order) using motion compensated prediction. For example, a decision can be made at the CU level whether to decode a picture area using inter-frame prediction or intra-frame prediction.

[0065] The encoder engine 106 and the decoder engine 116 (described in more detail below) can be configured to operate according to VVC. According to VVC, a video decoder (such as the encoder engine 106 and / or the decoder engine 116) partitions a picture into multiple decoding tree units (where the CTB of the luma sample and one or more CTBs of the chroma sample, together with the syntax of the sample, are referred to as CTUs). The video decoder can partition the CTU according to a tree structure (such as a quadtree-binary tree (QTBT) structure or a multi-type tree (MTT) structure). The QTBT structure eliminates the concept of multiple partition types, such as the separation between CU, PU, ​​and TU of HEVC. The QTBT structure includes two levels, including a first level according to quadtree partitioning and a second level according to binary tree partitioning. The root node of the QTBT structure corresponds to the CTU. The leaf nodes of the binary tree correspond to the decoding units (CUs).

[0066] In the MTT partitioning structure, blocks can be partitioned using quadtree partitioning, binary tree partitioning, and one or more types of ternary tree partitioning.

[0067] . A partitioning into three sub-blocks. In some examples, a ternary tree partitioning partitions a block into three sub-blocks without partitioning the original block by the center. The partitioning types in MTT (eg, quadtree, binary tree, and ternary tree) can be symmetric or asymmetric.

[0068] In some examples, the video coder may use a single QTBT or MTT structure to represent each of the luma and chroma components, while in other examples, the video coder may use two or more QTBT or MTT structures, such as one QTBT or MTT structure for the luma component and another QTBT or MTT structure for the two chroma components (or two QTBT and / or MTT structures for respective chroma components).

[0069] The video decoder may be configured to use quadtree segmentation, QTBT segmentation, MTT segmentation, or other segmentation structures in accordance with HEVC. For illustrative purposes, the description herein may refer to QTBT segmentation. However, it should be understood that the techniques of the present disclosure may also be applied to video decoders configured to use quadtree segmentation or other types of segmentation.

[0070] In some examples, one or more slices of a picture are assigned a slice type. Slice types include I slices, P slices, and B slices. I slices (intra-frame, independently decodable) are picture slices that are decoded only by intra-frame prediction and are therefore independently decodable because I slices only require data within the frame to predict any prediction unit or prediction block of the slice. P slices (unidirectionally predicted frames) are picture slices that can be decoded with intra-frame prediction and unidirectional inter-frame prediction. Each prediction unit or prediction block within a P slice is decoded with intra-frame prediction or inter-frame prediction. When inter-frame prediction is applied, the prediction unit or prediction block is predicted by only one reference picture, so the reference sample comes from only one reference area of ​​a frame. B slices (bidirectionally predicted frames) are picture slices that can be decoded with intra-frame prediction and inter-frame prediction (e.g., bi-prediction or uni-prediction). The prediction unit or prediction block of a B slice can be predicted bidirectionally from two reference pictures, where each picture contributes one reference area, and the sample sets of the two reference areas are weighted (e.g., with equal weights or with different weights) to produce a prediction signal for a bidirectionally predicted block. As described above, slices of a picture are independently coded. In some cases, a picture may be coded as only one slice.

[0071] As described above, intra prediction of a picture utilizes the correlation between spatially adjacent samples in the picture. There are multiple intra prediction modes (also referred to as "intra modes"). In some examples, intra prediction of a luminance block includes 35 modes, including a planar mode, a DC mode, and 33 angular modes (e.g., a diagonal intra prediction mode and an angular mode adjacent to the diagonal intra prediction mode). The 35 modes of intra prediction are indexed as shown in Table 1 below. In other examples, more intra modes may be defined, including prediction angles that may not yet be represented by the 33 angular modes. In other examples, the prediction angle associated with the angular mode may be different from the prediction angle used in HEVC.

[0072]

[0073] Table 1 - Specification of intra prediction modes and associated names

[0074] Inter prediction uses temporal correlations between pictures to derive motion compensated predictions for blocks of image samples. Using a translational motion model, the motion vector ( , ) indicates the position of the block in a previously decoded picture (reference picture), specifies the horizontal displacement of the reference block relative to the position of the current block, and Specifies the vertical displacement of the reference block relative to the position of the current block. In some cases, the motion vector ( , ) can be integer sample precision (also called integer precision), in which case the motion vector refers to an integer pixel grid (or integer pixel sampling grid) of the reference frame. In some cases, the motion vector ( , ) can be fractional sample precision (also known as fractional pixel precision or non-integer precision) to more accurately capture the motion of potential objects without being limited to the integer pixel grid of the reference frame. The precision of the motion vector can be represented by the quantization level of the motion vector. For example, the quantization level can be integer precision (e.g., 1 pixel) or fractional pixel precision (e.g., 1 / 4 pixel, 1 / 2 pixel, or other sub-pixel value). When the corresponding motion vector has fractional sample precision, interpolation is applied to the reference picture to derive the prediction signal. For example, the samples available at the integer positions can be filtered (e.g., using one or more interpolation filters) to estimate the values ​​at the fractional positions. The previously decoded reference picture is indicated by a reference index (refIdx) to the reference picture list. The motion vector and the reference index can be referred to as motion parameters. Two types of inter-frame prediction can be performed, including single prediction and dual prediction.

[0075] In inter prediction using bi-prediction, two motion parameter sets are used ( ) to generate two motion compensated predictions (from the same reference picture or possibly from different reference pictures). For example, in bi-prediction, each prediction block uses two motion compensated prediction signals and generates a B prediction unit. The two motion compensated predictions are then combined to obtain the final motion compensated prediction. For example, the two motion compensated predictions can be combined by averaging. In another example, weighted prediction can be used, in which case different weights can be applied to each motion compensated prediction. The reference pictures that can be used in bi-prediction are stored in two separate lists, denoted as List 0 and List 1. Motion parameters can be derived at the encoder using a motion estimation process.

[0076] In inter prediction using uni-prediction, a set of motion parameters is used ( ) Generates motion compensated prediction from a reference picture. For example, in uni-prediction, each prediction block uses at most one motion compensated prediction signal and generates a P prediction unit.

[0077] The PU may include data related to the prediction process (e.g., motion parameters or other suitable data). For example, when the PU is encoded using intra prediction, the PU may include data describing the intra prediction mode of the PU. As another example, when the PU is encoded using inter prediction, the PU may include data defining the motion vector of the PU. The data defining the motion vector of the PU may describe, for example, the horizontal component of the motion vector ( ), the vertical component of the motion vector ( ), the resolution of the motion vector (e.g., integer precision, quarter-pixel precision, or eighth-pixel precision), the reference picture to which the motion vector points, the reference index, the reference picture list of the motion vector (e.g., List 0, List 1, or List C), or any combination thereof.

[0078] After performing prediction using intra-frame and / or inter-frame prediction, the encoding device 104 may perform transformation and quantization. For example, after prediction, the encoder engine 106 may calculate a residual value corresponding to the PU. The residual value may include a pixel difference value between a current block (PU) of pixels being decoded and a prediction block (e.g., a predicted version of the current block) used to predict the current block. For example, after generating a prediction block (e.g., using inter-frame prediction or intra-frame prediction), the encoder engine 106 may generate a residual block by subtracting the prediction block generated by the prediction unit from the current block. The residual block includes a set of pixel difference values ​​that quantize the difference between the pixel values ​​of the current block and the pixel values ​​of the prediction block. In some examples, the residual block may be represented in a two-dimensional block format (e.g., a two-dimensional matrix or a pixel value array). In such an example, the residual block is a two-dimensional representation of pixel values.

[0079] Any residual data that may remain after performing the prediction is transformed using a block transform, which may be based on a discrete cosine transform, a discrete sine transform, an integer transform, a wavelet transform, other appropriate transform functions, or any combination thereof. In some cases, one or more block transforms (e.g., of size 32×32, 16×16, 8×8, 4×4, or other suitable sizes) may be applied to the residual data in each CU. In some embodiments, a TU may be used for a transform and quantization process implemented by the encoder engine 106. A given CU with one or more PUs may also include one or more TUs. As described in further detail below, a block transform may be used to transform residual values ​​into transform coefficients, which may then be quantized and scanned using a TU to produce serialized transform coefficients for entropy coding.

[0080] In some embodiments, after intra-prediction or inter-prediction decoding using the PU of the CU, the encoder engine 106 can calculate the residual data of the TU of the CU. The PU may include pixel data in the spatial domain (or pixel domain). The TU may include coefficients in the transform domain after applying the block transform. As described above, the residual data may correspond to the pixel difference between the pixel of the uncoded picture and the predicted value corresponding to the PU. The encoder engine 106 may form a TU, which includes the residual data of the CU, and then the TU may be transformed to generate the transform coefficients of the CU.

[0081] The encoder engine 106 can perform quantization of the transform coefficients. Quantization provides further compression by quantizing the transform coefficients to reduce the amount of data used to represent the coefficients. For example, quantization can reduce the bit depth associated with some or all of the coefficients. In one example, a coefficient with an n-bit value can be rounded down to an m-bit value during quantization, where n is greater than m.

[0082] Once quantization is performed, the encoded video bitstream includes quantized transform coefficients, prediction information (e.g., prediction mode, motion vectors, block vectors, etc.), segmentation information, and any other suitable data (such as other syntax data). The different elements of the decoded video bitstream can then be entropy encoded by the encoder engine 106. In some examples, the encoder engine 106 can scan the quantized transform coefficients using a predefined scan order to produce a serialized vector that can be entropy encoded. In some examples, the encoder engine 106 can perform adaptive scanning. After scanning the quantized transform coefficients to form a vector (e.g., a one-dimensional vector), the encoder engine 106 can entropy encode the vector. For example, the encoder engine 106 can use context adaptive variable length coding, context adaptive binary arithmetic coding, grammar-based context adaptive binary arithmetic coding, probability interval segmentation entropy coding, or another suitable entropy coding technique.

[0083] As previously described, the HEVC bitstream includes a set of NAL units, which include VCL NAL units and non-VCL NAL units. The VCL NAL unit includes decoded picture data that forms a decoded video bitstream. For example, a bit sequence that forms a decoded video bitstream is present in a VCL NAL unit. The non-VCL NAL unit may contain a parameter set with high-level information related to the encoded video bitstream and other information. For example, a parameter set may include a video parameter set (VPS), a sequence parameter set (SPS), and a picture parameter set (PPS). Examples of targets of parameter sets include bit rate efficiency, fault tolerance, and providing a system layer interface. Each slice references a single active PPS, SPS, and VPS to access information that the decoding device 112 can use to decode the slice. An identifier (ID) may be decoded for each parameter set, including a VPS ID, an SPS ID, and a PPS ID. The SPS includes an SPS ID and a VPS ID. The PPS includes a PPS ID and an SPS ID. Each slice header includes a PPS ID. Using the ID, an active parameter set may be identified for a given slice.

[0084] The PPS includes information applied to all slices in a given picture. Because of this, all slices in a picture refer to the same PPS. Slices in different pictures can also refer to the same PPS. The SPS includes information applied to all pictures in the same decoded video sequence (CVS) or bitstream. As previously described, a decoded video sequence is a series of access units (AUs) that start with a random access point picture (e.g., an instantaneous decoding reference (IDR) picture or a broken link access (BLA) picture, or other appropriate random access point picture) in the base layer and having certain properties (as described above), until and not including the next AU with a random access point picture in the base layer and having certain properties (or the end of the bitstream). The information in the SPS may not change with the picture in the decoded video sequence. The pictures in the decoded video sequence can use the same SPS. The VPS includes information applied to all layers within the decoded video sequence or bitstream. The VPS includes a syntax structure with syntax elements applied to the entire decoded video sequence. In some embodiments, the VPS, SPS, or PPS may be sent in-band with the coded bitstream. In some embodiments, the VPS, SPS, or PPS may be sent out-of-band in a transmission separate from the NAL units containing the coded video data.

[0085] The video bitstream may also include a supplemental enhancement information (SEI) message. For example, a SEI NAL unit may be part of the video bitstream. In some cases, the SEI message may contain information that is not required for the decoding process. For example, the information in the SEI message may not be necessary for the decoder to decode the video picture of the bitstream, but the decoder may use the information to improve the display or processing of the picture (e.g., decoded output). The information in the SEI message may be embedded metadata. In an illustrative example, the information in the SEI message may be used by a decoder-side entity to improve the visibility of the content. In some instances, some application standards may specify the presence of such SEI messages in the bitstream so that quality improvements can be brought to all devices that comply with the application standards (e.g., the transmission of frame packing SEI messages for frame-compatible planar stereoscopic 3DTV video formats, where SEI messages are carried for each frame of the video, the processing of recovery point SEI messages, the use of full-screen scan rectangle SEI messages in DVB, and many other examples).

[0086] The output 110 of the encoding device 104 may send NAL units, which constitute the encoded video bitstream data, to a decoding device 112 of a receiving device via a communication link 120. The input 114 of the decoding device 112 may receive the NAL units. The communication link 120 may include a channel provided by a wireless network, a wired network, or a combination of wired and wireless networks. The wireless network may include any wireless interface or combination of wireless interfaces and may include any suitable wireless network (e.g., the Internet or other wide area network, a packet-based network, a WiFi, or a wireless network). TM , Radio Frequency (RF), UWB, WiFi-Direct, Cellular, Long Term Evolution (LTE), WiMax TM The wired network may include any wired interface (e.g., optical fiber, Ethernet, Ethernet over power line, Ethernet over coaxial cable, digital signal line (DSL), etc.). The wired and / or wireless network may be implemented using various devices (such as base stations, routers, access points, bridges, gateways, switches, etc.). The encoded video bitstream data may be modulated according to a communication standard (such as a wireless communication protocol) and sent to a receiving device.

[0087] In some examples, the encoding device 104 may store the encoded video bitstream data in the storage 108. The output 110 may retrieve the encoded video bitstream data from the encoder engine 106 or from the storage 108. The storage 108 may include any of a variety of distributed or locally accessed data storage media. For example, the storage 108 may include a hard drive, a storage disk, a flash memory, a volatile or non-volatile memory, or any other suitable digital storage medium for storing the encoded video data. The storage 108 may also include a decoded picture buffer (DPB) for storing reference pictures used in inter-frame prediction. In another example, the storage 108 may correspond to a file server or another intermediate storage device that can store the encoded video generated by the source device. In this case, the receiving device including the decoding device 112 can access the stored video data from the storage device via streaming or downloading. The file server may be any type of server capable of storing the encoded video data and sending the encoded video data to the receiving device. Example file servers include web servers (e.g., for websites), FTP servers, network attached storage (NAS) devices, or local disk drives. The receiving device can access the encoded video data through any standard data connection (including an Internet connection). This can include a wireless channel (e.g., a Wi-Fi connection), a wired connection (e.g., DSL, cable modem, etc.), or a combination of both, which is suitable for accessing the encoded video data stored on a file server. The transmission of the encoded video data from storage 108 can be a streaming transmission, a download transmission, or a combination thereof.

[0088] The input 114 of the decoding device 112 receives the encoded video bitstream data, and the video bitstream data can be provided to the decoder engine 116, or to the storage 118 for later use by the decoder engine 116. For example, the storage 118 may include a DPB for storing reference pictures used in inter-frame prediction. A receiving device including the decoding device 112 may receive the encoded video data to be decoded via the storage 108. The encoded video code data may be modulated according to a communication standard (such as a wireless communication protocol) and sent to the receiving device. The communication medium for sending the encoded video data may include any wireless or wired communication medium, such as a radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form part of a packet-based network, such as a local area network, a wide area network, or a global network such as the Internet. The communication medium may include a router, a switch, a base station, or any other device that facilitates communication from a source device to a receiving device.

[0089] The decoder engine 116 may decode the encoded video bitstream data by entropy decoding (e.g., using an entropy decoder) and extracting elements of one or more decoded video sequences that constitute the encoded video data. The decoder engine 116 may then rescale the encoded video bitstream data and perform an inverse transform. The residual data is then passed to the prediction stage of the decoder engine 116. The decoder engine 116 then makes a prediction for the pixel block (e.g., PU). In some examples, the prediction is added to the output (residual data) of the inverse transform.

[0090] The decoding device 112 may output the decoded video to a video destination device 122, which may include a display or other output device for displaying the decoded video data to a consumer of the content. In some aspects, the video destination device 122 may be part of a receiving device that includes the decoding device 112. In some aspects, the video destination device 122 may be part of a separate device other than the receiving device.

[0091] In some embodiments, the video encoding device 104 and / or the video decoding device 112 may be integrated with the audio encoding device and the audio decoding device, respectively. The video encoding device 104 and / or the video decoding device 112 may also include other hardware or software necessary to implement the above-mentioned decoding technology, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combination thereof. The video encoding device 104 and the video decoding device 112 may be integrated as part of a combined encoder / decoder (decoder) in each device.

[0092] Figure 1 The example system shown is an illustrative example that can be used herein. The techniques for processing video data using the techniques described herein can be performed by any digital video encoding and / or decoding device. Although the techniques of the present disclosure are generally performed by a video encoding device or a video decoding device, these techniques can also be performed by a combined video encoder-decoder, commonly referred to as a "decoder (CODEC)". In addition, the techniques of the present disclosure can also be performed by a video preprocessor. The source device and the receiving device are merely examples of such decoding devices, wherein the source device generates decoded video data for transmission to the receiving device. In some examples, the source device and the receiving device can operate in a substantially symmetrical manner so that each device includes a video encoding and decoding component. Therefore, the example system can support one-way or two-way video transmission between video devices, such as for video streaming, video playback, video broadcasting, or video telephony.

[0093] Extensions to the HEVC standard include multi-view video coding extensions (referred to as MV-HEVC) and scalable video coding extensions (referred to as SHVC). MV-HEVC and SHVC extensions share the concept of layered coding, where different layers are included in the coded video bitstream. Each layer in the coded video sequence is addressed by a unique layer identifier (ID). The layer ID may be present in the header of the NAL unit to identify the layer associated with the NAL unit. In MV-HEVC, different layers may represent different views of the same scene in the video bitstream. In SHVC, different scalable layers are provided that represent the video bitstream at different spatial resolutions (or picture resolutions) or at different reconstruction fidelity. The scalable layer may include a base layer (layer ID = 0) and one or more enhancement layers (layer ID = 1, 2, ... n). The base layer may conform to the profile of the first version of HEVC and represent the lowest available layer in the bitstream. Compared to the base layer, the enhancement layer has increased spatial resolution, temporal resolution or frame rate and / or reconstruction fidelity (or quality). Enhancement layers are organized hierarchically and may or may not depend on lower layers. In some examples, a single standard decoder may be used to decode the different layers (e.g., all layers are encoded using HEVC, SHVC, or other decoding standards). In some examples, a multi-standard decoder may be used to decode the different layers. For example, the base layer may be decoded using AVC, while one or more enhancement layers may be decoded using SHVC and / or MV-HEVC extensions to the HEVC standard.

[0094] In general, a layer consists of a set of VCL NAL units and a corresponding set of non-VCL NAL units. A NAL unit is assigned a specific layer ID value. Layers can be hierarchical in that a layer can depend on a lower layer. A layer set refers to a self-contained set of layers represented within a bitstream, which means that the layers within a layer set can depend on other layers in the layer set during the decoding process, but do not depend on any other layers for decoding. Therefore, the layers in a layer set can form an independent bitstream that can represent video content. The set of layers in a layer set can be obtained from another bitstream through the operation of a sub-bitstream extraction process. A layer set can correspond to a set of multiple layers to be decoded when the decoder wishes to operate according to specific parameters.

[0095] As described above, for each block, a motion information set (also referred to herein as motion parameters) is available. The motion information set contains motion information for forward and backward prediction directions. The forward and backward prediction directions are the two prediction directions of the bidirectional prediction mode, in which case the terms "forward" and "backward" do not necessarily have a geometric meaning. Instead, "forward" and "backward" correspond to reference picture list 0 (RefPicList0 or L0) and reference picture list 1 (RefPicList1 or L1) of the current picture. In some examples, when only one reference picture list is available for a picture or slice, only RefPicList0 is available, and the motion information for each block of the slice is always forward.

[0096] In some cases, motion vectors and their reference indexes are used in the decoding process (e.g., motion compensation). Such motion vectors with associated reference indexes are represented as a unidirectional prediction motion information set. For each prediction direction, the motion information may contain a reference index and a motion vector. In some cases, for simplicity, the motion vector itself may be referenced in a manner assuming that it has an associated reference index. The reference index is used to identify the reference picture in the current reference picture list (RefPicList0 or RefPicList1). The motion vector has a horizontal component and a vertical component that provide an offset from a coordinate position in the current picture to a coordinate in the reference picture identified by the reference index. For example, a reference index may indicate a specific reference picture that should be used for a block in the current picture, and a motion vector may indicate the position of the best matching block (the block that best matches the current block) in the reference picture.

[0097] The picture order count (POC) can be used in video coding standards to identify the display order of pictures. Although there are cases where two pictures in a decoded video sequence may have the same POC value, this does not usually happen in a decoded video sequence. When multiple decoded video sequences are present in the bitstream, pictures with the same POC value may be closer to each other in decoding order. The POC value of a picture can be used for reference picture list construction, derivation of reference picture sets (such as in HEVC), and motion vector scaling.

[0098] In H.264 / AVC, each inter-frame macroblock (MB) can be partitioned in four different ways, including: one 16×16 MB partition; two 16×8 MB partitions; two 8×16 MB partitions; and four 8×8 MB partitions. Different MB partitions in one MB may have different reference index values ​​for each direction (RefPicList0 or RefPicList1). In some cases, when an MB is not partitioned into four 8×8 MB partitions, there can be only one motion vector for each MB partition in each direction. In some cases, when an MB is partitioned into four 8×8 MB partitions, each 8×8 MB partition can be further partitioned into sub-blocks, in which case each sub-block can have a different motion vector in each direction. In some examples, there are four different ways to get a sub-block from an 8×8 MB partition, including: one 8×8 sub-block; two 8×4 sub-blocks; two 4×8 sub-blocks; and four 4×4 sub-blocks. Each sub-block can have a different motion vector in each direction. Therefore, the motion vector exists at a level equal to or higher than the sub-block.

[0099] In AVC, for skip and / or direct mode in B slices, temporal direct mode can be enabled at the MB level or the MB partition level. For each MB partition, the motion vector of the block collocated with the current MB partition in RefPicList1[0] of the current block is used to derive the motion vector. Each motion vector in the collocated block is scaled based on the POC distance.

[0100] Spatial direct mode can also be implemented in AVC. For example, in AVC, direct mode can also predict motion information from spatial neighbors.

[0101] In HEVC, the largest coding unit in a slice is called a coding tree block (CTB). The CTB contains a quadtree whose nodes are coding units. In the HEVC Main profile, the size of the CTB can vary from 16×16 to 64×64. In some cases, an 8×8 CTB size can be supported. The coding unit (CU) can be the same size as the CTB and as small as 8×8. In some cases, each coding unit is decoded with one mode. When the CU is inter-coded, the CU can be further partitioned into 2 or 4 prediction units (PUs), or it can become just one PU when further partitioning is not applicable. When there are two PUs in a CU, they can be rectangles of half the size of the CU, or two rectangles of ¼ or ¾ the size of the CU.

[0102] When a CU is inter-coded, there is one motion information set for each PU. In addition, each PU is coded with a unique inter prediction mode to derive the motion information set.

[0103] For motion prediction in HEVC, there are two inter prediction modes, including merge mode and advanced motion vector prediction (AMVP) mode for prediction units (PUs). Skipping is considered a special case of merge. In AMVP or merge mode, a motion vector (MV) candidate list is maintained for multiple motion vector predictors. The motion vector (or vectors) of the current PU and the reference index in merge mode are generated by taking a candidate from the MV candidate list. In some examples, as described below, one or more stored local illumination compensation (LIC) flags may be included in the MV candidate list together with the stored motion vector.

[0104] In an example where the MV candidate list is used for motion prediction (and, where applicable, illumination compensation) of a block, the MV candidate list may be constructed by an encoding device and a decoding device, respectively. For example, the MV candidate list may be generated by an encoding device when encoding a block, and may be generated by a decoding device when decoding a block. Information related to motion information candidates in the MV candidate list (e.g., information related to one or more motion vectors, information related to one or more LIC flags that may be stored in the MV candidate list in some cases, and / or other information) may be signaled between the encoding device and the decoding device. For example, in merge mode, the index value of the stored motion information candidate may be signaled from the encoding device to the decoding device (e.g., in a syntax structure such as a picture parameter set (PPS), a sequence parameter set (SPS), a video parameter set (VPS), a slice header, a supplemental enhancement information (SEI) message sent in a video bitstream or separately from a video bitstream, and / or other signaling). The decoding device may construct an MV candidate list, and obtain one or more motion information candidates from the constructed MV candidate list using a signaled reference or index for motion compensated prediction. For example, the decoding device 112 may construct an MV candidate list and use the motion vector (and in some cases the LIC flag) from the index position for motion prediction of the block. In the case of the AMVP mode, in addition to the reference or index, the difference or residual value may also be signaled as a delta. For example, for the AMVP mode, the decoding device may construct one or more MV candidate lists and apply the delta value to one or more motion information candidates obtained using the signaled index value when performing motion compensated prediction of the block.

[0105] In some examples, the MV candidate list contains up to five candidates for merge mode and two candidates for AMVP mode. In other examples, different numbers of candidates may be included in the MV candidate list for merge mode and / or AMVP mode. The merge candidate may contain a motion information set. For example, the motion information set may include motion vectors corresponding to reference picture lists (list 0 and list 1) and reference indexes. If the merge candidate is identified by the merge index, the reference picture is used for prediction of the current block, and the associated motion vector is determined. However, in AMVP mode, for each potential prediction direction from list 0 or list 1, since the AMVP candidate contains only one motion vector, the reference index needs to be explicitly signaled to the MV candidate list together with the MVP index. In AMVP mode, the predicted motion vector can be further refined.

[0106] As can be seen above, the merge candidate corresponds to a complete set of motion information, while the AMVP candidate contains only one motion vector for a specific prediction direction and reference index. Candidates for both modes are similarly derived from the same spatial and temporal neighboring blocks.

[0107] In some examples, merge mode allows an inter-predicted PU to inherit the same one or more motion vectors, prediction direction, and one or more reference picture indexes from an inter-predicted PU that includes a motion data position selected from a set of spatially adjacent motion data positions and one of two temporally juxtaposed motion data positions. For AMVP mode, one or more motion vectors of the PU may be predictively decoded relative to one or more motion vector predictors (MVPs) from an AMVP candidate list constructed by the encoder and / or decoder. In some instances, for unidirectional inter prediction of a PU, the encoder and / or decoder may generate a single AMVP candidate list. In some instances, for bidirectional prediction of a PU, the encoder and / or decoder may generate two AMVP candidate lists, one using motion data from spatially and temporally adjacent PUs in a forward prediction direction and one using motion data from spatially and temporally adjacent PUs in a backward prediction direction.

[0108] Figure 2 2 is a block diagram illustrating a decoder system 200 that includes two amplifiers and stores deblocking (DB) filtered neighboring block data 235 and amplified DB filtered neighboring block data 240 in a neighboring block line buffer 205. The two amplifiers include an amplifier 220 and an amplifier 230. The decoder system 200 may be an example of a decoding device 112 and / or a decoder engine 116.

[0109] The decoder system 200 receives input video data 270. The input video data 270 may be encoded using an encoder such as the encoding device 104 and / or the encoder engine 106. The encoded video data 270 may include, for example, the encoded video bitstream data (e.g., NAL units) discussed with respect to the output 110 of the encoding device 104 and / or the input 114 of the decoding device 112. The decoder system 200 inputs the input video data 270 into a deblocking (DB) filter 210. In some examples, the input video data 270 may be at least partially decoded and / or processed by the decoder system 200 (and / or another aspect of the decoding device 112 and / or the decoder engine 116) when the decoder system 200 inputs the input video data 270 into the DB filter 210. For example, the decoder system 200 (and / or another aspect of the decoding device 112 and / or the decoder engine 116) may have been processed using an inverse discrete cosine transform (IDCT). Input video data 270 includes pixel data from a first block of a video frame and pixel data from a neighboring block of the video frame. The neighboring block may be located adjacent to the first block in the context of the video frame. In some examples, the neighboring block is located above the first block in the context of the video frame (e.g., Figure 4 ). In some examples, the neighboring block is located next to the first block in the context of the video frame (e.g., to the left or right of the first block). In some examples, the neighboring block is located below the first block in the context of the video frame. In some examples, the first block and / or the neighboring block can be a super block, a macro block, and / or a tile.

[0110] The decoder system 200 uses the DB filter 210 to filter the input video data 270 to remove boundary artifacts (e.g., boundary discontinuities) at the edges of decoded blocks, such as by smoothing the block edge artifacts. In some examples, to filter the input video data 270 using the DB filter 210, the decoder system 200 may analyze the input video data 270 using one or more heuristic-based analyses to determine the difference between boundary artifacts and legitimate details in the video data at the block edges, and control the conditional application of smoothing at the block edges. In some examples, the heuristic-based analysis may consider changes in brightness at and / or across block edges, changes in specific color channels (e.g., red, green, and / or blue) at and / or across block edges, whether the block edge is inside a larger block (e.g., a superblock, macroblock, and / or tile) or at one or more edges of a larger block, the presence of any decoded residuals for the block, any block motion data (e.g., motion vectors) associated with the block, any block motion compensation data associated with the block, or a combination thereof. Figure 2The graph representing the DB filter 210 in shows block edges using dashed lines to indicate the smoothing of visual artifacts at such block edges, thereby making the block edges less prominent when the video frame is viewed as a whole.

[0111] By filtering the input video data 270, the DB filter 210 generates DB-filtered block data 245 corresponding to the first block, and DB-filtered neighboring block data 235 corresponding to the neighboring block. The decoder system 200 passes the DB-filtered block data 245 from the DB filter 210 to a constrained directional enhancement filter (CDEF) 215. The decoder system 200 stores the DB-filtered neighboring block data 235 in a neighboring block line buffer 205. The decoder system 200 also passes the DB-filtered neighboring block data 235 to the amplifier 230 to generate amplified DB-filtered neighboring block data 240 using the amplifier 230. The decoder system 200 stores the amplified DB-filtered neighboring block data 240 in the neighboring block line buffer 205. The amplifier 230 amplifies the DB-filtered neighboring block data 235 by an amplification factor to generate amplified DB-filtered neighboring block data 240. In the illustrative example, the amplification factor is 2. In some examples, the magnification factor can be an integer value (e.g., 1, 2, 3, 4, etc.), a non-integer value (e.g., 1.0625, 1.125, 1.25, 1.5, 1.75, etc.), or another suitable magnification factor. Figure 2 The diagram of amplifier 230 in shows the amplification of the block by a factor of 2.

[0112] The decoder system 200 may filter the DB filtered block data 245 using the CDEF filter 215 to generate CDEF filtered block data 250. In order to filter the DB filtered block data 245 using the CDEF filter 215, the decoder system 200 may also retrieve the DB filtered adjacent block data 235 from the adjacent block row buffer 205, and use the DB filtered adjacent block data 235 and the DB filtered block data 245 as inputs to the CDEF filter 215 to generate the CDEF filtered block data 250. The CDEF filter 215 may remove ringing and base noise around sharp edges in the DB filtered block data 245. In some examples, the CDEF filter 215 may be a directional filter that follows the edge based on the direction of the edge. In some examples, the CDEF filter 215 may operate in 8 different directions. Representation Figure 2 The graph of the CDEF filter 215 in FIG. 2 shows example block edges oriented according to each of eight different directions.

[0113] In some examples, CDEF filter 215 performs filtering based on the following equation:

[0114]

[0115] According to the equation for CDEF filter 215, filtering the first block using CDEF filter 215 also uses the DB filtered block data 245 from neighboring blocks (eg, Figure 7 In some examples, the neighboring block row buffer may store at least two rows (e.g., rows or columns) of DB-filtered pixel data from the neighboring block (row 705 and row 710 in FIG. 1 ). In some examples, the neighboring block row buffer may store the DB-filtered pixel data from at least two rows of the neighboring block as DB-filtered neighboring block data 235, which may be retrieved by the decoder system 200 for use by the CDEF filter 215 when filtering the DB-filtered block data 245 to generate the CDEF-filtered block data 250.

[0116] Decoder system 200 passes CDEF-filtered block data 250 generated using CDEF filter 215 to amplifier 220 to generate amplified CDEF-filtered block data 255 using amplifier 220. Amplifier 220 amplifies CDEF-filtered block data 250 by an amplification factor to generate amplified CDEF-filtered block data 255. In the illustrative example, the amplification factor is 2. In some examples, the amplification factor may be 2, 3, 4, or another suitable amplification factor. Figure 2 The diagram of amplifier 220 in shows the enlargement of the block by a factor of 2. Amplifier 220 and / or amplifier 230 may use any(s) of the enlargement techniques, such as upscaling, upsampling, resampling, resizing, nearest neighbor interpolation, bilinear interpolation, bicubic interpolation, sinc resampling, Lanczos resampling, box sampling, mipmapping, Fourier transform-based interpolation, edge-directed interpolation, high quality scaling (HQX), vectorization, super-resolution,(s) deep convolutional neural network, or a combination thereof.

[0117] The decoder system 200 passes the amplified CDEF-filtered block data 255 to a loop restoration (LR) filter 225. The decoder system 200 also retrieves the amplified DB-filtered neighboring block data 240 from the neighboring block line buffer 205 and passes the amplified DB-filtered neighboring block data 240 to the LR filter 225. The decoder system 200 uses the LR filter 225 to filter the amplified CDEF-filtered block data 255 to generate output block data 260. In order to filter the amplified CDEF-filtered block data 255 using the LR filter 225, the decoder system 200 may also use the amplified DB-filtered neighboring block data 240 and the CDEF-filtered block data 255 as inputs to the LR filter 225 to generate output block data 260. The LR filter 225 may include one or more configurable filters and / or switchable filters, such as one or more Wiener filters and / or one or more self-guided filters. In some examples, the LR filter 225 and / or the filter(s) included by the LR filter 225 include one or more convolution filters configured to construct a kernel to restore lost quality of the input data (e.g., the upscaled CDEF filtered block data 255). In some examples, the LR filter 225 is used for denoising and / or for edge enhancement, for example using a configurable amount of blur (one or more) to remove DCT-based noise. Figure 2 The graph of the LR filter 225 in shows an exemplary removal (eg, smoothing) of noise.

[0118] In some examples, LR filter 225 performs filtering based on the following equation:

[0119]

[0120] According to these equations for LR filter 225, filtering the first block using LR filter 225 uses the amplified CDEF-filtered block data 255 from the first block in addition to the amplified CDEF-filtered block data 255 from the first block, and also uses the amplified CDEF-filtered block data 255 from the adjacent block (e.g., Figure 7 In some examples, the neighboring block row buffer may store at least three rows (e.g., rows or columns) of DB-filtered pixel data from the neighboring block amplified using amplifier 230 as amplified DB-filtered neighboring block data 240. In such examples, the amplified DB-filtered neighboring block data 240 may be retrieved by the decoder system 200 for use by the LR filter 225 in filtering the amplified CDEF-filtered block data 255 to generate the output block data 260.

[0121] The neighboring block row buffer 205 may store DB filtered pixel data of at least two rows from the neighboring block amplified using the amplifier 230 as amplified DB filtered neighboring block data 240. In such an example, the amplified DB filtered neighboring block data 240 may be retrieved by the decoder system 200, which may copy one or more rows of the amplified DB filtered neighboring block data 240 to simulate a third row of pixel data that was already present in the DB filtered neighboring block data 235 before the DB filtered neighboring block data 235 was amplified by the amplifier 230 to generate the amplified DB filtered neighboring block data 240 (e.g., see the copy 720 of row 710 to generate row 715). After the copying of the row(s), the amplified DB filtered neighboring block data 240 may be provided by the decoder system 200 as input to the LR filter 225 to filter the amplified CDEF filtered block data 255 to generate the output block data 260. In some examples, the decoder system 200 can duplicate one row of the amplified DB filtered neighboring block data 240 (e.g., without regard to amplification). In some examples, if the amplification factor of the amplifier 230 is 2, the decoder system 200 can duplicate two rows of the amplified DB filtered neighboring block data 240 before using the amplified DB filtered neighboring block data 240 with the repeated rows and the LR filter 225 together to filter the amplified CDEF filtered block data 255 to generate the output block data 260. Such row duplication can reduce the amount of data stored in the neighboring block row buffer 205 (e.g., from the DB filtered neighboring block data 235 and the amplified DB filtered neighboring block data 240) by a factor of one-third.

[0122] In some examples, decoder system 200 can pass the output of LR filter 225 (e.g., output block data 260) back to DB filter 210 for another loop through DB filter 210, CDEF filter 215, amplifier 220, LR filter 225, amplifier 230, and / or adjacent block line buffer 205. In this manner, output block data 260 can be used as at least a portion of input video data 270 (e.g., a portion representing the first block). This repetition of the filtering process may be referred to as a decoding loop, and is repeated in a sequential manner. Figure 2 2 is represented by the dashed arrow from LR filter 225 back to DB filter 210. In some examples, output block data 260 may refer to the output of multiple iterations of the filtering process in the decoding loop.

[0123] In some examples, at least one amplifier (e.g., amplifier 220 and / or amplifier 230) performs horizontal amplification instead of vertical amplification. In some examples, at least one amplifier (e.g., amplifier 220 and / or amplifier 230) performs vertical amplification instead of horizontal amplification. In some examples, at least one amplifier (e.g., amplifier 220 and / or amplifier 230) performs both horizontal and vertical amplification.

[0124] In some examples, in a decoder system 200 with 4:2:0 chroma subsampling, where a neighbor block is located above the first block in a video frame, the amount of data written to, stored in, and read from the neighbor block row buffer 205 in the un-enlarged domain (e.g., DB-filtered neighbor block data 235) can be determined as follows:

[0125]

[0126] In some examples, in a decoder system 200 with 4:2:0 chroma subsampling, where a neighbor block is located above the first block in a video frame, the amount of data (e.g., upscaled DB filtered neighbor block data 240) written to, stored in, and read from the neighbor block row buffer 205 in the upscaled domain can be determined as follows:

[0127]

[0128] In some examples, in a decoder system 200 with 4:2:0 chroma subsampling, where the neighbor block is located above the first block in a video frame, the total amount of data written to, stored in, and read from the neighbor block line buffer 205 (e.g., DB filtered neighbor block data 235 and upscaled DB filtered neighbor block data 240) can be determined as follows:

[0129]

[0130] In some examples, for a video frame configured to provide upscaled DB-filtered neighboring block data 240 to the LR filter 225 and without vertical tiles, the tile width is equal to the frame width. For a frame width (W) and an upscale ratio (upscale ratio) = 2, the amount of data written to, stored in, and read from the neighboring block row buffer 205 (e.g., DB-filtered neighboring block data 235 and upscaled DB-filtered neighboring block data 240) can be determined as follows:

[0131]

[0132] In some examples, for a frame rate (F frames per second), the minimum read / write memory bandwidth for writing data to and / or reading data from the neighboring block row buffer 205 is:

[0133]

[0134] Figure 3 is a block diagram showing a decoder system 300 that includes an amplifier 220 and stores deblocking (DB) filtered neighboring block data 235 in a neighboring block line buffer 205. Figure 2 Similar to the decoder system 200, Figure 3 The decoder system 300 includes a DB filter 210 , a CDEF filter 215 , an amplifier 220 , and an LR filter 225 . Figure 3 At least some of these elements of the decoder system 300 may be as described with respect to Figure 2 For example, similar to the decoder system 200, the decoder system 300 receives input video data 270, inputs the input video data 270 to the DB filter 210, outputs DB-filtered block data 245 from the DB filter 210 to be input to the CDEF filter 215, and outputs DB-filtered adjacent block data 235 from the DB filter 210 to be stored in the adjacent block line buffer 205.

[0135] and Figure 2 The decoder system 200 is different, Figure 3 Decoder system 300 includes a single amplifier (e.g., amplifier 220) rather than two separate amplifiers (e.g., amplifier 220 and amplifier 230). This aspect of decoder system 300 may provide several technical improvements over decoder system 200. For example, the hardware for decoder system 300 may be physically smaller than the hardware for decoder system 200 because amplifier 230 is not required in decoder system 300. Decoder system 300 may use less power for corresponding decoding operations than decoder system 200 because power does not need to be provided to amplifier 230 in decoder system 300. In some examples, removing amplifier 230 may provide decoder system 300 with 8,000 square micrometers (μm) of power on a circuit board compared to decoder system 200 on a circuit board. 2 ) size is reduced.

[0136] and Figure 2 The decoder system 200 is different, Figure 3The decoder system 300 stores the DB filtered neighboring block data 235 in the neighboring block row buffer 205 without storing the upscaled DB filtered neighboring block data 240 in the neighboring block row buffer 205. This aspect of the decoder system 300 can provide several technical improvements over the decoder system 200. In some examples, the decoder system 300 writes less data to the neighboring block row buffer 205, stores less data in the neighboring block row buffer 205, and retrieves less data from the neighboring block row buffer 205 as compared to the decoder system 200. In some examples, the use of the vertical raster scan order 435 and / or the vertical raster scan order 530 further allows the decoder system 300 to write less data to the neighboring block row buffer 205, store less data in the neighboring block row buffer 205, and retrieve less data from the neighboring block row buffer 205 as compared to the decoder system 200. For example, the example of decoder system 300 can achieve a 66.67% memory savings over the example of decoder system 200, e.g., by reducing the size of data written to, stored in, and read from neighboring block row buffer 205 from 12×8192×10 bits (122.88 kilobytes (KB)) to 4×8192×10 bits (40.96 KB), given a maximum tile width of 8192 bits and an enlargement ratio of 2. This allows decoder system 300 to use a smaller neighboring block row buffer 205 than decoder system 200.

[0137] Decoder system 300 reduces the minimum read and write bandwidth for reading from and writing to neighboring block row buffer 205 in decoder system 300 as compared to decoder system 200. In some examples, the use of vertical raster scan order 435 and / or vertical raster scan order 530 can also reduce the minimum read and write bandwidth for reading from and writing to neighboring block row buffer 205 in decoder system 300 as compared to decoder system 200. In some examples, DB-filtered neighboring block data 235 stored in neighboring block row buffer 205 by decoder system 300 includes 4×W pixels, where W is a tile width corresponding to at least one tile of a video frame. In some examples, the memory bandwidth of neighboring block row buffer 205 of decoder system 300 is 4×W×F pixels per second, where W is a tile width corresponding to at least one tile of a video frame, where F is a frame rate associated with a video, where the video includes the video frame. For example, the example of decoder system 300 may achieve a 66.67% reduction in the amount of read and write bandwidth used to read and write from neighboring block row buffer 205 in the example of decoder system 300 as compared to the example of decoder system 200, e.g., given a maximum tile width of 8192 bits, a frame rate of 60 frames per second, and an enlargement ratio of 2, by reducing the minimum read and write bandwidth used from 12×8192×60×10 bits / second (7.3728 megabytes per second (MBps)) to 4×8192×60×10 bits / second (2.4576 MBps). This allows decoder system 300 to use components and structures (e.g., memory cells of neighboring block row buffer 205, connections to neighboring block row buffer 205, and connections from neighboring block row buffer 205) that have lower bandwidth than corresponding components in decoder system 200. Using lower bandwidth components in decoder system 300 may consume less power than using higher bandwidth components in decoder system 200. The decoder system 300 may also save power on the decoder system 200 by writing less data to the adjacent block row buffer 205 and reading less data from the adjacent block row buffer 205 .

[0138] Similar to Figure 2 The decoder system 200, Figure 3The decoder system 300 may filter the DB filtered block data 245 using the CDEF filter 215 to generate the CDEF filtered block data 250. To filter the DB filtered block data 245 using the CDEF filter 215, the decoder system 200 may also retrieve the DB filtered neighboring block data 235 from the neighboring block line buffer 205, and use the DB filtered neighboring block data 235 and the DB filtered block data 245 as inputs to the CDEF filter 215 to generate the CDEF filtered block data 250. As discussed with respect to the CDEF filter 215 of the decoder system 200, the CDEF filter 215 of the decoder system 300 may generate the CDEF filtered block data 250. In some examples, in addition to the CDEF filtered block data 250, Figure 3 The CDEF filter 215 of the decoder system 300 may also output the DB filtered neighboring block data 235. In such an example, the decoder system 300 may pass both the DB filtered neighboring block data 235 and the CDEF filtered block data 250 from the CDEF filter 215 to the amplifier 220. In some examples, the decoder system 300 may pass the DB filtered neighboring block data 235 from the neighboring block line buffer 205 to the amplifier 220, as indicated using a dashed arrow from the neighboring block line buffer 205 to the amplifier 220.

[0139] The decoder system 300 passes the CDEF-filtered block data 250 and the DB-filtered neighboring block data 235 output by the CDEF filter 215 to the amplifier 220 to generate the amplified CDEF-filtered block data 255 and the amplified DB-filtered neighboring block data 340 using the amplifier 220. In some examples, the amplified DB-filtered neighboring block data 340 is similar to the amplified DB-filtered neighboring block data 240, but is amplified by the amplifier 220 instead of the amplifier 230. The amplifier 220 amplifies the CDEF-filtered block data 250 and the DB-filtered neighboring block data 235 by the amplification factor to generate the amplified CDEF-filtered block data 255 and the amplified DB-filtered neighboring block data 340, respectively. In the illustrative example, the amplification factor is 2. In some examples, the amplification factor may be 2, 3, 4, or another suitable amplification factor.

[0140] The decoder system 300 passes the amplified CDEF-filtered block data 255 and the amplified DB-filtered neighboring block data 340 to the LR filter 225. Instead of retrieving the amplified DB-filtered neighboring block data 240 from the neighboring block line buffer 205 in the decoder system 200, the decoder system 300 instead receives the amplified DB-filtered neighboring block data 340 from the amplifier 220 and passes the amplified DB-filtered neighboring block data 340 to the LR filter 225. The decoder system 300 uses the LR filter 225 to filter the amplified CDEF-filtered block data 255 to generate the output block data 260. In order to filter the amplified CDEF-filtered block data 255 using the LR filter 225, the decoder system 300 may also use the amplified DB-filtered neighboring block data 340 and the CDEF-filtered block data 255 as inputs to the LR filter 225 to generate the output block data 260.

[0141] In some examples, the decoder system 300 sets a first indication for a first block of data and a second indication for a neighboring block of data. In some examples, when both the first block of data and the neighboring block of data are part of the input video data 270, the decoder system 300 may set the first indication for the first block of data and the second indication for the neighboring block of data before the DB filter 210 receives the input video data 270. In some examples, the decoder system 300 may use elements of the DB filter 210 to set the first indication for the first block of data and the second indication for the neighboring block of data. The first indication and the second indication may be maintained and / or retained throughout filtering using the DB filter 210, filtering using the CDEF filter 215, amplifying using the amplifier 220, and / or filtering using the LR filter 225. The decoder system 300 may then perform the first indication and the second indication during operations performed by the decoder system 300 and performed ... Figure 3 , which data belongs to the first block (e.g., DB filtered block data 245, CDEF filtered block data 250, upscaled CDEF filtered block data 255, and / or at least a subset of output block data 260 corresponding to the first block) and which data belongs to neighboring blocks (e.g., DB filtered neighboring data 235, upscaled DB filtered neighboring data 340, and / or at least a subset of output block data 260 corresponding to neighboring blocks) at any point in the decoding process shown in . For example, the decoder system 300 can identify a first subset of output video data corresponding to a first block configured to be displayed as part of a decoded video.

[0142] In some examples, the decoder system 300 uses one or more flags to set a first indication for a first block of data and a second indication for a neighboring block of data. For example, the decoder system 300 may set a flag 305 variable to one of two binary values ​​(0, 1) to represent the first indication and the second indication, respectively. For example, in some examples, the decoder system 300 may set a first indication for a first block of data by applying a flag 305 having a first value to the data of the first block, and the decoder system 300 may set a second indication for the neighboring block of data by applying a flag 305 having a second value to the data of the neighboring block. In some examples, one of the two indications (e.g., the first indication and the second indication) may be the presence of the flag 305, and the other of the two indications may be the absence of the flag 305. Therefore, in some examples, the decoder system 300 may set a first indication for the first block of data by applying the flag 305 to the data of the first block, and the decoder system 300 may set a second indication for the neighboring block of data by not applying the flag 305 to the data of the neighboring block. In some examples, decoder system 300 may set a first indication for a first block of data by not applying flag 305 to the data of the first block, and decoder system 300 may set a second indication for a neighboring block of data by applying flag 305 to the data of the neighboring block.

[0143] In some examples, decoder system 300 can pass the output of LR filter 225 (e.g., output block data 260) back to DB filter 210 for another loop through DB filter 210, CDEF filter 215, amplifier 220, LR filter 225, and / or adjacent block line buffer 205. In this way, output block data 260 can be used as at least a portion of input video data 270 (e.g., a portion representing the first block). This repetition of the filtering process may be referred to as a decoding loop, and is repeated in a loop. Figure 3 2 is represented by the dashed arrow from LR filter 225 back to DB filter 210. In some examples, output block data 260 may refer to the output of multiple iterations of the filtering process in the decoding loop.

[0144] The decoder system 300 may output the output block data 260 for storage in at least one memory unit. The decoder system 300 may output the output block data 260 to a display such that the display displays the output block data 260 as part of a video frame, such as during playback of a video including the video frame using the display. The decoder system 300 may output the output block data 260 by, for example, sending the output block data 260 to a recipient device using the communication interface 960, such that in some examples, the recipient device displays the output block data 260 as part of the video frame, such as during playback of a video including the video frame using a display of the recipient device.

[0145] Figure 4 4 is a conceptual diagram 400 illustrating reading, processing and / or outputting pixel data from a super block and adjacent blocks according to a vertical raster scan order 435. The super block shown has a super block height 405 of 64 pixels and a super block width 410 of 64 pixels. 2 =4096, so the super block includes 4096 pixels in total. The super block is divided into 256 blocks 425, each of which has a dimension of 4 pixels by 4 pixels. The blocks 425 of the super block are each shown in white with a black outline. Although Figure 4 The example of FIG. 1 shows a superblock having dimensions of 64 pixels by 64 pixels, but it should be understood that the superblock may have other sizes, such as 16 pixels by 16 pixels, 32 pixels by 32 pixels, 128 pixels by 128 pixels, 256 pixels by 256 pixels, 512 pixels by 512 pixels, 1024 pixels by 1024 pixels, 2048 pixels by 2048 pixels, or other such dimensions. In addition, although Figure 4 The example shows block 425 as having dimensions of 4 pixels by 4 pixels, but it should be understood that block 425 can have other dimensions, such as 1 pixel, 2 pixels by 2 pixels, 3 pixels by 3 pixels, 5 pixels by 5 pixels, 6 pixels by 6 pixels, 7 pixels by 7 pixels, 8 pixels by 8 pixels, 9 pixels by 9 pixels, 10 pixels by 10 pixels, and so on.

[0146] The neighboring block data 420 includes a block 430 of the neighboring block. The block 430 is shown shaded using a black dot pattern on a white background with a black outline. The block 430 is 4 pixels wide and 2 pixels high. The super block boundary 415 is visible as a thick black outline between the block 425 of the super block and the block 430 of the neighboring block. In some examples, the 4 pixel by 2 pixel block 430 of the neighboring block represents a portion of a 4 pixel by 4 pixel block of the neighboring block (not shown). Although Figure 4The example shows block 425 as having dimensions of 4 pixels by 4 pixels, but it should be understood that block 430 can have other sizes, such as 1 pixel, 2 pixels by 2 pixels, 3 pixels by 2 pixels, 5 pixels by 2 pixels, 6 pixels by 2 pixels, 7 pixels by 2 pixels, 8 pixels by 2 pixels, 9 pixels by 2 pixels, 10 pixels by 2 pixels, and so on.

[0147] In some examples, use of the vertical raster scan order 435 (e.g., for reading input video data 270, filtering using the DB filter 210, filtering using the CDEF filter 215, amplifying using the amplifier 220, filtering using the LR filter 225, any other processing and / or output) allows the decoder system 300 to write less data to the neighboring block row buffer 205, store less data in the neighboring block row buffer 205, and retrieve less data from the neighboring block row buffer 205, as compared to the decoder system 200 and / or use of a horizontal raster scan order (e.g., the horizontal raster scan order 510). For example, because the horizontal raster scan order scans row by row from top to bottom, using the horizontal raster scan order may require that an entire row (e.g., row) of pixels of the width of the super block width 410 in the neighboring block data 420 be written to the neighboring block row buffer 205, stored in the neighboring block row buffer 205, and read from the neighboring block row buffer 205 (e.g., for reading the input video data 270, filtering using the DB filter 210, filtering using the CDEF filter 215, amplifying using the amplifier 220, filtering using the LR filter 225, any other processing and / or output). In contrast, because the vertical raster scan order 435 scans column by column from left to right, using the vertical raster scan order 435 may allow portions of rows (e.g., lines) of pixels, such as portions of only one or more blocks in the block 430 corresponding to the neighboring block data 420, to be written into, stored in, and read from the neighboring block row buffer 205 without requiring the entire row (e.g., line) of pixels of the width of the super block width 410 in the neighboring block data 420 to be written into, stored in, and read from the neighboring block row buffer 205. In other words, if the block 425 is read in the vertical raster scan order 435, the neighboring block row buffer 205 may store (one or more) portions of the neighboring block data 420 that are less than the entire super block width 410 at a time. Examples of (one or more) portions of the neighboring block data 420 that may be stored in the neighboring block row buffer 205 are outlined with dashed lines. In some examples, use of vertical raster scan order 435 with decoder system 300 can help produce a 66.67% memory savings over the example of decoder system 200 discussed above. In some examples, use of vertical raster scan order 435 with decoder system 300 can help produce a 66.67% reduction in the amount of read and write bandwidth used to read from and write to neighboring block row buffer 205 in the example of decoder system 300 as compared to the example of decoder system 200 discussed above.

[0148] Figure 5Ais a conceptual diagram 500 illustrating reading, processing, and / or outputting pixel data from a block according to a horizontal raster scan order 510. The horizontal raster scan order 510 may be used by decoder system 200 and / or decoder system 300 for reading input video data 270, filtering using DB filter 210, filtering using CDEF filter 215, amplification using amplifier 220, filtering using LR filter 225, amplification using amplifier 230, any other processing, and / or outputting of block data 260. The horizontal raster scan order 510 is illustrated in conceptual diagram 500 by numbering each block (e.g., block 425) of a super block from 0 to 255 according to the horizontal raster scan order 510.

[0149] Figure 5B is a conceptual diagram 520 illustrating reading, processing, and / or outputting pixel data from a block according to a vertical raster scan order 530. The vertical raster scan order 530 may be used by decoder system 200 and / or decoder system 300 for reading input video data 270, filtering using DB filter 210, filtering using CDEF filter 215, amplification using amplifier 220, filtering using LR filter 225, amplification using amplifier 230, any other processing, and / or outputting of block data 260. The vertical raster scan order 530 is illustrated in conceptual diagram 500 by numbering each block (e.g., block 425) of a super block from 0 to 255 according to the vertical raster scan order 530. Use of vertical raster scan order 530 by decoder system 300 may help provide technical improvements, such as a reduction in the amount of data to be written to, stored in, and / or read from neighboring block row buffer 205, and / or a reduction in the minimum bandwidth for writing to and / or reading from neighboring block row buffer 205, as compared to use of horizontal raster scan order 510 by decoder system 200 and / or decoder system 300. Vertical raster scan order 530 may be an example of vertical raster scan order 435. Vertical raster scan order 435 may be an example of vertical raster scan order 530.

[0150] Figure 6600 is a conceptual diagram showing pixel data for filtering a block 605 using a loop recovery (LR) filter 225. Block 605 is shown as a 4 pixel by 4 pixel block. LR filter 225 uses pixel data from 16 pixels of block 605 itself and pixels of one or more neighboring blocks 610 to filter the pixels of block 605 using LR filter 225. The pixels of block 605 are shown as white rectangles with black outlines. The pixels of one or more neighboring blocks 610 are shown as shaded rectangles with black outlines. In some examples, LR filter 225 uses all pixels of one or more neighboring blocks 610 shown in conceptual diagram 600 to filter the pixels of block 605. In some examples, LR filter 225 uses only some pixels of one or more neighboring blocks 610 shown in conceptual diagram 600 to filter the pixels of block 605, such as pixels of one or more neighboring blocks 610 located on one or more specified sides of block 605. In some examples, pixel data of block 605 and pixel data of one or more neighboring blocks 610 may be amplified (e.g., via amplifier 220 and / or amplifier 230) before applying LR filter 225, in which case each rectangle in conceptual diagram 600 may represent an area having multiple pixels rather than a single pixel. If the amplification factor is 2, the area may be a 2-pixel by 2-pixel area, or if the amplification factor is 3, the area may be a 3-pixel by 3-pixel area, and so on.

[0151] Figure 7 7 is a conceptual diagram 700 showing a copy 720 of a row of pixels 710 from a neighboring block row buffer 205 to generate an additional row of pixels 715 for use by a loop recovery (LR) filter 225. As previously described, in some examples, the LR filter 225 uses pixel data from a specified block, and 3 rows of pixels from one or more neighboring blocks adjacent to the specified block to filter the specified block using the LR filter 225. Row 705, row 710, and row 715 may be examples of the 3 rows of pixels from one or more neighboring blocks that are written to, stored in, and read from the neighboring block row buffer 205 for use by the LR filter 225. In some examples, row 705 is the row of the 3 rows that is closest to the block to be filtered using the LR filter 225, and row 715 is the row of the 3 rows that is farthest from the block currently being filtered using the LR filter 225. Row 705, row 710, and row 715 may be examples of data in the DB filtered neighboring block data 235. The rows with row 715 generated using the replicator 720 may be referred to as a modified row set.

[0152] Instead of writing, storing, and reading all three rows, the decoder system 300 only writes, stores, and reads two rows of pixels from one or more adjacent blocks, such as row 705 and row 710. The decoder system 300 can perform a copy 720 of row 710 to generate row 715. In the conceptual diagram 700, each of the three rows is shown with a pattern of alternating white rectangles and shaded rectangles, representing the pixel data in the corresponding row. In some examples, each white rectangle or shaded rectangle represents a pixel. The pixel data in row 710 is different from the pixel data in row 705. The pixel data in row 715 is different from the pixel data in row 705. The pixel data in row 715 matches the pixel data in row 710, which is the result of the decoder system 300 copying 720 row 710 to generate row 715.

[0153] One or more rows may be magnified horizontally and / or vertically, for example, using amplifier 220 and / or amplifier 230. In such an example, each of the white rectangles and / or shaded rectangles in the conceptual diagram may represent more than one pixel based on the magnification factor. For example, if the row is magnified vertically and horizontally according to a magnification factor of 2 (e.g., using amplifier 220 and / or amplifier 230), each of the white rectangles and / or shaded rectangles may represent an area of ​​2 pixels by 2 pixels. Row 705, row 710, and row 715 may be examples of data in the amplified DB filtered adjacent block data 240, and / or examples of data in the amplified DB filtered adjacent block data 340.

[0154] The use of copy 720 by the decoder system 300 can help provide technical improvements, such as a reduction in the amount of data to be written to, stored in, and / or read from the neighboring block row buffer 205, as compared to storing all 3 rows, and / or a reduction in the minimum bandwidth used to write to and / or read from the neighboring block row buffer 205.

[0155] Figure 8800. The decoder process 800 may be performed by a decoder system. In some examples, the decoder system may include, for example, a video source 102, an encoding device 104, an encoder engine 106, storage 108, an output 110, a communication link 120, a decoding device 112, an input 114, a decoder engine 116, storage 118, a video destination device 122, a decoder system 200, an adjacent block line buffer 205, a DB filter 210, a CDEF filter 215, an amplifier 220, an LR filter 225, an amplifier 230, a decoder system 300, a computing system 900, a processor 910, an apparatus, a non-transitory computer-readable medium storing instructions for execution by one or more processors, a mobile handset, a head mounted display (HMD), a wireless communication device, or a combination thereof.

[0156] At operation 805, the decoder system is configured and capable of reading first video data from a first block of a video frame. Examples of first video data from the first block of a video frame include output 110, input 114, input video data 270, DB filtered block data 245, block 425 of a super block having super block boundary 415, a block in conceptual diagram 500, a block in conceptual diagram 520, pixels of block 605, or a combination thereof. The first block may be an example of a block and / or super block as discussed herein. For example, the first block may be Figure 4 An example of a superblock.

[0157] At operation 810, the decoder system is configured and capable of retrieving adjacent video data from a line buffer. The adjacent video data is from an adjacent block adjacent to the first block in the video frame. An example of a line buffer is an adjacent block line buffer 205. Examples of adjacent video data include adjacent video data in output 110, adjacent video data in input 114, adjacent video data in input video data 270, DB filtered adjacent block data 235, adjacent block data 420 (or at least a portion thereof in adjacent block line buffer 205), pixels of adjacent block 610, row 705, row 710, row 715, or a combination thereof. Examples of adjacent blocks include adjacent blocks 610, each of which is adjacent to block 605.

[0158] In some examples, the decoder system receives the first video data and / or the adjacent video data from an image sensor, such as image sensor 130 or an image sensor of input device 945. In some examples, the decoder system includes the image sensor. In some examples, the decoder system includes an image sensor connector that couples and / or connects the image sensor to the rest of the decoder system (e.g., including a processor and / or memory of the decoder system). In some examples, the decoder system receives the first video data and / or the adjacent video data from the image sensor by receiving the image data from, on, and / or using the image sensor connector.

[0159] In some aspects, the first video data and / or the adjacent video data are encoded using a video encoder prior to reading the first video data and retrieving the adjacent video data. Examples of a video encoder include encoding device 104 and / or encoder engine 106.

[0160] In some aspects, the decoder system is configured and capable of using a deblocking (DB) filter (e.g., DB filter 210), and / or filtering the first video data before reading the first video data to read the first video data. For example, DB filtered block data 245 can be an example of the first video data read in operation 805. In some aspects, the adjacent video data is filtered using a DB filter before retrieving the adjacent video data from the line buffer. For example, DB filtered adjacent block data 235 can be an example of the adjacent video data retrieved in operation 810.

[0161] In some aspects, before retrieving the adjacent video data from the line buffer in operation 810, the decoder system is configured and capable of reading the adjacent video data from the adjacent block of the video frame and storing the adjacent video data in the line buffer. In some aspects, before storing the adjacent video data in the line buffer, the decoder system is configured and capable of filtering the first video data and the adjacent video data using a DB filter (e.g., DB filter 210). In other words, before retrieving the adjacent video data from the line buffer in operation 810, the decoder system may read the adjacent video data from the adjacent block of the video frame, filter the adjacent video data using a DB filter (e.g., DB filter 210), and store the adjacent video data in the line buffer.

[0162] In some aspects, when retrieved from the line buffer and / or stored in the line buffer, the adjacent video data is not upscaled. In these aspects, the adjacent video data stored in the line buffer and / or retrieved from the line buffer can be an example of DB filtered adjacent block data 235, rather than an example of upscaled DB filtered adjacent block data 240.

[0163] In some aspects, reading the first video data and retrieving the adjacent video data includes reading the first video data and retrieving the adjacent video data in a vertical raster scan order (such as vertical raster scan order 435 and / or vertical raster scan order 530). In some aspects, the adjacent video data retrieved from the line buffer includes a portion of one or more pixel rows corresponding to one or more columns along the vertical raster scan order, and a width of the portion of the one or more pixel rows is less than the entire width of the first block. For example, in Figure 4 In the context of , if the blocks 425 and the neighboring block data 420 are read in a vertical raster scan order 435, the neighboring block row buffer 205 may store (one or more) portions of the neighboring block data 420 (e.g., one or more blocks 430 and / or portions thereof) that are less than the entire super block width 410 at a time. In some aspects, reading the first video data and retrieving the neighboring video data includes reading the first video data and retrieving the neighboring video data in a horizontal raster scan order (such as the horizontal raster scan order 510).

[0164] At operation 815, the decoder system is configured and capable of processing the first video data and the adjacent video data using a constrained directional enhancement filter (CDEF) to generate filtered first video data. The adjacent video data used by the CDEF is retrieved from the line buffer. An example of a CDEF is CDEF 215. An example of the filtered first video data is the CDEF filtered block data 250.

[0165] At operation 820, the decoder system is configured and capable of amplifying the filtered first video data using an amplifier to generate amplified filtered first video data. Examples of the amplifier include amplifier 220. Examples of the amplified filtered first video data include amplified CDEF filtered block data 255.

[0166] At operation 825, the decoder system is configured and capable of amplifying the adjacent video data retrieved from the line buffer using an amplifier to generate amplified adjacent video data. Examples of amplifiers include amplifier 220. Examples of amplified adjacent video data include amplified DB filtered adjacent block data 340. In some aspects, after generating the filtered first video data, the adjacent video data is amplified. In some aspects, the amplifier receives the adjacent video data from the line buffer (e.g., from the adjacent block line buffer 205). In some aspects, the amplifier receives the adjacent video data from the CDEF (e.g., from the CDEF 215), which in turn receives the adjacent video data from the line buffer (e.g., from the adjacent block line buffer 205).

[0167] At operation 830 , the decoder system is configured and capable of processing the upscaled filtered first video data and the upscaled adjacent video data using a loop restoration (LR) filter to generate output video data. Examples of LR filters include LR filter 225. Examples of output video data include output block data 260.

[0168] The DB filter (e.g., DB filter 210), CDEF (e.g., CDEF filter 215), amplifier (e.g., amplifier 220), LR filter (e.g., LR filter 225), and / or line buffer (e.g., adjacent block line buffer 205) may be referred to as a decoding loop. In some aspects, the decoder system is configured and capable of running the output block data 260 through another pass of at least a portion of the decoding loop.

[0169] In some aspects, the decoder system is configured and capable of setting an indication for the first video data. The indication is retained by the decoder system during processing using CDEF and amplifier, and is therefore retained in the filtered first video data, the amplified filtered first video data, and the output video data. The decoder system can identify a first subset of the output video data corresponding to the first block and a second subset of the output video data corresponding to the adjacent block based on the presence of the indication in the output video data. For example, the decoder system can identify that the first subset of the output video data corresponds to the first block because the first subset includes the indicator, and the second subset of the output video data corresponds to the adjacent block because the second subset lacks the indicator. Examples of indicators include a first indication, a flag 305, or a variant of the flag 305 applied by the decoder system to the first video data (e.g., the input video data 270 and / or the DB filtered block data 245). In some aspects, at least a first subset of the output video data corresponding to the first block is configured to be displayed as part of the decoded video.

[0170] Similarly, the decoder system is configured and capable of setting an indication for adjacent video data. The indication is retained by the decoder system during processing using CDEF and amplifiers, and is therefore retained in the amplified adjacent video data and output video data. The decoder system can identify a first subset of output video data corresponding to the first block and a second subset of output video data corresponding to the adjacent block based on the presence of the indication in the output video data. For example, the decoder system can identify that the first subset of output video data corresponds to the first block because the first subset lacks the bracket indicator, and the second subset of output video data corresponds to the adjacent block because the second subset includes the indicator. Examples of indicators include a second indication applied to the adjacent video data by the decoder system, i.e., a flag 305. In some aspects, the decoder system can set different indications for the first video data and the adjacent video data, which can be retained by processing (e.g., by a DB filter, CDEF, amplifier, and / or LR filter), and the decoder system can identify the first subset and the second subset based on which indicator is included in each. In some aspects, at least a first subset of the output video data corresponding to the first block is configured to be displayed as part of the decoded video.

[0171] In some aspects, the decoder system is configured and capable of duplicating at least one row of pixels in the upscaled adjacent video data to generate modified upscaled adjacent video data, such as duplication 720 of row 710 to generate row 715. In response to generating the modified upscaled adjacent video data, the decoder system is configured and capable of processing the upscaled filtered first video data and the upscaled adjacent video data using the LR filter to generate output video data. Such duplication can reduce the amount written to the line buffer, stored in the line buffer, and / or retrieved from the line buffer because a smaller set of adjacent video data can be duplicated to increase the amount of adjacent video data after retrieval from the line buffer and before or after upscaling.

[0172] In some aspects, the neighboring block is located above the first block in the video frame, such as Figure 4 Neighboring block data 420 is shown above the superblock.

[0173] In some aspects, the adjacent video data comprises 4×W pixels, where W is a tile width corresponding to at least one tile of the video frame. In some aspects, the memory bandwidth of the line buffer is 4×W×F pixels per second, where W is a tile width corresponding to at least one tile of the video frame, where F is a frame rate associated with a video, where the video comprises the video frame.

[0174] In some aspects, the CDEF, amplifier, and LR filter are associated with an AOMedia Video 1 (AV1) format, an AV1 transcoder, an AV1 decoder, or a combination thereof.

[0175] In some aspects, the decoder system is configured and capable of outputting the output video data for storage in at least one memory (e.g., cache 912, memory 915, ROM 920, RAM 925, storage device 930, or a combination thereof). In some aspects, the decoder system is configured and capable of causing the output video data to be displayed using a display (e.g., output device 935). In some aspects, the decoder system is configured and capable of causing the output video data to be sent to a recipient device (e.g., computing system 900) using a communication interface (e.g., communication interface 940).

[0176] In some examples, an imaging system may include: a component for reading first video data from a first block of a video frame; a component for retrieving adjacent video data from a line buffer, wherein the adjacent video data is from an adjacent block adjacent to the first block in the video frame; a component for processing the first video data and the adjacent video data using a constrained directional enhancement filter (CDEF) to generate filtered first video data, wherein the adjacent video data used by the CDEF is retrieved from the line buffer; a component for amplifying the filtered first video data using an amplifier to generate amplified filtered first video data; a component for amplifying the adjacent video data retrieved from the line buffer using an amplifier to generate amplified adjacent video data; and a component for processing the amplified filtered first video data and the amplified adjacent video data using a loop recovery (LR) filter to generate output video data.

[0177] In some examples, components for reading the first video data include encoding device 104, output 110, decoding device 112, input 114, decoder engine 116, decoder system 200, DB filter 210, decoder system 300, computing system 900, or a combination thereof.

[0178] In some examples, components for retrieving adjacent video data include encoding device 104, output 110, decoding device 112, input 114, decoder engine 116, decoder system 200, adjacent block row buffer 205, DB filter 210, decoder system 300, computing system 900, or combinations thereof.

[0179] In some examples, components for processing first video data and adjacent video data using CDEF include decoding device 112, decoder engine 116, decoder system 200, adjacent block row buffer 205, CDEF 215, decoder system 300, computing system 900, or a combination thereof.

[0180] In some examples, components for upscaling the filtered first video data and upscaling the adjacent video data include a decoding device 112, a decoder engine 116, a decoder system 200, an adjacent block row buffer 205, an amplifier 220, a decoder system 300, a computing system 900, or a combination thereof.

[0181] In some examples, components for processing the upscaled filtered first video data and the upscaled adjacent video data using an LR filter include a decoding device 112, a decoder engine 116, a decoder system 200, an adjacent block row buffer 205, an LR filter 225, a decoder system 300, a computing system 900, or a combination thereof.

[0182] In some examples, the processes described herein (e.g., Figure 1 , Figure 2 , Figure 3 , Figure 4 , Figure 5A , Figure 5B , Figure 6 , Figure 7 The corresponding process Figure 8 The process 800 and / or other processes described herein) can be performed by a computing device or apparatus. In some examples, the processes described herein can be performed by and / or using the following: video source 102, encoding device 104, encoder engine 106, storage 108, output 110, communication link 120, decoding device 112, input 114, decoder engine 116, storage 118, video destination device 122, decoder system 200, decoder system 300, adjacent block line buffer, DB filter 210, CDEF filter 215, amplifier 220, LR filter 225, amplifier 230, a decoder system performing decoder process 800, computing system 900, processor 910, or a combination thereof.

[0183] The computing device may include any suitable device, such as a mobile device (e.g., a mobile phone), a desktop computing device, a tablet computing device, a wearable device (e.g., a VR headset, an AR headset, AR glasses, a networked watch or smart watch or other wearable device), a server computer, a vehicle or a computing device of a vehicle, a robotic device, a television, and / or any other computing device having the resource capability to perform the processes described herein. In some cases, the computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or (one or more) other components configured to perform the steps of the processes described herein. In some examples, the computing device may include a display, a network interface configured to transmit and / or receive data, any combination thereof, and / or (one or more) other components. The network interface may be configured to transmit and / or receive data based on an Internet Protocol (IP) or other types of data.

[0184] The components of the computing device may be implemented in circuits. For example, the components may include and / or be implemented using electronic circuits or other electronic hardware, which may include one or more programmable electronic circuits (e.g., microprocessors, graphics processing units (GPUs), digital signal processors (DSPs), central processing units (CPUs), and / or other suitable electronic circuits), and / or may include and / or be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein.

[0185] The process described herein is shown as a logical flow chart, block diagram or conceptual diagram, and its operation represents a sequence of operations that can be implemented in hardware, computer instructions or a combination thereof. In the context of computer instructions, the operation represents a computer executable instruction stored on one or more computer-readable storage media, and when executed by one or more processors, the instruction performs the operation. Generally, computer executable instructions include routines, programs, objects, components, data structures, etc. that perform specific functions or implement specific data types. The order in which the operations are described is not intended to be interpreted as limiting, but any number of the described operations can be combined in any order and / or in parallel to implement the process. In some examples, the execution of certain operations described herein can be in response to the execution of other operations described herein.

[0186] In addition, the processes described herein may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented by hardware or a combination thereof as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is collectively executed on one or more processors. As described above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program including a plurality of instructions executable by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0187] Fig. 9 is a schematic diagram showing an example of a system for implementing certain aspects of the present technology. Specifically, Fig. 9 An example of a computing system 900 is shown, which may be, for example, any computing device making up an internal computing system, a remote computing system, a camera, or any component in which components of a system communicate with each other using connection 905. Connection 905 may be a physical connection using a bus, or a direct connection into processor 910, such as in a chipset architecture. Connection 905 may also be a virtual connection, a networked connection, or a logical connection.

[0188] In some aspects, computing system 900 is a distributed system, where the functionality described in the present disclosure may be distributed across a data center, multiple data centers, a peer-to-peer network, etc. In some aspects, one or more of the described system components represent a plurality of such components, each component performing some or all of the functionality for which the described component is intended. In some aspects, a component may be a physical device or a virtual device.

[0189] The example system 900 includes at least one processing unit (CPU or processor) 910 and connections 905 that couple various system components including system memory 915 such as read only memory (ROM) 920 and random access memory (RAM) 925 to the processor 910. The computing system 900 may include a cache 912 of high-speed memory that is directly connected to the processor 910, is in close proximity to the processor 910, or is integrated as part of the processor 910.

[0190] Processor 910 may include any general purpose processor and hardware or software services, such as services 932, 934, and 936 stored in storage device 930, configured to control processor 910 as well as a dedicated processor where the software instructions are incorporated into the actual processor design. Processor 910 may essentially be a completely self-contained computing system, including multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0191] To enable user interaction, the computing system 900 includes an input device 945, which can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, voice, etc. The computing system 900 can also include an output device 935, which can be one or more of a plurality of output mechanisms. In some instances, a multimodal system can enable a user to provide multiple types of input / output to communicate with the computing system 900. The computing system 900 can include a communication interface 940, which can generally govern and manage user input and system output. The communication interface may use wired and / or wireless transceivers to perform or facilitate receiving and / or sending wired or wireless communications, including transceivers utilizing: audio jack / plug, microphone jack / plug, Universal Serial Bus (USB) port / plug, Apple® Lightning® port / plug, Ethernet port / plug, fiber optic port / plug, proprietary wired port / plug, BLUETOOTH® wireless signal transmission, BLUETOOTH® low energy (BLE) wireless signal transmission, IBEACON® wireless signal transmission, radio frequency identification (RFID) wireless signal transmission, near field communication (NFC) wireless signal transmission, dedicated short range communication (SRC) wireless signal transmission, wireless cellular communication (WLCS ... The communication interface 940 may include a DSRC wireless signal transmission, a 902.11 Wi-Fi wireless signal transmission, a wireless local area network (WLAN) signal transmission, a visible light communication (VLC), a world interoperability for microwave access (WiMAX), an infrared (IR) communication wireless signal transmission, a public switched telephone network (PSTN) signal transmission, an integrated services digital network (ISDN) signal transmission, a 3G / 4G / 5G / LTE cellular data network wireless signal transmission, an autonomous network signal transmission, a radio wave signal transmission, a microwave signal transmission, an infrared signal transmission, a visible light signal transmission, an ultraviolet light signal transmission, a wireless signal transmission along the electromagnetic spectrum, or some combination thereof. The communication interface 940 may also include one or more global navigation satellite system (GNSS) receivers or transceivers for determining the location of the computing system 900 based on one or more signals received from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the United States-based Global Positioning System (GPS), the Russian-based Global Navigation Satellite System (GLONASS), the China-based BeiDou Navigation Satellite System (BDS), and the European-based Galileo GNSS. There is no restriction to operating on any particular hardware arrangement, and therefore the basic features herein may be readily substituted for improved hardware or firmware arrangements as they are developed.

[0192] The storage device 930 may be a non-volatile and / or non-transitory and / or computer-readable storage device and may be a hard disk or other type of computer-readable medium that can store computer-accessible data, such as a magnetic cassette, a flash memory card, a solid-state storage device, a digital versatile disk, a cassette, a floppy disk, a collapsible disk, a hard disk, a magnetic tape, a magnetic stripe / strip, any other magnetic storage medium, a flash memory, a memristor memory, any other solid-state memory, a compact disc read-only memory (CD-ROM) disc, a rewritable disc (CD) disc, a digital video disc (DVD) disc, a Blu-ray disc (BDD) disc, a holographic disc, another optical medium, a secure digital (SD) card, a micro secure digital (microSD) card, a memory stick (Memory Stick), or a memory card. Stick®) card, a smart card chip, an EMV chip, a subscriber identity module (SIM) card, a mini / micro / nano / pico SIM card, another integrated circuit (IC) chip / card, a random access memory (RAM), a static RAM (SRAM), a dynamic RAM (DRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash EPROM (FLASHEPROM), a cache memory (L1 / L2 / L3 / L4 / L5 / L#), a resistive random access memory (RRAM / RERAM), a phase change memory (PCM), a spin transfer torque RAM (STT-RAM), another memory chip or cartridge memory, and / or a combination thereof.

[0193] Storage device 930 may include software services, servers, services, etc., which cause the system to perform functions when the code defining such software is executed by processor 910. In some aspects, hardware services that perform a particular function may include software components stored in a computer-readable medium in association with the necessary hardware components (such as processor 910, connection 905, output device 935, etc.) to perform the function.

[0194] As used herein, the term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing or carrying (one or more) instructions and / or data. A computer-readable medium may include a non-transitory medium in which data may be stored, and the non-transitory medium does not include a carrier wave and / or a transient electronic signal propagated wirelessly or through a wired connection. Examples of non-transitory media may include, but are not limited to, a disk or tape, an optical storage medium (such as a compact disk (CD) or a digital versatile disk (DVD)), a flash memory, a memory, or a memory device. A computer-readable medium may store thereon code and / or machine-executable instructions, which may represent any combination of a procedure, function, subroutine, program, routine, subroutine, module, software package, class, or instruction, data structure, or program statement. A code segment may be coupled to another code segment or hardware circuit by passing and / or receiving information, data, independent variables, parameters, or memory contents. Information, independent variables, parameters, data, etc. may be passed, forwarded, or sent using any suitable means including memory sharing, message passing, token passing, network transmission, etc.

[0195] In some aspects, computer readable storage devices, media, and memories may include cables or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer readable storage media explicitly excludes media such as energy, carrier signals, electromagnetic waves, and signals themselves.

[0196] Specific details are provided in the above description to provide a thorough understanding of the aspects and examples provided herein. However, it will be understood by those skilled in the art that these aspects can be practiced without these specific details. For clarity of explanation, in some instances, the present technology may be presented as including individual functional blocks, which include devices, device components, steps or routines in the method embodied in software, or a combination of hardware and software. In addition to those components shown in the figures and / or described in this article, additional components may also be used. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form so as not to confuse various aspects in unnecessary details. In other instances, known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary details to avoid confusing various aspects.

[0197] Individual aspects may be described above as processes or methods, which are depicted as flowcharts, flow diagrams, data flow diagrams, structure diagrams, or block diagrams. Although a flowchart may describe operations as a sequential process, many operations may be performed in parallel or concurrently. In addition, the order of operations may be rearranged. A process terminates when its operations are completed, but other steps may not be included in the diagram. A process may correspond to a method, function, procedure, subroutine, subprogram, etc. When a process corresponds to a function, its termination may correspond to the function returning to the calling function or main function.

[0198] The processes and methods according to the above examples can be implemented using computer executable instructions stored in a computer readable medium or otherwise obtainable from a computer readable medium. Such instructions may include, for example, instructions and data that cause or otherwise configure a general-purpose computer, a special-purpose computer, or a processing device to perform a certain function or group of functions. Some of the computer resources used may be accessed through a network. Computer executable instructions may be, for example, binary files, intermediate format instructions such as assembly language, firmware, source code, etc. Examples of computer readable media that can be used to store instructions, information, and / or information created during the methods according to the examples include disks or optical disks, flash memory, USB devices with non-volatile memory, networked storage devices, etc.

[0199] Devices implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments that perform the necessary tasks (e.g., a computer program product) may be stored in a computer-readable or machine-readable medium. (One or more) processors may perform the necessary tasks. Typical examples of form factors include personal computers of laptops, smartphones, mobile phones, tablet devices, or other small form factors, personal digital assistants, rack-mounted devices, stand-alone devices, etc. The functionality described herein may also be embodied in peripheral devices or plug-in cards. As a further example, such functionality may also be implemented between different chips or different processes executed in a single device on a circuit board.

[0200] Instructions, media for transmitting such instructions, computing resources for executing them, and other structures for supporting such computing resources are example components for providing the functionality described in this disclosure.

[0201] In the foregoing description, various aspects of the application are described with reference to specific aspects, but those skilled in the art will recognize that the application is not limited thereto. Therefore, although the illustrative aspects of the application have been described in detail herein, it should be understood that the concept of the present invention can be embodied and used differently in other ways, and the appended claims are intended to be interpreted as including such changes, unless limited by the prior art. The various features and aspects of the above-mentioned applications can be used individually or in combination. In addition, without departing from the broader spirit and scope of this specification, various aspects can be used in any number of environments and applications outside the environment and application described herein. Therefore, this specification and the accompanying drawings are regarded as illustrative rather than restrictive. For the purpose of illustration, the method is described in a specific order. It should be understood that in alternative aspects, the method can be performed in a different order than described.

[0202] One of ordinary skill will understand that the less than ("<") and greater than (">") symbols or terms used herein may be replaced by less than or equal to (" ”) and greater than or equal to (“ ”) symbol.

[0203] Where a component is described as being “configured to” perform certain operations, such configuration may be achieved, for example, by designing an electronic circuit or other hardware to perform the operation, by programming a programmable electronic circuit (e.g., a microprocessor or other suitable electronic circuit) to perform the operation, or any combination thereof.

[0204] The phrase "coupled to" refers to any component that is physically connected directly or indirectly to another component, and / or any component that communicates directly or indirectly with another component (e.g., connected to another component via a wired or wireless connection and / or other suitable communication interface).

[0205] Claim language or other language that lists "at least one of" a set and / or "one or more of" a set indicates that one member of the set or multiple members of the set (in any combination) satisfies the claim. For example, claim language that lists "at least one of A and B" means A, B, or A and B. In another example, claim language that lists "at least one of A, B, and C" means A, B, C, or A and B, or A and C, or A and B and C. The language "at least one of" a set and / or "one or more of" a set does not limit the set to the items listed in the set. For example, claim language that lists "at least one of A and B" can mean A, B, or A and B, and can additionally include items not listed in the A and B sets.

[0206] The various illustrative logic blocks, modules, circuits and algorithmic steps described in conjunction with the aspects disclosed herein can be implemented as electronic hardware, computer software, firmware or a combination thereof. In order to clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits and steps have been generally described above in terms of their functions. Whether this functionality is implemented as hardware or software depends on the specific application and design constraints imposed on the entire system. Those skilled in the art can implement the described functions in different ways for each specific application, but this implementation decision should not be interpreted as causing deviations from the scope of the present application.

[0207] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. These techniques may be implemented in any device such as a general-purpose computer, a wireless communication device mobile phone, or an integrated circuit device, which has multiple uses including applications in wireless communication devices, mobile phones, and other devices. Any features described as modules or components may be implemented together in an integrated logic device, or implemented separately as separate but interoperable logic devices. If implemented in software, the technology may be implemented at least in part by a computer-readable data storage medium including a program code, which includes instructions for executing one or more of the above methods when being run. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may include a memory or data storage medium, such as a random access memory (RAM) (such as a synchronous dynamic random access memory (SDRAM)), a read-only memory (ROM), a non-volatile random access memory (NVRAM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic or optical data storage medium, and the like. Additionally or alternatively, these techniques may be implemented at least in part by a computer-readable communication medium that carries or transmits program code in the form of instructions or data structures (e.g., propagated signals or waves) and that can be accessed, read, and / or executed by a computer.

[0208] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such processors may be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor; however, in another case, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Therefore, the term "processor" as used herein may refer to any of the foregoing structures, any combination of the foregoing structures, or any other structure or device suitable for implementing the techniques described in this application. In addition, in some aspects, the functions described in this application may be provided within a dedicated software module or hardware module configured for encoding and decoding, or incorporated in a combined video decoder (CODEC).

[0209] Illustrative aspects of the disclosure include:

[0210] Aspect 1: An apparatus for media processing, the device comprising: a memory; and one or more processors coupled to the memory, the one or more processors being configured to: read first video data from a first block of a video frame; retrieve adjacent video data from a line buffer, wherein the adjacent video data comes from an adjacent block adjacent to the first block in the video frame; process the first video data and the adjacent video data using a constrained directional enhancement filter (CDEF) to generate filtered first video data, wherein the adjacent video data used by the CDEF is retrieved from the line buffer; amplify the filtered first video data using an amplifier to generate amplified filtered first video data; amplify the adjacent video data retrieved from the line buffer using an amplifier to generate amplified adjacent video data; and process the amplified filtered first video data and the amplified adjacent video data using a loop recovery (LR) filter to generate output video data.

[0211] Aspect 2. The apparatus according to aspect 1, wherein after generating the filtered first video data, adjacent video data is upscaled.

[0212] Aspect 3. The apparatus according to any one of Aspect 1 or Aspect 2, wherein the at least one processor is configured to: filter the first video data using a deblocking (DB) filter to read the first video data, wherein the adjacent video data is filtered using the DB filter before retrieving the adjacent video data from the line buffer.

[0213] Aspect 4. An apparatus according to any one of Aspects 1 to 3, wherein at least one processor is configured to: read adjacent video data from adjacent blocks of a video frame before retrieving the adjacent video data from the line buffer; and store the adjacent video data in the line buffer before retrieving the adjacent video data from the line buffer.

[0214] Aspect 5. The apparatus according to aspect 4, wherein the at least one processor is configured to filter the first video data and the adjacent video data using a deblocking (DB) filter before storing the adjacent video data in the line buffer.

[0215] Aspect 6. The apparatus according to any one of aspects 1 to 5, wherein the adjacent video data is not upscaled when retrieved from the line buffer.

[0216] Aspect 7. The apparatus according to any one of aspects 1 to 6, wherein reading the first video data and retrieving the adjacent video data comprises reading the first video data and retrieving the adjacent video data in a vertical raster scan order.

[0217] Aspect 8. An apparatus according to Aspect 7, wherein the adjacent video data retrieved from the line buffer includes portions of one or more pixel rows corresponding to one or more columns along a vertical raster scan order, wherein a width of the portion of the one or more pixel rows is less than an entire width of the first block.

[0218] Aspect 9. An apparatus according to any one of Aspects 1 to 8, wherein at least one processor is configured to: set an indication for the first video data, wherein the indication is retained in the filtered first video data, the amplified filtered first video data, and the output video data; and based on the presence of the indication in the output video data, identify a first subset of the output video data corresponding to the first block and a second subset of the output video data corresponding to the adjacent block, wherein the first subset of the output video data corresponding to the first block is configured to be displayed as part of the decoded video.

[0219] Aspect 10. An apparatus according to any one of Aspects 1 to 9, wherein at least one processor is configured to: set an indication for adjacent video data, wherein the indication is retained in the upscaled adjacent video data during processing using CDEF, and in the output video data; and based on the presence of the indication in the output video data, identify a first subset of the output video data corresponding to the first block and a second subset of the output video data corresponding to the adjacent block, wherein the first subset of the output video data corresponding to the first block is configured to be displayed as part of the decoded video.

[0220] Aspect 11. An apparatus according to any one of Aspects 1 to 10, wherein at least one processor is configured to: copy at least one row of pixels in the upscaled adjacent video data to generate modified upscaled adjacent video data; and in response to generating the modified upscaled adjacent video data, process the upscaled filtered first video data and the upscaled adjacent video data using an LR filter to generate output video data.

[0221] Aspect 12. The apparatus according to any one of aspects 1 to 11, wherein the neighboring block is located above the first block in the video frame.

[0222] Aspect 13. The apparatus according to any one of aspects 1 to 12, wherein the adjacent video data comprises 4×W pixels, wherein W is a tile width corresponding to at least one tile of the video frame.

[0223] Aspect 14. An apparatus according to any one of Aspects 1 to 13, wherein the memory bandwidth of the line buffer is 4×W×F pixels per second, where W is a tile width corresponding to at least one tile of a video frame, and where F is a frame rate associated with a video, wherein the video includes the video frame.

[0224] Aspect 15. The apparatus according to any one of aspects 1 to 14, wherein the CDEF, the amplifier, and the LR filter are associated with an AOMedia Video 1 (AV1) format.

[0225] Aspect 16. The apparatus according to any one of aspects 1 to 15, wherein the at least one processor is configured to: output the output video data for storage in at least one memory.

[0226] Aspect 17. The apparatus according to any one of Aspects 1 to 16, further comprising: a display configured to display the output video data.

[0227] Aspect 18. The apparatus according to any one of Aspects 1 to 17, further comprising: a communication interface configured to send the output video data to a recipient device.

[0228] Aspect 19. The apparatus according to any one of Aspects 1 to 18, further comprising: a video encoder configured to encode the first video data and the adjacent video data before reading the first video data and retrieving the adjacent video data.

[0229] Aspect 20. The apparatus according to any one of aspects 1 to 19, wherein the apparatus comprises at least one of the following: a head mounted display (HMD), a mobile handset, or a wireless communication device.

[0230] Aspect 21. A method for video decoding, the method comprising: reading first video data from a first block of a video frame; retrieving adjacent video data from a line buffer, wherein the adjacent video data comes from an adjacent block adjacent to the first block in the video frame; processing the first video data and the adjacent video data using a constrained directional enhancement filter (CDEF) to generate filtered first video data, wherein the adjacent video data used by the CDEF is retrieved from the line buffer; amplifying the filtered first video data using an amplifier to generate amplified filtered first video data; amplifying the adjacent video data retrieved from the line buffer using an amplifier to generate amplified adjacent video data; and processing the amplified filtered first video data and the amplified adjacent video data using a loop recovery (LR) filter to generate output video data.

[0231] Aspect 22. The method according to aspect 21, wherein after generating the filtered first video data, adjacent video data is upscaled.

[0232] Aspect 23. The method according to any one of Aspect 21 or Aspect 22, further comprising: filtering the first video data using a deblocking (DB) filter to read the first video data, wherein the adjacent video data is filtered using the DB filter before retrieving the adjacent video data from the line buffer.

[0233] Aspect 24. The method according to any one of Aspects 21 to 23, further comprising: reading adjacent video data from adjacent blocks of the video frame before retrieving the adjacent video data from the line buffer; and storing the adjacent video data in the line buffer before retrieving the adjacent video data from the line buffer.

[0234] Aspect 25. The method according to aspect 24, further comprising: filtering the first video data and the adjacent video data using a deblocking (DB) filter before storing the adjacent video data in the line buffer.

[0235] Aspect 26. The method according to any one of aspects 21 to 25, wherein the adjacent video data is not upscaled when retrieved from the line buffer.

[0236] Aspect 27. The method according to any one of aspects 21 to 26, wherein reading the first video data and retrieving the adjacent video data comprises reading the first video data and retrieving the adjacent video data in a vertical raster scan order.

[0237] Aspect 28. A method according to Aspect 27, wherein the adjacent video data retrieved from the line buffer includes portions of one or more pixel rows corresponding to one or more columns along a vertical raster scan order, wherein a width of the portion of the one or more pixel rows is less than an entire width of the first block.

[0238] Aspect 29. The method according to any one of Aspects 21 to 28 further includes: setting an indication for the first video data, wherein the indication is retained in the filtered first video data, the amplified filtered first video data, and the output video data; and based on the presence of the indication in the output video data, identifying a first subset of the output video data corresponding to the first block and a second subset of the output video data corresponding to the adjacent block, wherein the first subset of the output video data corresponding to the first block is configured to be displayed as part of the decoded video.

[0239] Aspect 30. The method according to any one of Aspects 21 to 29 further includes: setting an indication for adjacent video data, wherein the indication is retained in the amplified adjacent video data during processing using CDEF, and in the output video data; and based on the presence of the indication in the output video data, identifying a first subset of the output video data corresponding to the first block and a second subset of the output video data corresponding to the adjacent block, wherein the first subset of the output video data corresponding to the first block is configured to be displayed as part of the decoded video.

[0240] Aspect 31. The method according to any one of Aspects 21 to 30 further includes: copying at least one row of pixels in the upscaled adjacent video data to generate modified upscaled adjacent video data; and in response to generating the modified upscaled adjacent video data, processing the upscaled filtered first video data and the upscaled adjacent video data using an LR filter to generate output video data.

[0241] Aspect 32. The method according to any one of Aspects 21 to 31, wherein the neighboring block is located above the first block in the video frame.

[0242] Aspect 33. The method according to any one of aspects 21 to 32, wherein the adjacent video data comprises 4×W pixels, wherein W is a tile width corresponding to at least one tile of the video frame.

[0243] Aspect 34. A method according to any one of Aspects 21 to 33, wherein the memory bandwidth of the line buffer is 4×W×F pixels per second, where W is a tile width corresponding to at least one tile of a video frame, and where F is a frame rate associated with a video, wherein the video includes the video frame.

[0244] Aspect 35. The method according to any one of Aspects 21 to 34, wherein the CDEF, amplifier, and LR filter are associated with the AOMedia Video 1 (AV1) format.

[0245] Aspect 36. The method according to any one of Aspects 21 to 35, further comprising: outputting the output video data to be stored in at least one memory.

[0246] Aspect 37. The method according to any one of Aspects 21 to 36, further comprising: causing the output video data to be displayed using a display.

[0247] Aspect 38. The method according to any one of aspects 21 to 37, further comprising: using the communication interface so that the output video data is sent to a recipient device.

[0248] Aspect 39. The method according to any one of aspects 21 to 38, wherein the first video data and the adjacent video data are encoded using a video encoder before reading the first video data and retrieving the adjacent video data.

[0249] Aspect 40. The method according to any one of aspects 21 to 39, wherein the method is performed using an apparatus comprising at least one of: a head mounted display (HMD), a mobile handset, or a wireless communication device.

[0250] Aspect 41: A non-temporary computer-readable medium having instructions stored thereon, which, when executed by one or more processors, cause the one or more processors to: read first video data from a first block of a video frame; retrieve adjacent video data from a line buffer, wherein the adjacent video data is from an adjacent block adjacent to the first block in the video frame; process the first video data and the adjacent video data using a constrained directional enhancement filter (CDEF) to generate filtered first video data, wherein the adjacent video data used by the CDEF is retrieved from the line buffer; amplify the filtered first video data using an amplifier to generate amplified filtered first video data; amplify the adjacent video data retrieved from the line buffer using an amplifier to generate amplified adjacent video data; and process the amplified filtered first video data and the amplified adjacent video data using a loop recovery (LR) filter to generate output video data.

[0251] Aspect 42: The non-transitory computer-readable medium according to Aspect 41, further comprising the operations according to any one of Aspects 2 to 20 and / or any one of Aspects 21 to 40.

[0252] Aspect 43: A device for video decoding, the device comprising: a component for reading first video data from a first block of a video frame; a component for retrieving adjacent video data from a line buffer, wherein the adjacent video data comes from an adjacent block adjacent to the first block in the video frame; a component for processing the first video data and the adjacent video data using a constrained directional enhancement filter (CDEF) to generate filtered first video data, wherein the adjacent video data used by the CDEF is retrieved from the line buffer; a component for amplifying the filtered first video data using an amplifier to generate amplified filtered first video data; a component for amplifying the adjacent video data retrieved from the line buffer using an amplifier to generate amplified adjacent video data; and a component for processing the amplified filtered first video data and the amplified adjacent video data using a loop recovery (LR) filter to generate output video data.

[0253] Aspect 44: The apparatus according to Aspect 43, further comprising means for performing the operations according to any one of Aspects 2 to 20 and / or any one of Aspects 21 to 40.

Claims

1. A device for video decoding, the device comprising: at least one memory; as well as at least one processor coupled to the at least one memory, the at least one processor being configured to: Reading first video data from a first block of a video frame; retrieving adjacent video data from a line buffer, wherein the adjacent video data is from an adjacent block adjacent to the first block in the video frame; processing the first video data and the neighboring video data using a constrained directional enhancement filter (CDEF) to generate filtered first video data, wherein the neighboring video data used by the CDEF is retrieved from the line buffer; amplifying the filtered first video data using an amplifier to generate amplified filtered first video data; amplifying the adjacent video data retrieved from the line buffer using the amplifier to generate amplified adjacent video data; as well as The upscaled filtered first video data and the upscaled adjacent video data are processed using a loop restoration (LR) filter to generate output video data. 2 . The apparatus of claim 1 , wherein after generating the filtered first video data, the adjacent video data is upscaled.

3. The apparatus of claim 1 , wherein the at least one processor is configured to: The first video data is filtered using a deblocking (DB) filter to read the first video data, wherein the neighboring video data is filtered using the DB filter before retrieving the neighboring video data from the line buffer.

4. The apparatus of claim 1 , wherein the at least one processor is configured to: reading the adjacent video data from the adjacent block of the video frame before retrieving the adjacent video data from the line buffer; and The adjacent video data is stored in the line buffer prior to retrieving the adjacent video data from the line buffer.

5. The apparatus of claim 4, wherein the at least one processor is configured to: The first video data and the neighboring video data are filtered using a deblocking (DB) filter before storing the neighboring video data in the line buffer.

6. The device of claim 1, wherein the adjacent video data is unamplified when retrieved from the line buffer.

7. The apparatus of claim 1, wherein reading the first video data and retrieving the adjacent video data comprises reading the first video data and retrieving the adjacent video data in a vertical raster scan order.

8. An apparatus according to claim 7, wherein the adjacent video data retrieved from the row buffer includes portions of one or more pixel rows corresponding to one or more columns along a vertical raster scan order, wherein a width of the portion of the one or more pixel rows is less than an entire width of the first block.

9. The apparatus of claim 1 , wherein the at least one processor is configured to: setting an indication for the first video data, wherein the indication is retained in the filtered first video data, the upscaled filtered first video data, and the output video data; and Based on the presence of the indication in the output video data, a first subset of the output video data corresponding to the first block and a second subset of the output video data corresponding to the neighboring block are identified, wherein the first subset of the output video data corresponding to the first block is configured to be displayed as part of a decoded video.

10. The apparatus of claim 1, wherein the at least one processor is configured to: setting an indication for the neighboring video data, wherein the indication is retained during processing using the CDEF, in the upscaled neighboring video data, and in the output video data; and Based on the presence of the indication in the output video data, a first subset of the output video data corresponding to the first block and a second subset of the output video data corresponding to the neighboring block are identified, wherein the first subset of the output video data corresponding to the first block is configured to be displayed as part of a decoded video.

11. The apparatus of claim 1 , wherein the at least one processor is configured to: duplicating at least one row of pixels in the upscaled adjacent video data to generate modified upscaled adjacent video data; and In response to generating the modified upscaled neighboring video data, the upscaled filtered first video data and the upscaled neighboring video data are processed using the LR filter to generate the output video data.

12. The device of claim 1, wherein the neighboring block is located above the first block in the video frame.

13. The apparatus of claim 1, wherein the adjacent video data comprises 4×W pixels, where W is a tile width corresponding to at least one tile of the video frame.

14. The apparatus of claim 1, wherein a memory bandwidth of the line buffer is 4×W×F pixels per second, where W is a tile width corresponding to at least one tile of the video frame, and where F is a frame rate associated with a video, wherein the video includes the video frame.

15. The apparatus of claim 1, wherein the CDEF, the amplifier, and the LR filter are associated with an AOMedia Video 1 (AV1) format.

16. The apparatus of claim 1, wherein the at least one processor is configured to: The output video data is output for storage in the at least one memory.

17. The apparatus according to claim 1, further comprising: A display is configured to display the output video data.

18. The apparatus according to claim 1, further comprising: The communication interface is configured to send the output video data to a receiving device.

19. The apparatus according to claim 1, further comprising: A video encoder is configured to encode the first video data and the adjacent video data before reading the first video data and retrieving the adjacent video data.

20. The apparatus of claim 1, wherein the apparatus comprises at least one of: a head mounted display (HMD), a mobile handset, or a wireless communication device.

21. A method for video decoding, the method comprising: Reading first video data from a first block of a video frame; retrieving adjacent video data from a line buffer, wherein the adjacent video data is from an adjacent block adjacent to the first block in the video frame; processing the first video data and the neighboring video data using a constrained directional enhancement filter (CDEF) to generate filtered first video data, wherein the neighboring video data used by the CDEF is retrieved from the line buffer; amplifying the filtered first video data using an amplifier to generate amplified filtered first video data; amplifying the adjacent video data retrieved from the line buffer using the amplifier to generate amplified adjacent video data; as well as The upscaled filtered first video data and the upscaled adjacent video data are processed using a loop restoration (LR) filter to generate output video data.

22. The method of claim 21, wherein after generating the filtered first video data, the adjacent video data is upscaled.

23. The method according to claim 21, further comprising: The first video data is filtered using a deblocking (DB) filter to read the first video data, wherein the neighboring video data is filtered using the DB filter before retrieving the neighboring video data from the line buffer.

24. The method according to claim 21, further comprising: reading the adjacent video data from the adjacent block of the video frame before retrieving the adjacent video data from the line buffer; as well as The adjacent video data is stored in the line buffer prior to retrieving the adjacent video data from the line buffer.

25. The method of claim 21, wherein the adjacent video data is unamplified when retrieved from the line buffer.

26. The method of claim 21, wherein reading the first video data and retrieving the adjacent video data comprises reading the first video data and retrieving the adjacent video data in a vertical raster scan order.

27. The method of claim 21, further comprising: setting an indication for the first video data, wherein the indication is retained in the filtered first video data, the upscaled filtered first video data, and the output video data; as well as Based on the presence of the indication in the output video data, a first subset of the output video data corresponding to the first block and a second subset of the output video data corresponding to the neighboring block are identified, wherein the first subset of the output video data corresponding to the first block is configured to be displayed as part of a decoded video.

28. The method of claim 21, further comprising: setting an indication for the neighboring video data, wherein the indication is retained during processing using the CDEF, in the upscaled neighboring video data, and in the output video data; as well as Based on the presence of the indication in the output video data, a first subset of the output video data corresponding to the first block and a second subset of the output video data corresponding to the neighboring block are identified, wherein the first subset of the output video data corresponding to the first block is configured to be displayed as part of a decoded video.

29. The method of claim 21, further comprising: Copying at least one row of pixels in the upscaled adjacent video data to generate modified upscaled adjacent video data; as well as In response to generating the modified upscaled neighboring video data, the upscaled filtered first video data and the upscaled neighboring video data are processed using the LR filter to generate the output video data.

30. The device of claim 21, wherein the neighboring block is located above the first block in the video frame.

Images