System and method for video decoding with dynamic noise reconstruction

Abstract

Systems and methods for decoding are described. The decoder system is configured to retrieve noise pixel data associated with at least one reconstructed video pixel from an array of particles stored within the on-chip memory. The decoder system is configured to apply noise data to the at least one reconstructed video pixel to generate at least one output video pixel. Noise data is determined based on noise pixel data.

Description

Technical Field

[0001] The present application relates to video decoding and decompression. More specifically, the present application relates to systems and methods for performing video decoding in which film grain or another type of noise is dynamically reconstructed and added to the decoded video. Background Art

[0002] Digital video capabilities can be incorporated into a wide range of devices, including digital televisions, digital direct broadcast systems, wireless broadcast systems, personal digital assistants (PDAs), laptop or desktop computers, tablet computers, e-book readers, digital cameras, digital recording devices, digital media players, video game devices, cellular or satellite radio phones, mobile phones (e.g., so-called "smart phones"), video teleconferencing devices, video streaming devices, etc. Such devices allow video data to be processed and output for consumption. Digital video data includes a large amount of data to meet the needs of consumers and video providers. For example, consumers of video data expect high fidelity, resolution, frame rate, etc. to have the highest quality video. A large amount of video data required to meet these needs burdens the communication network and equipment that process and store video data.

[0003] Digital video devices can implement video coding techniques to compress video data. Video coding can be performed according to one or more video codecs and / or coding formats. For example, video codecs and / or coding formats include Versatile Video Coding (VVC), Basic Video Coding (EVC), High Efficiency Video Coding (HEVC), VP8, VP9, ​​Advanced Video Coding (AVC), MPEG-2 Part 2 Coding (MPEG stands for Moving Picture Experts Group), etc., and proprietary video codecs / formats, such as AOMedia Video 1 (AV1) developed by the Open Media Alliance and SMPTE 421 (also known as VC-1). Video coding generally uses prediction methods (e.g., inter-frame prediction, intra-frame prediction, etc.) that exploit the redundancy present in video images or sequences. The goal of video coding techniques is to compress video data into a form that uses a lower bit rate while avoiding or minimizing the deterioration of video quality. The goal of video decoding techniques is to reconstruct the original video data as closely as possible from the compressed video data. As evolving video services become available, coding and decoding techniques with improved coding and decoding efficiency are needed.

[0004] In some cases, a video may include film grain or other types of noise. Film grain can provide a desired aesthetic effect in a video, and it can be desirable to preserve the film grain through the encoding and decoding processes. However, in some video codecs / formats, film grain can be lost or degraded during the encoding and decoding process. Furthermore, in some video codecs / formats, the random nature of film grain or other types of noise can make it difficult to compress a video with film grain as much as a similar video without film grain. Summary of the invention

[0005] Systems and techniques for video decoding are described, for example involving film grain or another type of noise reconstructed by a decoder and dynamically added to a decoded video. In some examples, the decoder generates a grain array corresponding to an encoded video frame divided into blocks. The decoder generates random offsets for the blocks of the encoded video frame. The decoder retrieves the noise pixel data from the grain array according to the random offsets based on a comparison of the location of the noise pixel data in a noise image with the location of the block in the encoded video frame. The decoder mixes a portion of the noise pixel data in the noise image to generate a mixed portion of the noise pixel data. The portion of the noise pixel data is located at a location in the noise image corresponding to a boundary between blocks in a plurality of blocks of the encoded video frame. The decoder adds at least the mixed portion of the noise pixel data to reconstructed video frame data to generate output video frame data, the reconstructed video frame data being generated based on the encoded video frame.

[0006] In one example, an apparatus for video decoding is provided. The apparatus includes at least one memory and at least one processor (e.g., implemented as a circuit) coupled to the memory. The at least one processor includes an on-chip memory storing a particle array. The at least one processor is configured and capable of: retrieving noise pixel data associated with at least one reconstructed video pixel from the particle array stored in the on-chip memory; and applying the noise data to the at least one reconstructed video pixel to generate at least one output video pixel, wherein the noise data is determined based on the noise pixel data.

[0007] In another example, a method of video decoding is provided. The method includes: retrieving noise pixel data associated with at least one reconstructed video pixel from a granular array stored in an on-chip memory as part of a processor; and applying the noise data to the at least one reconstructed video pixel to generate at least one output video pixel, wherein the noise data is determined based on the noise pixel data.

[0008] In another example, a non-transitory computer-readable medium having instructions stored thereon is provided, the instructions, when executed by at least one processor having on-chip memory, causes the at least one processor to: retrieve noise pixel data associated with at least one reconstructed video pixel from a granule array stored within the on-chip memory; and apply the noise data to the at least one reconstructed video pixel to generate at least one output video pixel, wherein the noise data is determined based on the noise pixel data.

[0009] In another example, an apparatus for video decoding is provided. The apparatus includes: means for retrieving noise pixel data associated with at least one reconstructed video pixel from a granule array stored in an on-chip memory as part of a processor; and means for applying the noise data to the at least one reconstructed video pixel to generate at least one output video pixel, wherein the noise data is determined based on the noise pixel data.

[0010] In some aspects, one or more of the methods, apparatuses, and computer-readable media described above further include: retrieving secondary noise pixel data from a granular array stored in an on-chip memory, the secondary noise pixel data being associated with at least one reconstructed video pixel, wherein the noise data is determined based on the noise pixel data and the secondary noise pixel data. In some aspects, one or more of the methods, apparatuses, and computer-readable media described above further include: blending the noise pixel data and the secondary noise pixel data to determine the noise data, wherein at least one position of at least one reconstructed video pixel in a video frame corresponds to an overlap region where a plurality of blocks of the video frame are arranged to overlap.

[0011] In some aspects, one or more of the above methods, apparatuses, and computer-readable media further include: generating a particle array using particle parameters, the particle parameters being determined based on the video data, wherein at least one reconstructed video pixel is configured as a pixel of the reconstructed video data. In some aspects, one or more of the above methods, apparatuses, and computer-readable media further include: storing the particle array in an on-chip memory.

[0012] In some aspects, one or more of the methods, apparatus, and computer-readable media described above further include: identifying a location of the noise pixel data in the particle array based on an offset to retrieve the noise pixel data, wherein the offset corresponds to a block of which the at least one reconstructed video pixel is a part. In some aspects, the offset is determined based on a randomly generated value. In some aspects, the offset is determined based on a frame size of a video frame, wherein the at least one reconstructed video pixel is configured to reconstruct at least one pixel of the video frame. In some aspects, the block is based on the frame size.

[0013] In some aspects, one or more of the methods, apparatuses, and computer-readable media described above further include: applying the noise data to at least one reconstructed video pixel while determining secondary noise data based on secondary noise pixel data, the secondary noise pixel data being retrieved from a granule array stored in an on-chip memory, the secondary noise pixel data being associated with at least a second reconstructed video pixel. In some aspects, one or more of the methods, apparatuses, and computer-readable media described above further include: applying the noise data to at least one reconstructed video pixel while retrieving the secondary noise pixel data associated with at least a second reconstructed video pixel from a granule array stored in an on-chip memory. In some aspects, one or more of the methods, apparatuses, and computer-readable media described above further include: determining the noise pixel data while retrieving the secondary noise pixel data associated with at least a second reconstructed video pixel from a granule array stored in an on-chip memory.

[0014] In some aspects, the noise pixel data is configured to be retrieved from a particle array stored in an on-chip memory on a pixel-by-pixel basis. In some aspects, the noise data is configured to be determined based on the noise pixel data on a pixel-by-pixel basis. In some aspects, the noise data is configured to be applied to at least one reconstructed video pixel on a pixel-by-pixel basis.

[0015] In some aspects, the particle array is a luma particle array, and wherein the noise pixel data comprises luma noise pixel data. In some aspects, the particle array is a chroma particle array, and wherein the noise pixel data comprises chroma noise pixel data.

[0016] In some aspects, one or more of the methods, apparatuses, and computer-readable media described above further include: applying a scaling factor to at least one output video pixel. In some aspects, one or more of the methods, apparatuses, and computer-readable media described above further include: rounding a pixel value of at least one output video pixel. In some aspects, one or more of the methods, apparatuses, and computer-readable media described above further include: clipping a pixel value of at least one output video pixel.

[0017] In some aspects, at least one memory is configured to store at least one reconstructed video pixel. In some aspects, one or more of the methods, apparatuses, and computer-readable media described above further include: outputting at least one output video pixel for storage in at least one memory. In some aspects, one or more of the methods, apparatuses, and computer-readable media described above further include: a display configured to display at least one output video pixel. In some aspects, one or more of the methods, apparatuses, and computer-readable media described above further include: a communication interface configured to send at least one output video pixel to a recipient device.

[0018] In some aspects, one or more of the methods, apparatus, and computer-readable media described above further comprises: encoding the video data to generate encoded video data; and decoding at least the encoded video data to generate at least one reconstructed video pixel.

[0019] In some aspects, the device is a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a head-mounted display (HMD) device, a wireless communication device, a mobile device (e.g., a mobile phone and / or a mobile handset and / or a so-called "smart phone" or other mobile device), a camera, a personal computer, a laptop computer, a server computer, a vehicle or a computing device or component of a vehicle, a part of another device or a combination thereof, and / or the device includes the above items. In some aspects, the device includes a camera or multiple cameras for capturing one or more images. In some aspects, the device also includes a display for displaying one or more images, notifications, and / or other displayable data. In some aspects, the device described above can include one or more sensors (e.g., one or more inertial measurement units (IMUs), such as one or more gyroscopes, one or more gyrometers, one or more accelerometers, any combination thereof, and / or other sensors).

[0020] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used alone to determine the scope of the claimed subject matter. The subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all drawings, and each claim.

[0021] The foregoing and other features and aspects will become more apparent by reference to the following description, claims and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Illustrative aspects of the present application are described in detail below with reference to the following drawings:

[0023] Figure 1 is a block diagram illustrating an example of a system including an encoding device and a decoding device according to some examples;

[0024] Figure 2 is a block diagram illustrating a codec system having an encoder that filters out film grain noise and a decoder that reconstructs film grain noise according to some examples;

[0025] Figure 3 is a conceptual diagram illustrating generation of noise pixel data corresponding to various blocks of a video frame based on corresponding offsets according to some examples;

[0026] Figure 4is a conceptual diagram illustrating the use of horizontal blending and vertical blending at overlapping portions of noise pixel data corresponding to different blocks of a video frame according to some examples;

[0027] Figure 5 is a conceptual diagram illustrating adding a noise image to a decoded video frame using an adder to generate an output frame according to some examples;

[0028] Figure 6 is a block diagram illustrating a decoder system according to some examples, wherein the decoder system generates noise data for an entire video frame before adding the noise data to the video frame;

[0029] Figure 7 is a block diagram illustrating a decoder system according to some examples, wherein the decoder system generates noise data on a pixel-by-pixel basis and adds the noise data to a video frame on a pixel-by-pixel basis;

[0030] Figure 8 is a block diagram illustrating generating noise pixel data based on the location of the noise pixel data in a particle image according to some examples;

[0031] Fig.9A is a conceptual diagram illustrating a timeline for horizontal mixing, vertical mixing, and noise addition using a decoder system, wherein the decoder system generates noise data for an entire video frame before adding the noise data to the video frame, according to some examples;

[0032] Fig. 9B is a conceptual diagram illustrating a timeline for horizontal mixing, vertical mixing, and noise addition using a decoder system, wherein the decoder system generates noise data on a pixel-by-pixel basis and adds the noise data to video frames on a pixel-by-pixel basis, according to some examples;

[0033] Fig.10 is a flowchart illustrating a codec process according to some examples; and

[0034] Fig.11 is a diagram illustrating an example of a computing system for implementing certain aspects described herein. DETAILED DESCRIPTION

[0035] Certain aspects of the present disclosure are provided below. Some of these aspects can be applied independently, and some of them can be applied in combination, which is clear to those skilled in the art. In the following description, for the purpose of explanation, specific details are set forth in order to provide a thorough understanding of various aspects of the application. However, it will be clear that various aspects can be practiced without these specific details. The drawings and description are not intended to be restrictive.

[0036] The following description provides only example aspects and is not intended to limit the scope, applicability or configuration of the present invention. On the contrary, the following description of the example aspects will provide an enabling description for implementing the example aspects for those skilled in the art. It should be understood that various changes may be made to the function and arrangement of the elements without departing from the spirit and scope of the present application as set forth in the appended claims.

[0037] A camera is a device that uses an image sensor to receive light and capture an image frame (such as a still image or a video frame). The terms "image", "image frame" and "frame" are used interchangeably herein. A camera can be configured with various image capture and image processing settings. Different settings result in images with different appearances. Some camera settings, such as ISO, exposure time, aperture size, f / stop (aperture), shutter speed, focus and gain, are determined and applied before or during the capture of one or more image frames. For example, settings or parameters can be applied to an image sensor to capture one or more image frames. Other camera settings can configure post-processing of one or more image frames, such as changes to contrast, brightness, saturation, sharpness, levels, curves or colors. For example, settings or parameters can be applied to a processor (e.g., an image signal processor or ISP) for processing one or more image frames captured by an image sensor.

[0038] Video encoding devices implement video compression techniques to efficiently encode and decode video data. Video compression techniques may include applying different prediction modes, including spatial prediction (e.g., intra-frame prediction or intra-prediction), temporal prediction (e.g., inter-frame prediction or inter-prediction), inter-layer prediction (across different layers of video data), and / or other prediction techniques for reducing or removing redundancy inherent in video sequences. The video encoder is able to partition each picture of the original video sequence into rectangular areas called video blocks or coding units (described in more detail below). These video blocks may be encoded using specific prediction modes.

[0039] The video block may be divided into one or more groups of smaller blocks in one or more ways. A block may include a coding tree block, a prediction block, a transform block, or other suitable blocks. Unless otherwise specified, references to "blocks" in general may refer to such video blocks (e.g., coding tree blocks, coding blocks, prediction blocks, transform blocks, or other suitable blocks or sub-blocks, as will be understood by one of ordinary skill in the art). In addition, each of these blocks may also be referred to interchangeably herein as a "unit" (e.g., coding tree unit (CTU), coding unit, prediction unit (PU), transform unit (TU), etc.). In some cases, a unit may indicate a coding logic unit encoded in a bitstream, and a block may indicate a portion of a video frame buffer for which the process is directed.

[0040] In some cases, a video may include film grain or other types of noise. Film grain can provide a desired aesthetic effect in a video, and it can be desirable to preserve the film grain through the encoding and decoding process. However, in some video codecs / formats, film grain can be lost or degraded during the encoding and decoding process. Furthermore, in some video codecs / formats, the random nature of film grain or other types of noise can make it difficult to compress a video with film grain as much as a similar video without film grain.

[0041] Systems and techniques for video decoding are described, for example involving film grain or another type of noise reconstructed by a decoder and dynamically added to a decoded video. In some examples, a decoder generates a grain array corresponding to an encoded video frame divided into blocks. The decoder generates random offsets for the blocks of the encoded video frame. The decoder retrieves the noise pixel data from the grain array according to the random offsets based on a comparison of the location of the noise pixel data in a noise image with the location of the block in the encoded video frame. The decoder mixes a portion of the noise pixel data in the noise image to generate a mixed portion of the noise pixel data. The portion of the noise pixel data is located at a location in the noise image corresponding to a boundary between blocks in a plurality of blocks of the encoded video frame. The decoder adds at least the mixed portion of the noise pixel data to reconstructed video frame data to generate output video frame data, wherein the reconstructed video frame data is generated based on the encoded video frame.

[0042] The codec systems and techniques described herein provide many technical improvements over existing codec systems. For example, the codec systems and techniques described herein allow film grain or other types of noise in decoded video to be reconstructed, thereby providing an improvement over codec systems in which film grain is lost or degraded during the encoding and / or decoding process. The codec systems and techniques described herein allow video to be compressed more than other codec systems. For example, by filtering out noise before encoding the video, generating grain parameters based on the filtered noise, reconstructing noise based on the grain parameters, and adding the reconstructed noise back to the decoded video, the codec systems and techniques described herein can compress the encoded video more heavily and still reconstruct film grain compared to codec systems that attempt to retain film grain during encoding. This increased compression can correspond to a reduced bit rate (in an illustrative example, a reduction of approximately 50%) compared to codec systems that attempt to maintain film grain during encoding. The codec systems and techniques described herein improve the efficiency of the reconstruction and addition of film grain by allowing the reconstruction and addition of film grain to occur dynamically (e.g., on a pixel-by-pixel basis), thereby improving the efficiency of codecs that wait for the entire frame of film grain data to be reconstructed before adding the film grain data to the decoded video. In some examples, the dynamic film grain reconstruction of the codec systems and techniques described herein can provide bandwidth savings of 778 megabytes per second (MBps) (given 1080p resolution and 120 frames per second (fps) frame rate) to 12458MBps (given 8K resolution and 120fps frame rate). In some examples, the dynamic film grain reconstruction of the codec systems and techniques described herein can reduce the memory read and write cycles used during decoding, and thus improve the speed of the decoded stream, especially for higher bit rate streams. In some examples, the speed and efficiency improvements described herein for codec systems and techniques provide improved usability for low latency use cases such as extended reality (XR) (e.g., virtual reality (VR), augmented reality (AR), and / or mixed reality (MR)).

[0043] Various aspects of the application will be described with reference to the accompanying drawings. Figure 11 is a block diagram showing an example of a system 100 including an encoding device 104 and a decoding device 112. The encoding device 104 may be part of a source device, and the decoding device 112 may be part of a receiving device. The source device and / or the receiving device may include an electronic device, such as a mobile or fixed telephone handset (e.g., a smart phone, a cellular phone, etc.), a desktop computer, a laptop or notebook computer, a tablet computer, a set-top box, a television, a camera, a display device, a digital media player, a video game console, a video streaming device, an Internet Protocol (IP) camera, or any other suitable electronic device. In some examples, the source device and the receiving device may include one or more wireless transceivers for wireless communication. The encoding techniques described herein are applicable to video encoding in various multimedia applications, including streaming video transmission (e.g., via the Internet), television broadcasting or transmission, encoding of digital videos stored on data storage media, decoding of digital videos stored on data storage media, or other applications. In some examples, the system 100 is capable of supporting one-way or two-way video transmission to support applications such as video conferencing, video streaming, video playback, video broadcasting, gaming, and / or video telephony.

[0044] The encoding device 104 (or encoder) can be used to encode video data using a video coding standard or protocol to generate a coded video bitstream. Examples of video coding standards include ITU-T H.261, ISO / IEC MPEG-1 Visual, ITU-T H.262 or ISO / IEC MPEG-2 Visual, ITU-T H.263, ISO / IEC MPEG-4 Visual, ITU-T H.264 (also known as ISO / IEC MPEG-4 AVC), including its scalable video coding (SVC) and multi-view video coding (MVC) extensions, and high-efficiency video coding (HEVC) or ITU-T H.265. There are various extensions of HEVC that handle multi-layer video coding, including range and screen content coding extensions, 3D video coding (3D-HEVC) and multi-view extensions (MV-HEVC) and scalable extensions (SHVC). HEVC and its extensions have been developed by the Joint Collaborative Team on Video Coding (JCT-VC) of the ITU-T Video Coding Experts Group (VCEG) and the ISO / IEC Moving Picture Experts Group (MPEG), and the Joint Collaborative Team on 3D Video Coding Extension Development (JCT-3V).

[0045] MPEG and ITU-TVCEG have also formed a joint exploration video team (JVET) to explore new coding tools for the next generation video coding standard, called Versatile Video Coding (VVC). The reference software is called the VVC Test Model (VTM) (or JEM (Joint Exploration Model)). The goal of VVC is to provide significant improvements in compression performance over the existing HEVC standard, thereby helping to deploy higher quality video services and emerging applications (e.g., such as 360° omnidirectional immersive multimedia, high dynamic range (HDR) video, etc.). VP9, ​​Alliance for Open Media (AOMedia) Video 1 (AV1) and Basic Video Coding (EVC) are other video codecs and / or coding formats that can apply the techniques described herein.

[0046] The techniques described herein can be applied to any of the existing video codecs (e.g., High Efficiency Video Coding (HEVC), Advanced Video Coding (AVC), or other suitable existing video codecs), and / or can be efficient coding tools for any video coding standard being developed and / or future video coding standards (such as, for example, VVC and / or other video coding standards under development or to be developed). For example, the examples described herein can be performed using video codecs such as VVC, HEVC, AVC, and / or their extensions. However, the techniques and systems described herein can also be applied to other coding standards, such as MPEG, JPEG (or other coding standards for still images), VP9, ​​AV1, their extensions, or other suitable codecs and / or coding formats that are already available or not yet available or not yet developed. Therefore, although the techniques and systems described herein can be described with reference to a specific video coding standard, those of ordinary skill in the art will recognize that the description should not be interpreted as being applicable only to a specific standard.

[0047] Many of the embodiments described herein provide examples using the JEM model, VVC, HEVC standards, and / or their extensions. However, the techniques and systems described herein may also be applicable to other coding standards, such as AVC, MPEG, JPEG (or other coding standards for still images), their extensions, or other suitable coding standards that are already available or not yet available or not yet developed. Therefore, although the techniques and systems described herein may be described with reference to a specific video coding standard, those of ordinary skill in the art will recognize that the description should not be interpreted as being applicable only to a specific standard.

[0048] refer to Figure 1, video source 102 may provide video data to encoding device 104. Video source 102 may be part of a source device, or may be part of a device other than a source device. Video source 102 may include a video capture device (e.g., a video camera, a camera phone, a video phone, etc.), a video archive containing stored video, a video server or content provider that provides video data, a video feed interface that receives video from a video server or content provider, a computer graphics system for generating computer graphics video data, a combination of such sources, or any other suitable video source.

[0049] The video data from video source 102 may include one or more input pictures or frames. A picture or frame is a still image that is, in some cases, part of a video. In some examples, the data from video source 102 may be a still image that is not part of a video. In HEVC, VVC, and other video coding specifications, a video sequence may include a series of pictures. A picture may include three sample arrays, denoted S L , S Cb and S Cr . S L is a two-dimensional array of brightness samples, S Cb is a two-dimensional array of Cb chrominance samples, and S Cr is a two-dimensional array of Cr chroma samples. Chroma samples may also be referred to herein as "chroma" samples. A pixel can refer to all three components (luminance and chrominance samples) of a given position in a picture array. In other cases, a picture may be monochrome and may include only a luminance sample array, in which case the terms pixel and sample can be used interchangeably. With respect to the example techniques described herein that refer to a single sample for illustrative purposes, the same techniques can be applied to pixels (e.g., all three sample components of a given position in a picture array). With respect to the example techniques described herein that refer to pixels for illustrative purposes (e.g., all three sample components of a given position in a picture array), the same techniques can be applied to a single sample.

[0050] There are two types of network abstraction layer (NAL) units in the HEVC standard, including video coding layer (VCL) NAL units and non-VCL NAL units. The VCL NAL unit includes a slice or slice segment (described below) of coded picture data, and the non-VCL NAL unit includes control information related to one or more coded pictures. In some cases, the NAL unit can be called a packet. The HEVC AU includes a VCL NAL unit containing coded picture data and a non-VCL NAL unit corresponding to the coded picture data (if present).

[0051] A NAL unit may contain a sequence of bits that form an encoded representation of video data (e.g., a coded video bitstream, a CVS of a bitstream, etc.), such as an encoded representation of a picture in a video. The encoder engine 106 generates an encoded representation of a picture by partitioning each picture into a plurality of slices. A slice is independent of other slices so that information in a slice is encoded without relying on data from other slices within the same picture. A slice includes one or more slice segments, including an independent slice segment and, if present, one or more dependent slice segments that depend on previous slice segments.

[0052] In HEVC, the slice is then divided into coding tree blocks (CTBs) of luma samples and chroma samples. The CTB of luma samples and one or more CTBs of chroma samples together with the syntax of the samples are called coding tree units (CTUs). CTUs can also be called "tree blocks" or "largest coding units" (LCUs). CTUs are the basic processing units for HEVC encoding. CTUs can be split into multiple coding units (CUs) of different sizes. A CU contains arrays of luma and chroma samples called coding blocks (CBs).

[0053] Luma and chroma CBs can be further split into prediction blocks. A PB is a sample block of a luma component or a chroma component that uses the same motion parameters for inter-frame prediction or intra-frame block copy (IBC) prediction (when available or enabled for use). A luma PB and one or more chroma PBs together with associated syntax form a prediction unit (PU). For inter-frame prediction, a motion parameter set (e.g., one or more motion vectors, reference indices, etc.) is signaled in the bitstream of each PU, and the motion parameter set is used for inter-frame prediction of the luma PB and one or more chroma PBs. Motion parameters can also be referred to as motion information. A CB can also be partitioned into one or more transform blocks (TBs). A TB represents a square block of samples to which a residual transform (e.g., the same two-dimensional transform in some cases) is applied for encoding the color component of the prediction residual signal. A transform unit (TU) represents a TB of luma and chroma samples and corresponding syntax elements. Transform coding is described in more detail below.

[0054] The size of the CU corresponds to the size of the coding mode and may be square in shape. For example, the size of the CU may be 8×8 samples, 16×16 samples, 32×32 samples, 64×64 samples, or any other appropriate size up to the size of the corresponding CTU. The phrase "N×N" is used herein to refer to the pixel dimensions of a video block in terms of vertical and horizontal dimensions (e.g., 8 pixels×8 pixels). The pixels in a block may be arranged in rows and columns. In some embodiments, a block may not have the same number of pixels in the horizontal direction as in the vertical direction. Syntax data associated with a CU may describe, for example, that a CU is partitioned into one or more PUs. The partitioning mode may differ between whether a CU is encoded in an intra-prediction mode or an inter-prediction mode. A PU may be partitioned into a non-square shape. Syntax data associated with a CU may also describe, for example, partitioning a CU into one or more TUs according to a CTU. The shape of a TU can be square or non-square.

[0055] According to the HEVC standard, a transform unit (TU) may be used to perform the transform. The TU may vary for different CUs. The TU may be sized based on the size of the PU within a given CU. The TU may be the same size as the PU or smaller than the PU. In some examples, a quadtree structure called a residual quadtree (RQT) may be used to subdivide the residual samples corresponding to the CU into smaller units. The leaf nodes of the RQT may correspond to the TU. The pixel difference values ​​associated with the TU may be transformed to generate transform coefficients. The transform coefficients may then be quantized by the encoder engine 106.

[0056] Once the picture of the video data is partitioned into CUs, the encoder engine 106 predicts each PU using a prediction mode. The prediction unit or prediction block is then subtracted from the original video data to obtain a residual (described below). For each CU, the prediction mode can be signaled inside the bitstream using syntax data. The prediction mode can include intra-frame prediction (or intra-picture prediction) or inter-frame prediction (or inter-picture prediction). Intra-frame prediction exploits the correlation between spatially adjacent samples within a picture. For example, with respect to using intra-frame prediction, each PU is predicted from adjacent image data in the same picture using (for example) DC prediction to find the average value of the PU, using plane prediction to fit a plane surface to the PU, using directional prediction to extrapolate from adjacent data, or using any other suitable type of prediction. Inter-frame prediction uses temporal correlation between pictures to derive motion compensated predictions for blocks of image samples. For example, with respect to using inter-frame prediction, each PU is predicted from image data in one or more reference pictures (before or after the current picture in output order) using motion compensated prediction. A decision on whether to use inter-picture prediction or intra-picture prediction to encode a picture area can be made, for example, at the CU level.

[0057] The encoder engine 106 and the decoder engine 116 (described in more detail below) can be configured to operate according to VVC. According to VVC, a video encoder (such as the encoder engine 106 and / or the decoder engine 116) partitions a picture into multiple coding tree units (CTUs) (wherein the CTB of a luma sample and one or more CTBs of a chroma sample together with the syntax of the sample are referred to as CTUs). The video encoder is capable of partitioning the CTU according to a tree structure (such as a quadtree-binary tree (QTBT) structure or a multi-type tree (MTT) structure). The QTBT structure removes the concept of multiple partition types, such as the separation between CU, PU, ​​and TU of HEVC. The QTBT structure includes two levels, including a first level partitioned according to quadtree partitioning and a second level partitioned according to binary tree partitioning. The root node of the QTBT structure corresponds to the CTU. The leaf nodes of the binary tree correspond to coding units (CUs).

[0058] In the MTT segmentation structure, blocks can be segmented using quadtree segmentation, binary tree segmentation, and one or more types of ternary tree segmentation. A ternary tree segmentation is a segmentation in which a block is split into three sub-blocks. In some examples, a ternary tree segmentation divides a block into three sub-blocks without dividing the original block by the center. The segmentation types in MTT (e.g., quadtree, binary tree, and ternary tree) can be symmetric or asymmetric.

[0059] In some examples, the video encoder can use a single QTBT or MTT structure to represent each of the luma and chroma components, while in other examples, the video encoder can use two or more QTBT or MTT structures, such as one QTBT or MTT structure for the luma component and another QTBT or MTT structure for the two chroma components (or two QTBT and / or MTT structures for respective chroma components).

[0060] The video encoder can be configured to use quadtree segmentation, QTBT segmentation, MTT segmentation, or other segmentation structures per HEVC. For illustrative purposes, the description herein may refer to QTBT segmentation. However, it should be understood that the technology of the present disclosure may also be applied to a video encoder configured to use quadtree segmentation or other types of segmentation.

[0061] In some examples, one or more slices of a picture are assigned a slice type. Slice types include I slices, P slices, and B slices. I slices (intra, independently decodable) are slices of a picture that are encoded only by intra prediction and are therefore independently decodable because an I slice requires only data within a frame to predict any prediction unit or prediction block of the slice. P slices (unidirectionally predicted frames) are slices of a picture that can be encoded with intra prediction or with unidirectional inter prediction. Each prediction unit or prediction block within a P slice is encoded with intra prediction or inter prediction. When inter prediction is applied, the prediction unit or prediction block is predicted by only one reference picture, so the reference sample comes from only one reference region of a frame. B slices (bidirectionally predicted frames) are slices of a picture that can be encoded with intra prediction and with inter prediction (e.g., bidirectional prediction or unidirectional prediction). The prediction unit or prediction block of a B slice can be bidirectionally predicted from two reference pictures, where each picture contributes one reference region and the sample sets of the two reference regions are weighted (e.g., with equal weights or with different weights) to produce a prediction signal for a bidirectionally predicted block. As explained above, slices of a picture are independently encoded. In some cases, a picture can be encoded as only one slice.

[0062] As described above, intra-picture prediction of a picture exploits the correlation between spatially adjacent samples within a picture. There are multiple intra-prediction modes (also referred to as "intra-modes"). In some examples, intra-prediction of a luma block includes 35 modes, including a planar mode, a DC mode, and 33 angular modes (e.g., a diagonal intra-prediction mode and an angular mode adjacent to the diagonal intra-prediction mode). The 35 modes of intra-prediction are indexed as shown in Table 1 below. In other examples, more intra-modes may be defined that include prediction angles that may not yet be represented by the 33 angular modes. In other examples, the prediction angle associated with the angular mode may be different from the prediction angle used in HEVC.

[0063] Table 1 Specification of intra prediction modes and associated names

[0064] Intra prediction mode Association Name 0 INTRA_PLANAR 1 INTRA_DC 2..34 INTRA_ANGULAR2..INTRA_ANGULAR34

[0065] Inter-picture prediction uses the temporal correlation between pictures in order to derive motion compensated prediction of image sample blocks. Using a translational motion model, the position of a block in a previously decoded picture (reference picture) is indicated by a motion vector (Δx, Δy), where Δx specifies the horizontal displacement of the reference block relative to the position of the current block and Δy specifies the vertical displacement of the reference block relative to the position of the current block. In some cases, the motion vector (Δx, Δy) can be integer sample precision (also called integer precision), in which case the motion vector points to an integer pixel grid (or integer pixel sampling grid) of a reference frame. In some cases, the motion vector (Δx, Δy) can have fractional sample precision (also called fractional pixel precision or non-integer precision) to more accurately capture the movement of underlying objects, without being limited to the integer pixel grid of a reference frame. The precision of the motion vector can be represented by the quantization level of the motion vector. For example, the quantization level can be integer precision (e.g., 1 pixel) or fractional pixel precision (e.g., 1 / 4 pixel, 1 / 2 pixel, or other sub-pixel value). When the corresponding motion vector has fractional sample precision, interpolation is applied to the reference picture to derive a prediction signal. For example, samples available at integer positions can be filtered (e.g., using one or more interpolation filters) to estimate values ​​at fractional positions. Previously decoded reference pictures are indicated by reference indices (refIdx) to reference picture lists. Motion vectors and reference indices can be referred to as motion parameters. Two types of inter-picture prediction can be performed, including unidirectional prediction and bidirectional prediction.

[0066] With inter prediction using bidirectional prediction, two sets of motion parameters (Δx0, y0, refIdx0 and Δx1, y1, refIdx1) are used to generate two motion compensated predictions (from the same reference image or possibly from different reference images). For example, with bidirectional prediction, each prediction block uses two motion compensated prediction signals and generates a B prediction unit. The two motion compensated predictions are then combined to obtain the final motion compensated prediction. For example, the two motion compensated predictions can be combined by averaging. In another example, weighted prediction can be used, in which case different weights can be applied to each motion compensated prediction. The reference pictures that can be used in bidirectional prediction are stored in two separate lists, denoted as list 0 and list 1. The motion parameters can be derived at the encoder using a motion estimation process.

[0067] In the case of inter prediction using unidirectional prediction, a set of motion parameters (Δx0, y0, refIdx0) is used to generate a motion compensated prediction from a reference picture. For example, with unidirectional prediction, each prediction block uses at most one motion compensated prediction signal and generates a P prediction unit.

[0068] The PU may include data related to the prediction process (e.g., motion parameters or other suitable data). For example, when the PU is encoded using intra prediction, the PU may include data describing the intra prediction mode for the PU. As another example, when the PU is encoded using inter prediction, the PU may include data defining the motion vector of the PU. The data defining the motion vector of the PU may describe, for example, the horizontal component of the motion vector (Δx), the vertical component of the motion vector (Δy), the resolution of the motion vector (e.g., integer precision, quarter-pixel precision, or eighth-pixel precision), the reference picture to which the motion vector points, the reference index, the reference picture list of the motion vector (e.g., list 0, list 1, or list C), or any combination thereof.

[0069] After performing prediction using intra-frame and / or inter-frame prediction, the encoding device 104 may perform transformation and quantization. For example, after prediction, the encoder engine 106 may calculate a residual value corresponding to the PU. The residual value may include a pixel difference between a current block (PU) of pixels being encoded and a prediction block (e.g., a predicted version of the current block) used to predict the current block. For example, after generating a prediction block (e.g., issuing an inter-frame prediction or an intra-frame prediction), the encoder engine 106 can generate a residual block by subtracting a prediction block generated by a prediction unit from the current block. The residual block includes a set of pixel difference values ​​that quantize the difference between the pixel values ​​of the current block and the pixel values ​​of the prediction block. In some examples, the residual block may be represented in a two-dimensional block format (e.g., a two-dimensional matrix or array of pixel values). In this example, the residual block is a two-dimensional representation of pixel values.

[0070] Any residual data that may remain after prediction is performed is transformed using a block transform, which may be based on a discrete cosine transform, a discrete sine transform, an integer transform, a wavelet transform, other suitable transform functions, or any combination thereof. In some cases, one or more block transforms (e.g., of size 32×32, 16×16, 8×8, 4×4, or other suitable sizes) may be applied to the residual data in each CU. In some embodiments, a TU may be used for a transform and quantization process implemented by the encoder engine 106. A given CU with one or more PUs may also include one or more TUs. As described in further detail below, a block transform may be used to transform residual values ​​into transform coefficients, which may then be quantized and scanned using the TUs to produce serial transform coefficients for entropy encoding.

[0071] In some embodiments, after intra-prediction or inter-prediction coding using the PU of the CU, the encoder engine 106 may calculate the residual data of the TU of the CU. The PU may include pixel data in the spatial domain (or pixel domain). The TU may include coefficients in the transform domain after applying the block transform. As previously described, the residual data may correspond to the pixel difference between the pixel of the uncoded picture and the predicted value corresponding to the PU. The encoder engine 106 may form a TU including the residual data of the CU, and then may transform the TU to generate the transform coefficients of the CU.

[0072] The encoder engine 106 may perform quantization of the transform coefficients. Quantization provides further compression by quantizing the transform coefficients to reduce the amount of data used to represent the coefficients. For example, quantization may reduce the bit depth associated with some or all of the coefficients. In one example, a coefficient having an n-bit value may be rounded down to an m-bit value during quantization, where n is greater than m.

[0073] Once quantization is performed, the encoded video bitstream includes quantized transform coefficients, prediction information (e.g., prediction mode, motion vectors, block vectors, etc.), segmentation information, and any other suitable data, such as other syntax data. The different elements of the encoded video bitstream can then be entropy encoded by the encoder engine 106. In some examples, the encoder engine 106 can scan the quantized transform coefficients using a predefined scan order to produce a serial vector that can be entropy encoded. In some examples, the encoder engine 106 can perform adaptive scanning. After scanning the quantized transform coefficients to form a vector (e.g., a one-dimensional vector), the encoder engine 106 can entropy encode the vector. For example, the encoder engine 106 can use context adaptive variable length coding, context adaptive binary arithmetic coding, grammar-based context adaptive binary arithmetic coding, probability interval segmentation entropy coding, or another suitable entropy coding technique.

[0074] As previously described, the HEVC bitstream includes a set of NAL units, including VCL NAL units and non-VCL NAL units. The VCL NAL unit includes coded picture data that forms a coded video bitstream. For example, a bit sequence that forms a coded video bitstream is present in a VCL NAL unit. In addition to other information, a non-VCL NAL unit may also contain a parameter set with advanced information related to the coded video bitstream. For example, a parameter set may include a video parameter set (VPS), a sequence parameter set (SPS), and a picture parameter set (PPS). Examples of the goals of a parameter set include bit rate efficiency, error resilience, and providing a system layer interface. Each slice references a single active PPS, SPS, and VPS to access information that a decoding device 112 can use to decode a slice. An identifier (ID) may be encoded for each parameter set, including a VPS ID, an SPS ID, and a PPS ID. The SPS includes an SPSID and a VPS ID. The PPS includes a PPS ID and an SPS ID. Each slice header includes a PPS ID. By using an ID, an active parameter set can be identified for a given slice.

[0075] The PPS includes information applied to all slices in a given picture. Therefore, all slices in a picture refer to the same PPS. Slices in different pictures can also refer to the same PPS. The SPS includes information applied to all pictures in the same coded video sequence (CVS) or bitstream. As previously described, a coded video sequence is a series of access units (AUs) that start with a random access point picture (e.g., an instantaneous decoding reference (IDR) picture or a broken link access (BLA) picture, or other appropriate random access point pictures) in the base layer with certain properties (described above) until and not including the next AU (or the end of the bitstream) with a random access point picture in the base layer with certain properties. The information in the SPS may not change from picture to picture within the coded video sequence. The pictures in the coded video sequence can use the same SPS. The VPS includes information applied to all layers within the coded video sequence or bitstream. The VPS includes a syntax structure with syntax elements applied to the entire coded video sequence. In some embodiments, the VPS, SPS, or PPS may be sent in-band with the coded bitstream. In some embodiments, the VPS, SPS, or PPS may be sent out-of-band in a transmission separate from the NAL units containing the encoded video data.

[0076] The video bitstream can also include a supplemental enhancement information (SEI) message. For example, a SEI NAL unit can be part of a video bitstream. In some cases, the SEI message can contain information that is not needed for the decoding process. For example, the information in the SEI message may not be necessary for the decoder to decode the video picture of the bitstream, but the decoder can use the information to improve the display or processing of the picture (e.g., decoded output). The information in the SEI message can be embedded metadata. In an illustrative example, the information in the SEI message can be used by a decoder-side entity to improve the visibility of the content. In some cases, certain application standards can force the presence of such SEI messages in the bitstream, so that quality improvements can be brought to all devices that comply with the application standards (e.g., the carrying of frame encapsulation SEI messages for frame-compatible planar stereoscopic 3DTV video formats, where SEI messages are carried for each frame of the video, handling recovery point SEI messages, using pan scan rectangle SEI messages in DVB, and many other examples).

[0077] The output 110 of the encoding device 104 may transmit the NAL units constituting the encoded video bitstream data to a decoding device 112 of a receiving device via a communication link 120. The input 114 of the decoding device 112 may receive the NAL units. The communication link 120 may include a channel provided by a wireless network, a wired network, or a combination of a wired and wireless network. The wireless network may include any wireless interface or combination of wireless interfaces and may include any suitable wireless network (e.g., the Internet or other wide area network, a packet-based network, a WiFi, or a wireless network). TM , Radio Frequency (RF), UWB, WiFi Direct, Cellular, Long Term Evolution (LTE), WiMax TM The wired network may include any wired interface (e.g., optical fiber, Ethernet, Ethernet over power line, Ethernet over coaxial cable, digital signal line (DSL), etc.). The wired and / or wireless network may be implemented using various devices such as base stations, routers, access points, bridges, gateways, switches, etc. The encoded video bitstream data may be modulated according to a communication standard (e.g., a wireless communication protocol) and sent to a receiving device.

[0078] In some examples, the encoding device 104 may store the encoded video bitstream data in a storage device 108. The output 110 may retrieve the encoded video bitstream data from the encoder engine 106 or from the storage device 108. The storage device 108 may include any of a variety of distributed or locally accessed data storage media. For example, the storage device 108 may include a hard drive, a storage optical disk, a flash memory, a volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded video data. The storage device 108 can also include a decoded picture buffer (DPB) for storing reference pictures used in inter-frame prediction. In another example, the storage device 108 can correspond to a file server or another intermediate storage device that can store the encoded video generated by the source device. In this case, the receiving device including the decoding device 112 can access the stored video data from the storage device via streaming or downloading. The file server can be any type of server that can store the encoded video data and send the encoded video data to the receiving device. Example file servers include web servers (e.g., for websites), FTP servers, network attached storage (NAS) devices, or local disk drives. The receiving device can access the encoded video data through any standard data connection (including an Internet connection). This can include a wireless channel (e.g., a Wi-Fi connection), a wired connection (e.g., DSL, cable modem, etc.), or a combination of both suitable for accessing encoded video data stored on a file server. The transmission of the encoded video data from the storage device 108 can be a streaming transmission, a download transmission, or a combination thereof.

[0079] The input 114 of the decoding device 112 receives the encoded video bitstream data, and the video bitstream data can be provided to the decoder engine 116, or to the storage device 118 for later use by the decoder engine 116. For example, the storage device 118 can include a DPB for storing reference pictures used in inter-frame prediction. A receiving device including the decoding device 112 can receive the encoded video data to be decoded via the storage device 108. The encoded video data can be modulated according to a communication standard (such as a wireless communication protocol) and sent to the receiving device. The communication medium for sending the encoded video data can include any wireless or wired communication medium, such as a radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium can form a part of a packet-based network (such as a local area network, a wide area network, or a global network such as the Internet). The communication medium can include a router, a switch, a base station, or any other device that can be used to facilitate communication from a source device to a receiving device.

[0080] The decoder engine 116 can decode the coded video bitstream data by entropy decoding (e.g., using an entropy decoder) and extracting elements of one or more coded video sequences that constitute the coded video data. The decoder engine 116 can then rescale the coded video bitstream data and perform an inverse transform on the coded video bitstream data. The residual data is then passed to the prediction stage of the decoder engine 116. The decoder engine 116 then predicts a pixel block (e.g., a PU). In some examples, the prediction is added to the output (residual data) of the inverse transform.

[0081] The decoding device 112 may output the decoded video to a video destination device 122, which may include a display or other output device for displaying the decoded video data to a consumer of the content. In some aspects, the video destination device 122 may be part of a receiving device that includes the decoding device 112. In some aspects, the video destination device 122 may be part of a separate device other than the receiving device.

[0082] In some embodiments, the video encoding device 104 and / or the video decoding device 112 may be integrated with an audio encoding device and an audio decoding device, respectively. The video encoding device 104 and / or the video decoding device 112 may also include other hardware or software necessary to implement the encoding techniques described above, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combination thereof. The video encoding device 104 and the video decoding device 112 may be integrated as part of a combined encoder / decoder (codec) in the respective devices.

[0083] Figure 1 The example system shown is an illustrative example that can be used here. The technology for processing video data using the technology described herein can be performed by any digital video encoding and / or decoding device. Although the technology of the present disclosure is usually performed by a video encoding device or a video decoding device, the technology can also be performed by a combined video encoder-decoder (commonly referred to as a "codec (CODEC)"). In addition, the technology of the present disclosure can also be performed by a video preprocessor. The source device and the receiving device are only examples of such encoding devices that generate encoded video data for the source device to be sent to the receiving device. In some examples, the source device and the receiving device can operate in a substantially symmetrical manner so that each of the devices includes a video encoding and decoding component. Therefore, the example system can support one-way or two-way video transmission between video devices, such as for video streaming, video playback, video broadcasting, or video telephony.

[0084] Extensions to the HEVC standard include a multi-view video coding extension called MV-HEVC and a scalable video coding extension called SHVC. The MV-HEVC and SHVC extensions share the concept of layered coding, where different layers are included in the coded video bitstream. Each layer in the coded video sequence is addressed by a unique layer identifier (ID). The layer ID may be present in the header of the NAL unit to identify the layer to which the NAL unit is associated. In MV-HEVC, different layers can represent different views of the same scene in a video bitstream. In SHVC, different scalable layers are provided to represent video bitstreams at different spatial resolutions (or picture resolutions) or at different reconstruction fidelity. Scalable layers may include a base layer (with layer ID = 0) and one or more enhancement layers (with layer ID = 1, 2, ..., n). The base layer may conform to the profile of the first version of HEVC and represent the lowest available layer in the bitstream. Compared to the base layer, the enhancement layer has an increased spatial resolution, temporal resolution or frame rate and / or reconstruction fidelity (or quality). The enhancement layer is hierarchically organized and may (or may not) depend on the lower layer. In some examples, a single standard codec may be used to encode different layers (e.g., all layers may be encoded using HEVC, SHVC, or other coding standards). In some examples, multiple standard codecs may be used to encode different layers. For example, a base layer may be encoded using AVC, while one or more enhancement layers may be encoded using SHVC and / or MV-HEVC extensions of the HEVC standard.

[0085] In general, a layer comprises a set of VCL NAL units and a corresponding set of non-VCL NAL units. NAL units are assigned specific layer ID values. Layers can be hierarchical in the sense that layers can depend on lower layers. A layer set refers to a self-contained set of layers represented in a bitstream, meaning that layers within a layer set can depend on other layers in the layer set during decoding, but do not depend on any other layer for decoding. Therefore, the layers in a layer set can form an independent bitstream that can represent video content. The set of layers in a layer set can be obtained from another bitstream by the operation of a sub-bitstream extraction process. A layer set can correspond to a set of layers to be decoded when a decoder wants to operate according to certain parameters.

[0086] As described above, for each block, a set of motion information (also referred to herein as motion parameters) can be available. The motion information set contains motion information for forward and backward prediction directions. The forward and backward prediction directions are the two prediction directions of the bidirectional prediction mode, in which case the terms "forward" and "backward" do not necessarily have a geometric meaning. Instead, "forward" and "backward" correspond to reference picture list 0 (RefPicList0 or L0) and reference picture list 1 (RefPicList1 or L1) of the current picture. In some examples, when only one reference picture list is available for a picture or slice, only RefPicList0 is available and the motion information for each block of the slice is always forward.

[0087] In some cases, motion vectors and their reference indexes are used in the encoding process (e.g., motion compensation). Such motion vectors with associated reference indexes are represented as a unidirectional prediction set of motion information. For each prediction direction, the motion information can contain a reference index and a motion vector. In some cases, for simplicity, it can be assumed that the motion vector itself is referenced in a manner with an associated reference index. The reference index is used to identify the reference picture in the current reference picture list (RefPicList0 or RefPicList1). The motion vector has horizontal and vertical components, which provide an offset from the coordinate position in the current picture to the coordinate in the reference picture identified by the reference index. For example, the reference index can indicate a specific reference picture that should be used for a block in the current picture, and the motion vector can indicate where the best matching block (the block that best matches the current block) in the reference picture is in the reference picture.

[0088] Picture Order Count (POC) can be used in video coding standards to identify the display order of pictures. Although there are cases where two pictures within a coded video sequence can have the same POC value, it does not usually occur within a coded video sequence. When there are multiple coded video sequences in the bitstream, pictures with the same POC value can be closer to each other in decoding order. The POC value of a picture can be used for reference picture list construction, derivation of reference picture sets such as in HEVC, and motion vector scaling.

[0089] In H.264 / AVC, each inter-frame macroblock (MB) can be partitioned in four different ways, including: one 16×16MB partition; two 16×8MB partitions; two 8×16MB partitions; and four 8×8MB partitions. Different MB partitions in one MB can have different reference index values ​​(RefPicList0 or RefPicList1) for each direction. In some cases, when the MB is not partitioned into four 8×8MB partitions, it can have only one motion vector for each MB partition in each direction. In some cases, when the MB is partitioned into four 8×8MB partitions, each 8×8MB partition can be further partitioned into sub-blocks, in which case each sub-block can have different motion vectors in each direction. In some examples, there are four different ways to obtain sub-blocks from an 8×8MB partition, including: one 8×8 sub-block; two 8×4 sub-blocks; two 4×8 sub-blocks; and four 4×4 sub-blocks. Each sub-block can have a different motion vector in each direction. Therefore, the motion vector exists at a level equal to or higher than the sub-block.

[0090] In AVC, temporal direct mode can be enabled at MB level or MB partition level for skip and / or direct mode in B slices. For each MB partition, the motion vectors of the blocks co-located with the current MB partition in RefPicList1[0] of the current block are used to derive the motion vectors. Each motion vector in the co-located block is scaled based on the POC distance.

[0091] Spatial direct mode can also be implemented in AVC. For example, in AVC, direct mode can also predict motion information from spatial neighbors.

[0092] In HEVC, the largest coding unit in a slice is called a coding tree block (CTB). The CTB contains a quadtree whose nodes are coding units. In the HEVC main profile, the size of the CTB can range from 16×16 to 64×64. In some cases, 8×8 CTB size can be supported. The coding unit (CU) can be the same size as the CTB and as small as 8×8. In some cases, each coding unit is encoded in one mode. When the CU is inter-coded, the CU can be further partitioned into 2 or 4 prediction units (PUs), or can become only one PU when further partitioning is not applied. When two PUs are present in a CU, it can be a half-sized rectangle or two rectangles with 1 / 4 or 3 / 4 the size of the CU.

[0093] When the CU is inter-coded, there is a motion information set for each PU. In addition, each PU is encoded with a unique inter prediction mode to derive a motion information set.

[0094] For motion prediction in HEVC, there are two inter prediction modes, including merge mode and advanced motion vector prediction (AMVP) mode for prediction units (PUs). Skip is considered a special case of merge. In AMVP or merge mode, a motion vector (MV) candidate list is maintained for multiple motion vector predictors. The motion vector (s) of the current PU and the reference index in merge mode are generated by taking a candidate from the MV candidate list. In some examples, as described below, one or more stored local brightness compensation (LIC) flags can be included in the MV candidate list along with the stored motion vectors.

[0095] In an example where the MV candidate list is used for motion prediction (and, where applicable, brightness compensation) of a block, the MV candidate list can be constructed separately by an encoding device and a decoding device. For example, the MV candidate list can be generated by the encoding device when encoding the block, and can be generated by the decoding device when decoding the block. Information related to motion information candidates in the MV candidate list (e.g., information related to one or more motion vectors, information related to one or more LIC flags that can be stored in the MV candidate list in some cases, and / or other information) can be signaled between the encoding device and the decoding device. For example, in merge mode, an index value to the stored motion information candidate can be signaled from the encoding device to the decoding device (e.g., in a syntax structure such as a picture parameter set (PPS), a sequence parameter set (SPS), a video parameter set (VPS), a slice header, a supplemental enhancement information (SEI) message sent in a video bitstream or separately from a video bitstream, and / or other signaling). The decoding device can construct the MV candidate list and obtain one or more motion information candidates from the constructed MV candidate list using the signaled reference or index for motion compensated prediction. For example, the decoding device 112 may construct an MV candidate list and use a motion vector (and in some cases a LIC tag) from an index position for motion prediction of a block. In the case of AMVP mode, in addition to a reference or index, a difference or residual value may also be signaled as a delta. For example, for AMVP mode, the decoding device is able to construct one or more MV candidate lists and apply the delta value to one or more motion information candidates obtained using the signaled index value when performing motion compensated prediction of the block.

[0096] In some examples, the MV candidate list contains up to five candidates for merge mode and two candidates for AMVP mode. In other examples, different numbers of candidates can be included in the MV candidate list for merge mode and / or AMVP mode. The merge candidate can include a motion information set. For example, the motion information set can include motion vectors corresponding to both reference picture lists (list 0 and list 1) and reference indices. If the merge candidate is identified by a merge index, the reference picture is used for prediction of the current block, and the associated motion vector is determined. However, in AMVP mode, for each potential prediction direction from list 0 or list 1, it is necessary to explicitly signal the reference index together with the MVP index to the MV candidate list, because the AMVP candidate only contains motion vectors. In AMVP mode, the predicted motion vector can be further refined.

[0097] As can be seen from above, the merge candidate corresponds to a complete set of motion information, while the AMVP candidate contains only one motion vector for a specific prediction direction and reference index.Candidates for both modes are similarly derived from the same spatial and temporal neighboring blocks.

[0098] In some examples, merge mode allows an inter-predicted PU to inherit the same motion vector(s), prediction direction, and reference picture index(s) from the following inter-predicted PU, wherein the following inter-predicted PU includes motion data positions selected from a set of spatially adjacent motion data positions and one of two temporally co-located motion data positions. For AMVP mode, one or more motion vectors of a PU can be predictively encoded relative to one or more motion vector predictors (MVPs) from an AMVP candidate list constructed by an encoder and / or decoder. In some cases, for unidirectional inter prediction of a PU, the encoder and / or decoder can generate a single AMVP candidate list. In some cases, for bidirectional prediction of a PU, the encoder and / or decoder can generate two AMVP candidate lists, one using motion data from spatially and temporally adjacent PUs in a forward prediction direction, and one using motion data from spatially and temporally adjacent PUs in a backward prediction direction.

[0099] Figure 2 2 is a block diagram showing a codec system 200 having an encoder 290 that filters out film grain noise and a decoder 295 that reconstructs film grain noise. The encoder 290 may be an example of the encoding device 104, the encoding engine 106, the computing system 1100, the processor 1110, or a combination thereof. The decoder 295 may be an example of the decoding device 112, the decoder engine 116, the computing system 1100, the processor 1110, or a combination thereof.

[0100] The encoder 290 receives an input video 205 having at least one video frame, for example, from a video source 102. The input video 205 includes film grain or other noise data. In some cases, film grain can provide a desired aesthetic effect in the video, and it can be expected that the film grain will be retained through the encoding and decoding process. However, the randomness and rapid changes of film grain and other noise data will increase the complexity of the video and thus reduce the efficiency of encoding and / or compression. For example, video compression is based on spatial and temporal prediction, and film grain and other noise data have a high degree of randomness in both spatial and temporal directions, making prediction difficult. Therefore, in order to reduce the complexity of the input video 205 and improve the efficiency of encoding and / or compression of the input video 205, the encoder 290 uses a denoiser 210 to denoise the input video 205 to generate a denoised video 215. The denoiser 210 can use any denoising algorithm or technique, such as spatial filtering, transform domain filtering, wavelet threshold processing or a combination thereof. The encoder 290 then passes the denoised video 215 to the encoding engine 220 to generate the encoded video 225. The encoding engine 220 can be an example of the encoding device 104, the encoding engine 106, the computing system 1100, the processor 1110, or a combination thereof. In some examples, the encoding engine 220 compresses the denoised video 215 to generate the encoded video 225.

[0101] The encoder 290 also includes a subtractor 250 that subtracts the denoised video 215 from the input video 205 to generate a residual 255. In some examples, the encoder 290 includes an analysis engine 240, wherein the analysis engine 240 receives the input video 205, the denoised video 215, data about the denoising process from the denoiser 210, and / or the residual 255. Based on the data, the analysis engine 240 performs an analysis of the structure and / or intensity of the noise removed from the input video 205 by the denoiser 210 to generate the denoised video 215. In some examples, the analysis engine 240 generates a map 245, wherein the map 245 maps the structure and / or intensity of the noise removed from the input video 205 by the denoiser 210 at various locations to corresponding locations in a video frame of the input video 205, the denoised video 215, and / or the encoded video 225. The encoder 290 includes a grain estimation engine 260, wherein the grain estimation engine 260 generates grain parameters 265 based on the mapping 245 from the analysis engine 240 and / or based on the residual 255. In some examples, the mapping 245 and / or the grain parameters 265 may include an autoregressive model of the grain removed from the input video 205 by the denoiser 210. In some examples, the mapping 245 and / or the grain parameters 265 may model the grain intensity as a function of the intensity. In some examples, the grain parameters 265 include an initial random state, and an indication of a bit depth, an indication of a grain scale, or a combination thereof.

[0102] The encoder 290 passes the encoded video 225 and the granular parameters 265 to the decoder 295. The decoder 295 includes a decoding engine 230, wherein the decoding engine 230 decodes the encoded video 225 to generate a decoded video 235. The decoding engine 230 can be an example of a decoding device 112, a decoder engine 116, a computing system 1100, a processor 1110, or a combination thereof. In some examples, the decoding engine 230 decompresses the encoded video 225 to generate a decoded video 235. The decoded video 235 is a reconstruction of the denoised video 215.

[0103] The decoder 295 also includes a granule generator 270 that generates granule pixel data 275 based on the granule parameters 265. In some examples, the granule pixel data 275 is a reconstruction of the noise removed from the input video 205 by the denoiser 210. In some examples, the granule pixel data 275 is a reconstruction of the residual 255. In some examples, the granule pixel data 275 is synthesized granule data that shares some properties (e.g., indicated by the granule parameters 265) with the noise removed from the input video 205 by the denoiser 210. The decoder 295 includes an adder 280 that adds the granule pixel data 275 to the decoded video 235 to generate an output video 285.

[0104] In some examples, the granule generator 270 generates an entire frame of granule pixel data 275 before the decoder 295 adds the granule pixel data 275 to the decoded video 235 using the adder 280 to generate the output video 285. An example of such a decoder 295 is shown as Figure 6 Part of the decoder system 600.

[0105] In some examples, the granule generator 270 dynamically generates a portion of a frame of granule pixel data 275 at a time, and the decoder 295 dynamically adds the granule pixel data 275 to the decoded video 235 at a time using the adder 280. Thus, the decoder 295 is able to dynamically generate the output video 285 a portion at a time, and eventually generate the entire frame of the output video 285 after the granule generator 270 eventually generates the entire frame of the granule pixel data 275. An example of such a decoder 295 is shown as Figure 7 Such a decoder 295 can provide efficiency improvements over a decoder that generates an entire frame of granule pixel data 275 before adding the granule pixel data 275 to the decoded video 235 to generate the output video 285 by allowing parallel granule generation and granule addition. For example, the adder 280 can add a first portion of the granule pixel data 275 to the decoded video 235 while the granule generator 270 is generating a second portion of the granule pixel data 275, such as Fig. 9B The conceptual diagram 950 is shown.

[0106] Figure 3 3 is a conceptual diagram 300 illustrating the generation of noise pixel data corresponding to various blocks of a video frame 350 based on corresponding offsets. The particle generator 270 (e.g., using the white noise generator 610 and / or the autoregressive filter 615) generates a particle array 305 based on the particle parameters 265. Figure 3 In the illustrative example of , particle array 305 has dimensions of 82 pixels (px) by 73 px. In some examples, particle array 305 can be larger or smaller than this size in either dimension.

[0107] The video frame 350 is divided into a plurality of blocks. Figure 3 In the illustrative example of , video frame 350 is divided into at least four blocks, including block 315, block 325, block 335, and block 345. In some examples, video frame 350 can be divided into more than four blocks or less than four blocks. Figure 3 In the illustrative example of , each of the blocks has dimensions of 32px×32px. In some examples, the blocks (eg, blocks 315, 325, 335, and 345) may be larger or smaller than this size in any dimension.

[0108] The granule generator 270 generates random offsets (e.g., using the random offset generator 635 and / or the random offset generator 710) that correspond to each block (e.g., block 315, block 325, block 335, and block 345) of the video frame 350, respectively. Based on the random offsets, the granule generator 270 retrieves noise pixel data 310 corresponding to block 315, noise pixel data 320 corresponding to block 325, noise pixel data 330 corresponding to block 335, and noise pixel data 340 corresponding to block 345 from the granule array 305. The set of random offsets for a given block may include horizontal and vertical offsets from an origin along the granule array 305. The origin may be a point of the granule array 305 with coordinates (0,0) at one of the corners of the granule array 305 (e.g., the upper left corner, the lower left corner, the upper right corner, or the lower right corner). The set of random offsets for a given block indicates the location along the granule array 305 where the granule generator 270 is to retrieve noise pixel data corresponding to the block.

[0109] For example, the granule generator 270 generates (e.g., using the random offset generator 635 and / or the random offset generator 710) a first set of random offsets corresponding to the block 315, wherein the first set of random offsets indicates the following positions along the granule array 305 from which the noise pixel data 310 corresponding to the block 315 is to be retrieved. The granule generator 270 generates a second set of random offsets corresponding to the block 325, wherein the second set of random offsets indicates the following positions along the granule array 305 from which the noise pixel data 320 corresponding to the block 325 is to be retrieved. The granule generator 270 generates a third set of random offsets corresponding to the block 335, wherein the third set of random offsets indicates the following positions along the granule array 305 from which the noise pixel data 330 corresponding to the block 335 is to be retrieved. The granule generator 270 generates a fourth set of random offsets corresponding to the block 345, wherein the fourth set of random offsets indicates the following positions along the granule array 305 from which the noise pixel data 340 corresponding to the block 345 is to be retrieved. exist Figure 3 In the illustrative example of , each set of noise pixel data (eg, noise pixel data 310, noise pixel data 320, noise pixel data 330, and noise pixel data 340) has a dimension of 34px×34px. Figure 4 As shown, an additional 2 pixels in each dimension (beyond the 32px×32px dimensions of the block) can be used for blending. In some examples, blocks (e.g., blocks 315, 325, 335, and 345) can be larger or smaller than this size in either dimension.

[0110] The granule generator 270 can arrange each set of noise pixel data (e.g., noise pixel data 310, noise pixel data 320, noise pixel data 330, and noise pixel data 340) together according to the same arrangement that the corresponding blocks (e.g., block 315, block 325, block 335, and block 345) in the video frame 350 have to generate the frame noise data 355. The frame noise data 355 can represent the granule pixel data 275, and the decoder 295 can add the granule pixel data 275 to the decoded video 235 using the adder 280 to generate the output video 285. In some examples, the noise pixel data (e.g., noise pixel data 310, noise pixel data 320, noise pixel data 330, and noise pixel data 340) can represent the granule pixel data 275, wherein the decoder 295 can add the granule pixel data 275 to the decoded video 235 using the adder 280 to generate the output video 285.

[0111] In some examples, the particle array 305 is a luma particle array 305 having particles on a luma channel, and the noise pixel data (e.g., noise pixel data 310, noise pixel data 320, noise pixel data 330, and noise pixel data 340) includes noise on the luma channel. In some examples, the particle array 305 is a chroma particle array 305, wherein the particles on one or more chroma channels correspond to one or more colors, and the noise pixel data (e.g., noise pixel data 310, noise pixel data 320, noise pixel data 330, and noise pixel data 340) includes noise on one or more chroma channels.

[0112] Figure 4 4 is a conceptual diagram 400 illustrating the use of horizontal blending 450 and vertical blending 455 at overlapping portions of noisy pixel data corresponding to different blocks of a video frame. For example, Figure 4 , each having a dimension of 34px×34px. The four sets of noisy pixel data include noisy pixel data 410 (labeled “A”) corresponding to a block 415 of a video frame (e.g., output 110, input 114, input video 205, denoised video 215, encoded video 225, decoded video 235, and / or video frame 350), noisy pixel data 420 (labeled “B”) corresponding to a block 425 of the video frame, noisy pixel data 430 (labeled “C”) corresponding to a block 435 of the video frame, and noisy pixel data 440 (labeled “D”) corresponding to a block 445 of the video frame. Figure 4 The four sets of noise pixel data (eg, noise pixel data 410, noise pixel data 420, noise pixel data 430, and noise pixel data 440) in may be Figure 3 An example of four sets of noise pixel data (e.g., noise pixel data 310, noise pixel data 320, noise pixel data 330, and noise pixel data 340) in , and vice versa. Similarly, Figure 4 The four blocks (e.g., block 415, block 425, block 435, and block 445) in may be Figure 3 An example of four blocks (e.g., block 315, block 325, block 335, and block 345) in , and vice versa.

[0113] Figure 4The four groups of noise pixel data (e.g., noise pixel data 410, noise pixel data 420, noise pixel data 430, and noise pixel data 440) in are arranged so that two adjacent pixel rows (e.g., pixel rows or pixel columns) at a given edge of a first group of noise pixel data overlap on two adjacent pixel rows (e.g., pixel rows or pixel columns) at a corresponding edge of a second group of noise pixel data arranged adjacent to the first group of noise pixel data. For example, the two rightmost columns of noise pixel data 410 overlap with the two leftmost columns of noise pixel data 420, the two rightmost columns of noise pixel data 430 overlap with the two leftmost columns of noise pixel data 440, the two bottommost columns of noise pixel data 410 overlap with the two topmost columns of noise pixel data 430, and the two bottommost columns of noise pixel data 420 overlap with the two topmost columns of noise pixel data 440.

[0114] The granule generator 270 can perform horizontal blending 450 (e.g., horizontal blending 640, horizontal and / or vertical blending performed by the horizontal / vertical blending engine 715) to blend overlapping pixel columns of the noise pixel data from different adjacent groups. For example, the granule generator 270 can perform horizontal blending 450 to blend the rightmost two columns of the noise pixel data 410 with the leftmost two columns of the noise pixel data 420, and blend the rightmost two columns of the noise pixel data 430 with the leftmost two columns of the noise pixel data 440.

[0115] Granular generator 270 can perform vertical blending 455 (e.g., vertical blending 645, horizontal and / or vertical blending performed by horizontal / vertical blending engine 715) to blend overlapping pixel rows of noisy pixel data from different adjacent groups. For example, granular generator 270 can perform vertical blending 455 to blend the two bottom-most columns of noisy pixel data 410 with the two top-most columns of noisy pixel data 430, and blend the two bottom-most columns of noisy pixel data 420 with the two top-most columns of noisy pixel data 430.

[0116] In some examples, particle generator 270 can perform horizontal mixing 450 before performing vertical mixing 455 (e.g., as Fig.9A In some examples, particle generator 270 can perform horizontal mixing 450 before performing vertical mixing 455 at least partially in parallel (e.g., as shown in FIG. Fig. 9B). In some examples, to perform horizontal mixing 450 and / or vertical mixing 455, particle generator 270 can add noise pixel data from overlapping lines of two sets of noise pixel data to each other. In some examples, to perform horizontal mixing 450 and / or vertical mixing 455, particle generator 270 can subtract noise pixel data from overlapping lines of two sets of noise pixel data from each other. In some examples, to perform horizontal mixing 450 and / or vertical mixing 455, particle generator 270 can multiply noise pixel data from overlapping lines of two sets of noise pixel data to each other. In some examples, to perform horizontal mixing 450 and / or vertical mixing 455, particle generator 270 can divide noise pixel data from overlapping lines of two sets of noise pixel data from each other. In some examples, to perform horizontal mixing 450 and / or vertical mixing 455, particle generator 270 can average noise pixel data from overlapping lines of two sets of noise pixel data to each other. In some examples, to perform horizontal mixing 450 and / or vertical mixing 455, particle generator 270 can maintain the mixing values ​​within a predetermined range, for example by reducing any mixing values ​​that exceed an upper threshold to an upper threshold and by raising any mixing values ​​that are less than a lower threshold to a lower threshold.

[0117] Figure 5 5 is a conceptual diagram 500 showing the addition of a noise image 550 to a decoded video frame 505 using an adder 280 to generate an output frame 560. The decoded video frame 505 is divided into at least four blocks having a size of 32px×32px, including a block 515, a block 525, a block 535, and a block 545. The noise image 550 of the decoded video frame 505 includes four groups of noise pixel data corresponding to the four blocks of the decoded video frame 505, including noise pixel data 510 corresponding to the block 515, noise pixel data 520 corresponding to the block 525, noise pixel data 530 corresponding to the block 535, and noise pixel data 540 corresponding to the block 545. The four groups of noise pixel data (e.g., noise pixel data 510, noise pixel data 520, noise pixel data 530, and noise pixel data 540) each have a dimension of 34px×34px and are arranged in the same order and / or arrangement as the four blocks (e.g., block 515, block 525, block 535, and block 545), with some overlap between adjacent groups of noise pixel data. The noise image 550 can include horizontal blending 450 and / or vertical blending 455 at the overlapping areas between adjacent groups of noise pixel data. In some examples, the noise image 550. In some examples, the particle generator 270 can generate the noise image 550 using piecewise linear interpolation using corresponding pixels of the decoded video frame 505.

[0118] The decoder 295 adds the noise image 550 to the decoded video frame 505 using the adder 280 to generate an output frame 560. The output frame 560 includes a first 32px×32px block region that includes noise pixel data 510 of the noise image 550 added to the block 515 of the decoded video frame 505 (e.g., with horizontal blending 450 and / or vertical blending 455 with adjacent (multiple) sets of noise pixel data). The output frame 560 also includes a second block area, a third block area, and a fourth block area, the second block area including noise pixel data 520 of a noise image 550 added to a block 525 of the decoded video frame 505 (e.g., having a horizontal blend 450 and / or a vertical blend 455 with adjacent (multiple) groups of noise pixel data), the third block area including noise pixel data 530 of a noise image 550 added to a block 535 of the decoded video frame 505 (e.g., having a horizontal blend 450 and / or a vertical blend 455 with adjacent (multiple) groups of noise pixel data), and the fourth block area including noise pixel data 540 of a noise image 550 added to a block 545 of the decoded video frame 505 (e.g., having a horizontal blend 450 and / or a vertical blend 455 with adjacent (multiple) groups of noise pixel data). In some examples, the decoder 295 is capable of scaling and / or rounding the noise image 550 before adding it to the decoded video frame 505. In some examples, decoder 295 can perform piecewise linear interpolation on noise image 550 using corresponding pixels of decoded video frame 505 before adding noise image 550 to decoded video frame 505 .

[0119] Figure 6 6 is a block diagram illustrating a decoder system 600 that generates noise data for an entire video frame before adding the noise data to the video frame. The decoder system 600 receives film grain (FG) parameters 605, which are examples of grain parameters 265. The decoder system 600 also receives a frame width and height 625 (e.g., frame dimensions) of the encoded video 225 and / or the decoded video 235. The decoded video 235 includes reconstructed pixels 655, which are also received by the decoder system 600.

[0120] The decoder system uses a white noise generator 610 and an autoregressive filter 615 to generate a color filter based on the FG parameters 605 in terms of luma (Y) and / or chroma (e.g., blue difference (C B ) and / or red difference (C R )) channel. In the illustrative example, the white noise generator 610 generates white noise of the particle array 620 based on the particle parameters 265 according to the following pseudo code 1:

[0121] Pseudocode 1

[0122]

[0123] In the pseudo code 1 of the white noise generator 610 provided above, LumaGrain represents a luma grain array. BitDepth and grain_scale_shift can represent the FG parameters 605.

[0124] In an illustrative example, the autoregressive filter 615 generates the granular array 620 from white noise (generated using the white noise generator 610) based on the FG parameters 605 according to the following pseudo code 2:

[0125] Pseudocode 2

[0126]

[0127]

[0128] In the pseudo code 2 of the autoregressive filter 615 provided above, ar_coeff_lag represents one of the FG parameters 605. In the example shown using pseudo code, given ar_coeff_lag equal to 3, each filtered particle value generated using the autoregressive filter 615 can depend on the maximum of the 24 white noise values ​​and / or the filtered particle values. The output of the autoregressive filter 615 is a particle array 620.

[0129] A noise image generator 630 of the decoder system 600 generates a noise image 675 based on the particle array 620, the frame width and height 625, and in some cases, the FG parameters 605. The noise image generator 630 includes a random offset generator 635 that generates random offsets for different groups of noise pixel data (e.g., as in the random offsets for noise pixel data 310, noise pixel data 320, noise pixel data 330, and noise pixel data 340) to identify which portions of the particle array 620 the noise pixel data should be retrieved from. The noise image generator 630 includes respective engines for horizontal blending 640 (e.g., for horizontal blending 450) and vertical blending 645 (e.g., for vertical blending 455). The horizontal blending 640 and the vertical blending 645 are capable of blending noise pixel data from overlapping lines of adjacent groups of noise pixel data, such as Figure 4 As shown. The noise image generator 630 writes the resulting mixed noise pixel data into a memory 650, which may include a double data rate (DDR) memory. The noise image generator 630 is capable of writing the mixed noise pixel data from the noise image and / or from the mixed noise slice into 32px×34px, 32px×2px, and / or 32px×32px portions at a time.

[0130] The decoder system 600 reads the noisy image 675 from the memory 650. The decoder system 600 uses a scaler and a scaling lookup table (LUT) 660 to scale the noisy image 675 based on the reconstructed pixels 655 and / or the frame width and height 625. The decoder system 600 uses a shifter / rounder / adder / slicer engine 665 to shift, round, add (e.g., as in the adder 280), and / or clip the noisy image 675 to the reconstructed pixels 655 to generate film grain pixels 670. The film grain pixels 670 are examples of pixels of the output video 285.

[0131] In some examples, the decoder system 600 performs sequential blending of film grain blocks (e.g., noise pixel data 310, noise pixel data 320, noise pixel data 330, noise pixel data 340, noise pixel data 410, noise pixel data 420, noise pixel data 430, noise pixel data 440, noise pixel data 510, noise pixel data 520, noise pixel data 530, noise pixel data 540) horizontally and vertically to generate a noise image 675 for the entire frame. The decoder system 600 writes the full frame width of the horizontally blended noise slices to the memory 650. The decoder system 600 performs 2 additional pixel writes after the horizontal blending, where the pixel writes for brightness are 0. 2 top row pixels are read and 2 pixels are written after vertical mixing of every 32 pixels in height. Similarly, the decoder system 600 reads 2 top row pixels and writes 2 pixels after vertical mixing of every 32 pixels in height. 1 additional pixel write, 1 top pixel read, and 1 pixel write are performed after every 16 pixels of vertical blending in height.

[0132] In some examples, the memory bandwidth of the memory 650 used by the decoder system 600 to generate the film grain pixels 670 can be determined using the following equation:

[0133]

[0134] In the above equation 1, W represents the width of the frame, H represents the height of the frame, B represents the bit depth, and F represents the frame rate in frames per second (fps). According to the above equation 1, the read / write memory bandwidth of the memory 650 of the decoder system 600 is 6644 Mbps for 4K resolution with a frame rate of 240 fps and a bit depth of 8 bits.

[0135] Figure 76 is a block diagram illustrating a decoder system 700 that generates noise data on a pixel-by-pixel basis and adds the noise data to a video frame on a pixel-by-pixel basis. The decoder system 700 uses FG parameters 605, a white noise generator 610, and an autoregressive filter 615 as described above with respect to the decoder system 600 to generate a grain array 620. The decoder system 700 includes a grain memory 705 that stores the grain array 620 in its entirety. The storage of the grain array 620 is the only write operation (or set of write operations) used by the decoder system 700 to generate film grain pixels 725. This represents a significant reduction in the number of write operations per frame compared to the various write operations to the memory 650 by the decoder system 600.

[0136] The decoder system 700 includes a random offset generator 710 and a horizontal / vertical blending engine 715 (e.g., for horizontal blending 640 and vertical blending 645), which the decoder system 700 uses to generate a noise image 720 using a granule array 620 stored in a granule memory 705. The decoder system 700 receives a frame width and height 625, sends a granule read request 730 to the granule memory 705 based on the frame width and height 625, and receives granule read data 735 from the granule memory 705 in response. In some examples, the decoder system 700 retrieves granule read data 735 from the granule memory 705 on a pixel-by-pixel basis as needed based on where a given noise pixel will be in the noise image 720. For example, if a given noise pixel is in a noise slice that corresponds to an overlap between multiple sets of pixel noise data (e.g., as shown in FIG. 1 ), the decoder system 700 may retrieve granule read data 735 from the granule memory 705 on a pixel-by-pixel basis as needed. Figure 4 7 (a noise slice of the horizontal blend 450 and / or the vertical blend 455 in the decoder system 700), the granule read request 730 can request granule read data 735 based on a corresponding random offset to each set of overlapping pixel noise data, wherein the decoder system 700 blends the granule read data 735 on a per-pixel basis using the horizontal / vertical blending engine 715. If a given noise pixel is not in a noise band corresponding to an overlap between multiple sets of pixel noise data, the granule read request 730 can request granule read data 735 based on a corresponding random offset to only one set of pixel noise data, which the decoder system 700 can use as part of the noise image 720. Figure 8 Pixel-by-pixel retrieval of pixel noise / grain data from the grain array 620 stored in the grain memory 705 is further shown in FIG.

[0137] The decoder system 700 uses a scaler and scaling lookup table (LUT) 660 to scale the noisy image 720 (or a portion thereof) based on the reconstructed pixels 655 and / or the frame width and height 625. The decoder system 700 uses a shifter / rounder / adder / slicer engine 665 to shift, round, add (e.g., as in adder 280), and / or clip the noisy image 720 (or a portion thereof) to the reconstructed pixels 655 to generate film grain pixels 725. Film grain pixels 725 are examples of pixels of the output video 285.

[0138] Figure 8 8 is a block diagram 800 illustrating the generation of noise pixel data based on the position of the noise pixel data in a particle image 850. The decoder system 700 can dynamically retrieve the noise pixel data from the particle array 620 stored in the particle memory 705 on the fly using random offsets and based on the position that the noise pixel data is set to occupy in the particle image 850. The decoder system 700 can dynamically process the noise pixel data on the fly using horizontal blending and / or vertical blending on a pixel-by-pixel basis, again based on the position that the noise pixel data is set to occupy in the particle image 850. In some examples, the decoder system 700 can retrieve and / or process the noise pixel data from one block of noise pixel data at a time. The particle image 850 includes four blocks of noise pixel data, labeled as block A, block B, block C, and block D. The noise pixel data block is an example of the multiple groups of noise pixel data described above (for example, noise pixel data 310, noise pixel data 320, noise pixel data 330, noise pixel data 340, noise pixel data 410, noise pixel data 420, noise pixel data 430, noise pixel data 440, noise pixel data 510, noise pixel data 520, noise pixel data 530, noise pixel data 540).

[0139] For example, the decoder system 700 can retrieve and process the noisy pixel 805 during the processing of block B by reading the noisy pixel data from block A at coordinates (32,0) and block B at coordinates (0,0), and horizontally blending the noisy pixels. The decoder system 700 can retrieve and process the noisy pixel 810 shown during the processing of block A by reading the noisy pixel data from block A at coordinates (10,0) without blending. The decoder system 700 can retrieve and process the noisy pixel 815 shown during the processing of block B by reading the noisy pixel data from block B at coordinates (18,6) without blending.

[0140] The decoder system 700 can retrieve and process the illustrated noisy pixel 820 during the processing of block D by reading the noisy pixel data from block A at coordinates (32, 32) and reading the noisy pixel data from block B at coordinates (0, 32), and horizontally mixing the noisy pixels to generate horizontally mixed intermediate pixel data Horz (A, B). For the illustrated noisy pixel 820, the decoder can also read the noisy pixel data from block C at coordinates (32, 0) and block D at coordinates (0, 0), and horizontally mix the noisy pixels to generate horizontally mixed intermediate pixel data Horz (C, D). For the illustrated noisy pixel 820, the decoder can then vertically mix the intermediate pixel data Horz (A, B) with the intermediate pixel data Horz (C, D).

[0141] The decoder system 700 can retrieve and process the illustrated noisy pixel 825 during processing of block C by reading the noisy pixel data from block A at coordinates (28, 33) and block C at coordinates (28, 1), and blending the noisy pixels vertically. The decoder system 700 can retrieve and process the illustrated noisy pixel 830 during processing of block D by reading the noisy pixel data from block D at coordinates (2, 12), without blending. The decoder system 700 can retrieve and process the illustrated noisy pixel 835 during processing of block B by reading the noisy pixel data from block A at coordinates (32, 0), reading the noisy pixel data from block B at coordinates (0, 0), and blending the noisy pixels horizontally. The decoder system 700 can retrieve and process the illustrated noisy pixel 840 during processing of block D by reading the noisy pixel data from block C at coordinates (32, 24), reading the noisy pixel data from block D at coordinates (0, 24), and blending the noisy pixels horizontally.

[0142] The decoder system 700 dynamically retrieves the noisy pixel data corresponding to a given reconstructed pixel, generates the noise data from the noisy pixel data (e.g., by rounding, shifting, clipping, and / or blending the noisy pixel data), and applies the noise data to the reconstructed pixel. In an illustrative example, the decoder system 700 performs these operations according to the following pseudo code 3:

[0143] Pseudocode 3

[0144]

[0145]

[0146]

[0147]

[0148] In pseudo code 3 above, the decoder system 700 checks one row of blocks within the frame at a time based on the value of lumaNum. Figure 8 In the context of , lumaNum==0 refers to a row of blocks including blocks A and B, while lumaNum==1 refers to a row of blocks including blocks C and D.

[0149] The decoder system 700 loops (e.g., using two for loops) along different pixel locations (indicated by coordinates x and y) in the current block within the frame. The frame has a height h and a width w. Each block may have a height (h+1) / 2 and a width (w+1) / 2. In some examples, the block may be 32 pixels by 32 pixels in luma space and / or may be 16 pixels by 16 pixels in chroma space.

[0150] For a given pixel, the decoder system 700 retrieves the value of the register RandomRegister_0 based on the parameter grain_seed. In some aspects, the parameter grain_seed is an example of the grain parameter 265 and / or the film grain parameter 605. The decoder system 700 determines the random value rand using the function get_random_number_0() further shown below in pseudo code 4. The decoder system 700 can access a luma grain array (LumaGrain) and one or more chroma grain arrays (CbGrain, CrGrain).

[0151] Within a given row of blocks, the decoder system 700 may reference the current block (indicated by a variable with the term "curr"), the left block to the left of the current block (indicated by a variable with the term "left"), the top block above the current block (indicated by a variable with the term "top"), and / or the top left block above and to the left of the current block (indicated by a variable with the term "top_left"). For example, if the current block is Figure 8 , the left block is block C, the top block is block B, and the upper left block is block A. Some of these may not apply depending on which block the decoder system 700 refers to as the current block. For example, if the current block is block B, the left block is block A, and there is no top block or upper left block.

[0152] The decoder system 700 generates offsets in the x and y dimensions using random value rand, which indicates the position of a particular noise pixel in the particle array relative to the position of the noise pixel data of a particular group or block (e.g., noise pixel data 310, noise pixel data 320, noise pixel data 330, noise pixel data 340, noise pixel data 410, noise pixel data 420, noise pixel data 430, noise pixel data 440, noise pixel data 510, noise pixel data 520, noise pixel data 530, or noise pixel data 540) of the noise pixel data of the particular group or block within the particle array. These offsets are determined per block. For example, the offsets in the x and y dimensions indicating the position of a particular noise pixel of the current block are indicated as offsetX_curr and offset_curr, respectively. The decoder system 700 can set the previously determined rand value (determined by the decoder when other blocks are set as the current block) to rand_left, rand_top, and / or rand_top_left, respectively. The decoder system 700 can use these random values ​​to generate offsets in the x and y dimensions of the left block (offset_left and offsetY_left), offsets in the x and y dimensions of the top block (offset_top and offsetY_top), and / or offsets in the x and y dimensions of the top left block (offset_top_left and offset_top_left). Figure 8 The decoder system 700 can determine offset_top, offset_top, offset_top_left, and / or offset_top_left if the block in the context of is C or D. The decoder system 700 can determine these based on the value in register RandomRegister_1 set according to grain_seed and function get_random_number_1(), which is further shown in pseudo code 4 below.

[0153] The decoder system 700 also determines plane offsets that indicate the location of noise pixel data (e.g., noise pixel data 310, noise pixel data 320, noise pixel data 330, noise pixel data 340, noise pixel data 410, noise pixel data 420, noise pixel data 430, noise pixel data 440, noise pixel data 510, noise pixel data 520, noise pixel data 530, or noise pixel data 540) for each corresponding group or block within the particle array. These offsets are determined per block. For example, the offsets in the x and y dimensions that indicate the location of the noise pixel data for the current group or block of the current block are indicated as planeOffsetX_curr and planeOffsetY_curr, respectively. The decoder system 700 may similarly determine the plane offsets in the x and y dimensions (planeOffsetX_left and planeOffsetY_left) of the left group or block of noisy pixel data, the plane offsets in the x and y dimensions (planeOffsetX_top and planeOffsetY_top) of the top group or block of noisy pixel data, and / or the plane offsets in the x and y dimensions (planeOffsetX_top_left and planeOffsetY_top_left) of the top left group or block of noisy pixel data, respectively. Each of these plane offsets may be Figure 2 Examples of offsets for noisy pixel data blocks, offsets generated using the random offset generator 635, and / or offsets generated using the random offset generator 710. Thus, in some examples, the decoder system 700 can, for example, use planeOffsetX_curr+offsetX_curr and planeOffsetY_curr+offsetY_curr, planeOffsetX_left+offsetX_left and planeOffsetY_left+offsetY_left, planeOffsetX_top+offsetX_top and planeOffsetY_top+offsetY_top, and / or planeOffsetX_top_left+offsetX_top_left and planeOffsetY_top_left+offsetY_top_left.

[0154] The decoder system 700 cycles through different values ​​of i and j corresponding to the x and y coordinates of different pixels to determine the particle values ​​of those pixels based on the noise pixel data of at least one group or block. For example, the decoder system 700 generates curr_g as the particle value of the pixel indicated by i, j for the noise pixel data of the current block or group based on the various plane offsets and / or offsets discussed above. Depending on which block is the current block, the decoder system 700 can also use the various plane offsets and / or offsets discussed above to generate left_g as the particle value of the pixel indicated by i, j for the noise pixel data of the left block or group, top_g as the particle value of the pixel indicated by i, j for the noise pixel data of the top block or group, and / or top_left_g as the particle value of the pixel indicated by i, j for the noise pixel data of the top left block or group.

[0155] The decoder system 700 then determines whether the current noisy pixel whose value the decoder system 700 is determining falls within an overlapping region where two or more blocks or groups of noisy pixel data overlap, such as in Figure 4 and Figure 8 Horizontal blending 450 is performed between two columns of pixels, or in Figure 4 and Figure 8 If yes, the decoder system 700 ensures that the noise pixel data for the pixel is obtained from each of the applicable blocks or groups of noise pixel data, mixes them together, shifts and / or clips and / or rounds the mixed values, and outputs the result as part of a noise image (noiseImage). The noise image can be the frame noise data 355, Figure 4 , the noise pixel data, the noise image 550, the film grain pixel 670, the film grain pixel 725, the grain image 850, or an example of a combination thereof shown in . For example, for the horizontal blend 450 (e.g., the noise pixel 825), the decoder system 700 blends curr_g with top_g. For the vertical blend 455 (e.g., the noise pixels 805, 840), the decoder system 700 blends curr_g with left_g. For the case with both the horizontal blend 450 and the vertical blend 455 (e.g., the noise pixel 820), the decoder system 700 blends curr_g, left_g, top_g, and top_left_g. For pixels that are not in the overlapping area (e.g., the noise pixels 810, 815, 830, 835), no blending is required, and thus the decoder is able to use curr_g without blending. The decoder system 700 continues to traverse its various loops to gradually continue to determine more and more values ​​of different pixels of the noiseImage over time. In some examples, the decoder system 700 eventually generates the entire noiseImage in this manner.

[0156] In an illustrative example, the functions get_random_number_0() and get_random_number_1() called in pseudo-code 3 above to generate random values ​​rand and rand_top, respectively, can operate according to pseudo-code 4 below:

[0157] Pseudocode 4

[0158]

[0159] It should be understood that the pseudo-code examples provided above in Pseudo-code 1, Pseudo-code 2, Pseudo-code 3, and / or Pseudo-code 4 are illustrative, non-limiting examples of techniques for generating particle array data, dynamically retrieving noise pixel data corresponding to a given reconstructed pixel, and / or generating noise data from noise pixel data (e.g., by rounding, shifting, clipping, and / or mixing noise pixel data). In some examples, the pseudo-codes may be modified, rearranged, added, or removed. In some examples, for example, one or more of the pseudo-code examples (e.g., Pseudo-code 1, Pseudo-code 2, Pseudo-code 3, and / or Pseudo-code 4) can be configured and / or arranged in a decoder system (and / or modified) so that one or more of the operations that the pseudo-code provides for execution (e.g., generating particle array data, dynamically retrieving noise pixel data corresponding to a given reconstructed pixel, and / or generating noise data from noise pixel data) can be executed in parallel with one or more other operations that the pseudo-code provides for execution, such as Fig. 9B shown.

[0160] Fig.9A 900 is a conceptual diagram showing a timeline 920 for horizontal mixing 905, vertical mixing 910, and noise addition 915 using a decoder system 600, wherein the decoder system 600 generates noise data for an entire video frame before adding the noise data to the decoded video frame. Because the decoder system 600 performs horizontal mixing 905 and vertical mixing 910 for the entire video frame to generate the entire noise image before adding the noise image to the decoded video frame, the horizontal mixing 905, vertical mixing 910, and noise addition 915 are performed sequentially rather than in parallel.

[0161] Fig. 9B9 is a conceptual diagram 950 showing a timeline 960 for horizontal mixing, vertical mixing, and noise addition using a decoder system 700, wherein the decoder system 700 generates noise data on a pixel-by-pixel basis and adds the noise data to the decoded video frame on a pixel-by-pixel basis. Because the decoder system 700 dynamically performs horizontal mixing 905 and vertical mixing 910 on the fly on a pixel-by-pixel basis, the decoder system 700 is able to perform horizontal mixing 905, vertical mixing 910, and noise addition 915 in parallel. For example, once horizontal mixing 905 is completed for pixel 0, the decoder system 700 is able to perform horizontal mixing 905 for pixel 1 while also performing vertical mixing 910 for pixel 0 in parallel. Thereafter, the decoder system 700 is able to perform horizontal mixing 905 for pixel 2 while also performing vertical mixing 910 for pixel 1 in parallel, and while also performing noise addition 915 for pixel 0 in parallel. Thereafter, decoder system 700 is able to perform horizontal blending 905 of pixel 3 while also performing vertical blending 910 of pixel 2 in parallel, and while also performing noise addition 915 of pixel 1 in parallel. This parallelization results in significant time savings for decoder system 700 as compared to decoder system 600.

[0162] Fig.10 1 is a flow chart illustrating a codec process 1000. The codec process 1000 may be performed by a codec system. In some examples, the codec system can include, for example, a video source 102, an encoding device 104, an encoding engine 106, a storage device 108, an output 110, a communication link 120, a decoding device 112, an input 114, a decoder engine 116, a storage device 118, a video destination device 122, a codec system 200, a denoiser 210, an encoding engine 220, a decoding engine 230, an analysis engine 240, a subtractor 250, a granularity estimation engine 260, a granularity generator 270, an adder 280, an encoder 290, a decoder 295, a decoder system 600, a decoder system 700, a computing system 1100, a processor 1110, an apparatus, a non-transitory computer-readable medium storing instructions for execution by one or more processors, a mobile handset, a head mounted display (HMD), a wireless communication device, or a combination thereof.

[0163] In operation 1005, the codec system is configured and capable of retrieving noise pixel data associated with at least one reconstructed video pixel from a granule array stored in an on-chip memory. Examples of granule arrays include granule pixel data 275, granule array 305, granule array 620, or a combination thereof. Examples of on-chip memory include storage device 118 and / or granule memory 705. Examples of noise pixel data include at least one pixel of any of the following: granule pixel data 275, noise pixel data 310, noise pixel data 320, noise pixel data 330, noise pixel data 340, frame noise data 355, noise pixel data 410, noise pixel data 420, noise pixel data 430, noise pixel data 440, noise pixel data 510, noise pixel data 520, noise pixel data 530, noise pixel data 540, noise image 675, granule read data 735, noise pixels 805-840, or a combination thereof. Examples of at least one reconstructed video pixel include at least one pixel of any of the following: decoded video 235, video frame 350, block 315, block 325, block 335, block 345, decoded video frame 505, block 515, block 525, block 535, block 545, reconstructed pixel 655, or a combination thereof.

[0164] In some aspects, the particle array is a luma particle array and the noise pixel data comprises luma noise pixel data. In some aspects, the particle array is a chroma particle array and the noise pixel data comprises chroma noise pixel data.

[0165] In some aspects, the codec system is configured and capable of using granularity parameters (e.g., granularity parameters 265, FG parameters 605) to generate a granularity array. The granularity parameters are determined based on video data (e.g., input video 205, denoised video 215, encoded video 225, and / or video frame 350). At least one reconstructed video pixel is configured to be a pixel of the reconstructed video data. In some aspects, the codec system is configured and capable of storing the granular array in an on-chip memory.

[0166] In some aspects, the codec system is configured and capable of identifying a location of noisy pixel data in the granular array based on an offset to retrieve the noisy pixel data. The offset corresponds to a block of which at least one reconstructed video pixel is a part. Examples of the offset include indicating Figure 3305 in the particle array 305. Other examples of offsets include offsets generated using random offset generator 635 and / or random offset generator 710, offsets including text "PlaneOffsetX" in pseudo code 3, offsets including text "PlaneOffsetY" in pseudo code 3, offsets including text "OffsetX" in pseudo code 3, offsets including text "OffsetY" in pseudo code 3, or combinations thereof. In some aspects, the offset is determined based on a randomly generated value (e.g., from random offset generator 635 and / or random offset generator 710). In some aspects, the offset is determined based on a frame size (e.g., frame width, frame height, or both) of a video frame (e.g., of input video 205, denoised video 215, encoded video 225, and / or video frame 350). Examples of frame sizes include frame width and height 625. At least one reconstructed video pixel is configured to reconstruct at least one pixel of a video frame. In some aspects, the block is based on a frame size. For example, the size and position(s) of the blocks may depend on the frame size.

[0167] In operation 1010, the codec system is configured and capable of applying noise data to at least one reconstructed video pixel to generate at least one output video pixel, wherein the noise data is determined based on the noise pixel data. Examples of noise data include at least one pixel of any of the following: grain pixel data 275, noise pixel data 310, noise pixel data 320, noise pixel data 330, noise pixel data 340, frame noise data 355, noise pixel data 410, noise pixel data 420, noise pixel data 430, noise pixel data 440, horizontal blending 450, vertical blending 455, noise pixel data 510, noise pixel data 520, noise pixel data 530, noise pixel data 540, horizontal blending 640, vertical blending 645, scaler, shifter / rounder / adder / limiter engine 665, film grain pixel 670, horizontal and / or vertical blending performed by horizontal / vertical blending engine 715, film grain pixel 725, noise pixels 805-840, or a combination thereof. Examples of at least one output video pixel include at least one pixel of: output video 285, output frame 560, noise addition 915, or a combination thereof.

[0168] In some aspects, the codec system is configured and capable of retrieving secondary noise pixel data from a particle array stored in an on-chip memory. The secondary noise pixel data is associated with at least one reconstructed video pixel. The noise data of operation 1010 is determined based on the noise pixel data and the secondary noise pixel data. In some aspects, the codec system is configured and capable of mixing the noise pixel data and the secondary noise pixel data to determine the noise data. At least one position of at least one reconstructed video pixel in a video frame corresponds to an overlapping region where multiple blocks of the video frame are arranged to overlap. For example, if the position of at least one reconstructed video pixel corresponds to an overlapping region, such as a pixel column in region 450 for horizontal mixing and / or a pixel row in region 455 for vertical mixing, the codec system is capable of retrieving noise pixel data corresponding to multiple blocks, and is capable of mixing the retrieved noise pixel data to generate the noise data of operation 1010. Additional examples include noisy pixel 805 (having noisy pixel data from blocks A and B mixed together), noisy pixel 820 (having noisy pixel data from blocks A, B, C, and D mixed together), noisy pixel 825 (having noisy pixel data from blocks A and C mixed together), and noisy pixel 825 (having noisy pixel data from blocks C and D mixed together).

[0169] In some aspects, the codec system is configured and capable of applying the noise data to at least one reconstructed video pixel while determining the secondary noise data based on the secondary noise pixel data. The secondary noise pixel data is retrieved from a granular array stored in an on-chip memory. The secondary noise pixel data is associated with at least a second reconstructed video pixel. For example, in Fig. 9B In the context of , the codec system is able to perform noise addition 915 on one pixel (e.g., pixel 0) while performing horizontal blending 905 on another pixel (e.g., pixel 2) and / or vertical blending 910 on another pixel (e.g., pixel 1).

[0170] In some aspects, the codec system is configured and capable of applying noise data to at least one reconstructed video pixel while retrieving secondary noise pixel data associated with at least a second reconstructed video pixel from a granular array stored in an on-chip memory. Fig. 9B In the context of , the codec system is able to perform noise addition 915 on one pixel (e.g., pixel 0) while retrieving secondary noisy pixel data for another pixel (e.g., pixel 1, pixel 2, or pixel 3) to be mixed with the noisy pixel data (via horizontal mixing 905 and / or vertical mixing 910).

[0171] In some aspects, the codec system is configured and capable of determining the noise pixel data while retrieving secondary noise pixel data associated with at least a second reconstructed video pixel from a granular array stored in an on-chip memory. Fig. 9B In the context of , the codec system is able to perform horizontal blending 905 on one pixel (e.g., pixel 2) while retrieving secondary noise pixel data for another pixel (e.g., pixel 3). Fig. 9B In the context of , the codec system is able to perform vertical blending 910 on one pixel (eg, pixel 1) while retrieving secondary noisy pixel data for another pixel (eg, pixel 2 or pixel 3).

[0172] In some aspects, the codec system is configured and capable of performing certain operations on a pixel-by-pixel basis. For example, the codec system can perform, on a pixel-by-pixel basis, retrieving noise pixel data from a granular array stored in an on-chip memory (as in operation 1005), determining noise data based on the noise pixel data (e.g., including blending), and / or applying noise data to at least one reconstructed video pixel (e.g., as in operation 1010).

[0173] In some aspects, the codec system is configured and capable of applying a scaling factor to at least one output video pixel, rounding a pixel value of at least one output video pixel, clipping a pixel value of at least one output video pixel, or some combination thereof (e.g., using a scaling lookup table (LUT) 660 and / or a shifter / rounder / adder / clipper engine 665).

[0174] In some aspects, at least one memory is configured to store at least one reconstructed video pixel. In some aspects, the codec system is configured and capable of storing at least one reconstructed video pixel in at least one memory. In some aspects, the codec system is configured and capable of outputting at least one output video pixel for storage in at least one memory. In some aspects, the codec system includes a display (e.g., video destination device 122, output device 1135) configured to display at least one output video pixel. In some aspects, the codec system includes a communication interface configured to send at least one output video pixel to a recipient device.

[0175] In some aspects, the codec system is configured and capable of encoding (e.g., using encoding device 104, encoding engine 106, encoder 290, and / or encoding engine 220) video data (e.g., video source 102, input video 205, and / or denoised video 215) to generate encoded video data (e.g., output 110, input 114, encoded video 225) and decoding (e.g., using decoding device 112, decoder engine 116, decoder 295, and / or decoding engine 230) at least the encoded video data to generate at least one reconstructed video pixel.

[0176] In some aspects, the codec system is associated with the AO Media Video 1 (AV1) format, an AV1 codec, an AV1 encoder, an AV1 decoder, or a combination thereof.

[0177] In some examples, an imaging system can include: components for retrieving noise pixel data associated with at least one reconstructed video pixel from a particle array stored in an on-chip memory that is part of a processor; and components for applying the noise data to the at least one reconstructed video pixel to generate at least one output video pixel, wherein the noise data is determined based on the noise pixel data.

[0178] In some examples, components for retrieving noise pixel data include decoding device 112, decoder engine 116, decoder 295, granule generator 270, decoder system 600 (e.g., via noise image granule read 675 and / or other granule reads from DDR / memory 650), DDR / memory 650, noise image generator 630, random offset generator 635, decoder system 700 (e.g., via granule read request 730 and / or granule read data 735), granule memory 705, computing system 1100, or a combination thereof.

[0179] In some examples, components for applying noise data to at least one reconstructed video pixel include a decoding device 112, a decoder engine 116, a decoder 295, a decoding engine 230, a particle generator 270, an adder 280, a decoder system 600, a noise image generator 630, an engine(s) for horizontal mixing 640, an engine(s) for vertical mixing 645, a scaling lookup table (LUT) 660, a shifter / rounder / adder / limiter engine 665, a decoder system 700, a horizontal / vertical mixing engine 715, a computing system 1100, or a combination thereof.

[0180] In some examples, the processes described herein (e.g., Figure 1 , Figure 2 , Figure 3 , Figure 5 , Figure 6 , Figure 7 , Figure 8 The corresponding process Fig.10 The process 1000 and / or other processes described herein) can be performed by a computing device or apparatus. In some examples, the processes described herein can be performed by and / or using video source 102, encoding device 104, encoding engine 106, storage device 108, output 110, communication link 120, decoding device 112, input 114, decoder engine 116, storage device 118, video destination device 122, codec system 200, denoiser 210, encoding engine 220, decoding engine 230, analysis engine 240, subtractor 250, granularity estimation engine 260, granularity generator 270, adder 280, encoder 290, decoder 295, decoder system 600, decoder system 700, computing system 1100, processor 1110, or a combination thereof.

[0181] The computing device can include any suitable device, such as a mobile device (e.g., a mobile phone), a desktop computing device, a tablet computing device, a wearable device (e.g., a VR headset, an AR headset, an AR glasses, a network-connected watch or smart watch, or other wearable device), a server computer, a vehicle or a computing device of a vehicle, a robotic device, a television, and / or any other computing device having the resource capability to perform the processes described herein. In some cases, the computing device or apparatus may include various components, such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more cameras, one or more sensors, and / or other (multiple) components configured to perform the steps of the processes described herein. In some examples, the computing device may include a display, a network interface configured to transmit and / or receive data, any combination thereof, and / or other (multiple) components. The network interface may be configured to transmit and / or receive data based on an Internet Protocol (IP) or other types of data.

[0182] The components of the computing device can be implemented in circuits. For example, the components can include and / or can be implemented using electronic circuits or other electronic hardware, which can include one or more programmable electronic circuits (e.g., microprocessors, graphics processing units (GPUs), digital signal processors (DSPs), central processing units (CPUs), and / or other suitable electronic circuits), and / or can include and / or be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein.

[0183] The processes described herein are shown as logic flow diagrams, block diagrams, or conceptual diagrams, and their operations represent a series of operations that can be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, operations represent computer executable instructions stored on one or more computer-readable storage media, and the instructions, when executed by one or more processors, perform operations. Typically, computer executable instructions include routines, programs, objects, components, data structures, etc. that perform specific functions or implement specific data types. The order in which the operations are described is not intended to be interpreted as limiting, and any number of the described operations can be combined in any order and / or in parallel to implement the process. In some examples, the execution of certain operations described herein can be responsive to the execution of other operations described herein.

[0184] In addition, the processes described herein may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed together on one or more processors, by hardware, or a combination thereof. As described above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program including multiple instructions executable by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0185] Fig.11 is a diagram illustrating an example of a system for implementing certain aspects of the present technology. In particular, Fig.11 An example of a computing system 1100 is shown, which can be any computing device, for example, constituting an internal computing system, a remote computing system, a camera, or any component thereof, wherein the components of the system communicate with each other using a connection 1105. The connection 1105 can be a physical connection using a bus, or a direct connection into the processor 1110, such as in a chipset architecture. The connection 1105 can also be a virtual connection, a networked connection, or a logical connection.

[0186] In some aspects, the computing system 1100 is a distributed system, wherein the functionality described in the present disclosure can be distributed within a data center, multiple data centers, a peer-to-peer network, etc. In some aspects, one or more of the described system components represent a number of such components, each component performing some or all of the functionality for which the component is described. In some aspects, a component can be a physical or virtual device.

[0187] The example system 1100 includes at least one processing unit (CPU or processor) 1110 and connections 1105 coupling various system components including system memory 1115, such as read only memory (ROM) 1120 and random access memory (RAM) 1125, to the processor 1110. The computing system 1100 can include a cache 1112 of high-speed memory directly connected to the processor 1110, in close proximity to the processor 1110, or integrated as part of the processor 1110.

[0188] Processor 1110 can include any general purpose processor and hardware services or software services, such as services 1132, 1134, and 1136 stored in storage device 1130, which are configured to control processor 1110 as well as a dedicated processor, where the software instructions are incorporated into the actual processor design. Processor 1110 can be essentially a completely self-contained computing system that includes multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors can be symmetric or asymmetric.

[0189] To enable user interaction, the computing system 1100 includes an input device 1145, which can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, voice, and the like. The computing system 1100 can also include an output device 1135, which can be one or more of a plurality of output mechanisms. In some examples, a multimodal system can enable a user to provide multiple types of input / output to communicate with the computing system 1100. The computing system 1100 can include a communication interface 1140, which can generally govern and manage user input and system output. The communication interface can use a wired and / or wireless transceiver to perform or facilitate receiving and / or sending wired or wireless communications, including utilizing an audio jack / plug, a microphone jack / plug, a universal serial bus (USB) port / plug, Ports / plugs, Ethernet ports / plugs, Fiber optic ports / plugs, Proprietary wired ports / plugs, Wireless signal transmission, Low power (BLE) wireless signal transmission, Wireless signal transmission, radio frequency identification (RFID) wireless signal transmission, near field communication (NFC) wireless signal transmission, dedicated short range communication (DSRC) wireless signal transmission, 1102.11 Wi-Fi wireless signal transmission, wireless local area network (WLAN) signal transmission, visible light communication (VLC), world interoperability for microwave access (WiMAX), infrared (IR) communication wireless signal transmission, public switched telephone network (PSTN) signal transmission, 3G / 4G / 5G / LTE cellular data network wireless signal transmission, ad-hoc network signal transmission, radio wave signal transmission, microwave signal transmission, infrared signal transmission, visible light signal transmission, ultraviolet light signal transmission, wireless signal transmission along the electromagnetic spectrum, or some combination thereof. The communication interface 1140 may also include one or more global navigation satellite system (GNSS) receivers or transceivers for determining the location of the computing system 1100 based on receiving one or more signals from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the United States-based Global Positioning System (GPS), the Russian-based Global Navigation Satellite System (GLONASS), the Chinese-based BeiDou Navigation Satellite System (BDS), and the European-based Galileo GNSS. There is no restriction to operate on any particular hardware arrangement, and thus the basic features herein may be easily replaced with improved hardware or firmware arrangements as such are developed.

[0190] The storage device 1130 can be a non-volatile and / or non-temporary and / or computer-readable storage device, and can be a hard disk or other type of computer-readable medium capable of storing data accessible by a computer, such as a magnetic tape cartridge, a flash memory card, a solid-state storage device, a digital versatile disk, a cassette, a floppy disk, a flexible disk, a hard disk, a magnetic tape, a magnetic stripe / slice, any other magnetic storage medium, a flash memory, a memristor memory, any other solid-state memory, a compact disk read-only memory (CD-ROM) disk, a rewritable compact disk (CD) disk, a digital video disk (DVD) disk, a Blu-ray disc (BDD) disk, a holographic disk, another optical medium, a secure digital (SD) card, a micro secure digital (microSD) card, a Memory Card, or a memory card. card, a smart card chip, an EMV chip, a subscriber identity module (SIM) card, a mini / micro / nano / pico SIM card, another integrated circuit (IC) chip / card, a random access memory (RAM), a static RAM (SRAM), a dynamic RAM (DRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash EPROM (FLASHEPROM), a cache memory (L1 / L2 / L3 / L4 / L5 / L#), a resistance random access memory (RRAM / ReRAM), a phase change memory (PCM), a spin transfer torque RAM (STT-RAM), another memory chip or carrier and / or a combination thereof.

[0191] Storage devices 1130 can include software services, servers, services, etc., which cause the system to perform functions when the code defining such software is executed by processor 1110. In some aspects, hardware services that perform a particular function can include software components stored in a computer-readable medium that are combined with the necessary hardware components (e.g., processor 1110, connection 1105, output device 1135, etc.) to perform the function.

[0192] As used herein, the term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or carrying (multiple) instructions and / or data. Computer-readable media may include non-transient media in which data can be stored and do not include carrier waves and / or temporary electronic signals that are propagated wirelessly or through wired connections. Examples of non-transient media may include, but are not limited to, disks or tapes, optical storage media (such as compact disks (CDs) or digital versatile disks (DVDs)), flash memories, memories, or memory devices. Computer-readable media may have codes and / or machine-executable instructions stored thereon, and codes and / or machine-executable instructions may represent any combination of programs, functions, subroutines, programs, routines, subroutines, modules, software packages, categories, or instructions, data structures, or program statements. Code segments may be coupled to another code segment or hardware circuit by transmitting and / or receiving information, data, independent variables, parameters, or memory contents. Any suitable means including memory sharing, message passing, token passing, network transmission, etc. may be used to transmit, forward, or send information, independent variables, parameters, data, etc.

[0193] In some aspects, computer readable storage devices, media, and memories can include wired or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer readable storage media explicitly excludes media such as energy, carrier signals, electromagnetic waves, and signals themselves.

[0194] Specific details are provided in the above description to provide a thorough understanding of the aspects and examples provided herein. However, it will be understood by those skilled in the art that these aspects can be practiced without these specific details. For clear explanation, in some cases, the present technology can be presented as including separate functional blocks, which include functional blocks including steps or routines in methods including devices, device components, software or a combination of hardware and software. In addition to those components shown in the drawings and / or described herein, additional components can be used. For example, circuits, systems, networks, processes and other components can be shown as components in block diagram form, so as not to obscure these aspects with unnecessary details. In other examples, known circuits, processes, algorithms, structures and techniques can be shown without unnecessary details, so as to avoid blurring these aspects.

[0195] The above various aspects can be described as a process or method depicted as a flow chart, flow diagram, data flow diagram, structure diagram or block diagram. Although a flow chart can describe an operation as a sequential process, many operations can be performed in parallel or simultaneously. In addition, the order of the operations can be rearranged. A process terminates when its operation is completed, but can have additional steps not included in the figure. A process can correspond to a method, function, procedure, subroutine, subprogram, etc. When a process corresponds to a function, its termination can correspond to the function returning to the calling function or main function.

[0196] The process and method according to the above example can be implemented using computer executable instructions stored in or otherwise available from a computer readable medium. Such instructions can include, for example, instructions and data that enable or otherwise configure a general-purpose computer, a special-purpose computer, or a processing device to perform a specific function or group of functions. The various parts of the computer resources used can be accessed through a network. Computer executable instructions can be, for example, binary files, intermediate format instructions such as assembly language, firmware, source code, etc. Examples of computer readable media that can be used to store instructions, information used, and / or information created during the method according to the described examples include disks or optical disks, flash memory, USB devices provided with non-volatile memory, networked storage devices, etc.

[0197] The equipment according to these disclosed implementation processes and methods can include hardware, software, firmware, middleware, microcode, hardware description language or any combination thereof, and can adopt any one of various form factors. When the program code or code segment (e.g., computer program product) for performing the necessary tasks is implemented with software, firmware, middleware or microcode, it can be stored in a computer-readable or machine-readable medium. (Multiple) processors can perform the necessary tasks. Typical examples of form factors include laptop computers, smart phones, mobile phones, tablet devices or other small form factor personal computers, personal digital assistants, rack-mounted devices, stand-alone devices, etc. The functions described herein can also be embodied in peripheral devices or add-on cards. As another example, such functions can also be implemented between different chips or different processes executed in a single device on a circuit board.

[0198] Instructions, media for transmitting these instructions, computing resources for executing these instructions, and other structures for supporting these computing resources are example components for providing the functionality described in this disclosure.

[0199] In the foregoing description, various aspects of the present application are described with reference to their specific aspects, but those skilled in the art will recognize that the present application is not limited thereto. Therefore, although the illustrative aspects of the present application have been described in detail herein, it will be understood that, in addition to being limited by the prior art, the inventive concept can be implemented and adopted differently in other ways, and the appended claims are intended to be interpreted as including such variations. The various features and aspects of the above-mentioned application can be used individually or in combination. In addition, without departing from the broader spirit and scope of this specification, various aspects can be utilized in any number of environments and applications outside the environment and application described herein. Therefore, the description and the accompanying drawings are considered to be illustrative rather than restrictive. For the purpose of illustration, the method is described in a specific order. It should be understood that, in alternative aspects, the method can be performed in an order different from the described order.

[0200] One of ordinary skill will understand that the less than ("<") and greater than (">") symbols or terms used herein can be replaced by less than or equal to ("≤") and greater than or equal to ("≥") symbols, respectively, without departing from the scope of the present specification.

[0201] Where a component is described as being “configured to” perform certain operations, such configuration can be achieved, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., a microprocessor or other suitable electronic circuits) to perform the operations, or any combination thereof.

[0202] The phrase "coupled to" refers to any component that is physically connected directly or indirectly to another component, and / or any component that communicates directly or indirectly with another component (e.g., connected to other components via a wired or wireless connection and / or other suitable communication interface).

[0203] Claim language or other language reciting "at least one of" a set and / or "one or more of" a set indicates that one member of the set or multiple members of the set (in any combination) satisfies the claim. For example, claim language reciting "at least one of A and B" means A, B, or A and B. In another example, claim language reciting "at least one of A, B, and C" means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language "at least one of" a set and / or "one or more of" a set does not limit the set to the items listed in the set. For example, claim language reciting "at least one of A and B" can mean A, B, or A and B, and can additionally include items not listed in the set of A and B.

[0204] Various illustrative logic blocks, modules, circuits and algorithmic steps described in conjunction with the various aspects disclosed herein can be implemented as electronic hardware, computer software, firmware or a combination thereof. In order to clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits and steps have been generally described above with respect to their functionality. Whether this function is implemented as hardware or software depends on the specific application and the design constraints imposed on the entire system. The technician can implement the described function in different ways for each specific application, but this implementation decision should not be interpreted as causing a departure from the scope of the application.

[0205] The technology described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such technology may be implemented in any of a variety of devices, such as a general-purpose computer, a wireless communication device handset, or an integrated circuit device, which has multiple uses including applications in wireless communication device handsets and other devices. Any features described as modules or components may be implemented together in an integrated logic device, or separately as discrete but interoperable logic devices. If the technology is implemented in software, the technology may be implemented at least in part by a computer-readable data storage medium including a program code, the program code including instructions for performing one or more of the methods described above when executed. The computer-readable data storage medium may form part of a computer program product, wherein the computer program product may include packaging materials. The computer-readable medium may include a memory or data storage medium, such as a random access memory (RAM) (such as a synchronous dynamic random access memory (SDRAM)), a read-only memory (ROM), a non-volatile random access memory (NVRAM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic or optical data storage medium, and the like. Additionally or alternatively, the techniques may be implemented at least in part by a computer-readable communication medium that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer, such as a propagated signal or wave.

[0206] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such a processor may be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor; but in an alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, a combination of one or more microprocessors and a DSP core, or any other such configuration. Therefore, the term "processor" as used herein may refer to any of the aforementioned structures, any combination of the aforementioned structures, or any other structure or device suitable for implementing the techniques described herein. In addition, in some aspects, the functionality described herein may be provided in a dedicated software module or hardware module configured for encoding and decoding, or incorporated into a combined video encoder-decoder (CODEC).

[0207] Illustrative aspects of the disclosure include:

[0208] Aspect 1: A device for video decoding, the device comprising: at least one memory; and at least one processor, including an on-chip memory storing a particle array, the at least one processor being configured to: retrieve noise pixel data associated with at least one reconstructed video pixel from the particle array stored in the on-chip memory; and apply the noise data to at least one reconstructed video pixel to generate at least one output video pixel, wherein the noise data is determined based on the noise pixel data.

[0209] Aspect 2. An apparatus according to Aspect 2, wherein at least one processor is configured to: retrieve secondary noise pixel data from a particle array stored in an on-chip memory, wherein the secondary noise pixel data is associated with at least one reconstructed video pixel, and wherein the noise data is determined based on the noise pixel data and the secondary noise pixel data.

[0210] Aspect 3. An apparatus according to Aspect 2, wherein at least one processor is configured to: mix noise pixel data and secondary noise pixel data to determine noise data, wherein at least one position of at least one reconstructed video pixel in a video frame corresponds to an overlapping region in which multiple blocks of the video frame are arranged as overlapping regions.

[0211] Aspect 4. An apparatus according to any one of Aspects 1 to 3, wherein at least one processor is configured to: generate a particle array using particle parameters, wherein the particle parameters are determined based on video data, and wherein at least one reconstructed video pixel is configured to be a pixel of the reconstructed video data.

[0212] Aspect 5. The apparatus according to any one of aspects 1 to 4, wherein at least one processor is configured to: store the particle array in an on-chip memory.

[0213] Aspect 6. An apparatus according to any one of Aspects 1 to 5, wherein at least one processor is configured to: identify the location of noise pixel data in the particle array based on an offset to retrieve the noise pixel data, wherein the offset corresponds to a block of which at least one reconstructed video pixel is a part.

[0214] Aspect 7. The apparatus according to aspect 6, wherein the offset is determined based on a randomly generated value.

[0215] Aspect 8. An apparatus according to any one of Aspects 6 to 7, wherein the offset is determined based on a frame size of a video frame, and wherein the at least one reconstructed video pixel is configured to reconstruct at least one pixel of the video frame.

[0216] Aspect 9. The apparatus of aspect 8, wherein the blocks are based on a frame size.

[0217] Aspect 10. An apparatus according to any one of Aspects 1 to 9, wherein at least one processor is configured to: apply noise data to at least one reconstructed video pixel while determining secondary noise data based on secondary noise pixel data, wherein the secondary noise pixel data is retrieved from a particle array stored in an on-chip memory, and the secondary noise pixel data is associated with at least a second reconstructed video pixel.

[0218] Aspect 11. The apparatus according to any one of aspects 1 to 10, wherein the at least one processor is configured to retrieve secondary noise pixel data associated with at least a second reconstructed video pixel from a granule array stored in an on-chip memory.

[0219] Aspect 12. An apparatus according to any one of aspects 1 to 11, wherein at least one processor is configured to retrieve secondary noise pixel data associated with at least a second reconstructed video pixel from a granular array stored in an on-chip memory.

[0220] Clause 13. The apparatus according to any one of clauses 1 to 12, wherein the noise pixel data is configured to be retrieved on a pixel-by-pixel basis from a particle array stored in an on-chip memory.

[0221] Clause 14. An apparatus according to any one of clauses 1 to 13, wherein the noise data is configured to be determined based on the noise pixel data on a pixel-by-pixel basis.

[0222] Clause 15. An apparatus according to any one of Clauses 1 to 14, wherein the noise data is configured to be applied to at least one reconstructed video pixel on a pixel-by-pixel basis.

[0223] Clause 16. The apparatus according to any one of clauses 1 to 15, wherein the particle array is a luminance particle array, and wherein the noise pixel data comprises luminance noise pixel data.

[0224] Aspect 17. The apparatus of any one of aspects 1 to 16, wherein the particle array is a chroma particle array, and wherein the noise pixel data comprises chroma noise pixel data.

[0225] Clause 18. The apparatus according to any one of clauses 1 to 17, wherein at least one processor is configured to: apply a scaling factor to at least one output video pixel.

[0226] Clause 19. The apparatus according to any one of Clauses 1 to 18, wherein at least one processor is configured to: round a pixel value of at least one output video pixel.

[0227] Clause 20. The apparatus according to any one of clauses 1 to 19, wherein at least one processor is configured to: clip a pixel value of at least one output video pixel.

[0228] Aspect 21. An apparatus according to any one of aspects 1 to 20, wherein at least one memory is configured to store at least one reconstructed video pixel.

[0229] Aspect 22. An apparatus according to any one of aspects 1 to 21, wherein at least one processor is configured to: output at least one output video pixel for storage in at least one memory.

[0230] Aspect 23. The apparatus according to any one of aspects 1 to 22, further comprising: a display configured to display at least one output video pixel.

[0231] Aspect 24. The apparatus according to any one of aspects 1 to 23, further comprising: a communication interface configured to send at least one output video pixel to a recipient device.

[0232] Aspect 25. An apparatus according to any one of Aspects 1 to 24, wherein at least one processor is configured to: encode video data to generate encoded video data; and decode at least the encoded video data to generate at least one reconstructed video pixel.

[0233] Aspect 26. An apparatus according to any one of Aspects 1 to 25, wherein the apparatus comprises at least one of a head mounted display (HMD), a mobile phone, or a wireless communication device.

[0234] Aspect 27. A method for video decoding, the method comprising: retrieving noise pixel data associated with at least one reconstructed video pixel from a granular array stored in an on-chip memory as part of a processor; and applying the noise data to the at least one reconstructed video pixel to generate at least one output video pixel, wherein the noise data is determined based on the noise pixel data.

[0235] Aspect 28. The method according to Aspect 27 further includes: retrieving secondary noise pixel data from a particle array stored in an on-chip memory, wherein the secondary noise pixel data is associated with at least one reconstructed video pixel, and wherein the noise data is determined based on the noise pixel data and the secondary noise pixel data.

[0236] Aspect 29. The method according to Aspect 28 further includes: mixing the noise pixel data and the secondary noise pixel data to determine the noise data, wherein at least one position of at least one reconstructed video pixel in the video frame corresponds to an overlapping area where multiple blocks of the video frame are arranged to overlap.

[0237] Aspect 30. The method according to any one of aspects 27 to 29 further comprises: generating a particle array using particle parameters, wherein the particle parameters are determined based on video data, wherein at least one reconstructed video pixel is configured as a pixel of the reconstructed video data.

[0238] Aspect 31. The method according to any one of aspects 27 to 30, further comprising: storing the particle array in an on-chip memory.

[0239] Aspect 32. The method according to any one of Aspects 27 to 31 further comprises: identifying a location of the noise pixel data in the particle array based on an offset to retrieve the noise pixel data, wherein the offset corresponds to a block of which at least one reconstructed video pixel is a part.

[0240] Aspect 33. A method according to aspect 32, wherein the offset is determined based on a randomly generated value.

[0241] Aspect 34. A method according to any one of aspects 32 to 33, wherein the offset is determined based on a frame size of a video frame, wherein the at least one reconstructed video pixel is configured to reconstruct at least one pixel of the video frame.

[0242] Aspect 35. The method of aspect 34, wherein the blocks are based on a frame size.

[0243] Aspect 36. The method according to any one of Aspects 27 to 35 further includes: while determining secondary noise data based on secondary noise pixel data, applying the noise data to at least one reconstructed video pixel, wherein the secondary noise pixel data is retrieved from a particle array stored in an on-chip memory, and the secondary noise pixel data is associated with at least a second reconstructed video pixel.

[0244] Aspect 37. The method according to any one of aspects 27 to 36, further comprising: retrieving secondary noise pixel data associated with at least a second reconstructed video pixel from a granule array stored in an on-chip memory.

[0245] Clause 38. The method according to any one of Clauses 27 to 37, further comprising: retrieving secondary noise pixel data associated with at least a second reconstructed video pixel from a granule array stored in an on-chip memory.

[0246] Aspect 39. A method according to any one of aspects 27 to 38, wherein the noise pixel data is configured to be retrieved on a pixel-by-pixel basis from a particle array stored in an on-chip memory.

[0247] Aspect 40. A method according to any one of aspects 27 to 39, wherein the noise data is configured to be determined based on the noise pixel data on a pixel-by-pixel basis.

[0248] Clause 41. A method according to any one of clauses 27 to 40, wherein the noise data is configured to be applied to at least one reconstructed video pixel on a pixel-by-pixel basis.

[0249] Aspect 42. The method of any one of aspects 27 to 41, wherein the particle array is a luminance particle array, and wherein the noise pixel data comprises luminance noise pixel data.

[0250] Aspect 43. A method according to any one of Aspects 27 to 42, wherein the particle array is a chrominance particle array, and wherein the noise pixel data includes chrominance noise pixel data.

[0251] Clause 44. The method according to any one of clauses 27 to 43, further comprising: applying a scaling factor to at least one output video pixel.

[0252] Clause 45. The method according to any one of clauses 27 to 44, further comprising: rounding a pixel value of at least one output video pixel.

[0253] Clause 46. The method according to any one of clauses 27 to 45, further comprising: clipping the pixel value of at least one output video pixel.

[0254] Aspect 47. A method according to any one of aspects 27 to 46, wherein at least one memory is configured to store at least one reconstructed video pixel.

[0255] Clause 48. The method according to any one of clauses 27 to 47, further comprising: outputting at least one output video pixel for storage in at least one memory.

[0256] Clause 49. The method according to any one of clauses 27 to 48, further comprising: causing a display to display at least one output video pixel.

[0257] Clause 50. The method according to any one of clauses 27 to 49, further comprising causing the communication interface to send at least one output video pixel to a recipient device.

[0258] Aspect 51. The method according to any one of aspects 27 to 50 further comprises: encoding the video data to generate encoded video data; and decoding at least the encoded video data to generate at least one reconstructed video pixel.

[0259] Aspect 52. A method according to any one of aspects 27 to 51, wherein the method is performed using an apparatus comprising at least one of a head mounted display (HMD), a mobile handset, or a wireless communication device.

[0260] Aspect 53: A non-transitory computer-readable medium having instructions stored thereon, which, when executed by at least one processor having on-chip memory, causes the at least one processor to: retrieve noise pixel data associated with at least one reconstructed video pixel from a particle array stored in the on-chip memory; and apply the noise data to the at least one reconstructed video pixel to generate at least one output video pixel, wherein the noise data is determined based on the noise pixel data.

[0261] Aspect 54: The non-transitory computer-readable medium according to aspect 53, further comprising operations according to any one of aspects 2 to 26 and / or any one of aspects 28 to 52.

[0262] Aspect 55: An apparatus for video decoding, the apparatus comprising: a component for retrieving noise pixel data associated with at least one reconstructed video pixel from a granular array stored in an on-chip memory as part of a processor; and a component for applying the noise data to at least one reconstructed video pixel to generate at least one output video pixel, wherein the noise data is determined based on the noise pixel data.

[0263] Aspect 56: The apparatus according to aspect 55, further comprising means for performing the operations according to any one of aspects 2 to 26 and / or any one of aspects 28 to 52.

Claims

1. A device for video decoding, the device comprising: at least one memory; as well as At least one processor including an on-chip memory storing the particle array, the at least one processor being configured to: retrieving noise pixel data associated with at least one reconstructed video pixel from a granular array stored in an on-chip memory; as well as Noise data is applied to the at least one reconstructed video pixel to generate at least one output video pixel, wherein the noise data is determined based on the noise pixel data.

2. The device according to claim 1, wherein: The at least one processor is configured to: Secondary noise pixel data is retrieved from a granular array stored in an on-chip memory, wherein the secondary noise pixel data is associated with at least one reconstructed video pixel, wherein the noise data is determined based on the noise pixel data and the secondary noise pixel data.

3. The device according to claim 2, wherein: The at least one processor is configured to: The noise pixel data and the secondary noise pixel data are mixed to determine the noise data, wherein at least one position of at least one reconstructed video pixel in a video frame corresponds to an overlap region where a plurality of blocks of the video frame are arranged to overlap.

4. The device according to claim 1, wherein: The at least one processor is configured to: The granule array is generated using granule parameters, wherein the granule parameters are determined based on video data, wherein the at least one reconstructed video pixel is configured to reconstruct a pixel of the video data.

5. The device according to claim 1, wherein: The at least one processor is configured to: The particle array is stored in the on-chip memory.

6. The device according to claim 1, wherein: The at least one processor is configured to: The location of the noisy pixel data in the granular array is identified based on an offset to retrieve the noisy pixel data, wherein the offset corresponds to a block of which the at least one reconstructed video pixel is a part.

7. The device according to claim 6, wherein: The offset is determined based on a randomly generated value.

8. The device according to claim 6, wherein: The offset is determined based on a frame size of a video frame, wherein the at least one reconstructed video pixel is configured to reconstruct at least one pixel of the video frame.

9. The device according to claim 8, wherein: The blocks are based on the frame size.

10. The device according to claim 1, wherein: The at least one processor is configured to: The noise data is applied to the at least one reconstructed video pixel while determining secondary noise data based on secondary noise pixel data retrieved from the granular array stored in the on-chip memory, the secondary noise pixel data being associated with at least a second reconstructed video pixel.

11. The device according to claim 1, wherein: The at least one processor is configured to: The noise data is applied to the at least one reconstructed video pixel while secondary noise pixel data associated with at least a second reconstructed video pixel is retrieved from the granular array stored in the on-chip memory.

12. The device according to claim 1, wherein: The at least one processor is configured to: The noise pixel data is determined while retrieving secondary noise pixel data associated with at least a second reconstructed video pixel from the granular array stored in the on-chip memory.

13. The device according to claim 1, wherein: The noise pixel data is configured to be retrieved from the particle array stored in the on-chip memory on a pixel-by-pixel basis.

14. The device according to claim 1, wherein: The noise data is configured to be determined based on the noisy pixel data on a pixel-by-pixel basis.

15. The device according to claim 1, wherein: The noise data is configured to be applied to the at least one reconstructed video pixel on a pixel-by-pixel basis.

16. The device according to claim 1, wherein The particle array is a luminance particle array, and wherein the noise pixel data comprises luminance noise pixel data.

17. The device according to claim 1, wherein: The particle array is a chroma particle array, and wherein the noise pixel data comprises chroma noise pixel data.

18. The device according to claim 1, wherein: The at least one processor is configured to: A scaling factor is applied to the at least one output video pixel.

19. The device according to claim 1, wherein: The at least one processor is configured to: A pixel value of the at least one output video pixel is rounded.

20. The device according to claim 1, wherein The at least one processor is configured to: The pixel value of the at least one output video pixel is clipped.

21. The device according to claim 1, wherein The at least one memory is configured to store the at least one reconstructed video pixel.

22. The device according to claim 1, wherein: The at least one processor is configured to: The at least one output video pixel is output for storage in the at least one memory.

23. The apparatus of claim 1, further comprising: A display is configured to display the at least one output video pixel.

24. The apparatus of claim 1, further comprising: A communication interface is configured to send the at least one output video pixel to a recipient device.

25. The device according to claim 1, wherein The at least one processor is configured to: encoding the video data to generate encoded video data; and At least the encoded video data is decoded to generate the at least one reconstructed video pixel.

26. The device according to claim 1, wherein The apparatus includes at least one of a head mounted display (HMD), a mobile handset, or a wireless communication device.

27. A method for video decoding, the method comprising: retrieving noise pixel data associated with at least one reconstructed video pixel from a granular array stored in an on-chip memory that is part of the processor; as well as Noise data is applied to the at least one reconstructed video pixel to generate at least one output video pixel, wherein the noise data is determined based on the noise pixel data.

28. The method according to claim 27, further comprising: Secondary noise pixel data is retrieved from the granular array stored in the on-chip memory, wherein the secondary noise pixel data is associated with the at least one reconstructed video pixel, wherein the noise data is determined based on the noise pixel data and the secondary noise pixel data.

29. The method according to claim 28, further comprising: The noise pixel data and the secondary noise pixel data are mixed to determine the noise data, wherein at least one position of at least one reconstructed video pixel in a video frame corresponds to an overlap region where a plurality of blocks of the video frame are arranged to overlap.

30. The method of claim 27, further comprising: The location of the noisy pixel data in the granular array is identified based on an offset to retrieve the noisy pixel data, wherein the offset corresponds to a block of which the at least one reconstructed video pixel is a part.

Images