Video compression using intra-loop sub-picture level controllable noise generation

Abstract

A system includes an encoder configured to compress video data using an in-loop noise generation process that generates noise at a sub-picture portion level of granularity, such as at a block level, in the compression loop. The encoder includes noise model and / or noise model input parameter information in an encoded bitstream. Further, a system includes a decoder configured to receive such a bitstream and decompress the video using an in-loop noise generation process that generates noise at a sub-picture portion level of granularity in the decompression loop.

Description

[0001] Priority

[0002] This application claims priority to and the benefit of U.S. Provisional Patent Application No. 62 / 904,576, filed September 23, 2019, the entire contents of which are incorporated herein by reference. TECHNICAL FIELD

[0003] The present disclosure relates generally to compressing and decompressing video data using in-loop noise generation controllable at the image portion level. BACKGROUND

[0004] Various devices, including but not limited to personal computer systems, desktop computer systems, laptops and notebooks, tablet or slate devices, digital cameras, digital video recorders, and mobile or smart phones, can include software and / or hardware that implements video processing methods. For example, a device can include a device (e.g., an integrated circuit (IC), such as a system on a chip (SOC), or a subsystem of an IC) that can receive and process digital video input from one or more sources and output processed video signals, such as pictures / frames or regions of pictures / frames, in accordance with one or more video processing methods. As another example, a software program that can receive and process digital video input from one or more sources and output processed video frames in accordance with one or more video processing methods can be implemented on a device.

[0005] In some video processing methods, to perform processing, each input video picture / frame is divided into rows and columns of blocks of pixels (e.g., 16x16 blocks of pixels). Each block of the input video picture / frame can be processed individually, and the processed blocks are combined to form an output video frame when processing is complete. This can be referred to as a block processing method.

[0006] A block processing method can include a plurality of processing steps or operations that are sequentially applied to each block in a video picture / frame. SUMMARY

[0007] In some embodiments, a system for encoding video image data is configured to compress pixel value data for a plurality of images included in a plurality of frames of video, where as part of the compression, the system performs in-loop noise estimation and generation at image portion level granularity. The supported multiple levels of granularity can include macroblocks, coding tree units (CTUs), coding units (CUs), prediction units (PUs), transform units (TUs), noise blocks or region units (which can overlap multiple macroblocks, CTUs, CUs, PUs, TUs, etc.), or other block coding units supported by the coding technology. In some embodiments, the techniques as described herein can be added as extensions to existing video coding standards (e.g., in new profiles) or can be included in new video coding standards, such as MPEG VVC / H.266, AOM’s AV2, or other such coding standards or technologies.

[0008] To perform the in-loop noise estimation and generation, the system is configured to determine a first noise pattern to be generated for a first portion of a given one of the images, determine a second noise pattern to be generated for a second portion of the same given one of the images, and generate the first and second noise patterns via an in-loop noise generator included in a compression loop used to compress the pixel value data, where the first and second noise patterns are different. Further, the system is configured to encode the compressed pixel value data for the plurality of images, where the encoded bitstream for the compressed pixel value data includes data indicative of a noise model or one or more noise parameters used to generate the first and second noise patterns. In some embodiments, the first and second portions of the given image can correspond to blocks, CTUs, CUs, PUs, noise blocks or region units, etc. that overlap with multiple blocks or CTUs, etc. The noise patterns can include a pattern of noise samples that are produced by one or more inputs used by the noise model to generate the noise pattern. The noise patterns generated at image portion level granularity can have the same size / shape as the portion (e.g., block, region, etc.) of the image for which it is generated.

[0009] In some embodiments, the first and second (or additional) noise patterns can be added to the residual values calculated in the compression loop, which are also added to the predicted pixel values. In some embodiments, the noise patterns can be combined with the predicted pixel values that have been adjusted based on the residual values at the block level before the adjusted predicted pixel values (e.g., predicted pixel values adjusted based on the residual values) are further filtered by one or more in-loop filters included in the compression loop, such as a deblocking filter, a sample adaptive offset (SAO) filter, an adaptive loop filter (ALF), a bilateral filter, or other in-loop filters. Conversely, in some embodiments, the first and second (or additional) noise patterns can be added at the block level after the residual-adjusted predicted pixel values have been filtered by one or more in-loop filters included in the compression loop, such as a deblocking filter, a sample adaptive offset (SAO) filter, an adaptive loop filter (ALF), a bilateral filter, or other in-loop filters. For example, each sample included in the noise patterns can be added to each corresponding sample of the block. The process can be repeated for each color component of multiple color planes. Further, in some embodiments, noise can be added to one or more color components without being added to other color components.

[0010] In some embodiments, a system for decoding an encoded bitstream of compressed video data is configured to perform in-loop noise estimation and generation at image portion level granularity as part of decompressing the compressed video data. For example, the system is configured to receive an encoded bitstream for a compressed image or video region, where the encoded bitstream includes data indicative of a noise model or noise parameters used to generate noise for two or more portions of a decompressed version of the compressed image or region. The system is further configured to generate, via an in-loop noise generator, a first noise pattern for a first portion of the portions and a second noise pattern for a second portion of the portions based on the data indicative of the noise model or noise parameters. Further, the system is configured to add the first noise pattern to predicted pixel values corresponding to the first portion of the decompressed version of the compressed image and to add the second noise pattern to predicted pixel values corresponding to the second portion of the decompressed version of the compressed image.

[0011] In some embodiments, the first noise pattern and the second (or additional) noise pattern can be added to the residual values calculated in the decompression loop, which are also added to the predicted pixel values. In some embodiments, the noise patterns can be combined with the predicted pixel values that have been adjusted based on the residual values at the block level before the adjusted predicted pixel values (e.g., predicted pixel values adjusted based on the residual values) are further filtered by one or more in-loop filters included in the decompression loop, such as a deblocking filter, a sample adaptive offset (SAO) filter, an adaptive loop filter (ALF), a bilateral filter, or other in-loop filters. Conversely, in some embodiments, the first noise pattern and the second (or additional) noise pattern can be added at the block level after the residual-adjusted predicted pixel values have been filtered by one or more in-loop filters included in the decompression loop, such as a deblocking filter, a sample adaptive offset (SAO) filter, an adaptive loop filter (ALF), a bilateral filter, or other in-loop filters. BRIEF DESCRIPTION OF DRAWINGS

[0012] Figure 1 An exemplary video compression loop of an encoder is shown, in accordance with some embodiments.

[0013] Figure 2 An exemplary video encoder including out-of-loop frame-level noise generation is shown, in accordance with some embodiments.

[0014] Figure 3 An exemplary video decoder including out-of-loop frame-level noise generation is shown, in accordance with some embodiments.

[0015] Figure 4 An exemplary video encoder including in-loop noise generation with sub-picture portion granularity control is shown, in accordance with some embodiments.

[0016] Figure 5 An exemplary video decoder including in-loop noise generation with sub-picture portion granularity control is shown, in accordance with some embodiments.

[0017] Figure 6 An exemplary video encoder including in-loop noise generation with sub-picture portion granularity control is shown, in accordance with some embodiments, in which the generated noise is added downstream of the loop filter.

[0018] Figure 7 An exemplary video decoder including in-loop noise generation with sub-picture portion granularity control is shown, in accordance with some embodiments, in which the generated noise is added downstream of the loop filter.

[0019] Figure 8An exemplary video encoder is shown that includes in-loop noise generation controlled at sub-picture portion granularity and includes controlled de-noising, according to some embodiments.

[0020] Figure 9 An exemplary video encoder is shown that includes in-loop noise generation controlled at sub-picture portion granularity and includes controlled de-noising, according to some embodiments.

[0021] Figure 10 An exemplary video decoder is shown that includes in-loop noise generation controlled at sub-picture portion granularity and includes controlled de-noising downstream of a loop filter, according to some embodiments.

[0022] Figures 11A-11B A compression process is shown that has in-loop noise estimation and generation at sub-picture portion level granularity, according to some embodiments.

[0023] Figure 12 An exemplary image frame divided into image blocks is shown, according to some embodiments.

[0024] Figure 13 An exemplary image frame showing noise blocks or region units overlapping with image blocks is shown, according to some embodiments.

[0025] Figure 14 An exemplary controlled de-noising process is shown, according to some embodiments.

[0026] Figure 15 An exemplary process for generating in-loop noise patterns in a decompression process is shown, according to some embodiments.

[0027] Figure 16 An exemplary computer system that can implement an encoder or decoder is shown, according to some embodiments.

[0028] This specification includes references to “one embodiment” or “an embodiment.” Occurrences of the phrases “in one embodiment” or “in an embodiment” do not necessarily all refer to the same embodiment. Particular features, structures, or characteristics can be combined in any suitable manner in one or more embodiments.

[0029] “Comprise,” means “including.” The term is open-ended. As used in the appended claims, this term does not foreclose addition of another structural or compositional element that can be “comprised” by the specified claim element. Exemplary claims with transitive “comprise” language that pass muster under 35 U.S.C. § 101 include the following claims: “A system comprising one or more processors...” Such a system is fully compliant with the patent statute.

[0030] “Configured to.” Various units, circuits, or other components can be described or claimed as “configured to” perform a task or tasks. In such contexts, “configured to” is used to connote structure by indicating that a unit / circuit / component includes structure (e.g., circuitry) that performs the task or tasks during operation. As such, the unit / circuit / component can be said to be configured to perform the task even when the specified unit / circuit / component is not currently operational (e.g., is not on). Understood, then, is that a unit / circuit / component configured to perform some task may

[0031] “First,” “second,” etc. As used herein, these terms are used as labels to refer to particular nouns that they are directly preceding. These terms do not necessitate any type of ordering (e.g., spatial, temporal, logical, etc.). For example, a buffer circuit can be described herein as performing a write operation of a “first” value and a “second” value. The terms “first” and “second” do not necessarily imply that the first value must be written before the second value.

[0032] “Based on.” As used herein, this term is used to describe one or more factors that affect a determination. This term does not foreclose additional, unrecited factors that can affect a determination. That is, a determination can be solely based on the recited factors or based at least in part on the recited factors. Consider the phrase “determine A based on B.” While in this case, B is a factor that affects the determination of A, such a phrase does not foreclose the determination of A from also being based on C. In other instances, a determination can be made based solely on B. DETAILED DESCRIPTION

[0033] Natural scenes typically do not include noise. However, video images or still images that capture natural scenes can include noise. For example, when processing dark content, such as images of natural scenes captured in low light conditions, noise is often captured in the image along with the natural scene. For example, thermal noise or noise introduced via the light sensor, etc. can be captured. Additionally, depending on the medium used to capture the natural image, other types of noise can be captured, such as film grain noise.

[0034] For various reasons, it can be desirable to preserve or recreate noise included in a captured image after the image has been compressed and decompressed. For example, film grain noise can be a desirable feature of a video in order to replicate an old movie, artistic intent, etc. However, current video compression algorithms either remove the noise during compression and decompression or are unable to realistically reproduce the noise in the decompressed version of the image or video.

[0035] For example, some video compression / decompression systems can add noise to a decompressed image at a full frame level granularity. This can be done by specifying a noise model that is typically associated with each frame of a sequence, e.g., using the Film Grain SEI mechanism in codecs such as MPEG-4 AVC / H.264 or HEVC / HEVC, and using that model to generate noise to be added to the current frame. The amount and type of noise to be added can be further modulated according to regional characteristics, e.g., the brightness of the region to which the noise is to be added. However, the noise model does not change, and the amount and type of noise typically varies within an image frame, such that a noise model that is one-size-fits-all at the frame level is unable to realistically recreate the noise included in the original image.

[0036] In some embodiments, a video encoder or decoder can include an in-loop noise estimator and / or an in-loop noise generator that selects a noise model and / or model parameters at a sub-image portion level granularity, where different noise models and / or parameters can be applied to different portions of an image within the same image frame. For simplicity, the term image frame is used herein. However, in some embodiments, similar processing can be performed on pictures, which can include interlaced images, such as field and frame pictures. In some embodiments, sub-image level noise can be added in a compression loop or a decompression loop upstream or downstream of a loop filter such as a deblocking filter or a sample adaptive offset (SAO) filter.

[0037] Further, in some embodiments, an optional denoising filter can be included in the compression or decompression loop to remove noise. In some embodiments, removing noise from the compression loop can improve compression efficiency by removing noise from residual values and thus reducing the number of bits required to encode or equivalently compress these residual values at a higher quality.

[0038] In some embodiments, a controlled denoising filter can be included in the compression / decompression loop to remove noise to improve compression efficiency. In some embodiments, the controlled denoising filter can be controlled such that the filter is adjusted to remove noise similar to the noise that is added at another location in the compression or decompression loop. For example, information indicative of the selected noise model / noise parameters used to generate the in-loop noise, such as noise parameters, can also be provided to the controlled denoising filter to select an appropriate filter to remove the generated noise.

[0039] Exemplary compression ring / encoder

[0040] For example, Figure 1 A compression loop 100 is shown that receives input video data 130. Certain portions of the input video data 130 are used to determine predicted pixel values via an intra-prediction module and / or a motion estimation / compensation module 102. The predicted values 142 are compared to the original pixel values in the input video data 130 to determine residual values 132 (e.g., the difference between the predicted pixel values and the pixel values of the input video data 130). The residual values 132 are further processed via a transform module 110 and a quantization module 104, and output as compressed image / video data 136. The transformed / quantized residual values (e.g., compressed image / video data 136) are entropy encoded via an entropy encoding module 114, and output as an encoded compressed bitstream 138.

[0041] The compression loop 100 also includes an inverse quantization module 116 and an inverse transform module 118. The inverse quantization module 116 and the inverse transform module 118 reconstruct the residual values so that they are similar to the residual values 132 prior to the transform module 110 and the quantization module 112. For example, the inverse quantization / inverse transform module undo the transform and quantization applied at the transform module 110 and the quantization module 112. In addition to the output of the inverse transform module 116, any distortion introduced due to the transform and quantization of the residual values can be included. The output of the inverse transform module 116 is combined with the predicted pixel values 140 from the motion compensation / intra-prediction module 102 to generate a reconstructed version of the original pixel values (e.g., decompressed pixel values 134). The decompressed pixel values 134 are further processed by in-loop filters, such as a deblocking filter 120 and a sample adaptive offset (SAO) filter 122. The reconstructed image frame including the decompressed pixel values 134 that have been filtered by the deblocking filter 120 and the SAO filter 122 are stored in a decoded picture buffer 124.

[0042] The images in the decoded picture buffer 124 are available as input to estimate intra-mode compression distortion. For example, the intra-mode estimation module 106 can utilize previously encoded regions from the current image also stored in the decoded picture buffer 124 to estimate distortion compared to uncompressed regions in the current image frame.

[0043] As used herein, intra mode compression refers to compression using predicted values determined within an image frame based on values of other pixels in the same image frame and residual values. For example, intra mode compression techniques can take a subset of pixel values of an image and explicitly include that subset. In some embodiments, the subset can be a single starting pixel value or set of pixel values. Further, intra prediction techniques use these starting pixel values to predict pixel values of neighboring pixels. Such a process can be iterative and can proceed block by block, where pixel values are predicted one block at a time. These predicted pixel values can then be compared to actual pixel values of the image to determine residual values. The residual values can then be encoded for use by a decoder to correct predicted pixel values generated using the same or similar prediction process.

[0044] In contrast, as used herein, inter prediction refers to compression techniques that utilize differences between image frames, such as due to temporal changes, to predict pixel values. For example, motion estimation and motion vectors can be used in inter prediction methods to estimate and indicate motion (e.g., temporal differences) of blocks or other subunits of an image between image frames. These motion vectors can then be used to predict pixel values of a subsequent image frame relative to a reference image frame, where the predicted pixel values compensate for motion between the image frames.

[0045] The motion compensation estimation module 108 can use images in the decoded picture buffer 124 to determine estimated motion and / or distortion between uncompressed image frames and reconstructed image frames, where the reconstructed image frames are compressed and decompressed using inter compression techniques, such as motion estimation and compensation.

[0046] The estimated distortion and / or compression efficiency determined by the intra mode estimation module 106 and the motion compensation estimation module 108 can be provided to the mode decision module 104, where the mode decision module 104 selects whether to use intra prediction techniques or inter prediction techniques to compress a given image region or even an entire frame.

[0047] In some embodiments, a device, such as a mobile device having a screen and a camera that is capturing an image or video being compressed, can also display the video image. For example, the reconstructed image frames stored in the decoded picture buffer 124 can be further provided to a display 126 for rendering on the display.

[0048] Traditionally, image and video encoding schemes perform prediction (either intra prediction or inter prediction) as described above on image regions being compressed and encoded. For example, such as a block region of size MxN. Previously encoded samples, such as other blocks in the same image, can be used for intra prediction techniques, while corresponding blocks in other image frames at different times can be used for inter prediction techniques.

[0049] In some embodiments, the transform applied to the residual values, such as via transform module 110, can include an inverse discrete cosine transform or other suitable transform.

[0050] In some embodiments, the in-loop filters, such as deblocking filter 120, SAO filter 122, and / or additional in-loop filters can be applied to a portion (e.g., block) of an image as well as to neighboring portions (e.g., neighboring blocks) of the image. The in-loop filters can reduce possible encoding / compression artifacts that can have been introduced by the encoding / compression process.

[0051] The encoded compressed bitstream 138 can include any information needed to reconstruct the portion (e.g., block) of the image being compressed and encoded, such as partition information, indications of the selected prediction modes and their related information, e.g., motion vectors and reference indices, types of transform(s) used, quantization parameters, and any remaining quantized coefficients, etc.

[0052] To some extent, the decoder follows the inverse process, first performing entropy decoding on the compressed bitstream to derive the encoding information (partitions, modes, motion vectors / mvs, quantized coefficients, etc.), followed by the prediction process. The inverse quantization and transform steps can be performed in parallel, and then the reconstructed residuals are added on top of the prediction information. Given the appropriate in-loop filtering mechanisms supported and used by the underlying codec, the reconstructed block can then be in-loop filtered.

[0053] While the above-described methods work well for most content, in some cases these methods can not perform well, which can result in poor encoding performance and severe encoding artifacts. This is because the described methods tend to assume that the signal is easily predictable and relatively noise-free or smooth. However, common image and video data can contain noise. Often, such noise is also not temporally correlated and can vary greatly from frame to frame. There can be different types of noise related to different components of the capture and image acquisition process, including but not limited to Gaussian and shot noise, film grain, salt and pepper noise, and quantization noise that can be a product of early compression processes. Noise can also be added as an “artistic” component during the content creation process, or can reduce artifacts such as banding that can already be present in the source, or can be introduced into the content due to additional processing steps such as bit-depth reduction or compression. The amount and characteristics of the noise signal in an image can vary from region to region, and can depend on multiple parameters, including intensity, motion, color, etc.

[0054] Some systems attempt to handle noise in image or video content by first performing a denoising step on the actual source material, then compressing / encoding the denoised content. The removed noise is also modeled in the encoder, then certain parameters that can allow the decoder to synthesize a similar noise pattern are signaled in the bitstream. The film grain modeling supplemental enhancement information (SEI) message supports such a model in both AVC and HEVC, while AV1 also has a similar process for modeling and adding noise to the signal after the decoding process. However, these models add noise outside the encoding loop. Furthermore, in these approaches, their parameters are global (i.e., at sequence or frame level), and no control parameters can be signaled at a local (e.g., block) level.

[0055] In these systems, two images, for example, with different noise characteristics are combined into a single image, for example, a montage, and subsequently encoding them as a single image would imply that different noise models cannot be considered for the regions corresponding to the two different sub-images. The noise is designed to resemble the overall characteristics of the noise in the entire image, and cannot be adjusted locally. In images with different degrees of noise, this can result in inaccurate noise modeling, leading to images that can be subjectively dissimilar to the original images before denoising.

[0056] Exemplary compression process with out-of-ring noise modeling

[0057] For example, Figure 2 An encoder 200 is shown that includes a compression loop similar to the compression loop 100. However, the encoder 200 includes out-of-loop noise generation via a noise modeling module 206 that is external to the compression loop 100. For example, the input video data 130 is filtered via a denoiser 202 before entering the compression loop 100. Additionally, the filtered video data is compared to the input video data 130 at a noise estimator 204 (out-of-loop) to determine frame-level noise parameters, such as which noise model to use to model the removed noise and which parameters to use in the selected model. This information is provided to the entropy encoding unit 114 to be included in the encoded bitstream 138. However, as previously mentioned, this frame-level information does not allow for sub-image portion noise generation control, and is added to the full frame after the frame has been reconstructed, rather than to the sub-image portion before the image frame is reconstructed.

[0058] In some encoding devices, also including a display, such as the display 126, the noise modeling module 206 can generate a noise frame that includes a frame-level noise pattern, and add the frame-level noise pattern to the reconstructed image frame from the decoded picture buffer 124 before providing the reconstructed image frame for rendering on the display 126. In some embodiments, the noise modeling module 206 can also receive (not shown) information about previous image frames from the decoded picture buffer 124 as input to the noise model.

[0059] Exemplary decompression process / decoder with in-ring noise modeling

[0060] Figure 3 An exemplary decoder including out-of-loop noise modeling is shown. Entropy decoding module 302 receives and entropy decodes the encoded bitstream 320. Inverse quantization module 306 and inverse transform 308 process the decoded residual values 322 (which can be similar to the residual values determined at the encoder that have been transformed and quantized such as via transform module 110 and quantization module 112) to generate dequantized, de-transformed residual values 324. Additionally, information 328 (e.g., initial pixel values, motion vectors, etc.) is provided to the motion compensated intra / prediction module 316. Information 328 can include an indication as to whether the image was compressed via intra-prediction or inter-prediction.

[0061] If compressed via intra-prediction, information 328 can include at least one or more initial pixel values for at least one pixel of the compressed image. The at least one or more pixel values can be starting values for predicting pixel values of other pixels via an intra-prediction process. Alternatively, if the image was compressed via inter-prediction, information 328 can include motion vectors (mvs). Motion compensated / intra-prediction module 316 can generate predicted pixel values 326, and predicted pixel values 326 can be combined with dequantized, de-transformed residual values 324 to produce decompressed pixel values 330. The decompressed pixel values 330 can be further filtered by one or more loop filters, such as deblocking filter 310 and sample adaptive offset (SAO) filter 312. The filtered decompressed pixel values are reconstructed into a decompressed image frame and stored in the decoded picture buffer 314. In some embodiments, the decompressed image frames from the decoded picture buffer 314 can be used by the motion compensated / intra-prediction module 316 to estimate motion between blocks (or other portions) of successive image frames.

[0062] The above-described decompression process can be similar to the various decoders described herein. However, Figure 3 Out-of-loop noise modeling is also shown, where decoded data 332 indicating a noise model / noise parameters used to generate a frame-level noise pattern is provided to the noise modeling module 304, which generates a noise pattern for the entire image frame outside of the decompression loop 300. At 336, the image frame-level noise is added to the reconstructed image frame from the decoded picture buffer 314 before being provided to be rendered on the display 318.

[0063] In-loop sub-picture portion level noise modeling

[0064] In some embodiments, unlike previous systems that remove noise from the original source and perform noise modeling outside the encoding loop, the compression / decompression process with in-loop noise generation enables adding noise within the image / video encoding loop. Such additions can also be done and controlled at the sub-image portion level (also referred to herein as block level), e.g., macroblock, coding tree unit (CTU), coding unit, transform unit, or other block coding unit supported by the encoding technique.

[0065] In particular, an additional coding block can be introduced inside a video encoder and thus a video decoder with the purpose of modeling and generating a "controlled" noise pattern that is added to the signal during reconstruction of an image block.

[0066] For example, Figure 4 A compression loop 400 is shown that is similar to the compression loop 100, except that the compression loop 400 includes optional noise reducers 404, 406, 408, and 410 and an in-loop noise modeling unit 402. The input video 130 is optionally denoised at the noise reducer 404 before being used to determine the residual values 132. In addition, the residual values 132 can be denoised via the noise reducer 406 before being transformed and quantized. Further, the uncompressed image frames from the decoded picture buffer 124 and the decompressed image frames can be denoised via the optional noise reducers 408 and 410 before being used by the motion estimation module 108.

[0067] However, unlike the noise modeling module 206 shown in Figure 2 the compression loop 100, the noise modeling module 402 is included as part of the compression loop 400, where the generated partial level noise pattern 412 is added to the predicted pixel values 140 of the image portion along with the inverse quantized, inverse transformed residual values of the image portion.

[0068] As another example, Figure 5 A decompression loop 500 is shown that is similar to the decompression loop 300, except that the decompression loop 500 includes a noise modeling unit 502 within the decompression loop so that noise can be added at the sub-image portion level granularity.

[0069] Unlike the compression loop and the decompression loop shown in Figure 4 and Figure 5 the noise generated in Figure 2 and Figure 3 is generated using frame level parameters and added outside the encoding loop. In contrast, in Figure 4 the compression loop 400 shown in Figure 5The noise generated in the decompression loop 500 shown is generated for each block or any other coding unit that the system can support and added to it. In both the compression loop 400 and the decompression loop 500, the addition and characteristics of the noise are controlled entirely by parameters signaled in the bitstream, where a signal can be signaled for each such coding unit. For example, the noise generation parameters can be signaled in a similar manner to how transform information is associated with and provided for each coding unit.

[0070] In particular, the compression loop 400 and the decompression loop 500 allow for the addition of “controlled noise” to each coding unit. Various parameters can be signaled that control the presence, type, and characteristics of the noise to be added. The coding units can be existing units such as CTUs, prediction units, transform units, and the like, or newly defined independent units that can be specified and controlled through high-level syntax elements similar to how these other units are controlled. For example, a CTU, in a similar manner to how it is partitioned into transform units, can also be partitioned into such units for noise addition. The partitioning can be done in a hierarchical manner. For example, tree-based methods can be used to partition, for example, quad-trees, binary-trees, ternary-trees, and the like. The maximum depth of such trees can be controlled through higher-level syntax elements, for example, at the sequence parameter set (SPS), picture parameter set (PPS), or slice header. Such units can also be larger than a CTU and span multiple CTUs to ensure better mixing of the noise across multiple blocks. Such units can be referred to as noise portion units.

[0071] For example, Figure 12 A portion of an image 1202 including multiple blocks 1204 is shown, and Figure 13 Noise block or region units 1302, 1304, 1306, and 1308 are shown, where each noise block or region unit overlaps with multiple blocks in the blocks 1204.

[0072] The in-loop noise generation process discussed herein can be applied to any color domain, e.g., YCbCr or RGB signals. Moreover, the in-loop noise generation process discussed herein can operate on a single plane, e.g., only luminance (Y), or all planes, with the control being the same for all planes or different for each color plane (e.g., color component of a color space). For example, noise can be added only on the luminance component, while no noise is added on the chroma planes, or noise can be modeled based on the luminance component and then the other color planes are scaled or adjusted appropriately (e.g., data channels reconstructing luminance vs. data channels reconstructing chroma before any noise is added; such values can be used to determine scaling factors for noise added in the chroma planes, e.g., based on the ratio of these two quantities) based on a fixed relationship or based on a relationship of luminance and chroma values before any noise is added. Different controls can also be applied for different types of content, e.g., whether the content is standard or high dynamic range content, video or full range, with standard / limited color gamut or wide color gamut, whether the content is categorized as natural or synthetic content, whether the content is regular content or point cloud projection content (e.g., in the context of a point cloud system based on the Video Point Cloud Compression (V-PCC) compression technology), etc.

[0073] For each noise block or region unit (e.g., block, CTU, etc.), the following can be signaled in the bitstream:

[0074] • Presence of noise. If there is no noise, there is no need to send additional parameters. This can be a flag or a parameter that also shows the type of noise to be added (as described in the next bullet).

[0075] • If there is noise, the type of noise to be added, and possibly the noise model used to generate the type of noise. For example, the noise to be added can follow a Gaussian or Laplacian model, generalized Gaussian, uniform, Gamma, Bernoulli noise, etc. All these noise models can be associated with a syntax element, e.g., block_noise_model. For example, if this parameter is set to 0, no noise is added. If the parameter is set to 1, a Gaussian model of noise will be used, if set to 2, a uniform noise model will be used, and so on.

[0076] • Depending on the noise model, additional parameters can be transmitted. For example, for a Gaussian model, the mean value of the noise and its standard deviation can be signaled. For noise models following a generalized power distribution model parameters such as alpha and beta parameters, in addition to the mean value, other things can be signaled. Another model can utilize a transform domain, e.g., a Discrete Cosine Transform (DCT) based noise generation process, while another model can use a recursive based model. Noise can also be generated using a properly defined polynomial / Linear Feedback Shift Register (LFSR) as well. LFSR methods are commonly used to generate uniform distribution and pseudo-random Gaussian distribution noise models. For such polynomials, the order and coefficients of the polynomial and the initial (seed value) required for the polynomial can be signaled. Such information can also be shared across neighboring blocks. In this case, the filter parameters can also be predicted from their neighborhood to reduce the overhead. For example, for a block, the noise to be added can be related to the noise added in one of its neighboring parts, e.g., the block to the left or the top. In this case, additional parameters can not need to be signaled or the parameters signaled for the current block can be "predicted" from the parameters of its neighboring parts. The most likely noise model can also be used based on the neighborhood to further reduce the overhead. The order of the locations of the noise can also be signaled or predetermined. For example, a raster scan or zigzag scan order can be used for the generation and addition of the noise. The noise block can also be considered as a "noise transform block" similar to a DCT or other transform block that generates a noise pattern in its appropriate designated domain and "transform model" (e.g., LFSR model). The same model can be used in the decoder and run in the same way, e.g., both the encoder and decoder noise models are included in the noise modeling module of the encoder and the noise modeling module of the decoder.

[0077] In some embodiments, the values of the noise to be added can be agnostic and independent of the signal to be added to. In alternative embodiments, the noise can also depend on the characteristics of the block. Such characteristics can include the overall data channel or pixels of its reconstruction (e.g., the noise to be added is modulated by the data channel or the current component value), based on the characteristics of the residual signal (again modulated by the value of each residual pixel; in this case, if the residual is 0, it can be desirable to perform the modulation using a "residual + fixed offset" value), or motion (e.g., low motion vs. high motion blocks can require different noise to be added. This can be done by controlling the noise generation and weighting of the motion vectors in the bitstream). Furthermore, different noise can be added for skip, intra, and affine mode blocks, etc. Such weights can also be adjusted depending on the color plane being processed (e.g., for luma vs. chroma components, different noise can be generated depending on the mode, motion, DC value of the block, etc., or such noise control parameters can be determined based on luma only to reduce complexity.

[0078] The noise to be added is determined at the encoder based on the characteristics of the source. Specifically, the source can be analyzed and the noise in the signal, if any, determined and then modeled. The encoder can do its best to recreate the noise close to its original characteristics, but can also just approximate it. This can be done using a variety of methods, such as using a least mean square estimation method, Wiener noise estimation, etc.

[0079] In some embodiments, the encoder can choose to denoise the content prior to performing this process, as it knows that it can resynthesize the noise as part of the encoding loop. This process is entirely optional. The denoising can occur directly on the source, e.g., in the image domain, or in the residual domain after any prediction has been performed. Such denoising in the prediction domain can also remove noise introduced by the prediction process (e.g., due to noisy references). The denoising can also be performed in both domains in a cascaded fashion. Such a step is clearly non-normative and is intended to reduce the energy of the residual signal that will be transformed and quantized. Any denoising method, including Wiener filtering, bilateral filtering, motion-compensated spatio-temporal filtering, wavelet or bandlet / edgelet / ridgelet filtering, Gaussian filtering, etc., or other commonly used filtering methods can be used for such processing.

[0080] Additional non-normative denoising can be applied via optional denoisers 408 and 410, as shown in Figure 4 For example, in the motion estimation process, optional denoising can be applied to both the source signal and the reference stored in the decoded picture buffer 124. This can help the motion estimation determine the best prediction of the actual signal in the scene (e.g., find the "actual / true" motion of the content) rather than noise, which is typically irrelevant. Any denoising method, including Wiener filtering, bilateral filtering, motion-compensated spatio-temporal filtering, wavelet or bandlet / edgelet / ridgelet filtering, Gaussian filtering, or other commonly used filtering methods can be used for such processing. In this case, the motion compensation and final prediction will use unfiltered data.

[0081] The noise can be added immediately or concurrently with adding the prediction and reconstructing (after inverse quantization and transform) the residual values. For example, the noise pattern can be added after the inverse transform 118, as shown in Figure 4However, this means that any subsequent processing steps, such as deblocking, Sample Adaptive Offset filtering (SAO), Adaptive Loop Filter (ALF), or any other in-loop filtering supported by the codec, will be applied to the signal that is the combination of the prediction, the reconstructed residual, and the reconstructed noise. This can be desirable under certain architectures, in which case it can also be desirable to control such filtering mechanisms based on adding noise to the current block and its neighboring portions. In this case, the characteristics of such in-loop filtering mechanisms can be altered so as to a) take the noise into account, b) ensure that they preserve the noise characteristics, and / or c) ensure that the noise mixes appropriately across block partitions.

[0082] For example, in existing deblocking schemes, the filtering strength is typically determined based on the similarity of pixels and block edges. Only the rows or columns to be filtered in these cases are evaluated. Since noise is added, it is preferable not to consider individual columns or rows, but to perform the evaluation of deblocking strength in a manner that considers multiple columns and rows. Effectively, the locations to be deblocked can be "subtly" and in-place denoised prior to deblocking strength determination. The strength determination can then be performed on such denoised pixels. Note that such denoising is only used for determining the deblocking strength, and the denoised values are not further used in the reconstruction process. Similar considerations can be made for other types of filters, such as SAO, ALF, in-loop bilateral filtering, etc. For example, in the case of SAO, the determination of the band or edge filter currently defined in HEVC also takes into account a similar denoising approach for the classification process. Similar considerations can also be made if other modes of SAO are to be used, such as the texture mode, the stripe detection, etc.

[0083] In some embodiments, intra prediction can be performed with or without added noise. This can be predetermined (no signaling) at the encoder and the decoder, or the decision can also be signaled in the bitstream. The signaling can be at a high level syntax structure, e.g., SPS, PPS, or slice header, or at a lower level, e.g., CTU or noise block level.

[0084] In some embodiments, the compression / decompression loop can function in an alternative mode in which noise is added downstream of the in-loop filter. For example, Figure 6 An encoder is shown in which noise is added downstream of the SAO filter 122. As another example, Figure 7 A decoder is shown in which the noise modeling module 702 is in the decompression loop, and noise is being added downstream of the SAO filter 312.

[0085] In some embodiments, it can be desirable to add a noise block after the in-loop filtering step. This can avoid filtering the added noise, which in some cases can lead to other types of artifacts. The final signal is still added to the reference buffer for future prediction. However, unlike the case before where intra prediction can be done with or without a noise model, the step at the in-loop filter stage is typically exempted from intra prediction for complexity and storage purposes. Therefore, it is proposed that the noise block is also excluded from any intra prediction (including neighboring block directional prediction and intra block copy prediction) stage in this case. However, the noise is still taken into account for inter / temporal prediction.

[0086] In some cases, the added noise can cause harm to inter prediction. This is because the system has to compensate not only for the inaccuracy of the prediction but also for the uncorrelated noise. The use of denoising methods, such as bilateral, Gaussian, Wiener or other filtering schemes, can be part of the prediction loop and remove the noise from the predicted signal. Such denoising methods are signaled and controlled with such signals in both the encoder and the decoder. For example, Figure 8 An encoder is shown that includes a controlled noise reduction module 804, and Figure 9 A decoder is shown that includes a controlled noise reduction module 904. Figure 10 A decoder is also shown that includes a controlled denoising module 1004 that removes noise from reconstructed images used by the motion compensation and intra prediction module 316.

[0087] In some embodiments, the signaling of such controlled denoising methods can now also be based on the type and characteristics of the added noise. Since both the encoder and the decoder know this information, the denoising can be adjusted accordingly according to these characteristics, for example, selecting the most suitable mode to remove the noise without needing to signal all the parameters needed to denoise “unknown” according to the characteristics of the noise. For example, if a certain type of noise (e.g., Gaussian noise with mean X) is added on a particular reference, then a bilateral filter can be preferred for that reference. For different references with different noise characteristics, a Gaussian or Wiener filter can be preferred. Instead of making an implicit decision on the filter, the most likely filter mode can also be selected according to such a mechanism, and then the index of the final filter to be used is encoded. This would then help to reduce the encoding overhead of the denoising processing filter.

[0088] In some embodiments, in-loop noise generation and addition can be combined with out-of-loop noise generation and addition, as shown in Figure 2 and Figure 3 This can be helpful in cases where it can be desirable to add noise on the reconstructed signal but avoid the noise appearing in the prediction loop. There are multiple ways to do this, for example:

[0089] • Switching noise patterns on a higher level structure, e.g. sequence, picture, tile or slice. For example, the noise can be only out-of-loop, or only in-loop.

[0090] • Allowing both modes to exist even at block level, but only one of them at a time. In this case, the encoder and decoder need to maintain a map of blocks that have added in-loop noise and later will add out-of-loop noise (e.g. for display purposes).

[0091] • Allowing both types of noise to exist, but also controlling at block level at the same time. In this case, in-loop noise can try to address certain types of artifacts, and can not need to have a significant energy to make sure that the noise does not impact inter-prediction too much. On the other hand, out-of-loop noise can be more pronounced. In this case, the noise model of in-loop and out-of-loop blocks needs to be considered jointly. In terms of in-loop noise addition, the out-of-loop noise can be "blind", i.e. not requiring metadata to be fed to the block to add noise, thus reducing complexity, however for some systems, the in-loop noise can also need to be considered for the adjustment and characteristics of the out-of-loop noise adder, e.g. where it is added and what characteristics it has.

[0092] • While the encoding module is based on pixel comparison operations, such as sum of absolute difference (SAD), sum of absolute transformed difference (SATD), sum of squared error (SSE), etc. can be used for many encoding decisions by the proposed encoder (as done by many existing encoders), it is preferable to utilize other matching criteria that can be more "noise" agnostic / robust. Using a modified SSIM approach that emphasizes the mean over the variance of the signal, this can prove to be more robust as a matching / decision criterion in the presence of noise, as well as transform domain criteria that emphasize the DC value over the lower (and more noise prone) AC coefficients in the context of a rate-distortion optimization model based on a Lagrangian parameter, the lambda parameter can also be adjusted appropriately depending on the distortion model and characteristics of the currently encoded block (motion, noise, DC, noiseless variance, etc.).

[0093] • Different decision rules can be applied to different color components or different content, e.g. SDR vs. HDR, which can be determined by the encoder, application or user. For example, the encoder can use different distortion criteria for chroma and luma.

[0094] Exemplary process to generate in-ring noise patterns

[0095] Figures 11A-11B An exemplary process for generating in-loop noise patterns is shown in accordance with some embodiments. Step 1102 is a step for determining block level noise as part of an in-loop compression process and includes sub-steps 1104-1122.

[0096] At 1104, a first image or a next image to be compressed is received. At 1106, a first portion (e.g., a block, a CTU, a noise block, or a region unit, etc.) of the first portion (or the next image) is selected to evaluate noise.

[0097] At 1106, a first noise pattern to be generated for the first portion is determined. The first noise pattern can be selected to approximate noise in the first portion of the image. At 1110, the first noise pattern is generated using a noise model and / or noise parameters determined at 1108.

[0098] At 1112, an additional (or next additional) portion (e.g., a block, a CTU, a noise block, or a region unit, etc.) of the first (or next image) is selected to evaluate noise.

[0099] At 1114, a noise pattern to be generated for the additional (or next additional) portion is determined. The noise pattern can be selected to approximate noise in the additional portion of the image. At 1116, the noise pattern is generated using a noise model and / or noise parameters determined at 1114.

[0100] At 1118, it is determined whether there are additional portions (e.g., a block, a CTU, a noise block, or a region unit, etc.) of the first (or next image) to be evaluated for noise. If so, the process continues at 1112 to compress the next additional portion of the first (or next) image. If there are no additional portions to evaluate for the image currently being evaluated, then at 1120, it is determined whether there are additional images to be evaluated. If so, the process continues at 1104 to evaluate the next image.

[0101] At 1124, predicted pixel values are determined for the current portion (e.g., a block, a CTU, a noise block, or a region unit, etc.) of the first (or next image). At 1126, the predicted pixel values are compared to uncompressed or original pixel values to determine residual values.

[0102] At 1128, the residual values and noise model and / or noise parameters determined at 1108 or 1114 for the given block being evaluated are encoded in the encoded bitstream. Note that noise generation and pixel value prediction and residual determination can be performed block-by-block or region-by-region.

[0103] Exemplary controlled de-noising process

[0104] Figure 14 An exemplary controlled denoising process is shown in accordance with some embodiments.

[0105] At 1402, a noise model for generating a noise pattern is selected based on various input parameters as described herein. At 1404, additional parameters that are input to the selected model are provided as input to the model selected at 1402. At 1406, a noise pattern is generated for a first (or next portion) of the image using the selected model and the selected input parameters.

[0106] Additionally, the selected model and the selected input are provided to a controlled denoising module, and at 1412, the controlled denoising module selects an in-loop filter for controlled denoising based on the noise model and input parameters selected at 1402 and 1404. At 1414, the controlled denoising module performs denoising (e.g., filtering) on the portion of the image that was noise generated at 1406.

[0107] At 1408, it is determined whether there are additional portions of the image to be evaluated; if so, the process returns to 1402. If not, the process continues to evaluate the next image.

[0108] Exemplary decoding process with in-ring sub-picture portion level noise generation

[0109] Figure 15 An exemplary process for generating in-loop noise patterns during decompression is shown in accordance with some embodiments.

[0110] At 1502, a decoder receives an encoded bitstream for a compressed image of a video, where the bitstream includes data indicating a noise model and / or noise parameters used to generate in-loop noise at an encoder.

[0111] At 1504, the decoder decodes the compressed bitstream. This can include decoding entropy encoded residual values and decoding encoded data indicating a noise model and / or noise parameters used to generate a noise pattern added in-loop during compression at the encoder.

[0112] At 1506, the decoder generates a noise pattern for a portion of the image being decompressed based on the noise model and noise parameters indicated for the respective portion (e.g., block, CTU, noise block, or region unit, etc.) included in the compressed bitstream. Further, at 1510, the decoder predicts pixel values for the portion (e.g., block, CTU, noise block, or region unit, etc.), and at 1512, applies the residual values to the predicted values to generate adjusted pixel values for the portion of the image.

[0113] At 1508, the generated noise pattern is added to the adjusted pixel values of the image portion. At 1516, a decompressed version of the compressed image portion including the generated noise pattern is provided. The decompressed image portion can be combined with other decompressed image portions to reconstruct a decompressed version of the compressed image. The decompressed version of the compressed image can be added to a decompressed picture buffer and can be used for motion estimation and / or provided for rendering at a display, such as a portion of a decompressed video.

[0114] Exemplary computer system

[0115] Figure 16 An example computer system 1600 that can implement an encoder or a decoder or any other component described herein (e.g., any of the components described above with reference to Figures 1-15 The computer system 1600 can be configured to perform any or all of the embodiments described above. In different embodiments, the computer system 1600 can be any of a variety of types of devices including, but not limited to: a personal computer system, desktop computer, laptop, notebook, tablet, all-in-one, slate, or netbook computer; a mainframe computer system; a mobile device; a consumer device; a video game console; a handheld video game device; an application server; a storage device; a television; a video recording device; a peripheral device such as a switch, modem, router; or any other type of computing or electronic device.

[0116] Various embodiments of the encoder or decoder described herein can be performed on one or more computer systems 1600, which can interact with various other devices. Note that according to various embodiments, any components, actions, or functionality described above with respect to Figures 1-15 any of the components, actions, or functionality described above can be implemented on a computer system configured to Figure 16one or more computers of the computer system 1600. In the illustrated embodiment, the computer system 1600 includes one or more processors 1610 coupled to a system memory 1620 via an input / output (I / O) interface 1630. The computer system 1600 further includes a network interface 1640 coupled to the I / O interface 1630, and one or more input / output devices 1650, such as a cursor control device 1660, a keyboard 1670, and one or more displays 1680. In some cases, it is contemplated that embodiments can be implemented using a single instance of computer system 1600 while in other embodiments multiple such systems, or multiple nodes making up a computer system 1600, can be configured to host different portions or instances of embodiments. For example, in one embodiment, some elements can be implemented via one or more nodes of computer system 1600 that are distinct from those nodes used to implement other elements.

[0117] In various embodiments, the computer system 1600 can be a uniprocessor system including one processor 1610, or a multiprocessor system including several processors 1610 (e.g., two, four, eight, or another suitable number). Processors 1610 can be any suitable processors capable of executing instructions. For example, in various embodiments, processors 1610 can be general- purpose or embedded processors implementing any of a variety of instruction set architectures (ISAs), such as the x86, PowerPC, SPARC, or MIPS ISAs, or any other suitable ISA. In multiprocessor systems, each of the processors 1610 can typically, but is not required to, implement the same ISA.

[0118] The system memory 1620 can be configured to store point cloud compression or point cloud decompression program instructions 1622 and / or sensor data accessible by the processors 1610. In various embodiments, the system memory 1620 can be implemented using any suitable memory technology, such as static random access memory (SRAM), synchronous dynamic RAM (SDRAM), nonvolatile / Flash-type memory, or any other type of memory. In the illustrated embodiment, the program instructions 1622 can be configured to implement an image sensor control application in conjunction with any of the above-described functionality. In some embodiments, the program instructions and / or data can be received, sent or stored upon different types of computer-accessible media or similar media separate from system memory 1620 or computer system 1600. While computer system 1600 is described as implementing the functionality of the functional blocks of the previous figures, any of the functionality described herein can be implemented via such a computer system.

[0119] In one embodiment, I / O interface 1630 can be configured to coordinate I / O communications between processor 1610, system memory 1620, and any peripheral devices in the device (including network interface 1640 or other peripheral device interfaces, such as input / output devices 1650). In some embodiments, I / O interface 1630 can perform any necessary protocol, timing, or other data conversion to convert data signals from one component (e.g., system memory 1620) into a format suitable for use by another component (e.g., processor 1610). In some embodiments, I / O interface 1630 can include support for devices attached, for example, via various types of peripheral buses (such as a variant of the Peripheral Component Interconnect (PCI) bus standard or the Universal Serial Bus (USB) standard). In some embodiments, the functionality of I / O interface 1630 can be divided into two or more separate components, such as a north bridge and a south bridge, for example. Furthermore, in some embodiments, some or all of the functionality of I / O interface 1630 (such as the interface to system memory 1620) can be incorporated directly into processor 1610.

[0120] The network interface 1640 can be configured to allow data to be exchanged between the computer system 1600 and other devices (e.g., carriers or proxy devices) attached to the network 1685, or between nodes of the computer system 1600. In various embodiments, the network 1685 can include one or more networks, including but not limited to a local area network (LAN) (e.g., an Ethernet or an intranet), a wide area network (WAN) (e.g., the Internet), a wireless data network, some other electronic data network, or some combination thereof. In various embodiments, the network interface 1640 can support communication via a wired or wireless general data network (such as any suitable type of Ethernet network), for example; via a telecommunications / telephone network (such as an analog voice network or a digital fiber optic communication network); via a storage area network (such as a Fibre Channel SAN), or via any other suitable type of network and / or protocol.

[0121] In some embodiments, input / output devices 1650 may include one or more display terminals, keyboards, keypads, touchpads, scanning devices, voice or optical recognition devices, or any other device suitable for inputting or accessing data by one or more computer systems 1600. Multiple input / output devices 1650 may be present in computer system 1600 or may be distributed across various nodes of computer system 1600. In some embodiments, similar input / output devices may be separate from computer system 1600 and may interact with one or more nodes of computer system 1600 through a wired or wireless connection, such as through network interface 1640.

[0122] likeFigure 16 As shown, memory 1620 can include program instructions 1622, which can be executable by the processor, to implement any of the elements or acts described above. In one embodiment, the program instructions can perform the methods described above. In other embodiments, different elements and data can be included. Note that the data can include any of the data or information described above.

[0123] Those skilled in the art will appreciate that the computer system 1600 is merely illustrative and is not intended to limit the scope of embodiments. In particular, the computer system and devices can include any combination of hardware or software that can perform the indicated functions, including computers, network devices, internet appliances, PDAs, wireless telephones, pagers, etc. The computer system 1600 can also be connected to any other devices in the network 1699 or memories in the memory 1620 that are not explicitly shown, or can operate as a stand-alone system. In addition, the functionality provided by the illustrated components can in some embodiments be combined in fewer components or distributed in additional components. Similarly, in some embodiments, the functionality of some of the illustrated components can not be provided and / or other additional functionality can be available.

[0124] Those skilled in the art will further appreciate that, while the various items are illustrated in the drawings and described above as separate items, these items or portions of them can be combined or integrated into other items. In some embodiments, the functions described herein can be implemented in hardware and / or in software (including firmware, resident software, micro-code, etc.) that can be executed by a processor, such as the processor 1610. As such, the terms "function" and "module" as used herein can refer to hardware, software, firmware, or any combination thereof. In some embodiments, the functions described herein can be implemented in one or more computer programs that are executable on various systems. The computer programs can be stored in any computer readable medium, all of which can be accessed by a general purpose or special purpose computer. By way of example, a computer readable medium can comprise RAM, ROM, EEPROM, CD-ROM, or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other suitable medium typically used for storing computer program software. The computer readable medium can be transitory or non-transitory.

[0125] In different embodiments, the methods described herein can be implemented in software, hardware, or a combination thereof. Moreover, the order of the blocks can be changed, and various elements can be added, reordered, combined, omitted, modified, etc. Various modifications and changes can be made as would be obvious to a person skilled in the art having the benefit of this disclosure. The various embodiments described herein are intended to be illustrative and not restrictive. Many variations, modifications, additions, and improvements are possible. Accordingly, plural instances can be provided for components described herein as a single instance. Boundaries between various components, operations and data stores are somewhat arbitrary, and particular operations can be performed by a single component, in several components, or across several components. Other implementations are within the scope of the claims. Finally, structures and functionality presented as discrete components in example configurations can be implemented as a combined structure or component. These and other variations, modifications, additions, and improvements can fall within the scope of embodiments as defined in the claims that follow.

Claims

1. A system configured to encode video image data, the system comprising: one or more processors; and one or more memories storing program instructions that, when executed on or across the one or more processors, cause the one or more processors to: compress pixel value data for a plurality of images included in a plurality of frames of a video, wherein to compress the pixel value data, the program instructions cause the one or more processors to: determine a first noise pattern to be generated for a first portion of a given one of the images; determine a second noise pattern to be generated for a second portion of the same given one of the images; generate the first noise pattern and the second noise pattern at an encoder and via an in-loop noise generator included in a compression loop used to compress the pixel value data, wherein the first noise pattern and the second noise pattern are different; and encode the compressed pixel value data for the plurality of images, wherein an encoded bitstream for the compressed pixel value data includes data indicative of one or more noise parameters or noise models used to generate the first noise pattern and the second noise pattern.

2. The system of claim 1, wherein the first portion of a given image corresponds to a block of the given image and the second portion corresponds to another block of the given image, wherein the program instructions, when executed on or across the one or more processors, implement in-loop block-level control of noise generation in the compression loop.

3. The system of claim 1 or 2, wherein the first noise pattern and the second noise pattern are added to prediction residuals used as part of an in-loop prediction compression process.

4. The system of claim 1 or 2, wherein the first noise pattern and the second noise pattern are added to prediction residuals used in the compression loop for inter-frame prediction between a given image and another image frame, wherein the inter-frame prediction is based at least in part on estimated motion between image frames.

5. The system of claim 1 or 2, wherein the first noise pattern and the second noise pattern are added to the prediction residuals used in the compression loop as part of an in-loop inter-frame prediction or intra-frame prediction process, prior to filtering of the prediction residuals via one or more in-loop filters of the compression loop.

6. The system of claim 1 or 2, wherein the first noise pattern and the second noise pattern are added to the prediction residuals used in the compression loop as part of an inter-frame prediction or intra-frame prediction process, after the prediction residuals have been filtered via one or more in-loop filters of the compression loop. ​ ​ 7. The system of claim 1 or 2, wherein the program instructions, when executed on or across the one or more processors, implement in-loop noise generation control that enables noise generation control based on whether a pixel value of an image is predicted via intra-frame prediction processing or inter-frame prediction processing.

8. The system of claim 1 or 2, wherein the program instructions, when executed on or across the one or more processors: generate the first noise pattern according to a first noise model, and generate the second noise pattern according to a second noise model, wherein the first noise model and the second noise model are selected from a plurality of noise models supported by the system.

9. The system of claim 8, wherein the first noise model and the second noise model are selected from a plurality of supported noise models supported by the system based at least in part on: an amount of motion associated with the first portion of the given image; a motion amount associated with the second portion of the given image; and wherein the first portion and the second portion of the given image have different associated motion amounts.

10. The system of claim 1 or 2, wherein the first portion or the second portion of a given image overlaps two or more prediction blocks of the given image, wherein prediction of pixel values of the pixel value data used to compress the given image is controlled at an image block level.

11. The system of claim 1 or 2, wherein the program instructions, when executed on or across the one or more processors: generate the first noise pattern and the second noise according to a same noise model using different input parameters associated with the first portion of a given image and the second portion of the given image.

12. The system of claim 1 or 2, wherein the program instructions, when executed on or across the one or more processors, cause the one or more processors to: predict pixel values of the plurality of images included in the plurality of frames of the video; determine residual values of the predicted pixel values based on a comparison of actual pixel values to predicted pixel values; and determine decompressed pixel values as part of the compression loop based on the predicted pixel values and the determined residual values, wherein the program instructions further cause the one or more processors to: filter noise from predicted pixel values such that noise filtered predicted pixel values are used with residual values and the generated first noise pattern and the generated second noise pattern to generate decompressed pixel values for a decompressed version of a given image as part of the compression loop, wherein the decompressed version of the given image is used in the compression loop to select or control a prediction mode of another image to be compressed.

13. The system of claim 1 or 2, wherein the program instructions, when executed on or across the one or more processors, cause the one or more processors to: ​ Noise is filtered from the decompressed version of the image used in the compression loop before the decompressed version of the image is used in a motion estimation process to determine estimated motion between image frames.

14. The system of claim 1 or 2, wherein the program instructions, when executed on or across the one or more processors, cause the one or more processors to: A filter model is selected to be used for filtering noise from predicted pixel values ​​used in the compression loop, wherein the filter model is selected based on one or more noise models selected to generate the first noise pattern and the second noise pattern.

15. The system of claim 1 or 2, wherein the program instructions, when executed on or across the one or more processors, cause the one or more processors to: Prior to compressing the pixel values ​​of the corresponding ones of the images, the pixel values ​​of the corresponding ones of the images are filtered for the corresponding ones of the images to remove noise.

16. The system of claim 1 or 2, wherein the program instructions, when executed on or across the one or more processors: applying the first noise pattern to a first color component of a color space of a given image; and applying the second noise pattern to a second color component of the color space of the given image, Such that the first noise pattern and the second noise pattern are applied to different color components of the given image.

17. The system of claim 16, wherein the first color component is a luma color component of a YCbCr color space, and the second color component is a chroma color component of a YCbCr color space.

18. The system of claim 16, wherein the first color component is a red, blue, or green component of an RGB color space, and the second color component is a different color component of the RGB color space.

19. A system configured to decode compressed video image data, the system comprising: one or more processors; as well as one or more memories storing program instructions that, when executed on or across the one or more processors, cause the one or more processors to: receiving an encoded bitstream of compressed images for a video, wherein the encoded bitstream includes data indicative of a noise model or noise parameters for generating a loop noise pattern at an encoder; generating, via an in-loop noise generator, a first noise pattern for a first portion of a decompressed version of the compressed image and a second noise pattern for a second portion of the decompressed version of the compressed image based on the data indicative of the noise model or noise parameters; adding the first noise pattern to predicted pixel values ​​corresponding to the first portion of the decompressed version of the compressed image; as well as The second noise pattern is added to predicted pixel values ​​corresponding to the second portion of the decompressed version of the compressed image.

20. The system of claim 19, wherein the first noise pattern and the second noise pattern are added to the predicted pixel values ​​at an image block level, wherein different noise pattern values ​​are added to different blocks of the decompressed version of the compressed image.

21. A system according to claim 19 or 20, wherein the first noise pattern and the second noise pattern are added to the predicted pixel value at a noise portion level, wherein each noise portion of the decompressed version of the compressed image overlaps with two or more blocks of the decompressed version of the compressed image, and wherein different noise pattern values ​​are added to different noise portions of the noise portions of the decompressed version of the compressed image.

22. The system of claim 19 or 20, wherein the first and second noise patterns are added to the predicted pixel values ​​before the predicted pixels are filtered by one or more in-loop filters of a decompression process.

23. A system according to claim 19 or 20, wherein the first noise pattern and the second noise pattern are added to adjusted predicted pixel values ​​that have been adjusted based on decoded residual pixel values ​​included in the bitstream, wherein the adjusted predicted pixel values ​​have been filtered by one or more in-loop filters of a decompression process before adding the first noise pattern and the second noise pattern.

24. The system of claim 19 or 20, wherein the program instructions, when executed on or across the one or more processors, cause the one or more processors to: predicting pixel values ​​of the image; Filter noise from predicted pixel values applying the residual value to the predicted pixel value to determine a decompressed pixel value, The noise filtered predicted pixel values ​​are used together with the residual pixel values ​​and the first generated noise pattern and the second generated noise pattern to generate a decoded version of the compressed image.

25. The system of claim 19 or 20, wherein the program instructions, when executed on or across the one or more processors, cause the one or more processors to: Before using the decompressed version of the other compressed image to determine estimated motion between image frames in a motion estimation process, noise is filtered from the decompressed version of the other compressed image included in adjacent frames of the video, wherein the noise is filtered from the decompressed version of the other compressed image using one or more filters selected based on a noise model, the noise model being selected to generate a first noise pattern and a second noise pattern for the decompressed version of the other compressed image.

26. A non-transitory computer-readable medium storing program instructions that, when executed by one or more processors, cause the one or more processors to: generating an encoded bitstream of an image, wherein the encoded bitstream comprises: compressed pixel value data of the image, the compressed pixel value data of the image being organized based on two or more corresponding noise data sets for a block of the image and a corresponding block used in a compression process for prediction, wherein the noise data indicates a noise model or one or more noise parameters used to generate two or more corresponding in-loop noise patterns for the block at an encoder.

27. A non-transitory computer readable medium storing program instructions that, when executed by one or more processors, cause the one or more processors to: decode an encoded bitstream of an image, wherein the encoded bitstream comprises: compressed pixel value data of the image, the compressed pixel value data of the image being organized based on two or more corresponding noise data sets for a block of the image and a corresponding block used in a compression process for prediction, wherein the noise data indicates a noise model or one or more noise parameters used to generate two or more corresponding in-loop noise patterns for the block at an encoder.

28. A mobile device, the mobile device comprising: a camera; a display; and the system of any one of claims 1 to 18.

29. A mobile device, the mobile device comprising: a display; and the system of any one of claims 19 to 25.

30. A method for image processing, comprising: operating the system of any one of claims 1 to 18 to compress a video image using loop-block level control of noise generation.

31. A method for image processing, comprising: operating the system of any one of claims 19 to 25 to decompress a compressed video bitstream using loop-block level control of noise generation.

32. A mobile device, the mobile device comprising: a camera; a display; and an encoder configured to generate the encoded bitstream of claim 26.

33. A mobile device, the mobile device comprising: a display; and a decoder configured to decode the encoded bitstream of claim 27.

34. A method for image processing, comprising: generating the encoded bitstream of claim 26.

35. A method for image processing, comprising: decoding the encoded bitstream of claim 27. ​ ​ ​ ​

Images