Video compression with in-loop sub-image level controllable noise generation

Abstract

The invention relates to video compression with in-loop sub-image level controllable noise generation. A system includes an encoder configured to compress video data using an in-loop noise generation process that generates noise in the compression loop at a sub-image portion level granularity, such as at a block level. The encoder includes a noise model and / or noise model input parameter information in an encoded bitstream. Further, a system includes a decoder configured to receive such a bitstream, and to decompress the video using an in-loop noise generation process that generates noise in the decompression loop at a granularity of a sub-image portion level.

Description

Related application citation

[0001] This application is a divisional application of the invention patent application with international application number PCT / US2020 / 052182, international application date of September 23, 2020, entry into the Chinese national phase date of March 22, 2022, Chinese national application number 202080066479.2, and invention title "Video Compression Using Intra-Ring Sub-Image Level Controllable Noise Generation".

[0002] This application claims priority and benefit to U.S. Provisional Patent Application No. 62 / 904,576, filed September 23, 2019, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure relates in general to compressing and decompressing video data using in-loop noise generation that is controllable at the image portion level. Background Technology

[0004] Various devices, including but not limited to personal computer systems, desktop computer systems, laptops and notebook computers, tablet computers or tablet devices, digital cameras, digital video recorders, and mobile phones or smartphones, may include software and / or hardware that implement video processing methods. For example, the device may include means (e.g., integrated circuits (ICs), such as system-on-a-chip (SoC), or subsystems of an IC) capable of receiving and processing digital video input from one or more sources and outputting processed video signals, such as pictures / frames or regions of pictures / frames, according to one or more video processing methods. As another example, software programs may be implemented on the device capable of receiving and processing digital video input from one or more sources and outputting processed video frames according to one or more video processing methods.

[0005] In some video processing methods, each input video image / frame is divided into rows and columns of pixel blocks (e.g., 16×16 pixel blocks) to perform processing. Each block of the input video image / frame can be processed individually, and the processed blocks are combined to form the output video frame when processing is complete. This can be called a block processing method.

[0006] Block processing methods may include multiple processing steps or operations that are sequentially applied to each block in a video image / frame. Summary of the Invention

[0007] In some implementations, the system for encoding video image data is configured to compress pixel value data of multiple images included in multiple frames of a video, wherein, as part of the compression, the system performs in-loop noise estimation and generation at the image portion level. Supported multi-level granularity may include macroblocks, coding tree units (CTUs), coding units (CUs), prediction units (PUs), transform units (TUs), noise blocks or region units (which may overlap with multiple macroblocks, CTUs, CUs, PUs, TUs, etc.), or other block coding units supported by the coding technology. In some implementations, the techniques described herein may be added as extensions to existing video coding standards (e.g., in new profiles) or may be included in new video coding standards such as MPEG VVC / H.266, AOM AV2, or other such coding standards or technologies.

[0008] To perform in-loop noise estimation and generation, the system is configured to determine a first noise pattern to be generated for a first portion of a given image in an image, determine a second noise pattern to be generated for a second portion of the same given image in an image, and generate the first and second noise patterns via an in-loop noise generator included in a compression loop for compressing pixel value data, wherein the first and second noise patterns are different. Furthermore, the system is configured to encode compressed pixel value data for multiple images, wherein the encoded bitstream for compressing the pixel value data includes data indicating a noise model or one or more noise parameters used to generate the first and second noise patterns. In some embodiments, the first and second portions of a given image may correspond to blocks, CTUs, CUs, PUs, noise blocks, or region units that overlap with multiple blocks or CTUs. The noise pattern may include a pattern of noise samples generated by one or more inputs used by the noise model to generate the noise pattern. The noise pattern generated at the image portion level may have the same size / shape as the portion (e.g., block, region, etc.) of the generated image.

[0009] In some implementations, a first noise pattern and a second (or additional) noise pattern can be added to the residual values ​​calculated in the compression loop, and these residual values ​​are also added to the predicted pixel values. In some implementations, the noise pattern can be combined with the predicted pixel values ​​already adjusted based on the residual values ​​at the block level before the adjusted predicted pixel values ​​(e.g., predicted pixel values ​​adjusted based on the residuals) are further filtered by one or more in-loop filters included in the compression loop, such as a deblocking filter, a sample adaptive offset (SAO) filter, an adaptive loop filter (ALF), a bilateral filter, or other in-loop filters. Conversely, in some implementations, the first noise pattern and the second (or additional) noise pattern can be added at the block level after the predicted pixel values ​​have been filtered by one or more in-loop filters included in the compression loop, such as a deblocking filter, a sample adaptive offset (SAO) filter, an adaptive loop filter (ALF), a bilateral filter, or other in-loop filters, to adjust the residuals. For example, each sample included in the noise pattern can be added to each corresponding sample in the block. This process can be repeated for each color component of multiple color planes. Furthermore, in some implementations, noise can be added to one or more color components without adding it to other color components.

[0010] In some implementations, a system for decoding coded bitstreams of compressed video data is configured to perform in-loop noise estimation and generation at the image portion level as part of decompressing the compressed video data. For example, the system is configured to receive a coded bitstream for compressing an image or video region, wherein the coded bitstream includes data indicating a noise model or noise parameters used to generate noise for two or more portions of a decompressed version of the compressed image or region. The system is further configured to generate a first noise pattern for a first portion of the portion and a second noise pattern for a second portion of the portion via an in-loop noise generator, based on the data indicating the noise model or noise parameters. Furthermore, the system is configured to add the first noise pattern to the predicted pixel values ​​of the first portion corresponding to the decompressed version of the compressed image, and to add the second noise pattern to the predicted pixel values ​​of the second portion corresponding to the decompressed version of the compressed image.

[0011] In some implementations, a first noise pattern and a second (or additional) noise pattern may be added to the residual values ​​calculated in the decompression loop, and these residual values ​​are also added to the predicted pixel values. In some implementations, the noise pattern may be combined with the predicted pixel values ​​that have already been adjusted based on the residual values ​​at the block level before the adjusted predicted pixel values ​​(e.g., predicted pixel values ​​adjusted based on the residuals) are further filtered by one or more in-loop filters included in the decompression loop, such as a deblocking filter, a sample adaptive offset (SAO) filter, an adaptive loop filter (ALF), a bilateral filter, or other in-loop filters. Conversely, in some implementations, the first noise pattern and the second (or additional) noise pattern may be added at the block level after the predicted pixel values ​​have been filtered by one or more in-loop filters included in the decompression loop, such as a deblocking filter, a sample adaptive offset (SAO) filter, an adaptive loop filter (ALF), a bilateral filter, or other in-loop filters, to adjust the residuals. Attached Figure Description

[0012] Figure 1 An exemplary video compression loop of an encoder according to some implementation schemes is shown.

[0013] Figure 2 An exemplary video encoder including out-of-loop frame-level noise generation is shown according to some implementations.

[0014] Figure 3 An exemplary video decoder including out-of-loop frame-level noise generation is shown according to some implementation schemes.

[0015] Figure 4 An exemplary video encoder, according to some embodiments, includes in-loop noise generation with sub-image portion granularity control.

[0016] Figure 5 An exemplary video decoder, according to some embodiments, includes in-loop noise generation with sub-image portion granularity control.

[0017] Figure 6 An exemplary video encoder, according to some embodiments, includes in-loop noise generation with sub-image portion granularity control, wherein the generated noise is added downstream of a loop filter.

[0018] Figure 7 An exemplary video decoder is shown according to some embodiments, including in-loop noise generation with sub-image portion granularity control, wherein the generated noise is added downstream of the loop filter.

[0019] Figure 8An exemplary video encoder according to some embodiments is shown, which includes in-loop noise generation controlled at the sub-image portion granularity and includes controlled denoising.

[0020] Figure 9 An exemplary video encoder is shown according to some embodiments, the exemplary video decoder including in-loop noise generation controlled at the sub-image portion granularity and including controlled denoising.

[0021] Figure 10 An exemplary video decoder according to some embodiments is shown, which includes in-loop noise generation controlled at the sub-image portion granularity and controlled denoising downstream of a loop filter.

[0022] Figures 11A to 11B The compression process with in-loop noise estimation and generation at the sub-image part level granularity is illustrated according to some embodiments.

[0023] Figure 12 An exemplary image frame divided into image blocks according to some implementation schemes is shown.

[0024] Figure 13 An exemplary image frame showing noise blocks or region units overlapping with image blocks, according to some embodiments, is shown.

[0025] Figure 14 An exemplary controlled denoising process according to some implementation schemes is shown.

[0026] Figure 15 An exemplary process for generating an in-loop noise pattern during decompression is shown according to some embodiments.

[0027] Figure 16 An example computer system that can implement an encoder or decoder according to some implementation schemes is shown.

[0028] This specification includes references to "one embodiment" or "implementation". The use of the phrase "in one embodiment" or "in an embodiment" does not necessarily refer to the same embodiment. Specific features, structures, or characteristics may be combined in any suitable manner consistent with this disclosure.

[0029] The term "comprising" is open-ended. As used in the appended claims, it does not exclude additional structures or steps. Consider the following cited claim: "An apparatus comprising one or more processor units..." Such claims do not exclude the inclusion of additional components (e.g., network interface units, graphics circuitry, etc.).

[0030] "Configured as" refers to various units, circuits, or other components that can be described or stated as being "configured as" to perform one or more tasks. In such a context, "configured as" is used to imply a structure by indicating that the unit / circuit / component includes a structure (e.g., a circuit) that performs this one or more tasks during operation. Thus, a unit / circuit / component is allegedly configured to perform the task even when the specified unit / circuit / component is currently inoperable (e.g., not switched on). Units / circuits / components used with the language "configured as" include hardware—e.g., circuits, memory storing program instructions that can be executed to perform the operation, etc. Referring to a unit / circuit / component as being "configured as" to perform one or more tasks is explicitly intended to not invoke 35 USC § 112(f) for that unit / circuit / component. Furthermore, "configured as" can include general structures (e.g., general circuits) manipulated by software and / or firmware (e.g., FPGAs or general-purpose processors executing software) in a manner capable of performing one or more tasks to be solved. "Configured to" may also include adjusting the manufacturing process (e.g., a semiconductor manufacturing facility) to manufacture equipment (e.g., an integrated circuit) suitable for performing one or more tasks.

[0031] "First," "second," etc. As used herein, these terms serve as labels for the nouns preceding them and do not imply any type of ordering (e.g., spatial, temporal, logical, etc.). For example, a buffer circuit may be described herein as performing write operations on a "first" value and a "second" value. The terms "first" and "second" do not necessarily imply that the first value must be written before the second value.

[0032] "Based on." As used herein, this term describes one or more factors that influence the determination. This term does not exclude additional factors influencing the determination. That is, the determination may be based solely on these factors or at least partially on them. Consider the phrase "A is determined based on B." In this case, B is the factor influencing the determination of A, and such phrases do not exclude the possibility that the determination of A may also be based on C. In other instances, A may be determined solely on B. Detailed Implementation

[0033] Natural scenes typically do not contain noise. However, video or still images capturing natural scenes may include noise. For example, when processing dark content, such as capturing images of natural scenes under low-light conditions, noise is often captured along with the natural scene in the image. For example, thermal noise or noise introduced via a light sensor may be captured. Additionally, depending on the medium used to capture the natural image, other types of noise, such as film grain noise, may be captured.

[0034] For various reasons, it may be necessary to retain or recreate noise included in a captured image after it has been compressed and decompressed. For example, film grain noise can be a desirable feature for video to reproduce old films, artistic intents, etc. However, current video compression algorithms either remove noise during compression and decompression or are unable to faithfully reproduce noise in the decompressed version of the image or video.

[0035] For example, some video compression / decompression systems can add noise to decompressed images at the full frame level. This can be done by specifying a noise model typically associated with each frame of the sequence, for example, using the Film Grain SEI mechanism in codecs such as MPEG-4 AVC / H.264 or HEVC / HEVC, and using that model to generate the noise to be added to the current frame. The amount and type of noise to be added can be further modulated based on regional characteristics, such as the brightness of the region to which noise is to be added. However, the noise model does not change, while the amount and type of noise typically vary within an image frame, making a noise model sized to fit all frame levels unable to accurately recreate the noise included in the original image.

[0036] In some implementations, a video encoder or decoder may include an in-loop noise estimator and / or an in-loop noise generator that selects noise models and / or model parameters at a sub-image portion level granularity, wherein different noise models and / or parameters may be applied to different portions of the image within the same image frame. For simplicity, the term image frame is used herein. However, in some implementations, similar processing may be performed on pictures, which may include interlaced images, such as field and frame pictures. In some implementations, sub-image level noise may be added in the compression or decompression loop upstream or downstream of loop filters such as deblocking filters or sample adaptive offset (SAO) filters.

[0037] Furthermore, in some embodiments, an optional denoising filter may be included in the compression or decompression loop to remove noise. In some embodiments, removing noise from the compression loop can improve compression efficiency by removing noise from the residual values ​​and thus reducing the number of bits required to encode or equivalently compress these residual values ​​at higher quality.

[0038] In some implementations, a controlled denoising filter may be included in the compression / decompression loop to remove noise and improve compression efficiency. In some implementations, the controlled denoising filter may be controlled such that it is adjusted to remove noise similar to that added at another location in the compression or decompression loop. For example, information such as noise parameters indicating the selected noise model / noise parameters used to generate the in-loop noise may also be provided to the controlled denoising filter to select an appropriate filter to remove the generated noise.

[0039] Exemplary compression ring / encoder For example, Figure 1 A compression loop 100 receiving input video data 130 is illustrated. Certain portions of the input video data 130 are used to determine predicted pixel values ​​via an intra-frame prediction module and / or a motion estimation / compensation module 102. The predicted values ​​142 are compared with the original pixel values ​​in the input video data 130 to determine a residual value 132 (e.g., the difference between the predicted pixel value and the pixel value in the input video data 130). The residual value 132 is further processed via a transform module 110 and a quantization module 104, and output as compressed image / video data 136. The transformed / quantized residual value (e.g., compressed image / video data 136) is entropy-coded via an entropy coding module 114 and output as an encoded compressed bitstream 138.

[0040] The compression loop 100 also includes an inverse quantization module 116 and an inverse transform module 118. The inverse quantization module 116 and the inverse transform module 118 reconstruct the residual values ​​such that they are similar to the residual values ​​132 prior to the transform module 110 and the quantization module 112. For example, the inverse quantization / inverse transform module undoes the transform and quantization applied at the transform module 110 and the quantization module 112. In addition to the output of the inverse transform module 116, any distortion introduced due to the transformation and quantization of the residual values ​​may be included. The output of the inverse transform module 116 is combined with the predicted pixel values ​​140 from the motion compensation / intra-frame prediction module 102 to generate a reconstructed version of the original pixel values ​​(e.g., decompressed pixel values ​​134). The decompressed pixel values ​​134 are further processed by in-loop filters, such as a deblocking filter 120 and a sample adaptive offset (SAO) filter 122. The reconstructed image frame, including the decompressed pixel values ​​134 filtered by the deblocking filter 120 and the SAO filter 122, is stored in the decoded image buffer 124.

[0041] The image in the decoded image buffer 124 can be used as input for estimating intra-frame mode compression distortion. For example, the intra-frame mode estimation module 106 can use previously coded regions from the current image, which are also stored in the decoded image buffer 124, to estimate the distortion compared to uncompressed regions in the current image frame.

[0042] As used herein, intra-frame mode compression refers to compression using predicted and residual values ​​determined within an image frame based on the values ​​of other pixels in the same image frame. For example, intra-frame mode compression techniques may employ and explicitly include a subset of the image's pixel values. In some implementations, the subset may be a single starting pixel value or a set of pixel values. Furthermore, intra-frame prediction techniques use these starting pixel values ​​to predict the pixel values ​​of neighboring pixels. Such a process can be iterative and can be performed block-by-block, where pixel values ​​are predicted one block at a time. These predicted pixel values ​​can then be compared to the actual pixel values ​​of the image to determine residual values. The residual values ​​can then be encoded for use by a decoder to correct predicted pixel values ​​generated using the same or similar prediction process.

[0043] Conversely, as used herein, inter-frame prediction refers to a compression technique that uses differences between image frames (such as due to temporal variations) to predict pixel values. For example, motion estimation and motion vectors can be used in inter-frame prediction methods to estimate and indicate the motion (e.g., temporal difference) of blocks or other sub-units of an image between image frames. These motion vectors can then be used to predict pixel values ​​in subsequent image frames relative to a reference image frame, where the predicted pixel values ​​compensate for the motion between image frames.

[0044] The motion compensation estimation module 108 can use the image in the decoded image buffer 124 to determine the estimated motion and / or distortion between the uncompressed image frame and the reconstructed image frame, wherein inter-frame compression techniques such as motion estimation and compensation are used to compress and decompress the reconstructed image frame.

[0045] The estimated distortion and / or compression efficiency determined by the intra-frame mode estimation module 106 and the motion compensation estimation module 108 can be provided to the mode decision module 104, whereby the mode decision module 104 selects whether to use intra-frame prediction or inter-frame prediction techniques to compress a given image region or even the entire frame.

[0046] In some implementations, a device, such as a mobile device having a screen and a camera that is capturing images or videos being compressed, may also display video images. For example, reconstructed image frames stored in the decoded image buffer 124 may be further provided to the display 126 for rendering on the display.

[0047] Traditionally, image and video coding schemes perform prediction (intra-frame prediction or inter-frame prediction) on the image regions to be compressed and encoded, as described above. For example, a block region of size M x N. Previously encoded samples, such as other blocks in the same image, can be used for intra-frame prediction techniques, while corresponding blocks in other image frames at different times can be used for inter-frame prediction techniques.

[0048] In some implementations, transformations applied to the residual values ​​via transformation module 110 may include inverse discrete cosine transform or other suitable transformations.

[0049] In some implementations, loop filters, such as deblocking filter 120, SAO filter 122, and / or additional loop filters, can be applied to a portion of the image (e.g., a block) and adjacent portions of the image (e.g., adjacent blocks). Loop filters can reduce possible coding / compression artifacts that may have been introduced during the coding / compression process.

[0050] The encoded compressed bitstream 138 may include any information required to reconstruct a portion (e.g., a block) of the image being compressed and encoded, such as partitioning information, an indication of the selected prediction mode and related information, such as motion vectors and reference indices, the type of one or more transforms used, quantization parameters, and any remaining quantization coefficients, etc.

[0051] To some extent, the decoder follows the inverse process, first performing entropy decoding on the compressed bitstream to derive encoded information (partitions, modes, motion vectors / mv, quantization coefficients, etc.), followed by a prediction process. Inverse quantization and transform steps can be performed in parallel, and the reconstructed residuals are then added to the prediction information. Given an appropriate in-loop filtering mechanism supported and used by the underlying codec, the reconstructed block can then be in-loop filtered.

[0052] While the methods described above apply to most content, they may not perform well in certain situations, leading to poor coding performance and severe coding artifacts. This is because the methods tend to assume that the signal is predictable and relatively noise-free or smooth. However, common image and video data can contain noise. Typically, such noise is not time-dependent and can vary significantly between frames. Different types of noise can exist, associated with different components of the capture and image acquisition process, including but not limited to Gaussian and shot noise, film grain, salt-and-pepper noise, and quantization noise that may be a product of earlier compression processes. Noise can also be added as an "artistic" component during content creation, or it may reduce artifacts that may already be present in the source, such as striping, or it may be introduced into the content due to additional processing steps such as bit depth reduction or compression. The amount and characteristics of noise signals in an image can vary by region and can depend on multiple parameters, including intensity, motion, color, etc.

[0053] Some systems attempt to process noise in image or video content by first performing a denoising step on the actual source material, and then compressing / encoding the denoised content. The removed noise is also modeled in the encoder, and then signaled in the bitstream to allow the decoder to synthesize certain parameters that resemble the noise pattern. Modeling supplementary enhancement information (SEI) messages using film grain modeling supports this type of model in both AVC and HEVC, while AV1 has a similar process for modeling the signal and adding noise after the decoding process. However, these models add noise outside the encoding loop. Furthermore, in these methods, their parameters are global (i.e., at the sequence or frame level), and it is not possible to signal any control parameters at the local (e.g., block) level.

[0054] In these systems, combining two images with different noise characteristics into a single image, such as a montage, and then encoding them as a single image, implies that it is impossible to consider different noise models for regions corresponding to the two different sub-images. The noise is designed to resemble the overall characteristics of noise throughout the image and cannot be locally adjusted. In images with varying degrees of noise, this can lead to inaccurate noise modeling, resulting in an image that may subjectively resemble the original image before noise removal.

[0055] An exemplary compression process using out-of-loop noise modeling For example, Figure 2 An encoder 200 is shown, comprising a compression loop similar to compression loop 100. However, encoder 200 includes out-of-loop noise generation via a noise modeling module 206 outside compression loop 100. For example, input video data 130 is filtered via denoiser 202 before entering compression loop 100. Additionally, at noise estimator 204 (outside the loop), the filtered video data is compared with input video data 130 to determine frame-level noise parameters, such as which noise model is used to model the removed noise and which parameters are used in the selected model. This information is provided to entropy coding unit 114 to be included in encoded bitstream 138. However, as previously stated, this frame-level information does not allow for sub-image portion noise generation control and is added to the full frame after the frame has been reconstructed, rather than to the sub-image portion before image frame reconstruction.

[0056] In some encoding devices, a display, such as display 126, is also included. The noise modeling module 206 can generate a noise frame including a frame-level noise pattern and add the frame-level noise pattern to the reconstructed image frame from the decoded image buffer 124 before providing the reconstructed image frame for rendering on display 126. In some embodiments, the noise modeling module 206 can also receive (not shown) information about previous image frames from the decoded image buffer 124 as input to the noise model.

[0057] An exemplary decompression process / decoder using out-of-loop noise modeling Figure 3 An exemplary decoder including out-of-loop noise modeling is shown. Entropy decoding module 302 receives the encoded bitstream 320 and entropy decodes the encoded bitstream. Inverse quantization module 306 and inverse transform 308 process the decoded residual values ​​322 (these decoded residual values ​​may be similar to residual values ​​determined at the encoder that have been transformed and quantized, for example, via transform module 110 and quantization module 112) to generate dequantized, detransformed residual values ​​324. Additionally, information 328 (e.g., initial pixel values, motion vectors, etc.) is provided to motion-compensated intra / prediction module 316. Information 328 may include an indication of whether the image is compressed via intra-prediction or inter-prediction.

[0058] If compression is performed via intra-frame prediction, information 328 may include at least one or more initial pixel values ​​for at least one pixel of the compressed image. These at least one or more pixel values ​​may be starting values ​​used to predict pixel values ​​for other pixels via the intra-frame prediction process. Alternatively, if the image is compressed via inter-frame prediction, information 328 may include motion vectors (MVS). The motion compensation / intra-frame prediction module 316 may generate predicted pixel values ​​326, and these predicted pixel values ​​326 may be combined with dequantized, detransformed residual values ​​324 to produce decompressed pixel values ​​330. The decompressed pixel values ​​330 may be further filtered by one or more loop filters, such as a deblocking filter 310 and a sample adaptive offset (SAO) filter 312. The filtered decompressed pixel values ​​are reconstructed into decompressed image frames and stored in a decoded image buffer 314. In some embodiments, the decompressed image frames from the decoded image buffer 314 may be used by the motion compensation / intra-frame prediction module 316 to estimate motion between blocks (or other portions) of consecutive image frames.

[0059] The decompression process described above can be similar to the various decoders described in this article. However, Figure 3 Out-of-loop noise modeling is also illustrated, wherein decoded data 332, indicating noise model / noise parameters for generating frame-level noise patterns, is provided to a noise modeling module 304, which generates noise patterns for the entire image frame outside of the decompression loop 300. At 336, image frame-level noise is added to the reconstructed image frame from the decoded image buffer 314 before being provided for rendering on the display 318.

[0060] Loop sub-image partial-level noise modeling In some implementations, unlike previous systems that removed noise from the original source and performed noise modeling outside the coding loop, compression / decompression processes with in-loop noise generation enable the addition of noise within the image / video coding loop. Such addition can also be done and controlled at the sub-image section level (also referred to herein as the block level), such as macroblocks, coding tree units (CTUs), coding units, transform units, or other block coding units that can be supported by coding techniques.

[0061] Specifically, additional coding blocks can be introduced within the video encoder and thus the video decoder, which aims to model and generate “controlled” noise patterns added to the signal during the reconstruction of image blocks.

[0062] For example, Figure 4 A compression loop 400 is shown, similar to compression loop 100, except that compression loop 400 includes optional denoisers 404, 406, 408, and 410, and an in-loop noise modeling unit 402. Input video 130 is optionally denoised at denoiser 404 before being used to determine residual value 132. Furthermore, residual value 132 can be denoised via denoiser 406 before being transformed and quantized. Additionally, uncompressed and decompressed image frames from decoded image buffer 124 can be denoised via optional denoisers 408 and 410 before being used by motion estimation module 108.

[0063] However, with Figure 2 Compared to the noise modeling module 206 outside the compression loop 100 shown, the noise modeling module 402 is included as part of the compression loop 400, wherein the generated partial-level noise pattern 412 is added to the predicted pixel value 140 of the image portion together with the residual values ​​of the inverse quantization and inverse transform of the image portion.

[0064] For example, Figure 5 The decompression loop 500 is shown, which is similar to the decompression loop 300, except that the decompression loop 500 includes a noise modeling unit 502 within the decompression loop, so that noise can be added at the sub-image part level.

[0065] and Figure 4 and Figure 5 The compression loop and decompression loop shown are different. Figure 2 and Figure 3 The noise generated in the code is essentially generated using frame-level parameters and added outside the coding loop. Conversely, in... Figure 4 The compression circuit 400 shown and Figure 5The noise generated in the decompression loop 500 shown is generated for each block or any other coding unit that the system can support, and added to it. In both the compression loop 400 and the decompression loop 500, the addition and characteristics of the noise are entirely controlled by parameters signaled in the bitstream, where signals can be sent for each such coding unit. For example, the noise generation parameters can be signaled in a manner similar to how transform information is associated with each coding unit and provided for each coding unit.

[0066] Specifically, the compression loop 400 and decompression loop 500 allow for the addition of "controlled noise" to each coding unit. Various parameters controlling the presence, type, and characteristics of the noise to be added can be signaled. Coding units can be existing units, such as CTUs, prediction units, transform units, etc., or newly defined independent units that can be specified and controlled through high-level syntax elements similar to how these other units are controlled. For example, a CTU, in a similar manner to how it is segmented into transform units, can also be segmented into such units for noise addition. Hierarchical segmentation can be used. For example, tree-based methods can be used to segment trees such as quadtrees, binary trees, ternary trees, etc. The maximum depth of such trees can be controlled by higher-level syntax units, such as at the Sequence Parameter Set (SPS), Picture Parameter Set (PPS), or slice header. Such units can also be larger than a CTU and span multiple CTUs to ensure better noise mixing across multiple blocks. Such units can be referred to as noise portion units.

[0067] For example, Figure 12 A portion of image 1202, comprising multiple blocks 1204, is shown, and Figure 13 Noise block or region units 1302, 1304, 1306 and 1308 are shown, wherein each noise block or region unit overlaps with multiple blocks in block 1204.

[0068] The in-loop noise generation process discussed in this paper can be applied to any color domain, such as YCbCr or RGB signals. Furthermore, the in-loop noise generation process can operate on a single plane, such as only luminance (Y), or all planes, with control over all planes being the same or different for each color plane (e.g., the color components of a color space). For example, noise can be added only to the luminance component while leaving the chrominance plane untouched, or noise can be modeled based on the luminance component, and then other color planes can be scaled or appropriately adjusted based on a fixed relationship or the relationship between luminance and chrominance values ​​before adding any noise (e.g., reconstructing the luminance data channel versus reconstructing the chrominance data channel before adding noise; such values ​​can be used to determine the scaling factor for noise added to the chrominance plane, such as a ratio of these two quantities). Different controls can also be applied to different types of content, such as whether the content is standard or high dynamic range content, video or full range, with standard / limited color gamut or wide color gamut, whether the content is classified as natural content or synthetic content, whether the content is regular content or point cloud projection content (e.g., in the context of a point cloud system based on video point cloud compression (V-PCC) technology), etc.

[0069] For each noise block or region unit (e.g., block, CTU, etc.), the following can be signaled in the bit stream: • Presence of noise. If there is no noise, no additional parameters need to be sent. This can be a flag or a parameter that also indicates the type of noise to be added (as described in the next bullet point).

[0070] • If noise is present, the type of noise to be added, and the noise model that might be used to generate that noise type. For example, the noise to be added could follow a Gaussian or Laplace model, generalized Gaussian, uniform, gamma, Berlin noise, etc. All of these noise models can be associated with syntax elements such as `block_noise_model`. For example, if this parameter is set to 0, no noise is added. If the parameter is set to 1, a Gaussian model of noise will be used; if it is set to 2, a uniform noise model will be used, and so on.

[0071] • Additional parameters can be sent based on the noise model. For example, for a Gaussian model, the mean and standard deviation of the noise can be signaled. For noise models following generalized power distribution model parameters such as alpha and beta, other parameters besides the mean can be signaled. One model can utilize the transform domain, such as a noise generation process based on the discrete cosine transform (DCT), while another model can use a recursive-based model. Noise can also be generated using a properly defined polynomial / linear feedback shift register (LSFR). The LFSR method is commonly used to generate uniform and pseudo-random Gaussian noise models. For such polynomials, the order and coefficients of the polynomial, as well as the initial (seed) values ​​required for the polynomial, can be signaled. This information can also be shared across neighboring blocks. In this case, filter parameters can also be predicted from its neighborhood to reduce overhead. For example, for a block, it can be signaled that the noise to be added is related to the noise added to one of its neighboring parts, such as the block to the left or top. In this case, it may not be necessary to signal additional parameters, or the parameters to be signaled for the current block can be "predicted" based on the parameters of its neighboring parts. Overhead can be further reduced by using the most probable noise model based on the neighborhood. The positional order of noise can also be signaled or predetermined. For example, raster scan or zigzag scan order can be used for noise generation and addition. Noise blocks can also be viewed as "noise transform blocks," similar to DCT or other transform blocks that generate noise patterns in their appropriate designated domain and "transform model" (e.g., LFSR model). The same model can be used in the decoder and operate in the same way; for example, the noise models of both the encoder and decoder are included in the noise modeling modules of the encoder and decoder, respectively.

[0072] In some implementations, the value of the noise to be added can be unknown and independent of the signal to be added. In alternative implementations, the noise can also depend on the characteristics of the block. Such characteristics may include the overall data channel or pixels it reconstructs (e.g., the noise to be added is modulated by the data channel or current component value), characteristics based on the residual signal (again modulated by the value of each residual pixel; in this case, if the residual is 0, it may be desirable to perform modulation using a "residual + fixed offset" value), or motion (e.g., low-motion and high-motion blocks may require different noise to be added. This can be done by controlling the noise generation and weighting of the motion vector in the bitstream). Furthermore, different noise can be added for skip, intra-frame, and affine mode blocks, etc. Such weights can also be adjusted according to the color plane being processed (e.g., for luma and chroma components, different noise can be generated based on mode, motion, block DC value, etc., or such noise control parameters can be determined based solely on luma to reduce complexity).

[0073] The noise to be added is determined at the encoder based on the characteristics of the source. Specifically, the source can be analyzed and noise (if present) in the signal can be identified and then modeled. The encoder can do its best to recreate the noise as close as possible to its original characteristics, or it can only approximate it. This can be done using various methods, such as least mean square estimation, Wiener noise estimation, etc.

[0074] In some implementations, the encoder may optionally denoise the content before performing this process, as it knows it can resynthesize the noise as part of the coding loop. This process is entirely optional. Denoising can occur directly at the source, such as in the image domain, or in the residual domain after any prediction has been performed. Such denoising in the prediction domain can also remove noise introduced through the prediction process (e.g., due to noise references). Denoising can also be performed in a cascaded manner in both domains. Such steps are obviously non-canonical and are designed to reduce the energy of the residual signal that will be transformed and quantized. Any denoising method, including Wiener filtering, bilateral filtering, motion-compensated spatiotemporal filtering, wavelet-based or bandlet / edgelet / ridgelet filtering, Gaussian filtering, or other commonly used filtering methods, can be used for this type of processing.

[0075] Additional non-standard noise reduction can be applied via optional noise reduction units 408 and 410, such as Figure 4 As shown. For example, during motion estimation, optional denoising can be applied to both the source signal and the reference stored in the decoded image buffer 124. This can help motion estimation determine the best prediction of the actual signal in the scene (e.g., finding the "real / true" motion of the content) rather than noise, which is generally irrelevant. Any denoising method, including Wiener filtering, bilateral filtering, motion-compensated spatiotemporal filtering, wavelet-based or bandlet / edgelet / ridgelet filtering, Gaussian filtering, or other commonly used filtering methods, can be used for this type of processing. In this case, motion compensation and the final prediction will use unfiltered data.

[0076] Noise can be added immediately or simultaneously with the prediction and reconstruction (after inverse quantization and transform) residuals. For example, a noise pattern can be added after inverse transform 118, such as... Figure 4As shown. However, this means that any subsequent processing steps, such as deblocking, Sample Adaptive Offset Filtering (SAO), Adaptive Loop Filtering (ALF), or any other in-loop filtering supported by the codec, will be applied to the signal as a combination of prediction, reconstruction residuals, and reconstruction noise. This may be desirable under certain architectures, in which case it may also be necessary to control such filtering mechanisms based on adding noise to the current block and its adjacent parts. In this case, the characteristics of such loop filtering mechanisms can be modified to a) account for noise, b) ensure that they preserve noise characteristics, and / or c) ensure that noise is properly mixed across block partitions.

[0077] For example, in existing deblocking schemes, filter intensity is typically determined based on the similarity between pixels and block edges. Only rows or columns to be filtered in these cases are evaluated. Due to the added noise, it's best not to consider individual columns or rows, but rather to perform deblocking intensity evaluation in a manner that considers multiple columns and rows. Effectively, "subtle" and in-situ denoising can be performed on the locations to be deblocked before the deblocking intensity is determined. Intensity determination can then be performed on such denoised pixels. Note that this denoising is only used to determine the deblocking intensity, and the denoised values ​​are not further used in the reconstruction process. Similar considerations can be made for other types of filters, such as SAO, ALF, loop bilateral filtering, etc. For example, in the case of SAO, the determination of band or edge filtering currently defined in HEVC also considers similar denoising methods used in the classification process. Similar considerations can also be made if other modes of SAO are to be used, such as texture modes, strip detection, etc.

[0078] In some implementations, intra-frame prediction can be performed with or without noise. This may be predetermined at the encoder and decoder (without signaling), or the decision may be signaled in the bitstream. Signaling can be at a high-level syntax structure, such as SPS, PPS, or slice headers, or at a lower level, such as CTU or noise block level.

[0079] In some implementations, the compression / decompression loop can function in an alternative mode where noise is added downstream of the loop filter. For example, Figure 6 An encoder is shown, in which noise is added downstream of the SAO filter 122. For example, Figure 7 The decoder is shown, in which the noise modeling module 702 is in the decompression loop, and noise is being added downstream of the SAO filter 312.

[0080] In some implementations, it may be desirable to add a noise block after the loop filtering step. This avoids filtering the added noise, which in some cases can lead to other types of artifacts. The final signal is still added to the reference buffer for future prediction. However, unlike the previous case where intra-frame prediction could be performed with or without a noise model, steps typically at the loop filter level are exempt from intra-frame prediction for complexity and storage purposes. Therefore, it is recommended that the noise block also be excluded from any intra-frame prediction stage (including adjacent block orientation prediction and intra-frame block copy prediction) in this case. However, the noise is still considered for interval / time prediction.

[0081] In some cases, added noise can impair inter-frame prediction. This is because the system must compensate not only for prediction inaccuracies but also for irrelevant noise. The use of denoising methods, such as bilateral, Gaussian, Wiener, or other filtering schemes, can be part of the prediction loop and remove noise from the predicted signal. Such denoising methods are signaled and controlled in both the encoder and decoder. For example, Figure 8 An encoder including a controlled noise reduction module 804 is shown, and Figure 9 A decoder including a controlled noise reduction module 904 is shown. Figure 10 A decoder is also shown, which includes a controlled denoising module 1004 for removing noise from the reconstructed image used by the motion compensation and intra-frame prediction module 316.

[0082] In some implementations, the signaling of such controlled denoising methods can now also be based on the type and characteristics of the added noise. Since both the encoder and decoder are aware of this information, denoising can be adjusted accordingly based on these characteristics, such as selecting the most suitable mode to remove noise, without needing to signal all the parameters required for denoising the "unknown" based on the characteristics of the noise. For example, if noise with certain characteristics (e.g., Gaussian noise with a mean of X) is added to a particular reference, then a bilateral filter might be preferred for that reference. For different references with different noise characteristics, Gaussian or Wiener filters might be preferred. Instead of making implicit decisions about the filters, the most likely filter mode can be selected based on such a mechanism, and then the index of the final filter to be used can be encoded. This would then help reduce the coding overhead of the denoising processing filters.

[0083] In some implementations, in-loop noise generation and addition can be combined with out-of-loop noise generation and addition, such as Figure 2 and Figure 3 As shown. This is helpful in situations where it may be necessary to add noise to the reconstructed signal but avoid noise in the prediction loop. There are several ways to do this, such as: • Switch noise patterns at higher-level structures, such as sequences, pictures, tiles, or slices. For example, noise may only exist outside the loop or only inside the loop.

[0084] • Even at the block level, both modes are allowed to exist, but only one at a time. In this case, the encoder and decoder need to maintain a block map with in-loop noise already added and out-of-loop noise to be added later (e.g., for display purposes).

[0085] • Both types of noise are permissible, but can also be controlled at the block level simultaneously. In this case, in-ring noise may attempt to resolve certain types of artifacts and may not require significant energy to ensure that the noise does not greatly affect inter-frame prediction. On the other hand, out-of-ring noise may be more significant. In this case, noise models for both in-ring and out-of-ring blocks need to be considered jointly. Regarding the addition of in-ring noise, out-of-ring noise can be "blind," meaning that metadata does not need to be fed into the block to add noise, thus reducing complexity. However, for some systems, the tuning and characteristics of the out-of-ring noise adder may also need to consider in-ring noise, such as the location where it is added and what characteristics it has.

[0086] • While the encoding module is based on pixel comparison operations, such as the sum of absolute differences (SAD), sum of absolute transform differences (SATD), and sum of squared errors (SSE), which are available for many encoding decisions used by the proposed encoder (as is done by many existing encoders), it is preferable to utilize other matching criteria that may be more “noise-agnostic” / robust. Using a modified SSIM method, which emphasizes the mean over the variance of the signal, this proves to be more robust as a matching / decision criterion in the presence of noise, and in the context of a rate distortion optimization model based on Lagrange parameters, it highlights the DC value rather than the lower (and more noisy) transform domain criterion of the AC coefficients. Furthermore, the λ parameter can be appropriately adjusted based on the distortion model and the characteristics of the currently encoded block (motion, noise, DC, noise-free variance, etc.).

[0087] • Different decision rules can be applied to different color components or different content, such as SDR versus HDR, which can be determined by the encoder, application, or user. For example, the encoder can use different distortion standards for chroma and luminance.

[0088] An exemplary process for generating an in-loop noise pattern Figures 11A to 11B An exemplary process for generating an in-loop noise pattern is shown according to some embodiments. Step 1102 is a step for determining block-level noise as part of an in-loop compression process and includes sub-steps 1104 to 1122.

[0089] At 1104, the first image or the next image to be compressed is received. At 1106, a first portion (e.g., block, CTU, noise block, or region unit, etc.) of the first portion (or the next image) is selected to evaluate the noise.

[0090] At 1106, a first noise pattern to be generated for the first portion is determined. The first noise pattern can be selected to approximate the noise in the first portion of the image. At 1110, the first noise pattern is generated using the noise model and / or the noise parameters determined at 1108.

[0091] At 1112, select an additional (or next additional) portion of the first (or next image) (e.g., block, CTU, noise block, or region unit, etc.) to evaluate the noise.

[0092] At 1114, determine the noise pattern to be generated for the additional (or next additional) portion. A noise pattern can be selected to approximate the noise in the additional portion of the image. At 1116, generate the noise pattern using the noise model and / or the noise parameters determined at 1114.

[0093] At 1118, it is determined whether there is an additional portion (e.g., block, CTU, noise block, or region unit, etc.) of the first (or next) image to be evaluated. If so, the process continues at 1112 to compress the next additional portion of the first (or next) image. If there is no additional portion to be evaluated for the currently evaluated image, at 1120, it is determined whether there is an additional image to be evaluated. If so, the process continues at 1104 to evaluate the next image.

[0094] At 1124, a predicted pixel value is determined for the current portion (e.g., block, CTU, noise block, or region unit, etc.) of the first (or next) image. At 1126, the predicted pixel value is compared with the uncompressed or original pixel value to determine the residual value.

[0095] At position 1128, and at positions 1108 or 1114, the residual values ​​and noise model and / or noise parameters determined for a given block being evaluated are encoded in the coded bitstream. Note that noise generation and pixel value prediction, as well as residual determination, can be performed block-by-block or region-by-region.

[0096] Exemplary controlled denoising process Figure 14 An exemplary controlled denoising process according to some implementation schemes is shown.

[0097] At 1402, a noise model for generating the noise pattern is selected based on various input parameters as described herein. At 1404, additional parameters, which are inputs to the selected model, are provided as inputs to the model selected at 1402. At 1406, a noise pattern is generated for the first (or next) portion of the image using the selected model and the selected input parameters.

[0098] Additionally, the selected model and the selected input are provided to the controlled denoising module, and at 1412, the controlled denoising module selects an in-loop filter for controlled denoising based on the noise model and input parameters selected at 1402 and 1404. At 1414, the controlled denoising module performs denoising (e.g., filtering) on ​​the image portion where noise was generated at 1406.

[0099] At point 1408, it is determined whether there is an additional portion of the image to be evaluated; if so, the process returns to point 1402. If not, the process continues to evaluate the next image.

[0100] An exemplary decoding process using in-ring sub-image partial-level noise generation Figure 15 An exemplary process for generating an in-loop noise pattern during decompression is shown according to some embodiments.

[0101] At 1502, the decoder receives an encoded bitstream of a compressed image for video, wherein the bitstream includes data indicating a noise model and / or noise parameters used to generate in-loop noise at the encoder.

[0102] At 1504, the decoder decodes the compressed bitstream. This may include decoding the entropy-encoded residuals and decoding encoded data of the noise model and / or noise parameters used to generate a noise pattern added within the loop during compression at the encoder.

[0103] At 1506, the decoder generates a noise pattern for the portion of the image being decompressed based on the noise model and noise parameters indicated for the corresponding portion (e.g., block, CTU, noise block, or region unit, etc.) included in the compressed bitstream. Furthermore, at 1510, the decoder predicts the pixel value of that portion (e.g., block, CTU, noise block, or region unit, etc.), and at 1512, applies the residual value to the predicted value to generate adjusted pixel values ​​for the image portion.

[0104] At 1508, the generated noise pattern is added to the adjusted pixel values ​​of the image portion. At 1516, a decompressed version of the compressed image portion, including the generated noise pattern, is provided. The decompressed image portion can be combined with other decompressed image portions to reconstruct a decompressed version of the compressed image. The decompressed version of the compressed image can be added to the decompressed image buffer and can be used for motion estimation and / or provided for rendering at the display, such as in a portion of a decompressed video.

[0105] Exemplary computer system Figure 16 An example computer system 1600 according to some embodiments is shown, which may implement an encoder or decoder or any other component described herein (e.g., referenced above). Figures 1 to 15 (Any component described). Computer system 1600 may be configured to perform any or all of the embodiments described above. In different embodiments, computer system 1600 may be any of a variety of types of devices, including but not limited to: personal computer systems, desktop computers, laptops, notebook computers, tablet computers, all-in-one computers, tablet computers or netbooks, mainframe computers, handheld computers, workstations, network computers, cameras, set-top boxes, mobile devices, consumer devices, video game controllers, handheld video game devices, application servers, storage devices, televisions, video recording devices, peripheral devices (such as switches, modems, routers), or any type of computing or electronic device in general.

[0106] The various implementations of the encoder or decoder described herein can be executed on one or more computer systems 1600, which can interact with various other devices. It should be noted that, depending on the various implementations, the above description is relative to... Figures 1 to 15 Any component, action, or functionality described can be implemented by being configured as Figure 16The computer system 1600 is implemented on one or more computers. In an illustrated embodiment, the computer system 1600 includes one or more processors 1610 coupled to system memory 1620 via an input / output (I / O) interface 1630. The computer system 1600 also includes a network interface 1640 coupled to the I / O interface 1630, and one or more input / output devices 1650, such as a cursor control device 1660, a keyboard 1670, and one or more displays 1680. In some cases, it is conceivable that an implementation may be carried out using a single instance of the computer system 1600, while in other implementations, multiple such systems or multiple nodes constituting the computer system 1600 may be configured to host different parts or instances of the implementation. For example, in one implementation, some elements may be implemented via one or more nodes of the computer system 1600 that are different from those nodes implementing other elements.

[0107] In various implementations, computer system 1600 may be a single-processor system including one processor 1610, or a multiprocessor system including several processors 1610 (e.g., two, four, eight, or another suitable number). Processor 1610 may be any suitable processor capable of executing instructions. For example, in various implementations, processor 1610 may be a general-purpose or embedded processor implementing any of a variety of instruction set architectures (ISAs) (such as x86, PowerPC, SPARC, or MIPS ISA or any other suitable ISA). In a multiprocessor system, each processor of processor 1610 may, but is not required to, implement the same ISA.

[0108] System memory 1620 may be configured to store point cloud compression or decompression program instructions 1622 and / or sensor data accessible by processor 1610. In various embodiments, system memory 1620 may be implemented using any suitable memory technology, such as static random access memory (SRAM), synchronous dynamic RAM (SDRAM), non-volatile / flash memory, or any other type of memory. In an illustrated embodiment, program instructions 1622 may be configured to implement an image sensor control application combining any of the functionalities described above. In some embodiments, program instructions and / or data may be received, transmitted, or stored on a different type of computer-accessible medium or similar medium separate from system memory 1620 or computer system 1600. Although computer system 1600 is described as implementing the functional blocks of the preceding figures, any of the functionalities described herein may be implemented via such computer system.

[0109] In one embodiment, I / O interface 1630 may be configured to coordinate I / O communication between processor 1610, system memory 1620, and any peripheral devices (including network interface 1640 or other peripheral device interfaces, such as input / output devices 1650) within the device. In some embodiments, I / O interface 1630 may perform any necessary protocol, timing, or other data conversions to convert data signals from one component (e.g., system memory 1620) into a format suitable for use by another component (e.g., processor 1610). In some embodiments, I / O interface 1630 may include support for devices attached, for example, via various types of peripheral buses (e.g., variants of the Peripheral Component Interconnect (PCI) bus standard or the Universal Serial Bus (USB) standard). In some embodiments, the functionality of I / O interface 1630 may be divided among two or more separate components, such as a north bridge and a south bridge. Furthermore, in some embodiments, some or all of the functionality of I / O interface 1630 (such as an interface to system memory 1620) may be directly incorporated into processor 1610.

[0110] Network interface 1640 may be configured to allow data exchange between computer system 1600 and other devices (e.g., bearers or agent devices) attached to network 1685, or between nodes of computer system 1600. In various embodiments, network 1685 may include one or more networks, including but not limited to local area networks (LANs) (e.g., Ethernet or enterprise networks), wide area networks (WANs) (e.g., the Internet), wireless data networks, some other electronic data network, or some combination thereof. In various embodiments, network interface 1640 may support communication via wired or wireless general-purpose data networks (such as any suitable type of Ethernet network), for example; via telecommunications / telephone networks (such as analog voice networks or digital fiber optic communication networks); via storage area networks (such as Fibre Channel SANs), or via any other suitable type of network and / or protocol.

[0111] In some implementations, input / output device 1650 may include one or more display terminals, keyboards, keypads, touchpads, scanning devices, voice or optical recognition devices, or any other device suitable for inputting or accessing data by one or more computer systems 1600. Multiple input / output devices 1650 may be present in computer system 1600 or distributed across various nodes of computer system 1600. In some implementations, similar input / output devices may be separate from computer system 1600 and may interact with one or more nodes of computer system 1600 via wired or wireless connections (such as via network interface 1640).

[0112] like Figure 16 As shown, memory 1620 may contain program instructions 1622, which may be executable by a processor to perform any of the elements or actions described above. In one embodiment, the program instructions may perform the methods described above. In other embodiments, different elements and data may be included. Note that the data may include any data or information described above.

[0113] Those skilled in the art will understand that computer system 1600 is merely illustrative and not intended to limit the scope of embodiments. Specifically, computer systems and devices may include any combination of hardware or software capable of performing the indicated functions, including computers, network devices, internet devices, personal digital assistants, wireless telephones, pagers, etc. Computer system 1600 may also be connected to other devices not shown, or conversely, may operate as a stand-alone system. Furthermore, the functionality provided by the illustrated components may, in some embodiments, be combined into fewer components or distributed across additional components. Similarly, in some embodiments, the functionality of some of the illustrated components may not be provided, and / or other additional functions may be available.

[0114] Those skilled in the art will also recognize that while various items are shown as being stored in memory or on storage devices during use, these items, or portions thereof, may be transferred between memory and other storage devices for memory management and data integrity purposes. Alternatively, in other embodiments, some or all of these software components may be executed in memory on another device and communicate with the illustrated computer system via inter-computer communication. Some or all of system components or data structures may also be stored (e.g., as instructions or structured data) on a computer-accessible medium or portable article of manufacture for reading by a suitable drive, various examples of which are described above. In some embodiments, instructions stored on a computer-accessible medium separate from computer system 1600 may be transmitted to computer system 1600 via a transmission medium or signal (such as electrical, electromagnetic, or digital signals transmitted via communication media such as networks and / or wireless links). Various embodiments may also include receiving, transmitting, or storing instructions and / or data implemented according to the above description on a computer-accessible medium. Generally, computer-accessible media can include non-transitory computer-readable storage media or memory media, such as magnetic or optical media, like discs or DVD / CD-ROMs, and volatile or non-volatile media, such as RAM (e.g., SDRAM, DDR, RDRAM, SRAM, etc.), ROM, etc. In some embodiments, computer-accessible media can include transmission media or signals, such as electrical signals, electromagnetic signals, or digital signals transmitted via communication media such as networks and / or wireless links.

[0115] In various implementations, the methods described herein can be implemented in software, hardware, or a combination thereof. Furthermore, the order of the blocks of the method can be changed, and various elements can be added, reordered, combined, omitted, modified, etc. Various modifications and changes will be apparent to those skilled in the art who benefit from this disclosure. The various implementations described herein are intended to be illustrative and not restrictive. Many variations, modifications, additions, and improvements are possible. Thus, multiple examples may be provided for a component described herein as a single example. The boundaries between various components, operations, and data repositories are somewhat arbitrary, and specific operations are shown in the context of a particular exemplary configuration. Other allocations of functionality are contemplated, which may fall within the scope of the appended claims. Finally, the structures and functions of discrete components presented in exemplary configurations can be implemented as combined structures or components. These and other variations, modifications, additions, and improvements may fall within the scope of the implementations as defined in the following claims.

Claims

1. A method for decoding compressed video image data, the method comprising: Receives encoded bitstreams of compressed images used for video; A first noise pattern and a second noise pattern are generated via an in-loop noise generator. The first noise pattern is to be added to a first portion of the decompressed version of the compressed image, and the second noise pattern is to be added to a second portion of the decompressed version of the compressed image. At least a portion of the second portion is different from the first portion of the decompressed version of the compressed image. The first noise pattern is added to the predicted pixel values ​​of the first portion corresponding to the decompressed version of the compressed image; as well as The second noise pattern is added to the predicted pixel values ​​of the second portion corresponding to the decompressed version of the compressed image; The first noise pattern and the second noise pattern are added to the predicted pixel values ​​that have been adjusted based on the decoded residual pixel values ​​included in the bit stream, wherein the adjusted predicted pixel values ​​have been filtered by one or more in-loop filters in the decompression process before the first noise pattern and the second noise pattern are added.

2. The method of claim 1, wherein the encoded bitstream includes data indicating a noise model or noise parameters to be used to generate noise for the first and second portions of the decompressed version of the compressed image.

3. The method according to claim 2, further comprising: Based on the data indicating the noise model or the noise parameters, the first noise pattern and the second noise pattern are generated via the in-loop noise generator.

4. The method of claim 1, wherein the first noise pattern and the second noise pattern are added to the predicted pixel value at the image block level, wherein different noise pattern values ​​are added to different blocks of the decompressed version of the compressed image.

5. The method of claim 1, wherein the first noise pattern and the second noise pattern are added to the predicted pixel value at a noise portion level, wherein each noise portion of the decompressed version of the compressed image overlaps with two or more blocks of the decompressed version of the compressed image, and wherein different noise pattern values ​​are added to different noise portions of the noise portions of the decompressed version of the compressed image.

6. The method of claim 1, wherein the first noise pattern and the second noise pattern are added to the predicted pixel value before the predicted pixel is filtered by one or more in-loop filters in the decompression process.

7. The method according to claim 1, further comprising: Predict the pixel values ​​to be used for the compressed image; Filter noise from predicted pixel values; as well as The residual value is applied to the predicted value to determine the decompressed pixel value. The predicted values ​​from the noise filtering, along with the residual pixel values ​​and the first and second noise patterns, are used to generate the decompressed version of the compressed image.

8. The method according to claim 1, further comprising: Before using a decompressed version of another compressed image to determine the estimated motion between the images during motion estimation, noise is filtered from the decompressed version of the other compressed image included in adjacent frames of the video, wherein noise is filtered from the decompressed version of the other compressed image using one or more filters selected based on a noise model, the noise model being selected to generate a first noise pattern and a second noise pattern for the decompressed version of the other compressed image.

9. A system configured to decode compressed video image data, the system comprising: One or more processors; as well as One or more memories storing program instructions that, when executed on or throughout the one or more processors, cause the one or more processors to: Receives encoded bitstreams of compressed images used for video; A first noise pattern and a second noise pattern are generated via an in-loop noise generator. The first noise pattern is to be added to a first portion of the decompressed version of the compressed image, and the second noise pattern is to be added to a second portion of the decompressed version of the compressed image. At least a portion of the second portion is different from the first portion of the decompressed version of the compressed image. The first noise pattern is added to the predicted pixel values ​​of the first portion corresponding to the decompressed version of the compressed image; as well as The second noise pattern is added to the predicted pixel values ​​of the second portion corresponding to the decompressed version of the compressed image; The first noise pattern and the second noise pattern are added to the predicted pixel value at a noise portion level, wherein each noise portion of the decompressed version of the compressed image overlaps with two or more blocks of the decompressed version of the compressed image, and wherein different noise pattern values ​​are added to different noise portions of the noise portion of the decompressed version of the compressed image.

10. The system of claim 9, wherein the encoded bitstream includes data indicating a noise model or noise parameters to be used to generate noise for the first and second portions of the decompressed version of the compressed image, and wherein the program instructions, when executed on or throughout the one or more processors, cause the one or more processors to generate the first noise pattern and the second noise pattern via the in-loop noise generator based on the data indicating the noise model or noise parameters.

11. The system of claim 9, wherein the first noise pattern and the second noise pattern are added to the predicted pixel value at the image block level, wherein different noise pattern values ​​are added to different blocks of the decompressed version of the compressed image.

12. The system of claim 9, wherein the first noise pattern and the second noise pattern are added to the predicted pixel value before the predicted pixel is filtered by one or more in-loop filters in the decompression process.

13. The system of claim 9, wherein the first noise pattern and the second noise pattern are added to an adjusted predicted pixel value that has been adjusted based on the decoded residual pixel value included in the bitstream, wherein the adjusted predicted pixel value has been filtered by one or more in-loop filters of the decompression process before the first noise pattern and the second noise pattern are added.

14. The system of claim 9, wherein the program instructions, when executed on or throughout the one or more processors, cause the one or more processors to: Predict the pixel values ​​to be used for the compressed image; Filter noise from predicted pixel values; and The residual value is applied to the predicted value to determine the decompressed pixel value. The predicted values ​​from the noise filtering, along with the residual pixel values ​​and the first and second noise patterns, are used to generate the decompressed version of the compressed image.

15. The system of claim 9, wherein the program instructions, when executed on or throughout the one or more processors, cause the one or more processors to: Before using a decompressed version of another compressed image to determine the estimated motion between the images during motion estimation, noise is filtered from the decompressed version of the other compressed image included in adjacent frames of the video, wherein noise is filtered from the decompressed version of the other compressed image using one or more filters selected based on a noise model, the noise model being selected to generate a first noise pattern and a second noise pattern for the decompressed version of the other compressed image.

16. A method for encoding video image data, the method comprising: Select the first part of the video image; Determine the first noise pattern to be added to the first portion of the image; Select a second portion of the image, at least a portion of which is different from the first portion of the image; Determine the second noise pattern to be added to the second part of the image; The first noise pattern and the second noise pattern are generated via a noise generator included in a compression loop for compressing pixel value data for the image, wherein the first noise pattern and the second noise pattern are different; as well as The pixel value data for the image is compressed based at least in part on the first noise pattern and the second noise pattern; The first noise pattern and the second noise pattern are to be added to the adjusted predicted pixel value, which has been adjusted based on the residual pixel value, wherein the adjusted predicted pixel value has been filtered by one or more in-loop filters of the compression process before the first noise pattern and the second noise pattern are added.

17. The method of claim 16, further comprising: The compressed pixel value data for the image is encoded, wherein the encoded bitstream for the compressed pixel value data includes data indicating a noise model or one or more noise parameters.

18. The method of claim 16, wherein the first noise pattern and the second noise pattern are added to the prediction residual used in the compressed loop as part of an intra-loop prediction or inter-loop prediction process.

19. A method for decoding compressed video image data, the method comprising: Receives encoded bitstreams of compressed images used for video; A first noise pattern for a first portion of the decompressed version of the compressed image and a second noise pattern for a second portion of the decompressed version of the compressed image are generated via an in-loop noise generator. The first noise pattern is added to the predicted pixel values ​​of the first portion corresponding to the decompressed version of the compressed image; as well as The second noise pattern is added to the predicted pixel values ​​of the second portion corresponding to the decompressed version of the compressed image.

20. The method of claim 19, wherein the encoded bitstream includes data indicating a noise model or noise parameters to be used to generate noise for the first and second portions of the decompressed version of the compressed image.

21. The method of claim 20, further comprising: Based on the data indicating the noise model or the noise parameters, the first noise pattern and the second noise pattern are generated via the in-loop noise generator.

22. The method of claim 19, wherein the first noise pattern and the second noise pattern are added to the predicted pixel value at the image block level, wherein different noise pattern values ​​are added to different blocks of the decompressed version of the compressed image.

23. The method of claim 19, wherein the first noise pattern and the second noise pattern are added to the predicted pixel value at a noise portion level, wherein each noise portion of the decompressed version of the compressed image overlaps with two or more blocks of the decompressed version of the compressed image, and wherein different noise pattern values ​​are added to different noise portions of the noise portions of the decompressed version of the compressed image.

24. The method of claim 19, wherein the first noise pattern and the second noise pattern are added to the predicted pixel value before the predicted pixel is filtered by one or more in-loop filters in the decompression process.

25. The method of claim 19, wherein the first noise pattern and the second noise pattern are added to an adjusted predicted pixel value that has been adjusted based on the decoded residual pixel value included in the bitstream, wherein the adjusted predicted pixel value has been filtered by one or more in-loop filters of the decompression process before the first noise pattern and the second noise pattern are added.

26. The method of claim 19, further comprising: Predict the pixel values ​​to be used for the compressed image; Filter noise from predicted pixel values; as well as The residual value is applied to the predicted value to determine the decompressed pixel value. The predicted values ​​from the noise filtering, along with the residual pixel values ​​and the first and second noise patterns, are used to generate the decompressed version of the compressed image.

27. The method of claim 19, further comprising: Before using a decompressed version of another compressed image to determine the estimated motion between the images during motion estimation, noise is filtered from the decompressed version of the other compressed image included in adjacent frames of the video, wherein noise is filtered from the decompressed version of the other compressed image using one or more filters selected based on a noise model, the noise model being selected to generate a first noise pattern and a second noise pattern for the decompressed version of the other compressed image.

28. A system configured to decode compressed video image data, the system comprising: One or more processors; as well as One or more memories storing program instructions that, when executed on or throughout the one or more processors, cause the one or more processors to: Receives encoded bitstreams of compressed images used for video; A first noise pattern for a first portion of the decompressed version of the compressed image and a second noise pattern for a second portion of the decompressed version of the compressed image are generated via an in-loop noise generator. The first noise pattern is added to the predicted pixel values ​​of the first portion corresponding to the decompressed version of the compressed image; as well as The second noise pattern is added to the predicted pixel values ​​of the second portion corresponding to the decompressed version of the compressed image.

29. The system of claim 28, wherein the encoded bitstream includes data indicating a noise model or noise parameters to be used to generate noise for the first and second portions of the decompressed version of the compressed image, and wherein the program instructions, when executed on or throughout the one or more processors, cause the one or more processors to: generate the first noise pattern and the second noise pattern via the in-loop noise generator based on the data indicating the noise model or noise parameters.

30. The system of claim 28, wherein the first noise pattern and the second noise pattern are added to the predicted pixel value at the image block level, wherein different noise pattern values ​​are added to different blocks of the decompressed version of the compressed image.

31. The system of claim 28, wherein the first noise pattern and the second noise pattern are added to the predicted pixel value at a noise portion level, wherein each noise portion of the decompressed version of the compressed image overlaps with two or more blocks of the decompressed version of the compressed image, and wherein different noise pattern values ​​are added to different noise portions of the noise portions of the decompressed version of the compressed image.

32. The system of claim 28, wherein the first noise pattern and the second noise pattern are added to the predicted pixel value before the predicted pixel is filtered by one or more in-loop filters in the decompression process.

33. The system of claim 28, wherein the first noise pattern and the second noise pattern are added to an adjusted predicted pixel value that has been adjusted based on the decoded residual pixel value included in the bitstream, wherein the adjusted predicted pixel value has been filtered by one or more in-loop filters of the decompression process before the first noise pattern and the second noise pattern are added.

34. The system of claim 28, wherein the program instructions, when executed on or throughout the one or more processors, cause the one or more processors to: Predict the pixel values ​​to be used for the compressed image; Filter noise from predicted pixel values; and The residual value is applied to the predicted value to determine the decompressed pixel value. The predicted values ​​from the noise filtering, along with the residual pixel values ​​and the first and second noise patterns, are used to generate the decompressed version of the compressed image.

35. The system of claim 28, wherein the program instructions, when executed on or throughout the one or more processors, cause the one or more processors to: Before using a decompressed version of another compressed image to determine the estimated motion between the images during motion estimation, noise is filtered from the decompressed version of the other compressed image included in adjacent frames of the video, wherein noise is filtered from the decompressed version of the other compressed image using one or more filters selected based on a noise model, the noise model being selected to generate a first noise pattern and a second noise pattern for the decompressed version of the other compressed image.

36. A method for encoding video image data, the method comprising: Select the first part of the video image; Determine the first noise pattern to be generated for the first portion of the image; Select a second portion of the image, at least a portion of which is different from the first portion of the image; Determine the second noise pattern to be generated for the second portion of the image; The first noise pattern and the second noise pattern are generated via a noise generator included in a compression loop for compressing pixel value data for the image, wherein the first noise pattern and the second noise pattern are different; as well as The pixel value data for the image is compressed based at least in part on the first noise pattern and the second noise pattern.

37. The method of claim 36, further comprising: The compressed pixel value data for the image is encoded, wherein the encoded bitstream for the compressed pixel value data includes data indicating a noise model or one or more noise parameters.

38. The method of claim 36, wherein the first noise pattern and the second noise pattern are added to the prediction residual used in the compressed loop as part of an intra-loop prediction or inter-loop prediction process.

Images