Combined loop filtering for image processing

Abstract

In an image processing device (i.e., an encoder or decoder), the number of loop filter stages is reduced by combining bilateral loop filtering (or Hadamard loop filtering) with sample adaptive offset filtering (SAO) or adaptive loop filtering (ALF). This avoids implementation problems associated with too many loop filter stages and provides approximately the same compression efficiency gain as having a separate loop filter stage.

Description

Technical Field

[0001] The present invention relates generally to image and video processing, and in particular to a system and method for performing combined loop filtering operations on image data. Background Art

[0002] Technology, especially the advent of sophisticated wireless mobile communications devices, most of which include one or more cameras, has made digital imaging ubiquitous in everyday life in many parts of the world. Video is prevalent, not only from traditional sources such as broadcast and cable TV distribution systems, but also from computers, tablet devices, smartphones, and even watches. In addition, news and entertainment sources, as well as social media networks that are redefining interpersonal communities, are saturated with high-resolution color digital images. The sheer volume of imaging has been a driving factor in the development of state-of-the-art network technologies, digital storage, and camera and display technologies.

[0003] Significant advances have been made in image processing algorithms as well as specialized image processing hardware. However, even as bandwidth and data speeds of networks, the capacity of storage devices, etc. continue to advance, intelligently processing digital images to reduce their storage size and facilitate their efficient transmission over networks remains an important area of ​​research and development. This applies to both single images and videos, the latter consisting of sequences of single images.

[0004] When viewed on a screen, an image (whether a still image or a video frame) consists of pixels, each of which has a red, green, and blue value (RGB). However, when encoding and decoding video sequences, images are not usually represented using RGB, but are often represented in another color space. Common image color spaces include (but are not limited to): YCbCr; ICtCp; and variations of these, such as non-constant brightness YCbCr and constant brightness YCbCr. Consider the example of YCbCr, which includes three components: luminance (Y) and blue difference (Cb) and red difference (Cr) chrominance components. The Y component roughly represents brightness or intensity, which is full resolution, while the resolution of the chrominance components Cb and Cr is usually smaller. A typical example is a high-definition (HD) video sequence containing 1920×1080 RGB pixels, which is usually represented by a Y component of 1920×1080 resolution, a Cb component of 960×540, and a Cr component of 960×540. The elements in these components are called samples. In the example of high-definition video, there are therefore 1920×1080 samples in the Y component, so there is a direct relationship between samples and pixels. In this context, the terms pixel and sample are synonymous and can be used interchangeably. For the Cb and Cr components, there is no direct relationship between samples and pixels; a single Cb sample typically affects several pixels.

[0005] In many video coding standards, the components Y, Cb and Cr are further divided into blocks. As an example, in Advanced Video Coding (AVC), an image is divided into macroblocks of 16×16 Y samples and 8×8 Cb and Cr samples, which represent the same 16×16 pixel area.

[0006] In High Efficiency Video Coding (HEVC), an image is divided into Coding Tree Units (CTUs). A CTU consists of an N×N block of luma samples, M×M chroma samples for Cb, and M×M chroma samples for Cr. As an example, N=64 and M=32. A CTU can be divided into four square blocks, each of which can be recursively divided into four square blocks. This forms a partition tree with the CTU as the root and square blocks called coding units (CUs) as leaves.

[0007] In the Versatile Video Coding (VVC) standard, an image is divided into coding tree units (CTUs). A CTU consists of an N×N block of luma samples, M×M chroma samples for Cb, and M×M chroma samples for Cr. As a typical example, N=128 and M=64. As in the case of HEVC, the CTU can then be divided into smaller blocks; however, these blocks do not have to be square. As an example, a block can be divided into two smaller blocks using horizontal partitioning, where the width of the divided block is the same as the width of the original block, but the height is half the height of the original block. This partitioning can be done recursively, forming a partition tree, where the CTU is the root and the blocks on the leaves are called coding units (CUs). These CUs can be further divided into transform units or TUs. In the decoder, the samples of the TU are first predicted, using samples from previously decoded blocks in the same image (intra-frame prediction), or using samples from blocks in previously decoded images (inter-frame prediction), or a combination of both.

[0008] It is known in the art that bilateral filtering of image data directly after forming a reconstructed image block can facilitate video compression, such as P. Wennersten, J. Y. Wang, K. Andersson, R. and J.Enhorn, “Bilateral Filtering for Video Coding,” IEEE Visual Communications and Image Processing (VCIP), December 2017, available at http: / / www.jacobstrom.com / publications / Wennersten et al VCIP2017.pdfobtained, the entire disclosure of which is incorporated herein by reference. Wennersten et al. showed that the use of bilateral filters can reduce bitrate while maintaining visual quality. The Bjontegaard metric provides a way to compare codecs or encoding techniques on the same data and produces a metric called BD-rate, where a negative incremental BD-rate number of -1% means that the bitrate has been reduced by 1% while maintaining the same visual quality. For the filter in the Wennersten paper, the incremental BD-rate is -0.5% when the runtime for random access increases by 3% (encoding) and 0% (decoding). The runtime is the time required to encode or decode a sequence, and a low runtime increase is expected. Since the filtering in this case occurs directly after block reconstruction, this type of bilateral filtering is referred to as "post-reconstruction bilateral filtering" in this article.

[0009] An alternative to filtering directly after forming the reconstructed block is to delay filtering until the entire image has been reconstructed, at which point the entire image (or part of it) can be filtered. J. P. Wennersten, J. Enhorn, and R. This approach is described in the standardization contribution “CE1-related: Multiplication-free bilateral loop filter”, presented as JVET-N0493 at the 14th JVET meeting, Geneva, Switzerland, March 19-27, 2019, available as a zip file from http: / / phenix.it-sudparis.eu / jvet / doc_end_user / documents / 14 _ Geneva / wg11 / JVET-N0493-v4.zip Download, the entire disclosure of which is incorporated herein by reference. This filtering scheme is generally referred to as a loop filter, or in-loop filter, because the result of the filtering is looped back to be used for prediction of frames that have not yet been encoded / decoded. Therefore, this type of bilateral filtering is referred to as "bilateral in-loop filtering" herein.

[0010] As an alternative to the bilateral filter, filtering can also be performed in the Hadamard domain, as described by S. Ikonin, V. Stepin, A. Karabutov and J. Chen in the standardization contribution “CE1: Hadamard transformdomain filter (CE1-2)”, published as JVET-N0478 at the 14th JVET meeting, Geneva, Switzerland, March 19-27, 2019, available as a compressed file from http: / / phenix.it-sudparis.eu / ivet / doc_ end_user / documents / 14_Geneva / we11 / JVET-N0478-v3.zipDownload, the entire disclosure of which is incorporated herein by reference. Hadamard domain filtering can be performed directly after reconstruction of each block (post-reconstruction Hadamard filtering) or as a loop filter (Hadamard loop filtering) as in the case of a bilateral filter.

[0011] Both post-reconstruction filtering and loop filtering have disadvantages. In the case of post-reconstruction filtering, an undesirable delay is introduced from the time when the unfiltered samples are generated to the time when they have been filtered. This is problematic because the filtered samples may be needed to make predictions for neighboring blocks. A known technique to minimize this disadvantage is to avoid filtering smaller blocks and avoid using pixels outside of the blocks. J. It is described by P. Wennersten, J. Enhorn, K. Reuze, D. Rusanovskyy and M. Karczewicz in the standardization contribution “CE1: Bilateral filter tests”, presented as JVET-N0489 at the 14th JVET meeting, Geneva, Switzerland, March 19-27, 2019, available as a zip file from http: / / phenix.it-sudparis.eu / jvet / doc_end_user / documents / 14_Geneva / wg11 / JVET- N0489-v2.zip Download, the entire disclosure of which is incorporated herein by reference. Unfortunately, however, this reduces the compression efficiency in terms of BD rate from -0.5% to about -0.35% (note that a larger negative number indicates better compression efficiency).

[0012] Loop filtering can achieve better gains, for example, about -0.65%, but introduces another drawback, namely that it includes a separate stage in which all samples must be touched. The 5.0 draft of the VVC standard already includes three loop filters: a deblocking filter, a filter called sample adaptive offset (SAO), and a filter called adaptive loop filter (ALF). Having many sequential filters makes hardware implementation difficult because they are usually applied at least partially in parallel. As a simplified example, since the image is reconstructed from top to bottom, once the upper half of the image has been reconstructed, SAO filtering can be started on the upper half. Once completed, SAO filtering of the lower half can be performed in parallel with ALF filtering of the upper half. Finally, the lower half of the image is processed with ALF. This can save time well if the filtering time for the two halves of the image is roughly the same. However, it may be the case that the SAO processing of the upper half of the image is very slow, while the ALF is very fast, and vice versa. If this happens, it becomes almost impossible to save time and the implementation must accelerate ALF and SAO separately, which is typically expensive in terms of silicon surface area.

[0013] It will be appreciated that this situation gets worse as the number of loop filters increases. It is therefore questionable whether the gain in compression efficiency provided by bilateral loop filtering is worth the additional implementation cost.

[0014] The background technology section of this document is provided to place the embodiments of the present invention in a technical and operational context to help those skilled in the art understand their scope and utility. The methods described in the background technology section can be considered, but these methods are not necessarily methods that have been previously conceived or considered. Unless explicitly stated, any statement herein is not considered to be prior art by virtue of being included in the background technology section.

[0015] The following papers provide additional background information and are incorporated herein by reference in their entirety:

[0016] J. “Non-CE: Reduced complexity bilateral filter”, JVET-M0885, 13th JVET conference, Marrakech, Morocco, 9-18 January 2019. The paper is available as a zip file from http: / / phenix.it-sudparis.eu / jvet / doc_end_user / documents / 13_Marrakech / wg11 / JVET- M0885-v4.zip download

[0017] J. P.Wennersten, J.Enhorn, D.Liu, K.Andersson, R. “CE2related: Reduced complexity bilateral filter”, JVET-K0274, 11th JVET Conference, Ljubljana, Slovenia, July 10-18, 2018. The paper is available as a zip file from http: / / phenix.it- sudparis.eu / jvet / doc_end_user / documents / 11_Ljubliana / wg11 / JVET-K0274-v5.zip download

[0018] S. Ikonin, V. Stepin, J. Chen, “Non-CE: Hadamard transform domain filter”, JVET-M0468, 13th JVET Conference, Marrakech, Morocco, January 9-18, 2019. The paper is available as a zip file from http: / / phenix.it-sudparis.eu / ivet / doc_end_user / documents / 13_Marrakech / wg11 / JVET-M0468-v2.zip download Summary of the invention

[0019] A brief summary of the disclosure is presented below to provide a basic understanding to those skilled in the art. This summary is not a detailed review of the disclosure and is not intended to identify key / important elements of embodiments of the invention or to delineate the scope of the invention. The sole purpose of this summary is to present some of the concepts disclosed herein in a simplified form as a preface to a more detailed description presented later.

[0020] According to one or more embodiments described and claimed herein, the number of loop filter stages is reduced by combining bilateral loop filtering (or Hadamard loop filtering) with sample adaptive offset filtering (SAO) or adaptive loop filtering (ALF). This avoids implementation issues associated with too many loop filter stages and provides approximately the same compression efficiency gain as having separate loop filter stages.

[0021] One embodiment relates to a method of applying multiple different filter operations to image data. The image data is divided into one or more partitions. For each partition of the image data, a first filtering operation is applied to a current partition of the image data to generate one of first filtered image data and first incremental data; a second filtering operation is applied to the current partition of the image data to generate one of second filtered image data and second incremental data; the outputs of the first filtering operation and the second filtering operation for the current partition are combined to generate combined filtered image data; and the combined filtered image data for the current partition is cropped.

[0022] Another embodiment relates to an image processing device adapted to apply multiple different filtering operations to image data. The image processing device includes a processor adapted to divide the image data into one or more partitions. For each partition of the image data, the processor is adapted to: apply a first filtering operation to a current partition of the image data to generate one of first filtered image data and first incremental data; apply a second filtering operation to the current partition of the image data to generate one of second filtered image data and second incremental data; combine the outputs of the first filtering operation and the second filtering operation for the current partition to generate combined filtered image data; and crop the combined filtered image data for the current partition. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The present invention will now be described more fully hereinafter with reference to the accompanying drawings, in which embodiments of the present invention are shown. However, the present invention should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided so that the present disclosure will be comprehensive and complete, and the scope of the present invention will be fully conveyed to those skilled in the art. Throughout the text, similar reference numerals represent similar elements.

[0024] Figure 1is a block diagram / flow chart of a conventional method for filtering image data for a decoder.

[0025] Figure 2 is a block diagram / flow chart of a method for combined image data filtering.

[0026] Figure 3 is a block diagram / flow chart of an iterative method for combined image data filtering suitable for hardware implementation.

[0027] Figure 4 is a block diagram / flow chart of a separate iterative method for combined image data filtering suitable for software implementation.

[0028] Figure 5 is a block diagram / flow chart of an iterative method for individually combining image data filtering suitable for software implementation.

[0029] Figure 6 A block diagram / flow chart of a parallel concurrent method for combined image data filtering implemented on multiple CPUs or CPU cores.

[0030] Figure 7 is a block diagram / flow chart of a conventional method for filtering image data for an encoder.

[0031] Figure 8 is a block diagram / flow chart of a method of combined image data filtering according to one embodiment.

[0032] Fig. 9 is a block diagram / flow chart of a method of combined image data filtering according to another embodiment.

[0033] Fig.10 It is a block diagram of an image processing device.

[0034] Fig.11 is a flow chart of a method of applying a plurality of different filter operations to image data. DETAILED DESCRIPTION

[0035] This application claims priority to U.S. Patent Application No. 62 / 865,533, filed on June 24, 2019, the entire contents of which are incorporated herein by reference.

[0036] For the purpose of simplicity and illustration, the present invention is described primarily with reference to exemplary embodiments. In the following description, a large number of specific details are set forth in order to provide a complete understanding of the present invention. However, it will be apparent to those skilled in the art that the present invention can be performed without the limitation of these specific details. In this specification, well-known methods and structures are not described in detail to avoid unnecessarily obscuring the present invention. Although at least some embodiments herein may be described as being applicable to certain contexts and / or wireless network types for illustrative purposes, these embodiments are also equally applicable to other contexts and / or wireless network types that are not explicitly described.

[0037] This article uses the example of combining a bilateral loop filtering operation with an SAO filtering operation to explain embodiments of the present invention. However, the present invention is not limited to these specific examples. In general, the bilateral loop filtering operation can be combined with an ALF filtering operation, a deblocking filter, or any other filter. In addition, the combined filter does not need to be a bilateral loop filter. For example, using the teachings of this article, a person skilled in the art can combine a Hadamard filter and an SAO, a Hadamard and an ALF, or even an SAO and an ALF.

[0038] Figure 1 A conventional loop filter implementation 10 for a decoder is shown, such as (referenced above) and those described by Ikonin. In this case, the output of one of the filters 12 is used as the input to the other filter 14. This is the case for SAO and ALF in the current version of the VVC draft: the output of SAO is used as the input to ALF. Figure 1 In , a bilateral filter 12 with clipping is first in the decoding chain and the output is fed to a SAO filter 14 which also includes clipping.

[0039] As mentioned above, this leads to implementation problems. Although it is possible to start processing a portion of an image with a first filter 12, and then process that portion with a second filter 14 after the first filter 12 has processed that portion, this can lead to implementation inefficiencies if the two filters 12, 14 operate on different parts of the image (or portion of an image) at different speeds.

[0040] Figure 2 A decoder filter implementation 20 according to an embodiment of the present invention is depicted. In the embodiment 20, both the first filter 22 and the second filter 24 (here again the bilateral filter 22 and the SAO 24 are used as examples) receive the same image data. Figure 2As shown, the input data may be image data after deblocking, but in general, it may be any input data, such as image data before any loop filtering, or image data after ALF. The outputs of the two filters 22, 24 are combined (e.g., in a summer 26), and the result is then subjected to a clipping operation 28.

[0041] Note that the first filter 22 and the second filter 24 operate in parallel because both filters 22, 24 operate on the same input image data. As further detailed herein, the image data may include an entire image, a large portion of an image (e.g., a CTU), a smaller portion of an image (e.g., a CU or TU), a smaller portion of an image (e.g., a group of pixels or even a single pixel). In addition, although the first filter 22 and the second filter 24 operate in parallel, they may not necessarily operate at the same time. That is, the same hardware may be reused for each of the first filtering operation 22 and the second filtering operation 24 (e.g., the output of one filtering operation 22, 24 may be saved while another filtering operation 24, 22 is performed, and then their outputs are combined 26 and cropped 28).

[0042] In more detail, assume that the input data is pixel intensity I(x, y), where I(x, y) represents the brightness value (Y value in YCbCr color space) of the sample at pixel location (x, y). Bilateral filtering of such samples is expressed as:

[0043] I BIF (x,y)=BIF(I(x,y),I(x-1,y),I(x+1,y),I(x,y-1),I(x,y+1),...)

[0044] Equation 1

[0045] Where I(x-1, y), I(x+1, y), etc. are the intensity values ​​of samples around the pixel position I(x, y). To simplify the notation, it is abbreviated as:

[0046] I BIF (x,y)=BIF(I(x,y)) Equation 2

[0047] Similarly, the SAO filtered version of I(x, y) is expressed as:

[0048] I SAO (x,y)=SAO(I(x,y)) Equation 3

[0049] Although SAO filtering usually also depends on surrounding samples and parameters, these are not shown for simplicity.

[0050] Figure 1 The traditional filtering arrangement shown in will produce:

[0051] I BIF (x,y)=BIF(I(x,y)) Equation 4

[0052] I BIFC (x, y) = clip(I BIF (x, y)) Equation 5

[0053] I SAO (x, y) = SAO(I BIFC (x, y)) Equation 6

[0054] I SAOC (x, y) = clip(I SAO (x, y)) Equation 7

[0055] The function clip ensures that the sample is still within its legal range, for example, [0, 1023] for 10-bit data. BIFC and I SAOC I BIF and I SAO An example of a clip function may be clip(x) = max(0, min(1023, x)), although in general, the output may be clipped to any range. For example, in one embodiment, the minimum clipping value c is signaled from the encoder. min and the maximum clipping value c min Notify the decoder and then use clip(x)=max(c min ,min(c max , x)) performs clipping.

[0056] As mentioned above, both filters 22, 24 receive the same input samples I(x, y):

[0057] I BIF (x,y)=BIF(I(x,y)) Equation 8.1

[0058] I SAO (x,y)=SAO(I(x,y)) Equation 8.2

[0059] In one embodiment, combining 26 these filter outputs includes calculating the difference between each filtered sample and the input sample:

[0060] ΔI BIF (x, y) = I BIF (x, y)-I(x, y) Equation 9

[0061] ΔI SAO (x, y) = I SAO(x, y)-I(x, y) Equation 10

[0062] In this embodiment, the combined value I COMB It is just the input sample plus two differences:

[0063] I COMB (x, y) = I (x, y) + ΔI BIF (x,y)+ΔI SAO (x, y) Equation 11

[0064] Generally, filters 22, 24 may output image data (eg, I BIF (x, y), I SAO (x, y)), referred to herein as "filtered image data". Alternatively, the filters 22, 24 may directly output the difference values ​​(eg, ΔI BIF (x, y), ΔI SAO (x, y)), referred to herein as "delta data". Of course, in the latter case, equations 9 and 10 are not necessary, and equation 11 will operate directly on the outputs of filters 22, 24.

[0065] The final value is obtained by combining the values ​​I COMB The result of trimming by 28 is:

[0066] I COMBC (x, y) = clip(I COMB (x, y)) Equation 12

[0067] It should be noted that if more than two filters are combined, clipping should not be applied until the outputs of all filters are combined.

[0068] Note that if both filters 22, 24 output filtered image data, then it is not necessary to generate two delta data values. Instead, either delta may be added to the filtered image of the other filter:

[0069] I COMB (x, y) = I BIF (x,y)+ΔI SAO (x, y) Equation 13

[0070] I COMB (x, y) = ΔI BIF (x,y)+I SAO (x, y) Equation 14

[0071] As described above, many image processing filters are implemented so that they output delta data rather than filtered image data. In the case of the bilateral filter in the paper, the calculation of the final filtered pixel value is given by Equation 10 in the document:

[0072] I F =I C +((cm sum +4)>>3)(from Equation 10)

[0073] Among them I C is the input pixel value, I F is the filtered pixel value (output), Cm sum is the quantity calculated in the previous step. Converting it to the notation used in this article, it becomes:

[0074] I BIF (x, y) = I (x, y) + ((cm sum +4)>>3) Equation 14.1

[0075] Comparing this equation with Equation 9, it is clear that ΔI BIF (x, y) must be equal to ((cm sum +4)>>3). Therefore, for the embodiments of the present invention and The combination of the bilateral filter does not require the use of equation 14.1 to calculate I BIF (x, y), and then immediately subtract I(x, y) from it to obtain the desired quantity, ΔI BIF (x, y). Instead, the calculation would be ΔI BIF (x, y) = ((cm sum +4)>>3), this will be the output of the bilateral filter.

[0076] Generally, the filtering operation can output the filtered image data I FILTER (x, y), referred to herein as "filtered image data". Alternatively, the filtering operation may output an image such as ΔI FILTER The difference between (x, y) is referred to herein as "delta data". Specifically, both the bilateral filter and the SAO filter can be implemented to output delta data instead of filtered image data. For completeness and clarity, in the former case, equations 8.1 and 8.2 are rewritten as:

[0077] ΔI BIF (x,y)=ΔBIF(I(x,y)) Equation 14.2

[0078] ΔI SAO (x, y) = ΔSAO (I (x, y)) Equation 14.3

[0079] where ΔBIF(I(x, y)) represents the bilateral filter version of the output delta data, e.g. In the case of ΔBIF(I(x, y))=((cm sum +4)>>3). Similarly, ΔSAO(I(x,y)) represents a version of the SAO filter that outputs only delta data (i.e., the offset for each pixel, rather than the offset plus the input). In this case, equations 9 and 10 can be skipped, and the combined value I can be calculated directly from the filter output using equation 11 COMB .

[0080] Figure 2 Embodiment 20 allows to calculate I with great flexibility COMBC (x, y). As an example, we can first calculate ΔI BIF (x, y), and then later calculate ΔI SAO (x, y). Another possible way is to first calculate ΔI SAO (x, y), and then later calculate ΔI BIF (x, y). This flexibility is valuable, for example, if one filter is faster on one part of the data and another filter is faster on another part of the data. Figure 2 In Example 20, both can proceed at full speed, and filtering will never be slower than the slowest filter. This is in equations 4 to 7 ( Figure 1 ) is not possible with this arrangement. For simplicity, we assume a very simple filter where ΔI BIF (x, y) = 4, and ΔI SAO (x, y) = -5. According to formulas 4 to 7, for an input value of I(x, y) = 1020, we obtain:

[0081] I BIF (x, y)=BIF(I(x, y))=1020+4=1024 Equation 15

[0082] I BIFC (x, y) = clip(I BIF (x, y)) = clip (1024) = 1023 Equation 16

[0083] I SAO =SAO(I BIFC (x, y)) = 1023 - 5 = 1018 Equation 17

[0084] I SAOC =clip(I SAO (x, y)) = clip (1018) = 1018 Equation 18

[0085] However, a decoder that computes these in reverse order will get a different result:

[0086] I SAO =SAO(I(x, y))=1020-5=1015 Equation 19

[0087] I SAOC =clip(I SAO (x, y)) = clip (1015) = 1015 Equation 20

[0088] I BIF (x, y) = BIF(I SAOC (x, y)) = 1015 + 4 = 1019 Equation 21

[0089] I BIFC (x, y) = clip(I BIF (x, y)) = clip (1019) = 1019 Equation 22

[0090] Although the error is only one intensity level, this will induce a drift in the decoding process, possibly leading to unbounded errors. Therefore, the decoding must be bit-accurate and the decoder cannot choose between Equations 15 to 18 and Equations 19 to 22. In fact, the situation in reality is worse than this oversimplification, because the functions BIF(I(x, y)) and SAO(I(x, y)) depend on the input in a nonlinear way. This means that, even without clipping, it is impossible to reverse the order of Equations 15 and 17. However, in Figure 2 In performing the parallel filter operations depicted in and described herein with respect to equations 8.1 to 12, it is possible to rearrange the order of the calculations.

[0091] Figure 3 An embodiment 30 of a decoder that may be well suited for hardware implementation is described. In this embodiment, the hardware implementation obtains the next pixel (or group of pixels) 32 and applies a first filtering operation 34 (e.g., a bilateral filter) and a second filtering operation 36 (e.g., SAO). The Δ BIF (x, y) and ΔI SAO (x, y), and the outputs of the filters 34, 36 are combined 37 and clipped 38, for example using I COMBC =clip(I(x, y)+ΔI SAO (x,y)+ΔI BIF (x, y)). Then, the value I COMBCwritten to memory or passed on for further processing. The process is repeated 39 over all available pixels (or groups of pixels) in the image data 32. This iterative process avoids desynchronizing the two filters 34, 36 from each other and makes the clock frequency of the system easier to scale.

[0092] From a hardware perspective, another advantage of embodiment 30 is that both BIF filter 34 and SAO 36 use the same number of surrounding samples. The proposed bilateral filter 34 uses the following samples as input:

[0093] I BIF (x,y)=BIF(I(x,y),I(x+1,y),I(x-1,y),I(x,y+1),I(x,y-1),I(x+1,y+1),I(x+1,y-1),I(x-1,y+1),I(x-1,y-1)).

[0094] SAO filter 36 accesses the same pixel:

[0095] I SAO (x,y)=SAO(I(x,y),I(x+1,y),I(x-1,y),I(x,y+1),I(x,y-1),I(x+1,y+1),I(x+1,y-1),I(x-1,y+1),I(x-1,y-1)).

[0096] In more detail, SAO selects one of several filters, primarily a three-sample filter:

[0097] SAO BO (I(x,y))

[0098] SAO 135 (I(x-1,y-1),I(x,y),I(x+1,y+1))

[0099] SAO 90 (I(x,y-1),I(x,y),I(x,y+1))

[0100] SAO 45 (I(x+1,y+1),I(x,y),I(x-1,y-1))

[0101] SAO 0 (I(x-1,y),I(x,y),I(x+1,y))

[0102] Thus, SAO never accesses pixels other than those utilized by BIF, and vice versa - BIF never accesses pixels that SAO cannot utilize. Thus, the set of pixels 32 is compatible between filters 34, 36. For hardware implementation, this means that memory only needs to be read once and stored once, i.e., no buffers are required, which take up expensive silicon surface area.

[0103] Figure 4 An embodiment 40 of a decoder that has found a certain suitability is described, in which a first filtering operation 42 and a second filtering operation 45 are performed by executing software on a computing device (e.g., a digital signal processor (DSP) or a central processing unit (CPU)). In this case, the filtering operations 42, 45 are typically performed on larger partitions of the image data, such as CTUs, which are typically 128×128 samples. For many reasons, it may be desirable or required that the first filtering operation 42 and the second filtering operation 45 be performed sequentially rather than simultaneously. Structurally, the filtering operations 42, 45 are still performed in parallel because each filtering operation receives the same image data as input. However, the two filtering operations 42 and 45 independently loop through the image data at different times, each storing its output. These outputs are then obtained as part of a combining operation 47. The combined output is then clipped 48 and passed for downstream processing.

[0104] As an example, the CTUs of image data 41, 44 may include image data after deblocking. The process 40 may iterate over all CTUs 43 of the image data individually, performing bilateral filtering 42 and storing the output. The process 40 then loops over all CTUs 46 of the same image data again, this time performing SAO filtering 45 and storing the output. The outputs of the filtering operations 42, 45 are then retrieved and combined 47 before cropping 48.

[0105] One reason why it may be advantageous to separate the first filtering operation 42 and the second filtering operation 45 in time is that it is easy to write software code to implement the filtering operations 42 and 45, such as using single instruction multiple data (SIMD) code. As the name implies, SIMD executes one instruction on multiple data instances (such as pixels), providing a powerful means for developing very high-performance image processing software. However, in any particular computing device, the number of registers available is limited, and it may not be possible to accommodate two filters, such as bilateral filtering 42 and SAO filtering 45, in the same SIMD routine. Another reason is that the first filtering operation 42 and the second filtering operation 45 may not be interoperable in execution. For example, bilateral filtering 42 occurs on the entire image data, while SAO filtering 45 is turned off for some CTUs. Therefore, SAO filtering 45 can exit early and skip the entire CTU, which is difficult to do if the filtering operations 42 and 45 are combined at a per-pixel level.

[0106] Going further, the two filters may traverse the CTU in different ways in order to have an efficient implementation. As an example, a CTU is divided into smaller blocks called CUs, which in turn are divided into smaller blocks called TUs. A typical size of a CTU may be 128×128 pixels, while a TU may be 8×8 pixels. For a bilateral filter, parameters such as the filter strength are constant over the TU. Therefore, the implementation can be very efficient if the bilateral filter processes a CTU by iterating over each TU independently, i.e., the filter strength can be changed at the beginning of the subroutine without having to change it for every pixel. However, if the TU structure is ignored and the CTU is processed row by row, it would be necessary to check in each pixel what the filter strength should be. This would make it very difficult to implement efficiently in SIMD code. For example, in In the paper, the filter strength is determined by parameters k1 and k2, which depend on qp (always the same within a TU) and on the TU size (naturally always the same within a TU). In contrast, the SAO filter does not depend on the TU size, so it can be more efficient to process CTUs line by line, which can bring cache advantages because images are usually arranged line by line in memory.

[0107] Figure 5An alternative embodiment 50 is depicted in . In this embodiment, a second filtering operation 53 (e.g., SAO) is always performed after the first filtering operation 52 (e.g., bilateral filter). For each CTU 51, bilateral filtering 52 is first performed on the entire CTU 51 using the deblocked image data as input. The output of the bilateral filter 52 is then stored. Then, an SAO filtering operation 53 is performed on the same CTU 51 using the deblocked image data as input. For each pixel in the CTU 51, once the SAO filtering operation 53 is completed, the pixel is combined 54 with the corresponding pixel from the output of the bilateral filtering operation 52, cropped 55, and then stored. After the above operations are performed on each pixel in the CTU 56, the method is continued on the next CTU 51. An advantage of this embodiment 50 is that it is similar to Figure 4 Compared to the complete image in embodiment 40, only one CTU 51 worth of the output of the bilateral filter 52 needs to be stored. Figure 4 As in the embodiment 50, it is also possible to traverse the pixels in different ways in the two filtering operations 52, 53. For example, the first filtering operation 52 may traverse the CTU 51 TU by TU, while the second filtering operation 53 may traverse the CTU 51 row by row.

[0108] Figure 6 Yet another embodiment 60 of a decoder is depicted in which separate processors, also referred to as CPUs (or separate cores in a single multi-core CPU), each perform filtering operations 62, 64 on the entire image data. In this embodiment 60, the image data is "partitioned" into only one partition, the entire image. The data is then processed by two computational engines, e.g., CPU0 performs bilateral filtering 62 on the entire image data, and CPU1 performs SAO filtering 64 on the entire image data. The outputs of the filtering operations 62, 64 are then combined 66, cropped 68, and passed downstream for further processing. In some embodiments with very large image data, each processor may iteratively perform its filtering 62, 64 on very large partitions that are smaller than the entire image data.

[0109] The filtering required to encode the image data is similar to the filtering used in decoding, with some exceptions, such as the parameters of some filters (such as SAO) must be estimated. Examples of these parameters include which filter to use SAO BO 、SAO 0 、SAO 45 、SAO 90 Still SAO 135 , and determine the offset strength.

[0110] Figure 7A conventional encoding method 70 is depicted in which bilateral filtering and cropping 72 are performed on input image data (e.g., after deblocking). The filtered data is then used by an SAO filter parameter estimation function 74 to estimate parameters for an SAO filter 76. The SAO filter 76 receives the filtered image data from the bilateral filtering operation 72 and the parameters from the estimation function 74. The SAO filtering operation 76 includes cropping. The output of the SAO filtering operation 76 is then sent downstream, such as to ALF filtering. This method is similar to Figure 1 Implementation 10 suffers from the same drawbacks, for example, the difference in execution speed of the two filtering operations 72, 76 may lead to inefficiencies.

[0111] Figure 8 An embodiment 80 of combined filtering for an encoder according to an embodiment of the present invention is depicted. Figure 2 As in FIG. 1 , both filtering operations 82, 86 operate on the same input image data and are therefore performed in parallel (although not necessarily simultaneously). An SAO filter parameter estimation function 84 also operates on the input image data and provides parameters for the SAO filtering operation 86. The outputs of the first filtering operation 82 and the second filtering operation 86 are combined 88 and clipped 89 before being passed downstream for further processing.

[0112] In some aspects, it can be improved Figure 8 Embodiment 80. After processing, the combined image will contain bilateral filtering, but Figure 8 In the embodiment 80 of the figure, the SAO parameter estimation block does not know the result of the bilateral filtering. As an example, assume that the input intensity value I(x, y) = 500 in the pixel is too low compared to the original value 510, and its ideal value should be increased by 10 intensity levels. The bilateral filter can be able to select ΔI BIF (x, y) = 10 to fully correct this. However, the SAO filter only receives input I(x, y), which is 10 levels lower, but can also be corrected by choosing ΔI SAO (x, y) = 10 to correct it. The result will then be the combined value I COMB =I(x, y)+ΔI BIF (x,y)+ΔI SAO (x, y)=500+10+10=520, which is 10 levels higher instead of 10 levels lower, that is, overcorrection occurs, which is no better than the unfiltered pixel value I(x, y).

[0113] Fig. 9 An embodiment 90 is described that addresses this drawback. In embodiment 90, the input image data (e.g., after deblocking) is first bilaterally filtered 92 and then sent to the SAO parameter estimation function 94. This is similar to Figure 8 This is in contrast to the depicted embodiment 80, in which input image data is used as input to SAO parameter estimation 84. By using the bilaterally filtered output as input, the SAO parameter estimation process 94 is aware of the corrections applied by bilateral filtering 92. Furthermore, in this arrangement, SAO filtering operation 96 uses input image data (i.e., the same data as bilateral filtering operation 92) as input. This is in contrast to Figure 7 Unlike the conventional encoder shown, Figure 7 The filter in uses the output of the bilateral filter 72 as input to the SAO filtering operation 76.

[0114] It should be noted that with the use of Figure 1 and Figure 7 Compared with the traditional method shown in Figure 2 and Fig. 9 , the performance of the two filtering operations 92, 96 may be slightly degraded when using the arrangement depicted in . This is due to the fact that the SAO filtering uses different inputs in its estimation step and in its filtering step. However, according to measurements performed according to the invention, this performance degradation is very small; the BD rate drops from -0.43% to -0.42%. The greater flexibility gained by using embodiments of the invention should obviously offset this performance degradation.

[0115] Another point to note is that even if the decoder uses a filter that produces a difference, such as ΔBIF(I(x, y)) in equation 14.2 above, this type of filter cannot be used when providing input to the SAO parameter estimation process 94. The reason is that the SAO process 94 expects image data that can be directly compared to the original image data. Therefore, in one embodiment of the present invention, the decoder can use a bilateral filter 92 that outputs delta data ΔBIF(I(x, y)), but the encoder can use a bilateral filter 92 that outputs filtered image data BIF(I(x, y)) when producing input to the parameter estimation process 94 of the subsequent filter 96.

[0116] Although not explicitly described, Fig. 9 The architecture is easily optimized for use in hardware (with Figure 3 Similar), software (with Figure 4 or Figure 5 similar) or a separate CPU to perform the entire (or large) image processing (similar to Figure 6 Those skilled in the art will appreciate that these embodiments can be directly extended to the case of an encoder.

[0117] The steps of combining 26, 37, 47, 54, 66, 88, 98 the filter outputs are not limited to addition as shown in equations 9 to 11. In one embodiment, the combination may be a multiplication. In this embodiment, the ratios of the filtered image data to the image data are calculated and the image data is multiplied by these ratios. For example:

[0118] r BIF (x, y) = I BIF (x, y) / I(x, y) Equation 23

[0119] r SAO (x, y) = I SAO (x, y) / I(x, y) Equation 24

[0120] I COMB (x, y) = I (x, y) * r BIF (x,y)*r SAO (x, y) Equation 25

[0121] In one embodiment, to ensure that division by zero operations are avoided, a constant offset α is used in the numerator and denominator.

[0122] r BIF (x, y) = (I BIF (x,y)+α) / (I(x,y)+α) Equation 23'

[0123] r SAO (x, y) = (I SAO (x,y)+α) / (I(x,y)+α) Equation 24'

[0124] I COMB (x, y) = I (x, y) * r BIF (x,y)*r SAO (x, y) Equation 25

[0125] In this way, for example, when I(x, y) is close to or equal to 0, the ratio r BIF (x, y) will approach a stable value, rather than being unstable or undefined.

[0126] Fig.10A block diagram of an image processing device 100 is depicted, which can implement the embodiments described in the present invention and the claims herein. The image processing device 100 may include customized hardware that is optimized for performing image processing functions including filtering operations. Alternatively, the image processing device 100 may include a general-purpose computing device that is programmed to perform image processing functions including filtering operations. In either case, the image processing device 100 may implement an image or video encoder, decoder, or both, as well as other functions. In one embodiment, the image processing device 100 may include one or more processors 101, a memory 102, an input / output (I / O) circuit 103, a user interface 104, and / or a display 105. These components are connected in a data communication relationship via one or more buses 106. In some cases, some of these components may not exist, or may be combined in various combinations with another or other components not shown. These components may be located in a single physical entity or in more than one physical entity. Any actions taken by processor 101 described herein may be performed by processor 101 alone, or by processor 101 in conjunction with one or more components shown or not shown in the figure (e.g., a digital signal processor (DSP), a graphics coprocessor, etc.).

[0127] Processor 101 may include any one or more sequential state machines operable to execute machine instructions stored in memory 102 as a machine-readable computer program, such as one or more hardware-implemented state machines (e.g., in discrete logic, FPGA, ASIC, etc.); programmable logic together with appropriate firmware; one or more stored programs, general-purpose processors (e.g., microprocessors or digital signal processors (DSPs)), or any combination of the above. In some embodiments, the processor is a multi-core processor that includes two or more processing cores or instruction execution channels and is operable to perform two or more image filtering operations substantially simultaneously. Generally, processor 101 executes instructions, codes, computer programs, or scripts, which it can access from memory 102 or from one or more devices (not shown) accessed via I / O circuits 103. Although only one processor 101 is shown, multiple processors may also be present.

[0128] The memory 102 may include any non-transitory machine-readable medium known in the art or that may be developed, including but not limited to magnetic media (e.g., floppy disks, hard disks, etc.), optical media (e.g., CD-ROMs, DVD-ROMs, etc.), solid-state media (e.g., SRAM, DRAM, DDRAM, ROM, PROM, EPROM, flash memory, solid-state optical disks, etc.), etc. The memory may store image data to be processed, intermediate filtering operation outputs, and combined filtering operation outputs, as well as instructions for the processor 101. Although depicted as separate entities, those skilled in the art will appreciate that many processors 101 include multiple forms of memory 102 (e.g., registers, caches, CAMs, etc.).

[0129] I / O circuit 103 provides connection and data communication / transmission with any number of external devices or interfaces. I / O circuit 103 may employ one or more modems, modem banks, Ethernet devices, universal serial bus (USB) interface devices, serial interfaces, token ring devices, fiber distributed data interface (FDDI) devices, wireless local area network (WLAN) devices, radio transceiver devices (e.g., 3G, 4G or 5G wireless cellular network interface devices, Worldwide Interoperability for Microwave Access (WiMAX) devices) and / or other known devices for connecting to a network. I / O circuit 103 may enable processor 101 to communicate with the Internet, or one or more telecommunications networks, or other networks, and processor 101 may receive information from or output information to these networks.

[0130] Although many stand-alone, dedicated image processing devices 100 may have minimal or no user interface, in some embodiments, the image processing device 100 includes one or more user interfaces 104. These may include, for example, a keyboard, keypad, switches, dials, mouse, trackball, voice recognizer, touch screen, etc.

[0131] The display 105 may include any suitable display unit for displaying information suitable for the image processing device 100. In addition, the display 105 may be implemented as an additional user interface 104 device, such as a touch screen, a touch pad, a touch screen panel, etc. Touch screen technology allows the use of the display 105 as an input device to remove or enhance a keyboard and / or a mouse as the main input device for interacting with the content provided on the display 105. In one embodiment, for example, the display 105 may be implemented as a liquid crystal display (LCD) or other type of suitable visual interface. The display 105 may include, for example, a touch-sensitive color display screen.

[0132] Those skilled in the art will appreciate that, in any particular implementation, the image processing device 100 may include Fig.10Only some of the components and interfaces depicted in Fig.10 It is not intended to describe an exhaustive list of circuits, and the image processing device 100 may include numerous other functions and features.

[0133] Fig.11 A method 110 of applying a plurality of different filter operations to image data is depicted. For example, the method 110 may be executed as computer software on a processor 101 of an image processing device 100. The image data is divided into one or more data partitions (box 111). For each partition of the image data, a first filtering operation is applied to a current partition of the image data to generate one of first filtered image data and first incremental data (box 112). A second filtering operation is applied to the current partition of the image data to generate one of second filtered image data and second incremental data (box 113); the outputs of the first filtering operation and the second filtering operation for the current partition are combined to generate combined filtered image data (box 114). The combined filtered image data for the current partition is cropped (box 115). The process is repeated iteratively until all partitions of the image data have been processed (box 116).

[0134] Embodiments of the present invention present many advantages over filtering methods known in the prior art. Combining the first filtering operation with the second filtering operation reduces implementation problems associated with too many loop filter stages that may lose synchronization. In addition, it provides compression efficiency gains that are approximately the same as having separate loop filter stages. Embodiments of the present invention also make it possible to choose how tightly coupled the combined filter is in a decoder or encoder implementation. For example, if a bilateral filter is combined with SAO filtering, a hardware implementation may choose to implement decoding or encoding entirely in lockstep, so that each sample is filtered using a bilateral filter and SAO together, and then goes to the next sample, etc. However, a software implementation may prefer to apply a bilateral filter on the entire CTU first, for example, which can be done efficiently with SIMD instructions, and then apply SAO to the same CTU. It may even be possible to apply SAO first and then apply a bilateral filter, with exactly the same results, which is critical to avoiding decoder drift. In addition, a CPU implementation may decide to filter the entire image with a bilateral filter on one CPU core while performing SAO on the entire image on another CPU core.

[0135] Any appropriate steps, methods, features, functions or benefits disclosed herein may be performed by one or more functional units or modules of one or more virtual devices. Each virtual device may include a plurality of these functional units. These functional units may be implemented via a processing circuit, which may include one or more microprocessors or microcontrollers and other digital hardware (may include a digital signal processor (DSP), dedicated digital logic, etc.). The processing circuit may be configured to execute program codes stored in a memory, which may include one or more types of memory, such as a read-only memory (ROM), a random access memory (RAM), a cache memory, a flash memory device, an optical storage device, etc. The program code stored in the memory includes program instructions for executing one or more telecommunications and / or data communication protocols and program instructions for executing one or more technologies described herein. In some implementations, the processing circuit may be used to cause the corresponding functional unit to perform a corresponding function according to one or more embodiments of the present disclosure.

[0136] Generally, unless a different meaning is clearly given and / or implied from the context, all terms used in this article will be interpreted according to their common meaning in the relevant technical field. As used herein, the term "suitable for" means setting, organization, configuration or arrangement to operate in a specific way; the term is synonymous with "designed to". As used herein, the terms "approximately", "substantially", etc. include and illustrate mechanical tolerances, measurement errors, random variations and similar sources of imprecision. Unless otherwise explicitly stated, all references to "one / an / element, equipment, component, device, step, etc." should be openly interpreted as referring to at least one instance in an element, equipment, component, device, step, etc. Unless a step must be explicitly described as after or before another step and / or implicitly a step must be after or before another step, the steps of any method disclosed herein do not have to be performed in the exact order disclosed. Where appropriate, any feature of any embodiment disclosed herein may be applied to any other embodiment. Similarly, any advantage of any embodiment may be applicable to any other embodiment, and vice versa. By description, other purposes, features and advantages of the attached embodiments will be apparent.

[0137] The term unit may have the conventional meaning in the field of electronics, electrical devices and / or electronic devices, and may include, for example, electrical and / or electronic circuits, devices, modules, processors, memories, logical solid-state and / or discrete devices, computer programs or instructions for performing corresponding tasks, procedures, calculations, output and / or display functions, etc., such as those described herein.

[0138] Some embodiments contemplated herein are more fully described with reference to the accompanying drawings. However, other embodiments are included within the scope of the subject matter described herein. The disclosed subject matter should not be construed as being limited to the embodiments set forth herein; rather, these embodiments are provided merely as examples to convey the scope of the subject matter to those skilled in the art. Of course, the present invention may be implemented in other ways than those specifically set forth herein without departing from the essential features of the present invention. The proposed embodiments are to be considered in all respects to be illustrative and not restrictive, and all changes that fall within the meaning and equivalent range of the appended claims are intended to be included therein.

Claims

1. A method (110) of applying a plurality of different filter operations (22, 24, 34, 36, 42, 45, 52, 53, 62, 64, 82, 86, 92, 96) to image data, characterized in that: dividing (111) the image data into one or more partitions (32, 51); and For each partition (32, 51) of the image data: applying (112) a first filtering operation (22, 34, 42, 52, 62, 82, 92) to a current partition (32, 51) of the image data to generate one of first filtered image data and first delta data comprising a difference between the image data and the first filtered image data; applying (113) a second filtering operation (24, 36, 45, 53, 64, 86, 96) to a current partition (32, 51) of the image data to generate one of second filtered image data and second delta data comprising a difference between the image data and the second filtered image data; For each partition of the image data, combining (114) outputs of a first filtering operation and a second filtering operation (22, 24, 34, 36, 42, 45, 52, 53, 62, 64, 82, 86, 92, 96) for the current partition (32, 51) to generate combined filtered image data by: If the first filtering operation (22, 34, 42, 52, 62, 82, 92) generates the first filtered image data, calculating the first delta data as a difference between the image data and the first filtered image data; If the second filtering operation (24, 36, 45, 53, 64, 86, 96) generates the second filtered image data, calculating the second delta data as a difference between the image data and the second filtered image data; as well as summing the image data, the first incremental data, and the second incremental data; as well as The combined filtered image data of the current partition (32, 51) is cropped (115).

2. The method (110) according to claim 1, further comprising: One or more parameters (84, 94) for the second filtering operation (86, 96) are estimated based on an output of the first filtering operation (82, 92); and wherein applying (113) the second filtering operation (86, 96) includes applying the second filtering operation (86, 96) using the one or more estimated parameters (84, 94).

3. The method (110) according to claim 2, wherein: One of the estimated parameters (84, 94) establishes a range of partitions in the image data to which the second filtering operation (86, 96) is applied.

4. The method (110) according to claim 1, wherein: The first filtering operation and the second filtering operation (22, 24, 34, 36, 42, 45, 52, 53, 62, 64, 82, 86, 92, 96) are performed in an image or video decoder.

5. The method (110) according to any one of claims 1 to 4, wherein: Each partition includes the entire image data; Applying (112) the first filtering operation (62) includes applying the first filtering operation (62) to the entire image data using the first processing unit CPU0; and Applying (113) the second filtering operation (64) comprises applying the second filtering operation (64) to the entire image data using the second processing unit CPU1.

6. The method (110) according to any one of claims 1 to 4, wherein: Each partition includes a coding tree unit CTU (41, 44, 51); Applying (112) a first filtering operation (42, 52) includes executing software that applies the first filtering operation (42, 52) to a current CTU (41, 51) of image data at a first time; Applying (113) a second filtering operation (45, 53) includes executing software applying the second filtering operation (45, 53) to a current CTU (44, 51) of image data at a second time different from the first time.

7. The method (110) according to claim 6, wherein: Software executing the application (112, 113) of one of a first filtering operation and a second filtering operation (42, 45) to a current CTU (41, 44) of image data includes executing a single instruction multiple data (SIMD) instruction on a computing device having insufficient registers to simultaneously execute both filtering operations (42, 45).

8. The method (110) according to any one of claims 1 to 4, wherein: Each partition includes a group of pixels; Applying (112) the first filtering operation (34) includes applying the first filtering operation (34) to the current pixel group using hardware; and Applying (113) the second filtering operation (36) includes applying the second filtering operation (36) to the current pixel group using hardware.

9. The method (110) according to any one of claims 1 to 4, wherein: Each partition of the image data comprises one or more pixels, and wherein the first filtering operation (22, 34, 42, 52, 62, 82, 92), the second filtering operation (24, 36, 45, 53, 64, 86, 96), the combining (26, 37, 47, 54, 66, 88, 98) operation and the cropping (28, 38, 48, 55, 68, 89, 99) operation are performed on each partition of the image data before processing the next partition of the image data.

10. The method (110) according to any one of claims 1 to 4, wherein: Combining the first filtered image data and the second filtered image data for the current segment comprises: for each segment of the image data, calculating a first ratio of the first filtered image data to the image data; calculating a second ratio of the second filtered image data to the image data; and The image data is multiplied by the first ratio and the second ratio.

11. The method (110) according to claim 10, wherein: The first ratio and the second ratio are calculated as ratios of the respective first filtered image data and second filtered image data shifted by a constant value to the image data shifted by the constant value.

12. The method (110) according to claim 10, wherein: Cropping (28, 38, 48, 55, 68, 89, 99) the combined filtered image data includes ensuring that the data is within a predetermined range.

13. The method (110) according to claim 12, wherein the clipping function is: clip(x) = max(0, min(2 n -1, x)), where n is the bit length of the pixel data, min(x) is a function that selects the minimum of its two parameters, and max(x) is a function that selects the maximum of its two parameters.

14. The method (110) according to any one of claims 1 to 4, wherein: The first filtering operation (22, 34, 42, 52, 62, 82, 92) and the second filtering operation (24, 36, 45, 53, 64, 86, 96) can include one or more of: post-reconstruction bilateral filtering, bilateral loop filtering, post-reconstruction Hadamard filtering, bilateral Hadamard filtering, deblocking filtering, sample adaptive offset SAO filtering, and adaptive loop filter ALF filtering.

15. The method (110) according to claim 14, wherein: The first filtering operation (22, 34, 42, 52, 62, 82, 92) is a bilateral filtering operation; and The second filtering operation (24, 36, 45, 53, 64, 86, 96) is a SAO filtering operation.

16. The method (110) of claim 15, wherein: The method (110) is performed after a deblocking filter operation and before an ALF filter operation.

17. An image processing device (100) comprising a processor (101) and a memory, the memory containing instructions executable by the processor (101) such that the image processing device (100) is operable to apply a plurality of different filter operations (22, 24, 34, 36, 42, 45, 52, 53, 62, 64, 82, 86, 92, 96) to image data, characterized in that The processor (101) is adapted to: dividing (111) the image data into one or more partitions (32, 51); and For each partition (32, 51) of the image data: applying (112) a first filtering operation (22, 34, 42, 52, 62, 82, 92) to a current partition (32, 51) of the image data to generate one of first filtered image data and first delta data comprising a difference between the image data and the first filtered image data; applying (113) a second filtering operation (24, 36, 45, 53, 64, 86, 96) to a current partition (32, 51) of the image data to generate one of second filtered image data and second delta data comprising a difference between the image data and the second filtered image data; For each partition of the image data, combining (114) outputs of a first filtering operation and a second filtering operation (22, 24, 34, 36, 42, 45, 52, 53, 62, 64, 82, 86, 92, 96) for the current partition (32, 51) to generate combined filtered image data by: If the first filtering operation (22, 34, 42, 52, 62, 82, 92) generates the first filtered image data, calculating the first delta data as a difference between the image data and the first filtered image data; If the second filtering operation (24, 36, 45, 53, 64, 86, 96) generates the second filtered image data, calculating the second delta data as a difference between the image data and the second filtered image data; as well as summing the image data, the first incremental data, and the second incremental data; as well as The combined filtered image data of the current partition (32, 51) is cropped (115).

18. The image processing device (100) according to claim 17, wherein: The processor (101) is further adapted to estimate one or more parameters (84, 94) for the second filtering operation (86, 96) based on an output of the first filtering operation (82, 92); and wherein the processor (101) is adapted to apply (113) the second filtering operation (86, 96) by using the one or more estimated parameters (84, 94).

19. The image processing device (100) according to claim 18, wherein: One of the estimated parameters (84, 94) establishes a range of segments of the image data to which the second filtering operation (86, 96) is applied.

20. The image processing device (100) according to claim 17, wherein: The first filtering operation and the second filtering operation (22, 24, 34, 36, 42, 45, 52, 53, 62, 64, 82, 86, 92, 96) are performed in an image or video decoder.

21. The image processing device (100) according to any one of claims 17 to 20, wherein: Each partition includes the entire image data; The processor (101) is adapted to apply (112) a first filtering operation (62) to the entire image data using a first processing unit CPU0; and The processor (101) is adapted to apply (113) a second filtering operation (64) to the entire image data using the second processing unit CPU1.

22. The image processing device (100) according to any one of claims 17 to 20, wherein: Each partition includes a coding tree unit CTU (41, 44, 51); The processor (101) is adapted to apply (112) the first filtering operation (42, 52) by executing software for applying the first filtering operation (42, 52) to a current CTU (41, 51) of image data at a first time; and The processor (101) is adapted to apply (113) the second filtering operation (45, 53) to a current CTU (44, 51) of image data by executing software that applies the second filtering operation (45, 53) to the current CTU (44, 51) at a second time different from the first time.

23. The image processing device (100) according to claim 22, wherein: Software executing the application (112, 113) of one of a first filtering operation and a second filtering operation (42, 45) to a current CTU (41, 44) of image data includes executing a single instruction multiple data (SIMD) instruction on a computing device having insufficient registers to simultaneously execute both filtering operations (42, 45).

24. The image processing device (100) according to any one of claims 17 to 20, wherein: Each partition includes a group of pixels; The processor (101) being adapted to apply (112) a first filtering operation (34) comprises: applying the first filtering operation (34) to a current pixel group using hardware; and The processor (101) is adapted to apply (113) a second filtering operation (36) including applying the second filtering operation (36) to the current pixel group using hardware.

25. The image processing device (100) according to any one of claims 17 to 20, wherein: Each partition of the image data comprises one or more pixels, and wherein the processor is adapted to perform the first filtering operation (22, 34, 42, 52, 62, 82, 92), the second filtering operation (24, 36, 45, 53, 64, 86, 96), the combining (26, 37, 47, 54, 66, 88, 98) operation and the cropping (28, 38, 48, 55, 68, 89, 99) operation on each partition of the image data before processing the next partition of the image data.

26. The image processing device (100) according to any one of claims 17 to 20, wherein: The processor (101) is adapted to combine the first filtered image data and the second filtered image data for the current segment by: for each segment of the image data, calculating a first ratio of the first filtered image data to the image data; calculating a second ratio of the second filtered image data to the image data; and The image data is multiplied by the first ratio and the second ratio.

27. The image processing device (100) according to claim 26, wherein: The first ratio and the second ratio are calculated as ratios of the respective first filtered image data and second filtered image data shifted by a constant value to the image data shifted by the constant value.

28. The image processing device (100) according to claim 26, wherein: The processor (101) is adapted to crop (28, 38, 48, 55, 68, 89, 99) the combined filtered image data by ensuring that the data is within a predetermined range.

29. The image processing device (100) according to claim 28, wherein the cropping function is: clip(x) = max(0, min(2 n -1, x)), where n is the bit length of the pixel data, min(x) is a function that selects the minimum of its two parameters, and max(x) is a function that selects the maximum of its two parameters.

30. The image processing device (100) according to any one of claims 17 to 20, wherein: The first filtering operation (22, 34, 42, 52, 62, 82, 92) and the second filtering operation (24, 36, 45, 53, 64, 86, 96) can include one or more of: post-reconstruction bilateral filtering, bilateral loop filtering, post-reconstruction Hadamard filtering, bilateral Hadamard filtering, deblocking filtering, sample adaptive offset SAO filtering, and adaptive loop filter ALF filtering.

31. The image processing device (100) according to claim 30, wherein: The first filtering operation (22, 34, 42, 52, 62, 82, 92) is a bilateral filtering operation; and The second filtering operation (24, 36, 45, 53, 64, 86, 96) is a SAO filtering operation.

Images