Method of processing image data

Abstract

A computer-implemented method for processing image data representing at least one image wherein the image data comprises at least one input array of pixels wherein each pixel of the at least one input array of pixels is associated with a pixel value, the method comprises the steps of recursively performing a hierarchical multi-scale decomposition of the image data, decomposing the image data into a multi-level hierarchy of pixel arrays, where, for each scale level of the multi-level hierarchy, at least one input pixel array is decomposed into a low frequency pixel array and at least one high frequency pixel array.

Description

Technical Field

[0001] The present invention generally relates to a method for processing image data, and in particular to a method for denoising and / or de-flickering image data, especially low-light image data. Background Art

[0002] Digital images are inevitably affected by noise, i.e., artifacts that do not originate from the original scene content, which may degrade the visual quality of the image, especially in low-light images. In a sequence of images, such as in video images, global light variations between frames may cause image flicker. The problem of noise and flicker reduction in images has been studied for a long time but is rather complex, as noise reduction methods may also result in a loss of image quality, such as loss of sharpness at edges, known as blurring, and / or the introduction of artifacts.

[0003] There are many different filtering methods, such as mean filtering or Wiener filtering, which are spatial domain linear filtering methods. Another example is bilateral filtering, which is a relatively widely used method for image denoising. This is a non-linear method that has the advantage of preserving edges relatively well. In this method, the intensity value of each pixel is replaced by a weighted average of the intensity values ​​of nearby pixels.

[0004] A problem associated with these methods is that the known methods require considerable processing and / or computing power, which is particularly a problem for relatively large images. Due to the required computing power, these methods may be relatively slow, especially for video images.

[0005] It is therefore an object of the present invention to solve or at least alleviate one or more of the above problems. In particular, the present invention aims to provide an improved method of denoising and / or de-flickering image data which is relatively fast and efficient. Summary of the invention

[0006] To this end, a computer-implemented method for processing image data is provided, characterized by the features of claim 1. In particular, the image data represents at least one image, and thus may be a single image or a time sequence of images, such as video images. The image data may, for example, contain low-light image data. The image data comprises at least one input pixel array. A single image may be represented by a single input pixel array, while a sequence of images may be represented by a time sequence pixel array. Each pixel of the at least one input pixel array is associated with a pixel value. The pixel value may be a single-dimensional pixel value, for example representing the light intensity or depth of the pixel in the image. Alternatively, the pixel value may be a multi-dimensional pixel value, for example the RGB intensity of the pixel in the image. The method for processing the image data comprises a step of recursively performing a hierarchical multiscale decomposition of the image data into a multi-level hierarchy of pixel arrays, such that at each scale level of the multi-level hierarchy, the at least one input pixel array is decomposed into a low-frequency pixel array and at least one high-frequency pixel array. The hierarchical multiscale decomposition may comprise a wavelet decomposition, such as a Haar wavelet decomposition, or a pyramid decomposition, or any other suitable multiscale decomposition, including performing a discrete spectral transform. The recursiveness of the multi-scale decomposition is preferably only applicable to low-frequency pixel arrays, as known to the skilled person: preferably only a low-frequency pixel array of a first scale level is further decomposed into a low-frequency pixel array and a high-frequency pixel array of at least one next scale level.

[0007] The method further comprises a step of forming a first group of pixel arrays of each scale level of the multi-level hierarchy by selecting a plurality of low-frequency pixel arrays and / or high-frequency pixel arrays, in particular selecting a plurality of groups of low-frequency pixel arrays of the scale level of the multi-level hierarchy from a temporal sequence of input pixel arrays, or a plurality of low-frequency pixel arrays and at least one high-frequency pixel array of the scale level of the multi-level hierarchy of at least one input pixel array. The pixel arrays of the first group may allow group processing of the selected pixel arrays. The selection of the pixel arrays to form the first group may depend on the type of processing required: denoised image data and / or de-flickered image data.

[0008] The method further includes the step of performing a first edge-preserving convolution on the low-frequency pixel array of the pixel array of the multi-level hierarchy at each scale level of the multi-level hierarchy. In an innovative way, the first edge-preserving convolution uses weighted filtering, wherein the filtering weight of a certain pixel in the low-frequency pixel array depends simultaneously or jointly on the difference or distance between each pixel array of the first pixel array group at the scale level of the pixel value associated with the pixel, which means that the corresponding pixel values ​​in each pixel array of the first pixel array group are considered simultaneously for group processing. The distance between the pixel values ​​should be understood as a mathematical distance, such as, but not necessarily limited to, the Euclidean distance. The dependency of the distance may include a function, such as a weight that decreases as the distance increases. Alternatively, any other functional dependency may be used. The dependency may include a weight that increases as the distance increases, combined with a weight that decreases as the distance increases starting from a threshold distance. Compared with traditional bilateral filtering, this innovative weighted filtering uses weights based on multiple spatially and temporally related pixel arrays, and simultaneously processes low-frequency and high-frequency pixel arrays in a hierarchical decomposition, meaning that it is joint. The weights in traditional bilateral filtering only consider the spatial proximity and intensity differences of nearby pixels in the image or its associated pixel array itself.

[0009] The method ultimately includes the step of reconstructing an output pixel array by recursively performing an inverse transform of the hierarchical multi-scale decomposition on the filtered low-frequency pixel array and the high-frequency pixel array. The output pixel array may thus provide de-noised and / or de-flickered image data. Since the method depends on a judicious combination of efficient hierarchical image decomposition with innovative weighted filtering, the method is relatively fast and requires less computational and processing time and / or capacity than known methods, which is particularly advantageous for low-light image data, and more specifically, for low-light video image data.

[0010] The first pixel array group may be formed by selecting low-frequency pixel arrays at the scale level in the multi-level hierarchy of the temporal sequence of the input pixel array. The temporal sequence of the input pixel array is particularly suitable for image data including video image data, wherein a relatively high number of image frames are captured in the temporal sequence, for example in a time window [t min ,t max ] around the reference time t 0 Each image frame in such a time sequence can be represented by an input pixel array I(x, y, t), thereby forming a time sequence of input pixel arrays. Each input pixel array of the input pixel array sequence is then decomposed into a low-frequency pixel array C in a recursive and multi-scale hierarchical manner. LF (x, y, t) and at least one high-frequency pixel array C HF(x, y, t). The selection step to form the first group is performed at each scale level and may include the low-frequency pixel array at that scale level, for example, in the time window [t min ,t max ] around the reference time t 0 for different times t.

[0011] In the time window [t min ,t max ] around the reference time t 0 The filter weight w of a pixel in the low-frequency pixel array of the time series input pixel array at time t is given by the following formula, for example: where σ t Is a parameter related to the amplitude of flicker and / or noise. Absolute value |C LF (x,y,t)-C LF (x,y,t 0 )| is the distance on which the weight depends. The effect of the negative exponential function is that when the distance |C LF (x,y,t)-C LF (x,y,t 0 )| increases, the weight decreases. Other functions can also be used, depending on the desired dependencies and effects. Since the selection step and filtering are applied at each scale level in the multi-level hierarchy, this weight can vary at each scale level. This weight can then be taken into account when performing the first edge-preserving convolution, resulting in a filtered low-frequency pixel array C′ LF : Where [t min ,t max ] is a time window over which filtering is applied. The time window may vary depending on the scale level of the multi-level hierarchy at which the first edge-preserving deconvolution is performed. In particular, the time window may be larger for higher scale levels of the multi-scale decomposition because the resolution at these higher scale levels is lower. Filtering image data comprising a time series of image frames using the above method and the above filtering weights may effectively provide an output pixel array representing filtered image data, thereby minimizing flicker between image frames of the time series.

[0012] Alternatively, the first pixel array group may be formed by selecting a low-frequency pixel array and at least one high-frequency pixel array, which belong to a multi-level hierarchy of scales of the at least one input pixel array. The input pixel array I(x, y) is decomposed into low-frequency pixel arrays C in a recursive and multi-level hierarchy. LF (x, y) and at least one, and preferably more, high frequency pixel array C HF(x,y). For min ,t max ], a hierarchical multi-scale decomposition may be performed by time. The step of selecting to form a first group, performed by scale level and time, may include a low-frequency pixel array at said scale level at time t, and all high-frequency pixel arrays at said scale level at time t. Such selection of the first group of incoming pixel arrays may be particularly effective for denoising images.

[0013] The weight of the filtering preferably depends on the distance between the pixels in the neighborhood around the pixel and the pixel value associated with the pixel. The neighborhood can have the same size in the scale level of the low-frequency pixel array and the at least one high-frequency pixel array. The size of the neighborhood can be determined as a compromise between computing time and image quality improvement.

[0014] Low frequency pixel array C LF The filter weight W(i,j) of a pixel in (x,y) can be expressed as: where D is the multidimensional Euclidean distance given by: Where k is the index of the high-frequency pixel array of the scale level being filtered, and (x+i,y+j) represents the neighboring pixels around pixel (x,y). Again, the effect of the negative exponential function is that the weights decrease with increasing distance, but other functions can also be used depending on the desired dependencies and effects. The performance of edge-preserving convolution on the low-frequency pixel array of this scale level at a given weight can be obtained as a filtered low-frequency pixel array C′ LF :

[0015] When the first pixel array group includes a low-frequency pixel array and at least one high-frequency pixel array at the scale level, the method may further include the step of performing edge-preserving convolution on at least one high-frequency pixel array at the scale level of the pixel array of the multi-level level using weighted filtering at each scale level of the multi-level level. In this case, the weight of the filtering of the pixel simultaneously (meaning jointly) depends on the distance between each pixel array of the first pixel array group at the scale level of the pixel value associated with the pixel. In this case, performing edge-preserving convolution on at least one high-frequency pixel array at the scale level can obtain at least one filtered high-frequency pixel array C′ HF : where σ l is a parameter that depends on the scale level l, σl σ l =α l ·σ and σ depends on the average noise amplitude, where α is a constant, for example α≈0.48.

[0016] When performing edge-preserving convolution on at least one high-frequency pixel array, the step of reorganizing the output pixel array I'(x, y) is performed by filtering the low-frequency pixel array C' LF (x,y) and the filtered high-frequency pixel array This is accomplished by recursively performing an inverse transform of the hierarchical multiscale decomposition. Filtering the image data using the above method and the above filtering weights can provide an output pixel array representing the filtered image data, wherein the noise in the image data has been minimized in an efficient manner. The efficiency, in particular the reduction in computation time and required computing power, is at least partially due to the weights used in the filtering being directly based on the multiscale decomposition of the low frequency pixel array and the at least one high frequency pixel array, thereby reducing the number of operations that need to be performed.

[0017] The filtering may further preferably include at least one factor configured to adjust the weight of each low-frequency pixel array and at least one high-frequency pixel array in the first group. The factor may be a constant weight and depends on the noise level and the type of multi-scale decomposition. The factor may be a factor K specifically for low-frequency pixel array filtering. LF , and a factor K for at least one high-frequency pixel array HF .

[0018] The method may further include a step of forming a second pixel array group by selecting low-frequency pixel arrays of the scale level of the multi-level hierarchy of the temporal sequence of the input pixel array, and performing a second edge-preserving convolution step on the filtered low-frequency pixel array. In this way, different selections can be performed according to different purposes, for example, the first selection step forms a first group for a first image processing, and the second selection step forms a second group that may be different from the first group for a second image processing.

[0019] The first pixel array group may, for example, include a low-frequency pixel array and at least one high-frequency pixel array at the scale level, belonging to a multi-level hierarchy of at least one input pixel array, preferably belonging to a temporal sequence of the input pixel array, and the second pixel array group may include a low-frequency pixel array at a multi-level hierarchy of the temporal sequence of the input pixel array. The first pixel array group and the second pixel array group may at least partially include the same pixel array. In particular, the low-frequency pixel array at a given scale level may simultaneously belong to part of the first group and the second group. In this way, the low-frequency pixel array may be subjected to two edge-preserving convolutions with different weights.

[0020] In a preferred embodiment of the method, a first edge-preserving convolution may be performed on a low-frequency pixel array at the scale level of the multi-level hierarchy of pixel arrays and on at least one high-frequency pixel array at the scale level of the multi-level hierarchy of pixel arrays, as described above. In particular, the edge-preserving convolution uses weighted filtering, wherein the filtering weight of a pixel in the low-frequency pixel array or at least one high-frequency pixel array simultaneously (meaning jointly) depends on the distance between the pixel value associated with the pixel and each pixel array in the first pixel array group at the scale level. In particular, the weight may, for example, take into account neighboring pixels in the low-frequency and high-frequency pixel arrays at a given scale level at a given point in time. A second edge-preserving convolution may then be performed on the filtered low-frequency pixel array at the scale level of the multi-level hierarchy of pixel arrays, using weighted filtering, wherein the filtering weight of a pixel in the low-frequency pixel array or at least one high-frequency pixel array simultaneously (meaning jointly) depends on the distance between the pixel value associated with the pixel and each pixel array in the second pixel array group at the scale level. In particular, the weight may take into account the distance between the pixel value associated with the pixel and each pixel array in the second pixel array group at the scale level at time t 0 The pixel values ​​of the time series of the pixel array within a predetermined time range nearby. It is possible to consider neighboring pixels in the weights of the second edge-preserving convolution, but it may increase the calculation time without necessarily improving the image quality. The reorganization step of the output pixel array can be completed by recursively performing an inverse transform of the hierarchical multi-scale decomposition on the low-frequency pixel array filtered by the second edge-preserving convolution and the high-frequency pixel array filtered by the first edge-preserving convolution. Performing the first and second group formation steps, as well as the first and second edge-preserving deconvolutions in this manner, the image data can be first denoised, especially on a single image, and then the flicker of the image data in the time series can be reduced.

[0021] The method may further comprise a post-processing step of the image data, in particular of the reconstructed output pixel array. This step may preferably comprise weighted filtering of the reconstructed output pixel array, wherein the weight of the filtering of a pixel depends on the difference between the pixel value associated with the pixel and the local average. The selection of such a post-processing step may depend on a hierarchical multi-scale decomposition. Certain decompositions may result in multi-scale induced artifacts, such as the Gibbs phenomenon, which may need to be corrected in a post-processing step. The post-processing step of the image data may be performed according to a computer-implemented method, which may be regarded as an independent invention. The image data is represented as an image pixel array, wherein each pixel of the image pixel array is associated with a certain value. The method comprises a step of determining an average pixel array by convolving the image pixel array with an average kernel, such as the output pixel array I'(x,y) average kernel of the method described previously. The average kernel may be, for example, a Gaussian blur 3x3 kernel, for example or any other known averaging kernel. The method further comprises the step of determining a difference δ of a neighborhood in the image pixel array, for example between I'(x+i,y+j) and the average pixel array μ(x,y). The method further comprises the step of determining a weighted filtering result of the difference δ. The method for post-processing the image data finally comprises adding the weighted filtering result to the average pixel array μ(x,y), thereby obtaining the post-processed image data I res (x, y). For example, a weighted filter may be applied to the reorganized output pixel array I'(x, y), so that the post-processed pixel array may be represented as: where σ is a parameter that depends on the multiscale decomposition, in particular on the type of decomposition and the number of scale levels, and δ(x+i,y+j)=I'(x,y)-μ(x,y), where μ(x,y) is the local average obtained by convolving I'(x,y) with the averaging kernel, e.g.,

[0022] The method may also include the step of pre-filtering the image data. Pre-filtering the image data may include normalizing the levels of the image data and / or removing statistical outliers of pixels in the image data, such as those due to dead, burned-in or locked pixels. The step of pre-filtering the image data may be performed using any known pre-filtering method.

[0023] Alternatively, and more preferably, the pre-filtering of the image data may be performed according to a computer-implemented method, which may be regarded as an invention in its own right. The image data is represented as an array of image pixels, wherein each pixel of the array of image pixels is associated with a certain value. The method comprises the step of determining an average pixel array by convolving the array of image pixels with an average kernel. The average kernel may be, for example, a Gaussian blur 3x3 kernel, e.g. or any other known averaging kernel. The method further comprises a step of determining a variant pixel array v, such as the matrix M mentioned above, by convolving the absolute value of the difference |I(x,y)-μ(x,y)| between the image data and the average pixel array μ(x,y). The method then comprises a step of determining a corrected difference δ' of the difference δ between the image data and the average pixel array, the difference δ=I(x,y)-μ(x,y). The corrected difference δ' comprises an exponential function of the difference, which depends on the variant pixel array, so that the corrected difference comprises a reduced value of values ​​outside the distribution determined relative to the average pixel array and the variant pixel array. The method for pre-filtering image data finally comprises a step of adding the corrected difference δ' to the average pixel array μ(x,y), thereby regressing the noise band to normalized statistics and obtaining pre-filtered image data I'(x,y).

[0024] The correction difference value δ' may preferably include a linear response of values ​​within a distribution determined by the mean pixel array and the variation pixel array so that the center pixel value may remain unchanged.

[0025] The corrected difference value δ' may include a response factor ρ configured to be adjustable so that the corrected difference value includes a reduced value or an amplified value relative to a value within a distribution determined by the average pixel array and the variation pixel array. In particular, the response factor ρ may be selected to be When ρ = 1, a linear response with respect to δ is obtained under local variation v; When ρ < 1, under local variation v, a decreasing response with respect to δ is obtained; When ρ>1, an enhanced response with respect to δ is obtained under local variation v.

[0026] The correction difference δ' can advantageously be expressed as: where N is a constant: and and ρ is the response factor as described previously. The modified difference is a function of the difference δ, comprising two exponential functions. The interaction between the two exponential functions allows describing in a single function different behaviors inside and outside the distribution determined by the mean pixel array and the variant pixel array. Without the modified function, similar behaviors would be described and programmed by multiple different functions, depending on a domain, or in programming terms, by loops and conditional functions on the domain. The modified difference avoids such loops and conditional functions and can simplify and speed up the calculations, thereby speeding up the pre-filtering of image data without losing image quality.

[0027] The above-described computer-implemented method for pre-filtering image data can be used independently for processing the image data as described above. However, the pre-filtering method can also be advantageously integrated into the image data processing method as a separate pre-filtering step prior to the recursive execution of the hierarchical multi-scale decomposition of the image data, or as part of one or more edge-preserving deconvolutions in the processing method.

[0028] According to a further aspect of the present invention, a controller, a computer program product and a computer readable storage medium are provided for executing the method described above, having the features of claims 12, 13 and 14, thereby providing one or more of the advantages mentioned above. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 shows a schematic flow chart illustrating a preferred embodiment of a computer-implemented method for processing image data according to an aspect of the present invention;

[0030] Figure 2A series of graphs are shown representing the effects of a preferred embodiment of a computer-implemented method for pre-filtering image data according to another aspect of the present invention;

[0031] Figure 3 A schematic diagram is shown, illustrating Figure 1 Step 110 of the method;

[0032] Figure 4a and 4b A schematic diagram is shown respectively, illustrating Figure 1 Step 120 and step 140 of the method;

[0033] Figure 5 A schematic diagram is shown, illustrating Figure 1 Step 160 of the method;

[0034] Figure 6 A computing system suitable for performing the steps of the method is shown according to an exemplary embodiment. DETAILED DESCRIPTION

[0035] Figure 1 A schematic flow chart is shown illustrating a preferred embodiment of a computer-implemented method for processing image data. After acquiring image data, such as an image frame, or more preferably, multiple image frames, such as a time series of image frames, such as video image data, an optional pre-filtering step 100 can be performed. An image frame can be represented by a pixel array, and the time series of image frames can be represented as a time series of pixel arrays. In the one or more pixel arrays, a one-dimensional or multi-dimensional value can be associated with each pixel, representing, for example, the intensity, depth or RGB value of the pixel. Pre-filtering the image data 100 can standardize the shape and level of the image data and remove statistical outliers, such as outliers caused by dead spots, burnt-out or locked pixels. Such pre-filtering can be performed by any known method. More preferably, a new computer-implemented method can be applied for pre-filtering of image data, which can be regarded as an independent invention and will be combined with it later. Figure 2 In the next step 110, the input pixel array is recursively decomposed into pixel arrays of multiple hierarchical levels, so that at each scale level of the multiple hierarchical levels, at least one input pixel array is decomposed into a low-frequency pixel array and at least one high-frequency pixel array, which is subsequently combined with Figure 3 Steps 120, 130, 140 and 150 will be described in more detail in Figure 4a and 4bDetailed explanation is given in . These steps involve a selection or grouping step 120, 140 to form a first and a second pixel array group, followed by an edge-preserving deconvolution 130, 150, including taking into account the weights of the pixel arrays of the first or second group. These steps are repeated 125, 145 at each scale level of the multi-level decomposition. In a first processing portion 135, at least one pixel array may be denoised, and in a second processing portion 155, flicker between image frames in a temporal sequence of image frames may be reduced. In step 160, the filtered pixel arrays of steps 130 (165) and 150 are recombined by recursively performing an inverse transform of the hierarchical multi-scale decomposition to obtain at least one output pixel array. Finally, and optionally, at least one output pixel array of step 160 may be post-processed in step 170 to prevent multi-scale induced artifacts, such as Gibbs artifacts, from appearing in the recombined image. Some of these steps may be performed simultaneously with other steps, and / or the order of the steps may be reversed. For example, pre-filtering of the image data may also be performed after the hierarchical multi-level decomposition.

[0036] Figure 2 A series of graphs are shown, showing the effects of a preferred embodiment of a computer-implemented method for pre-filtering image data according to a further aspect of the present invention. According to the method, an average pixel array μ(x,y) is determined by convolving an input image pixel array I(x,y) with an average kernel M. Then, a variant pixel array v is determined by convolving a difference δ of the input image pixel array with an absolute difference |I(x,y)-μ(x,y)| of the average pixel array using an average kernel, and the average kernel used may be the matrix M mentioned above. In an innovative manner, a corrected difference δ' of the difference δ between the image data and the average pixel array is determined, the difference δ=I(x,y)-μ(x,y). The corrected difference δ' comprises an exponential function of the difference, which depends on the variant pixel array, so that the corrected difference comprises a reduced value 101 relative to the difference at values ​​outside the distribution determined by the average pixel array and the variant pixel array. Instead of adding the difference value δ to the average pixel array, a modified difference value δ' is added to the average pixel array μ(x,y), thereby regressing the noise band to normalized statistics and obtaining a pre-filtered image frame. The modified difference value δ' may preferably be selected as a linear response 102 of values ​​included in the distribution determined by the average pixel array and the variation pixel array. More preferably, the modified difference value δ' may include a response factor ρ configured to adjust so that the value of the modified difference within the distribution determined by the average pixel array and the variation pixel array includes a reduced value or an amplified value relative to the difference value. In particular, in Figure 2In the figure, graphs 103, 104, 106 represent the corrected difference value δ' as a function of the uncorrected difference value δ, with variance v=1. In the upper graph 103, the response factor is ρ=1, so that for values ​​within the distribution determined by the average pixel array and the variation pixel array, there is a linear response 102 corresponding to δ, and for values ​​outside the distribution determined by the average pixel array and the variation pixel array, there is a reduced value 101 of the difference value. In the middle graph 104, the response factor is ρ<1, so that for values ​​within the distribution determined by the average pixel array and the variation pixel array, there is a reduced response 105 corresponding to δ, and for values ​​outside the distribution determined by the average pixel array and the variation pixel array, there is a reduced value 101 of the difference value. In the lower graph 106, the response factor is ρ>1, so that for values ​​within the distribution determined by the average pixel array and the variation pixel array, there is an amplified response 107 corresponding to δ, and for values ​​outside the distribution determined by the average pixel array and the variation pixel array, there is a reduced value 101 of the difference value.

[0037] Figure 3 A schematic diagram is shown, illustrating Figure 1 Step 110 of the method shown in . In step 110, a multi-level hierarchical decomposition of a pixel array, denoted as an input pixel array I(x,y), is performed for each image frame. This means that at each scale level 111, 112, 113 of the multi-level hierarchy, the input pixel array is decomposed into a low-frequency pixel array 115 and at least one high-frequency pixel array 116. This is performed in a recursive manner on the low-frequency pixel arrays 115, i.e. the low-frequency pixel array 115 of the first level 111 can be further decomposed into a low-frequency array 115 and one or more high-frequency pixel arrays 116 of the next level 112. Similarly, the low-frequency pixel array 115 of the second level 112 can be further decomposed into a low-frequency pixel array 115 and one or more high-frequency pixel arrays 116 of the third level 113. The hierarchical multi-scale decomposition may include a wavelet decomposition, such as a Haar wavelet decomposition, or a pyramid decomposition, or any other suitable multi-scale decomposition, including a discrete spectral transform of the input pixel array and the low-frequency pixel arrays.

[0038] Figure 4a and Figure 4b They are shown separately Figure 1 Schematic diagram of step 120 and step 140 of the method shown. Step 120, such as Figure 4aAs shown, for each scale level 111, 112, 113, it is performed separately, including selecting a plurality of low-frequency pixel arrays and / or high-frequency pixel arrays at each scale level to form a first pixel array group at the scale level. The first pixel array group 121, 122, 123 may preferably include a low-frequency pixel array 115 and at least one high-frequency pixel array 116 at the corresponding scale level 111, 112, 113 of the multi-level hierarchy. In the time window [t min ,t max ], for example frames taken at times t-1, t and t+1, where t is a reference time, this selection step 120 is performed not only at each scale level, but also at each frame in time, just as the hierarchical multi-level decomposition is performed for each frame in time. In a next step 130, a first edge-preserving convolution is performed on the low-frequency pixel array 115 and the at least one high-frequency pixel array 116 of the pixel arrays of the corresponding scale levels 111, 112 of the multi-level hierarchy, using weighted filtering. The weight of the filtering of the pixel depends simultaneously (meaning jointly) on the distance between the pixel value associated with the pixel and each pixel array of the first pixel array group of the scale level. Therefore, the pixel values ​​of the high-frequency pixel array and the low-frequency pixel array of the first group are considered simultaneously at the corresponding scale level, i.e. considered jointly. The weight of the filtering may also depend on the difference between the pixel value associated with the pixel and the pixels in the neighborhood around the pixel. The filtering may also include at least one factor for adjusting the weight of each low-frequency pixel array and the at least one high-frequency pixel array in the first group. This factor can be a constant weight and depends on the noise level and the type of multiscale decomposition. This factor can be a factor K specific to low frequency pixel array filtering LF , and a factor K for at least one high-frequency pixel array HF Therefore, the first edge-preserving convolution 130 can generate a filtered low-frequency pixel array C' for each scale level. LF and at least one filtered high frequency pixel array C' HF Steps 120 and 130 may be repeated 125 at each scale level. Forming the first selection of the pixel array into a first group, and the associated first edge-preserving convolution, may be viewed as an image denoising step 135 .

[0039] In the next step 140, if Figure 4bAs shown, multiple pixel arrays are selected to form a second group 141, 142. The second pixel array group may include low-frequency pixel arrays at scale levels 111, 112, 113 of multiple hierarchical levels in the time series of the input pixel array. The time window for this second group 141, 142 may be different, depending on the scale level 111, 112, 113: the higher the scale level of the decomposition, the larger the selected time window may be. For example, at scale level 111, the second group 141 may include low-frequency pixel arrays at t-1, t and t+1, while at scale level 112, the second group 142 may include additional low-frequency pixel arrays at earlier and / or later times. At scale level 113, the second group (not shown) may also include more low-frequency pixel arrays. In the next step 150, a second edge-preserving convolution is performed on the filtered low-frequency pixel arrays 115 of the corresponding scale levels 111, 112, 113 at time t, using weighted filtering. The filtering weight of a pixel depends simultaneously (meaning jointly) on the distance between the pixel values ​​associated with the pixel in each pixel array of the second pixel array group 141, 142 at the corresponding scale level 111, 112, 113. The pixel values ​​in the neighborhood around the pixel are preferably not taken into account in the weight, as this does not appear to have a substantial impact on the image quality. Figure 4a and Figure 4b As shown, the first group 121 and the second group 141 may at least partially contain the same pixel array, in particular, the low-frequency pixel array at the respective level of the frame at time t. Therefore, the second edge-preserving convolution may be performed on the filtered low-frequency pixel array C' LF , which is the result of step image 130. Therefore, the second edge-preserving convolution 150 can produce a doubly filtered low-frequency pixel array C" at each scale level LF The selection step 140 and deconvolution step 150 may be repeated again 145 at each scale level, which may be performed simultaneously or sequentially with step 125. This second selection of pixel arrays forming a second group and its associated second edge-preserving convolution may be considered an image deblinking step 155.

[0040] Figure 5 A schematic diagram is shown, illustrating Figure 1 In this step 160, the output pixel array I'(x, y) is reassembled by filtering the low frequency pixel array C" filtered by the second edge preserving convolution 150. LF and the high frequency pixel array C' filtered by the first edge preserving convolution 130 HFThe inverse transform of the hierarchical multi-scale decomposition is recursively performed to complete. The above-mentioned high-frequency pixel arrays are not selected to form the second group and are not considered in the second edge-preserving convolution, so they are directly transferred from step 130 to step 160 through arrow 165. As mentioned above, the low-frequency pixel array is preferably filtered twice: the first time through the first edge-preserving convolution, and then through the second edge-preserving convolution. The recursiveness of this step 160 is performed only on the low-frequency pixel array. Specifically, starting from the highest scale level, such as scale level 113, the filtered low-frequency pixel array and at least one high-frequency pixel array of the level 113 are recombined into a low-frequency pixel array of a lower scale level, in particular scale level 112, wherein the low-frequency pixel array and at least one high-frequency pixel array of the scale level 112 can be recombined into a low-frequency pixel array of scale level 111. The above method can therefore obtain the final processed image data, in which noise and flicker are significantly reduced without reasonably losing image details or significantly increasing image blur.

[0041] Figure 6A suitable computing system 800 is shown, including circuits so that the steps according to the described embodiments can be performed. The computing system 800 can generally be constructed as a suitable general-purpose computer and includes a bus 810, a processor 802, a local memory 804, one or more optional input interfaces 814, one or more optional output interfaces 816, a communication interface 812, a storage element interface 806, and one or more storage elements 808. The bus 810 can include one or more conductors to communicate between the components of the computing system 800. The processor 802 can include any type of conventional processor or microprocessor for interpreting and executing programming instructions. The local memory 804 can include a random access memory (RAM) or other type of dynamic storage device for storing information and instructions for execution by the processor 802, and / or a read-only memory (ROM) or other type of static storage device for storing static information and instructions for use by the processor 802. The input interface 814 may include one or more conventional mechanisms that allow an operator or user to input information to the computing device 800, such as a keyboard 820, a mouse 830, a stylus, a voice recognition and / or biometric mechanism, a camera, etc. The output interface 816 may include one or more conventional mechanisms to output information to the operator or user, such as a display 840, etc. The communication interface 812 may include any transceiver-like mechanism, such as one or more Ethernet interfaces, to enable the computing system 800 to communicate with other devices and / or systems, such as with other computing devices 881, 882, 883. The communication interface 812 of the computing system 800 may be connected to another computing system via a local area network (LAN) or a wide area network (WAN), such as the Internet. The storage element interface 806 may include a storage interface, such as a serial advanced technology attachment (SATA) interface or a small computer system interface (SCSI), for connecting the bus 810 to one or more storage elements 808, such as one or more local disks, such as SATA disk drives, and controlling the reading and writing of data to these storage elements 808. Although the above storage element 808 is described as a local disk, in general, any other suitable computer-readable medium, such as removable disks, optical storage media such as CD-ROM or DVD-ROM disks, solid-state drives, flash memory cards, etc., may be used.

[0042] In this application, the term "circuitry" may refer to one or more or all of the following: (a) hardware circuit implementation only, such as implementation only in analog and / or digital circuits, and (b) a combination of hardware circuitry and software, such as (where applicable): (i) a combination of analog and / or digital hardware circuitry and software / firmware, and (ii) any portion of the hardware processor (including a digital signal processor), software and memory that together enable a device (such as a mobile phone or server) to perform its functions, and (c) Hardware circuits and / or processors, such as a microprocessor or portion of a microprocessor, require software (such as firmware) to operate, but the software may not be present when it is not required for operation. This definition of circuitry applies to all uses of the term in this application, including in any claims. As a further example, the term circuitry as used in this application also covers implementations of only a hardware circuit or processor (or multiple processors) or portions of a hardware circuit or processor and their (or their) accompanying software and / or firmware. The term circuitry also covers, such as and if applicable to a particular claim element, a baseband integrated circuit or processor integrated circuit of a mobile device, or a similar integrated circuit in a server, cellular network device, or other computing or network device.

[0043] Although the present invention has been described with reference to specific embodiments, it will be apparent to those skilled in the art that the present invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be implemented by various changes and modifications without departing from its scope. Therefore, the present embodiments should be considered illustrative and not restrictive in all respects, and the scope of the invention is indicated by the appended claims, rather than by the foregoing description, and all changes within the meaning and equivalent range of the claims are therefore intended to be included. In other words, it is contemplated that any and all modifications, variations or equivalents are covered, which are within the scope of the basic underlying principles, and their basic attributes are claimed in this patent application. In addition, readers of this patent application should understand that the words "comprise" or "include" do not exclude other elements or steps, the words "one" or "one" do not exclude pluralities, and a single element, such as a computer system, processor or other integrated unit, may perform the functions of multiple devices listed in the claims. Any figure mark in the claims should not be interpreted as limiting the related claims. The terms "first", "second", "third", "a", "b", "c", etc., when used in the description or claims, are to distinguish similar elements or steps, and do not necessarily describe the order or time sequence. Likewise, the terms "upper", "lower", "above", "below", etc. are introduced for descriptive purposes and do not necessarily indicate relative positions. It should be understood that the terms used are interchangeable where appropriate, and that embodiments of the invention are capable of operating in other sequences or directions different from those described or illustrated above in accordance with the invention.

Claims

1. A computer-implemented method for processing image data, the image data representing at least one image, wherein the image data comprises at least one input pixel array I(x, y, t), wherein each pixel of the at least one input pixel array is associated with a pixel value, the method comprising the following steps: - recursively perform a hierarchical multi-scale decomposition on the image data to decompose it into pixel arrays of multiple hierarchical levels, wherein at each scale level of the multiple hierarchical levels, at least one input pixel array is decomposed into a low-frequency pixel array C LF (x,y) and at least one high-frequency pixel array For each scale level of the multi-level hierarchy of pixel arrays, a first pixel array group for that scale level is formed by selecting one of the following: o A plurality of low-frequency pixel arrays C of the scale level at the multi-level level in the time series of the input pixel array I (x, y, t) LF (x,y,t); o The low-frequency pixel array C of the scale level of the multi-level hierarchy in the at least one input pixel array LF (x,y) and at least one high-frequency pixel array combination of; - for each scale level of the multi-hierarchical hierarchy, performing a first edge-preserving convolution on a low-frequency pixel array of the scale level of the pixel arrays of the multi-hierarchical hierarchy using a weighted filter, wherein a filter weight of a pixel depends simultaneously on a distance between each pixel array of a first group of pixel arrays of the scale level and a pixel value associated with the pixel; - Reconstructing the output pixel array by recursively performing an inverse transform of the hierarchical multi-scale decomposition on the filtered low-frequency pixel array and the high-frequency pixel array.

2. The method according to claim 1, wherein the first pixel array group is formed by selecting a plurality of low-frequency pixel arrays at the scale level of the multi-level hierarchy in the time series of input pixel arrays.

3. The method according to claim 2, wherein the time series of the input pixel array has an influence on the low-frequency pixel array C LF The weight w of the pixel filter in (x, y, t) is calculated in the time window [t min ,t max ] around the reference time t0 is given as: where σ t is a parameter related to the amplitude of flicker and / or noise.

4. The method according to claim 1, wherein the first pixel array group is formed by selecting a low-frequency pixel array and at least one high-frequency pixel array, which low-frequency pixel array and high-frequency pixel array belong to a scale level of a multi-level hierarchy of the at least one input pixel array.

5. A method according to any one of the preceding claims, wherein the weight of the filtering also depends on the distance between pixels in a neighbourhood around the pixel and the pixel value associated with the pixel.

6. The method according to claim 4 and 5, wherein the low frequency pixel array C LF The filter weight W of a pixel in (x,y) depends on the distance given by Where k is the index of the high-frequency pixel array at the scale level where filtering is performed, and (x+i, y+j) represents the neighboring pixels surrounding the pixel (x, y).

7. The method according to any one of the preceding claims 4-6 further includes performing edge-preserving convolution on at least one high-frequency pixel array of the multi-level hierarchy of pixel arrays at each scale level of the multi-level hierarchy using weighted filtering, wherein the filtering weight of the pixel also depends on the distance between each pixel array of the first pixel array group at the scale level and the pixel value associated with the pixel.

8. The method of claim 7, wherein the step of reorganizing the output pixel array is accomplished by recursively performing an inverse transformation of the hierarchical multi-scale decomposition on the filtered low-frequency pixel array and the filtered high-frequency pixel array.

9. The method according to any one of the preceding claims 4 to 8, wherein the weights of the filtering include at least one factor for adjusting the weights of each low-frequency pixel array and at least one high-frequency pixel array of the first group.

10. The method according to any one of the preceding claims 4-9 further includes the step of forming a second pixel array group at the scale level: by selecting a low-frequency pixel array at the scale level of a multi-level hierarchy of the time series of the input pixel array, and including performing a second edge-preserving convolution step on the filtered low-frequency pixel array.

11. The method according to claim 10, wherein a first edge-preserving convolution step is performed on a low-frequency pixel array of a multi-level scale level of the pixel array and at least one high-frequency pixel array of the multi-level scale level of the pixel array, and wherein a second edge-preserving convolution step is performed on the filtered low-frequency pixel array of the multi-level scale level of the pixel array by weighted filtering, wherein the filtering weight of the pixel depends simultaneously on the distance between the pixel value associated with the pixel in each pixel array of the second pixel array group, and wherein the step of reorganizing the output pixel array is accomplished by recursively performing an inverse transform of the hierarchical multi-scale decomposition on the low-frequency pixel array filtered by the second edge-preserving convolution and the high-frequency pixel array filtered by the first edge-preserving convolution.

12. A controller, comprising at least one processor and at least one memory, wherein the memory contains computer program code, and the at least one memory and the computer program code are configured to, together with the at least one processor, enable the controller to execute the method according to any one of the preceding claims 1-11.

13. A computer program product comprising computer executable instructions for executing the method according to any one of the preceding claims 1 to 11 when the program is run on a computer.

14. A computer-readable storage medium comprising computer-executable instructions, wherein when the program is run on a computer, the computer-executable instructions are used to execute the method according to any one of the preceding claims 1 to 11.

Images