Simultaneous and consistent handling of image data and associated noise models in image processing and image composition

Abstract

To provide an image processing method for maintaining a consistent noise model over the entire image.SOLUTION: A method includes a step of obtaining input data 100 including input raw image data 110 to be processed, for storage and / or transmission. The input raw image data 110 includes a value y of pixels of an image sensor to be used to obtain the image data. The method further includes a step 130 of processing the input raw image data 110, and a step of outputting the processed image data by providing output data 140. The step of obtaining the input data 100 includes a step of obtaining an input noise model 120 from the input data 100. The step 130 of processing the input raw image data 110 includes at least one preprocessing operation or image compositing operation.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to the field of image processing, and more particularly to a method for processing image data including noise and information, comprising the steps of: acquiring input data for storage and / or transmission, the input raw image data including input raw image data to be processed, wherein the input raw image data includes pixel values ​​of an image sensor used to acquire the image data; processing the input raw image data; and outputting the processed image data by providing output data. [Background technology]

[0002] Generally speaking, the present invention is situated in the context of image sensors, which are increasingly being incorporated into a growing number of devices. Image data can be easily and inexpensively generated in large quantities, and sophisticated processing algorithms enable a wide range of applications.

[0003] The overall performance of processing algorithms generally depends heavily on the quality of the image data. It is often best to start with raw image data, i.e., unmodified image data from the acquisition device, and perform any type of processing and analysis from there. Operations including, but not limited to, the conversion of image data from a color filter array to RGB data or the application of standard irreversible compression can result in significant loss of information and quality, and such operations should be avoided. For this reason, the applicant has developed an image compression technique disclosed in European Patent Application EP 3 820 150, in which the recovered image data is statistically equivalent to uncompressed raw sensor data.

[0004] Raw image data often contains imperfections, such as non-uniform response or faulty pixels in the image sensor, insufficient illumination, or imperfect optical elements that cause distortion and vignetting. Numerous methods have been developed in the prior art to address such problems in order to obtain better results from image processing and analysis, and in addition, to obtain images that are subjectively better looking. While these methods have been quite successful in improving the appearance of individual images, they have failed to preserve the statistical properties of the image data because, as is generally recognized, the noise in raw image data contains signal-dependent components that behave non-linearly. For this reason, applying linear corrections such as scaling pixel values ​​or replacing pixel values ​​with the mean of their neighbors corrects the local characteristics of the noise, resulting in noise that is inconsistent across the entire image.

[0005] For example, U.S. Patent No. 7,683,948 discloses a method for defective pixel replacement in image processing, in which a broken pixel is detected and replaced with the mean, average, maximum, or other statistical function of selected surrounding pixels. Applying the mean, average, or median results in a replaced pixel value with a noise level below the expected value, which becomes apparent when processing several consecutive images simultaneously.

[0006] The document titled "Flat-field correction technique for digital detectors" by Seibert et al. discloses a method for correcting the non-uniform response of a sensor. This method is based on measuring the response curve for each individual pixel. Then, the curve is approximated by a linear model, generating two parameters for each pixel. Next, normalization with respect to the pixel-specific response curve is used to achieve flat-field correction. This technique helps improve the appearance of the image but almost certainly degrades the statistical properties of the image data. More recently, U.S. Patent No. 9,143,709 has adapted this method to be suitable for image sensors with non-linear responses, but the above-mentioned problem regarding inconsistent noise remains.

[0007] The Digital Negative (DNG) specification by Adobe Inc. also recognizes the need to correct defective pixels and peripheral dimming. This is addressed by providing operations represented by so-called opcodes that are specially parameterized for those specific purposes. The said opcodes are FixVignetteRadial, FixBadPixelsConstant, and FixBadPixelsList. However, similar to the above prior art, these implementations also do not attempt to preserve consistent noise in the above sense. Specifically, FixVignetteRadial functions by applying a radially varying normalization and is equivalent to the above prior art method, and FixBadPixelsConstant and FixBadPixelsList are aimed at correcting defective pixels using interpolation, which is equivalent to replacing the pixel value with a local average.

[0008] Some prior arts also use noise models in image processing. For example, the DNG specification allows information about noise profiles to be stored in the DNG file format. In this case, the main purpose is to use the noise profile for noise removal. However, most image data does not have any attached information about noise characteristics, and there may not even be an appropriate noise model for various amounts of correction and preprocessing. Noise removal methods that directly estimate noise parameters from image data, such as those proposed in the literature "Variance Stabilization for Noisy+Estimate Combination in Iterative Poisson Denoising" by Azzari and Foi, have been developed. However, the direct estimation of noise models from image data is often insufficient, and such techniques cannot be used with high reliability for general image processing purposes.

[0009] Noise models also bring benefits to applications other than noise removal, such as in the field of medical imaging using X-rays. In that case, it is possible to investigate through simulation the potential for reducing patients' radiation exposure for various image diagnostic methods. Here, a small X-ray dose is accompanied by the sacrifice of a decrease in the signal-to-noise ratio (SNR), and as a result, the SNR can only be realistically simulated by using a noise model. A method for simulating dose reduction through a combination of scaling and noise injection is disclosed in the literature "A Technique for Simulating the Effect of Dose Reduction on Image Quality in Digital Chest Radiography" by Veldkamp et al., in which a noise model is used for the noise injection. However, similar to the above-described applications, the use of noise models remains extremely limited in this prior art application as well.

[0010] Finally, an increasing number of applications rely on machine learning techniques for processing and / or interpreting image data. In more classical image processing, the central challenge was generally the invention and development of appropriate algorithms, but a further challenge in the context of machine learning is obtaining large amounts of high-quality training data.

[0011] To obtain highly reliable results using machine learning, the training image set must contain samples of all expected variations in the target environment, and the amount of training data increases exponentially along with the parameter space. This problem has been partially mitigated by introducing data augmentation, i.e., by adding synthetic images, which are modified versions of existing data, to the training set. Examples of such modifications include geometric transformations (rotation, translation, scaling, and inversion), as well as manipulation of contrast and color, or the injection of noise. Such types of data augmentation by prior art do not involve the use of noise models.

[0012] In summary, conventional methods for correcting image imperfections and generating composite images suffer from the drawback of focusing on the image's appearance, thereby ignoring one of the key characteristics of raw image data: the very specific relationship between the signal and the noise arising from fundamental physical principles and the technical nature of the acquisition device. Breaking this relationship prevents processing algorithms from reliably using noise and / or measurement uncertainty contained within the raw image data, introduces confusing elements into machine learning training sets, and makes it virtually impossible to find proper and consistent normalization of image data from various sensors. [Overview of the project] [Problems that the invention aims to solve]

[0013] Overcoming the aforementioned drawbacks is an objective of the present invention. Generally, one objective of the present invention is to provide an image processing method that maintains a consistent noise model across the entire image. A further objective of the present invention is to enable users of image data to obtain image data with clearly defined statistical properties, and to enable them to utilize these statistical properties in their own calculations and analyses based on this image data.

[0014] A further object of the present invention is to provide an image preprocessing method that corrects various imperfections present in the acquisition hardware or acquisition conditions.

[0015] Another object of the present invention is to provide a method for generating synthetic image data from existing image data such that the generated synthetic image data is sufficiently suitable for training machine learning algorithms, validating image processing under challenging conditions, and other similar purposes. [Means for solving the problem]

[0016] In this regard, the present invention proposes a method characterized by the features enumerated in claim 1 and enabling the achievement of the objectives specified above. Specifically, the method according to the present invention comprises the step of acquiring input data, A step of obtaining an input noise model from the input data, wherein the input noise model is adapted to reflect noise present in the input raw image data, and the input raw image data and the input noise model together form the input data in a manner that links the input raw image data to the input noise model. It is distinguished from conventional technology by including, The step of processing the aforementioned input raw image data is: ° Processing operation to correct the pixel values ​​of the input raw image data, °Processing operation to apply flat-field correction to the input raw image data, ° Processing operation to apply correction for photosensitivity non-uniformity to the input raw image data, ° Processing operation that applies binning and / or averaging operations to the input raw image data, ° Processing operation to perform normalization of the input raw image data, °Processing operations that apply rounding and / or truncation, and / or integer quantization, to the input raw image data, °In particular, a processing operation that performs image synthesis by creating and / or adding a composite image based on the input raw image data, and / or modulating the input raw image data. It includes at least one of the following: The processing step described above is °A step of determining an output noise model that is adapted to reflect the noise present in the output data, based on the input noise model and in accordance with the processing operation applied to the input raw image data, °The processing operation applied to the input raw image data, and the step of generating output raw image data that is statistically consistent with the output noise model based on the input noise model and / or the output noise model. It is further distinguished from conventional technology by including, Furthermore, the step of outputting the processed image data is A step of storing and / or transmitting the output raw image data and the output noise model, wherein the output raw image data and the output noise model together form the output data in such a manner that the output raw image data is linked to the output noise model. Includes, To make the aforementioned processing suitable for pipeline processing, the output data is used as input data and processed either by any one of the processing operations alone or by any combination of the processing operations. This distinguishes it from conventional technology.

[0017] In this way, image data and the noise model are considered as inseparable entities that pass through the image processing pipeline together. Each processing step in the pipeline can modify either the image data or the noise model, or both. Therefore, the processed image data will in all cases have statistical properties consistent with the output noise model. The noise model further allows the user of the image data to obtain uncertainty about individual pixel values, and thus perform their own calculations and analyses based on these pixel values.

[0018] Furthermore, the present invention provides a method for flat-field correction, which includes, by at least some of the processing operations, replacement of dead pixel values, correction of light sensitivity non-uniformity, and correction of non-uniform illumination and vignetting. Such image preprocessing is performed in a manner that allows the noise model of the input data to remain valid after preprocessing, i.e., remain applicable to the output data.

[0019] The present invention also provides a method for generating synthetic image data, particularly synthetic image data that follows the same noise model as the input data. Specifically, a method is provided for generating synthetic images that mimic a reduction in exposure time or light collection.

[0020] The method according to the present invention can be realized in various embodiments.

[0021] In one particularly preferred embodiment, the input noise model and / or the output noise model is represented by a mapping of the average pixel value of a given pixel value of an image sensor used to acquire image data to the pixel value standard deviation, a mapping of the average pixel value to the pixel value variance, or a mapping of the average pixel value to the signal-to-noise ratio.

[0022] More preferably, the input noise model and / or the output noise model are adapted to represent noise in the input raw image data and / or the output raw image data that follows a Poisson-Gaussian distribution.

[0023] In another particularly preferred embodiment, generating output raw image data that is statistically equivalent to the input raw image data and consistent with the output noise model is ensured by applying a transformation representing noise-matched scaling behavior to a given pixel value of the image sensor used to acquire the image data, the transformation of which can perform pixel value reduction.

[0024] Other features and advantages of the present invention are mentioned in the dependent claims and in the description below, which discloses the invention in more detail with reference to the figures.

[0025] The attached figures illustrate and schematicly illustrate the principle of the present invention and several embodiments. [Brief explanation of the drawing]

[0026] [Figure 1] This is a schematic diagram of the noise-matched image processing according to the present invention. [Figure 2] This figure shows one embodiment of the noise-matched image processing method according to the present invention, which combines several features to optimize the performance of an image processing pipeline starting from initial input data consisting of raw input image data and an input noise model. [Figure 3] This diagram schematically illustrates how, in some embodiments of the noise-matching image processing method according to the present invention, two or more processing operations acting on input data can be combined into a single processing operation that performs the combined function of each of the said processing operations. [Modes for carrying out the invention]

[0027] The present invention will be described in detail below with reference to the figures described above.

[0028] The concept of the present invention is shown in FIG. 1. Although the present invention relates to the processing of raw image data 110, in contrast to established methods, the raw image data 110 is only part of the input data 100 for the processing algorithm 130. The processing algorithm 130 requires a noise model 120 as a second input. The raw image data 110 and the noise model 120, which together form the input data 100, are then processed together to generate output data 140 consisting of output raw image data 150 and output noise model 160. The processing 130 generates output raw image data 150 such that every pixel of its output is consistent with the output noise model 160. An example of noise-consistent processing 130 is shown below. In some processing algorithms, the output noise model 160 is the same as the input noise model 120. The reason the output image data 150 is denoted as raw is that it is statistically equivalent to the actual raw image data coming from the sensor, and the output noise model 160 is applied to this output image data 150. Thus, the output data 140 is adapted to serve as the input data 100 for another step of the noise-consistent processing 130, and thus the entire processing pipeline can be built based thereon.

[0029] Noise model The noise model is an essential building block of the present invention. For example, in accordance with the EMVA1288 standard, hereinafter, as the noise model, the standard deviation σ y of the value y of a given pixel is associated with its average value μ y by the mapping σ y (μ y ) is considered. Generally, the average value μ y is determined according to the flux and wavelength of the photons colliding with the pixel, while the standard deviation σ y is determined only according to the average value μ y . Hereinafter, the mapping σ y (μ y ) introduced above is focused on, but the pixel variance

[0030]

Number

[0031] The mean value μ y Dependence and signal-to-noise ratio

[0032]

number

[0033] However, μ y.dark σ is the black level, i.e., the average pixel value in the absence of light, etc. y (μ y There exists an equivalent mapping that can be used to extract ).

[0034] The mapping / model described above relates to so-called time noise, which quantifies the variation in pixel values ​​when acquisition is repeated under the same conditions. In principle, each pixel of a given image sensor has its own characteristic mapping σ y (μ y While it is possible to have (variables), recent image sensors have achieved extremely good uniformity, and therefore, in many cases, using the same model for all pixels of the sensor yields a good approximation. However, image sensors can exhibit spatial noise, which quantifies the difference in the average values ​​of different pixels exposed at the same illumination. Spatial noise can often be reduced or even eliminated using preprocessing. A defective pixel can be considered an extreme form of spatial noise.

[0035] Depending on the type of image data to be processed, mapping σ y (μ y A properly parameterized version of this can become a digital representation of the noise model. This example, usable as input noise model 120 or output noise model 160, ·Finite point set {(μ i ,σ i )}, from there, by linear interpolation or nonlinear interpolation, or by any other suitable approximation of the set of points, a continuous model σ y (μy ) can be obtained, or ·σ y (μ y A set of coefficients used to construct a polynomial that approximates ), or • Any type of function in which the free parameters are combined with values ​​specific to those parameters. That is the case.

[0036] For the purpose of noise-matching processing 130, the input raw image data 110 is combined with a digital representation of the input noise model 120 to form input data 100. In various embodiments of the present invention described below, this combination is • Store the raw image data together with the noise model in a suitable (single) container file, such as a Tagged Image File Format (TIFF) file, or - The noise model is stored in a so-called sidecar file, i.e., a (second) file, which is combined with the (first) file in which the raw image data is stored, and the sidecar file is adapted to store additional (meta)data not supported by the raw image data file, or • Store the noise model in a database that links specific raw image data, stored within a corresponding (single) raw image data file, to its associated noise model. This can be achieved by any of the following various methods or combinations thereof.

[0037] In any case, according to the present invention, storing a noise model means storing a digital representation of the model's parameters. In some embodiments of the present invention, these parameters are passed to a function having a fixed definition. In other embodiments, the implementation consists of a catalog of functions, and the specific function to be used is selected by an additional identification parameter. While some embodiments of the present invention utilize a noise model in the very general sense, as discussed above, other embodiments specialize in the Poisson-Gaussian noise model, which is commonly used for linear image sensors where the sensor output is directly proportional to the amount of incident light. The Poisson-Gaussian noise model is,

[0038]

number

[0039] Given by, However, σ y.dark μ represents the standard deviation of pixel values ​​in the absence of light and is often called readout noise. y.dark {K,μ} is the average pixel value in the absence of light, and K is the gain coefficient. Therefore, the Poisson-Gaussian noise model has three parameters {K,μ}. y.dark ,σ y.dark It is represented by}.

[0040] It should be noted that the Poisson-Gaussian noise model is applicable to raw image data output by the vast majority of image sensors. The Poisson-Gaussian noise model is a formal representation of the statistical quantity of naturally occurring pixel value fluctuations during image acquisition using linear sensors. Despite its ubiquitous nature, the importance of the Poisson-Gaussian noise model is generally ignored in prior art for image data correction and processing.

[0041] Noise-matched processing The image data processing that is the subject of this invention can be divided into two categories. The first category is image preprocessing, which is primarily aimed at correcting imperfections in the acquisition hardware or acquisition conditions, or at standardizing the image data in a particular way. The second category relates to the synthesis of image data, which can then be used to test and improve the reliability of image processing algorithms, or to enrich training datasets in the context of machine learning.

[0042] While both processing categories exist in conventional techniques, they fail to focus on the underlying noise model, nor do they attempt to generate output data consistent with the noise model, nor do they consider how the processing affects image statistics, and therefore the noise model. In conventional techniques, only processing algorithms specifically related to noise, such as denoising and deconvolution, sometimes use noise models in the specific and limited ways described in the introduction. Even in these cases, the focus of conventional techniques has always been on the single operation at hand, without attempting to generalize the method to the entire processing pipeline, nor without generating output data that remains consistent with any noise model.

[0043] However, images with a consistent and accurate noise model across the entire image allow for far more reliable processing, especially when the noise model is taken into account in the processing algorithm. Conversely, images that do not follow any consistent noise model tend to confuse algorithms with their unexpected statistics. This is especially true for deep learning and neural network-based algorithms that consider all aspects of image data, including noise and noise correlation, when analyzing them.

[0044] Two important special considerations in this context are that a) when the input is simply a single pixel value and its noise model, not every operation can necessarily increase the signal-to-noise ratio for that pixel, and reduction can occur, and b) the signal-to-noise ratio generally increases with higher illumination (and thus larger pixel values), but the improvement is neither linear nor proportional. Based on this, it is possible to mimic a reduction in illumination corresponding to the product of multiplying the initial average pixel value by a coefficient q, where 0 < q < 1, and each can be achieved. To do so, the initial pixel value y is replaced with a corrected pixel value y' = q·y + δ(y,q), where δ is a (pseudo) random number sampled from a normal distribution with mean 0 and variance

[0045] [Number]

[0046] and has, where

[0047] [Number]

[0048] is the standard deviation σ of the noise of the output raw image data 150 y (qy) is a coefficient that reduces by the amount of the actual pixel value reduction by the coefficient q. Note that q < q' < 1. This transformation represents a noise-consistent scaling operation, which is represented by the symbol S(y,q), i.e., S(y,q) = q·y + δ(y,q), and its usage in the context of the present invention will become clearer below. In fact, this transformation can be used to perform a more refined operation in the context of image correction when correctly applied.

[0049] In a specific case of the Poisson-Gaussian noise model, the variance used for the pseudo random number δ(y,q) is

[0050] [Number]

[0051] It is given by.

[0052] Defective Pixel Correction In conventional techniques, defective pixels are generally corrected by interpolation using correctly functioning neighboring pixels, either in the hardware of the imaging device or in the software as part of the processing pipeline. As a result of interpolation, the resulting pixel value does not fluctuate as much as that of a functioning pixel, and the interpolated pixel can be clearly identified by a different light sensitivity curve.

[0053] Since the noise of defective pixels is below the expected value, their statistics can be made consistent with the noise model by adding the noise. This is achieved using noise-matched scaling. For example, the value y C The value y L and y R Please consider the defective pixels that should be replaced by using the left and right neighbors that have this value y. C In the formula y' C =S(y L +y R -μ y.dark (q=1 / 2) By substituting this, the above equation generally reproduces the same value as obtained by using interpolation, and in addition, y is consistent with the noise model. C It also reproduces the standard deviation.

[0054] Flat-field correction Apertures, lenses, and filters often result in spatially non-uniform "capture" of incident photons, such as vignetting, i.e., when a uniformly lit scene is imaged, it appears darker towards the edges. In conventional techniques, a standard way to correct this is to use a reference flat-field image N(i,j), where i and j represent pixel coordinates, y'(i,j) = y(i,j) / N(i,j) is achieved by normalizing through

[0055] In this case, the flat field image should be as noise-free as possible, which can be achieved by fitting a mathematically smooth function to the reference image or by averaging a number of reference images. Additionally, since one usually wants to preserve the relatively bright parts (usually the center) in the original image, N(i,j) should be normalized for the range 0 < N(i,j) ≤ 1.

[0056] This method can be used to normalize the image brightness, but it is no longer possible to use the same noise model for the entire image because previously relatively dark parts now have the same brightness as previously relatively bright parts of the image but have a larger relative noise than the previously relatively bright parts of the image.

[0057] In the present invention, noise-consistent flat field correction is achieved by using noise-dependent scaling with a pixel-dependent coefficient q(i,j) to darken relatively bright parts of the image. All considerations of noise-consistent scaling, i.e., regarding the use of the equation y' = q·y + δ(y,q) to determine the corrected pixel values, still apply, and the values of q(i,j) for making the noise-consistent flat field correction reliable, etc., are

[0058]

Number

[0059] are

[0060] Correction of photosensitivity non-uniformity Light sensitivity non-uniformity is a specific type of non-uniform illuminance resulting from varying detection efficiencies of individual pixels. The difference is typically around 1% to 2%, and can be caused by differences in absorption within the active area of ​​a pixel, among other factors.

[0061] The changing detection efficiency is equivalent to non-uniform illumination or exposure and can be corrected in the same way as flat-field correction by using the spatially varying relative detection efficiency, which is determined through careful calibration of the image sensor used to acquire the image data, as N(i,j).

[0062] Binning and averaging The noise-matching processing method presented above ensures that the output raw image data 150 remains consistent with the input noise model 120. In many use cases, there is a need or interest in performing processing that results in an output noise model 160 that, while modified, remains applicable to the entire output raw image data 150. For example, binning may be used at the expense of spatial resolution, resulting in an increased signal-to-noise ratio. The key here is to update the noise model so that the next processing step can distinguish between binned data with a consistently modified noise model and cropped data where the noise model remains the same and therefore the noise model is inconsistent with that of the binned data.

[0063] One implementation of binning is to take the sum of groups consisting of N=n·m pixels, thereby determining the value for each group.

[0064]

number

[0065] This consists of having an output pixel having, where the subscript i is a label indicating the spatial position within the block being binned.

[0066] Output noise model σ Y (μ Y ) must be updated to take into account the new dark conditions. Specifically, Y has a contribution of N times the black level, i.e., μ Y.dark =Nμ y.dark Therefore, the readout noise in the output model is

[0067]

number

[0068] That is the case.

[0069] In another implementation of the binning operation, the output pixel values

[0070]

number

[0071] It is calculated as follows.

[0072] This means that the black level of the output noise model is the same as the black level of the input noise model, and therefore from 0 to μ y.dark This has the advantage that the range of rarely used pixel values ​​in between is not expanded unnecessarily.

[0073] In contrast to the mean-based binning operation described below, the two binning operations described above map integers to integers and therefore do not carry the risk of precision loss associated with rounding or truncating fractional parts.

[0074] Another version of the binning operation averages the input pixel values.

[0075]

number

[0076] By taking this, the output pixel value can be obtained.

[0077] This is particularly useful when the digital representation of the output raw image data 150 is a bounded data type such as integer values ​​with a fixed bit depth, because it avoids clipping of large sums of values. In the case of averaging, the black levels of the data remain unchanged, but the output noise is smaller than the input noise.

[0078]

number

[0079] Specifically,

[0080]

number

[0081] Therefore, in the case of the Poisson-Gaussian noise model, the gain parameter of the output model is equal to K / N.

[0082] Similarly, pixels that are in the same position but exposed separately can be summed or averaged. In this case, the output raw image data 150 has the same dimensions as the input raw image data 110, but the output noise model 160 must be modified as in the case of spatial summing or spatial averaging. For example, averaging N exposures to obtain pixel values

[0083]

number

[0084] However, in this case the subscript i is a label for the exposure number, and the value y i This is the output pixel value in the image data.

[0085]

number

[0086] An output image can be obtained that has the same position as . In the exposure-averaged output noise model,

[0087]

number

[0088] Therefore, in the case of the Poisson-Gaussian noise model, the output gain parameter is equal to K / N.

[0089] Normalization Some embodiments of noise matching processing use different parameters as input raw image data 110.

[0090]

number

[0091] Having a sequence of images from several million different devices, each with a Poisson-Gaussian noise model, the goal of processing is to generate output raw image data 150 formed by the sequence of output images, which follows a single output noise model 160. This determines which noise model has the largest normalized readout noise.

[0092]

number

[0093] To determine whether it has, and each pixel value y of the input image i i Regarding

[0094]

number

[0095] However, δ i The mean is 0 and the variance is

[0096]

number

[0097] The output pixel values ​​ε are (pseudo) random numbers sampled from a normal distribution having the following characteristics: i This can be achieved by substituting the following. In that case, the parameters of the common output noise model 160 are {K=1,μ ε.dark =0,σ ε.dark =σ ε}

[0098] integer quantization In some of the operations described above, floating-point arithmetic or floating-point random numbers are used for correction, and the change to the input pixel value may be small. The overall correction may actually be less than 1 (unity); for example, if a 2% reduction should be applied to a pixel value of 10, the effective corrected pixel value may be 9.8. In general, such small corrections can be applied by storing the corrected value as a non-integer representation (e.g., fixed-point, floating-point, or rational number). However, in many use cases, it is desirable to have an output in integer representation. Simple (naive) implementations such as truncation and rounding may result in a difference between the desired correction coefficient and the effective correction applied. For example, truncation or rounding may introduce an unintended shift, also known as bias, to the statistical mean of the image data. To avoid this, a preferred embodiment of the method according to the present invention involves noise-matched rounding, i.e., dithering before the actual rounding operation, i.e., the addition of appropriately selected random numbers Δ which differ for each pixel, i.e. y' = round(y + Δ) To provide.

[0099] Of particular interest are random numbers following a uniform distribution between -1 / 2 and +1 / 2, and random numbers following a triangular distribution between -1 and +1, i.e., a distribution formed by adding two independent samples of the aforementioned uniform distribution, because both distributions introduce zero bias. Furthermore, the latter introduces quantization noise that is uncorrelated with Δ. In some applications, it may be beneficial to use other random distributions for Δ.

[0100] The quantization operation introduces a small amount of extra noise, thereby reducing the readout noise of the output noise model.

[0101]

number

[0102] ,however

[0103]

number

[0104] Δ is the variance of the distribution of random numbers, and it needs to be fitted according to this. Due to this additional noise, and to minimize the introduction of processing errors, a preferred implementation of the processing pipeline according to the present invention is to perform quantization only once, as the final operation of the pipeline.

[0105] Image synthesis using exposure scaling and gain scaling. The normalization procedure presented above is well-suited for machine learning and algorithm testing because it allows for the collection of data from numerous sources without increasing the number of noisy models that the algorithm must be able to handle or tolerate.

[0106] To further increase the amount of training or test data, images with a reduced signal-to-noise ratio can be simulated within the same noise model. This noise model can be a model of a single set of the original input data 100, or a model of a set of normalized input data generated by following the procedure described above. A composite image having properties equivalent to a reduction of exposure time or detection efficiency by a coefficient q can then be generated by applying a noise-matched scaling S(y,q), as defined in more detail above, to each pixel of the input raw image data 110 having a value y.

[0107] In other situations, it is desirable to increase the noise level while keeping the signal constant. In the case of the Poisson-Gaussian input noise model, this can be achieved in two ways. The trivial way is to use a mean value of 0 and a desired standard deviation σ δ This involves adding a (pseudo) random value δ to all pixel values ​​of the input image data 110. This substantially corresponds to increasing the readout noise parameter, which in the output noise model is

[0108]

number

[0109] This should be changed. Another method combines normalization with exposure scaling by coefficient q, followed by applying a new gain coefficient K'=K / q, and input black level μ y.dark Adding it again, that is

[0110]

number

[0111] This consists of the following. Through this transformation, the output raw image data 150 has the same mean value as the input raw image data 110, and the output noise model has a new parameter {K'=K / q,μy.dark , σ y.dark ' = σ y.dark has / q}.

[0112] It is clear from the above description that the processing step 130 of the method according to the present invention generally, that is, in all cases, on the one hand, based on the input noise model 120 and in accordance with the processing operation applied to the input raw image data 110, includes a step of determining an output noise model 160 adapted to reflect the noise present in the output data 140, and on the other hand, includes a step of generating output raw image data 150 that is statistically consistent with the output noise model 160 based on the processing operation applied to the input raw image data 110 and the input noise model 120 and / or the output noise model 160. Needless to say, these two steps are mutually dependent.

[0113] In fact, the processing is an operation (or a series of operations) that is noise-consistent, and thus occurs within the input noise model, that is, by using the input noise model as the output noise model, or the processing is an operation (or a series of operations) that is not noise-consistent, and thus the input noise model is simultaneously corrected to generate an output noise model that is consistent with the processed image data. It is either one of them.

[0114] In the case of the first solution described above, the processing step 130 preferably includes a step of applying a transformation S(y, q) representing a noise-consistent operation to the value y of a given pixel of the image sensor used for acquiring the image data, where q is a pixel value reduction coefficient and 0 < q < 1, and the processing step 130 uses an output noise model 160 that is the same as the input noise model 120.

[0115] In the case of the second solution described above, the processing step 130 preferably includes a step of applying noise mismatch operation to a given pixel value y of an image sensor used to acquire image data, wherein the processing step 130 uses an output noise model 160 that is different from the input noise model 120 and is statistically consistent with the noise of the output raw image data 150 of the output data 140.

[0116] One way to verify that the acquired image data is consistent with the associated noise model is to acquire the set of images under identical conditions so that any difference between pairs of individual images can be attributed to noise. Preferably, the pixel values ​​of a single image should cover the majority of all possible pixel values. Then, for each pixel position i, the mean value μ is calculated using all images in the set. i and standard deviation σ i The more images there are in the set, the higher the mean value μ. i and standard deviation σ i The estimation becomes better. Finally, point (μ i ,σ i ) is the function σ of the noise model y (μ y ) check whether it approximates well. Similarly, to check that the output noise model of the processing operation is consistent with the output noise model, pass each individual image in the set through the processing operation and measure the mean and standard deviation (μ') for each pixel position in the output image data. i ,σ' i This can be verified by extracting all points (μ'). i ,σ' i ) is the function σ' of the output noise model. y (μ' y If the output noise model is well approximated by ), it is considered consistent with the output image data.

[0117] Figure 2 illustrates one embodiment of a noise-matched image processing method according to the present invention, which combines several features to optimize the overall performance of an image processing pipeline starting from initial input data 100 consisting of input raw image data 110 and an input noise model 120. In the context of the present invention, the input data 100 must consist of both input raw image data 110 and a noise model 120, so the two arrows 311 and 312 connecting subsequent blocks 310, 320, 330, 340, 350, 360, and 370, each schematically representing a processing operation, symbolize that both raw image data and noise model information are propagated through each processing operation. This input data is processed through one or more operations 310, 320, 330, 340, 350, 360, and 370. These operations handle the input data (image data and noise model) and supply processed output raw image data 150 and an output noise model 160 associated with this processed output raw image data, and these two parts together form output data 140. The availability of the noise model in operations 310, 320, 330, 340, 350, 360, and 370 enhances the performance of the operations. Optionally, a specific processing operation 385, i.e., a specific element of the processing pipeline, can utilize both image data and noise data, but it is possible to output only image data, as schematically shown in Figure 2 by a single arrow 311 exiting from the block corresponding to this specific processing operation 385. This specific processing operation 385 may be added at the end of the processing pipeline, and this specific processing operation 385 can be, for example, a deconvolution. However, prior to this particular processing operation 385, both raw image data and noise model data are available at each intermediate step, and therefore the output data 365 from the preceding processing operation becomes the input data 365 for the next processing operation.Optionally, several additional processing operations 390, which may also be added at the end of the processing pipeline and may be classification or segmentation, may not require and / or output noise model data, but may still benefit from a high-quality processing pipeline up to that point in the noise-matched image processing method according to the present invention, thereby making the final result 395 more precise, more adequately qualified, and more reliable.

[0118] Figure 3 schematically illustrates several embodiments of the noise-matching image processing method according to the present invention, symbolized in Figure 3 by blocks 520, 521, and 522. Two or more processing operations acting on the input data 100 can be combined into a single processing operation 510 that performs the combined function of each individual processing operation 520, 521, and 522, such as generating output data 140 consisting of both output raw image data 150 and output noise model 160. As an example, pixel linearization and vignetting correction can be combined into a single processing operation 510. In Figure 3, the two arrows connecting consecutive blocks 100, 510, and 140 symbolize that, in this case as well, both raw image data and noise model information are propagated through each processing operation combined into the single processing operation.

[0119] Finally, the present invention also relates to computer program means stored in a computer-readable medium adapted to implement the methods described above, as well as to devices equipped with such computer program means. For example, such devices may be microprocessors, field-programmable gate arrays, image sensors, mobile phones, smartphones equipped with digital cameras in particular, digital photographic devices, digital video cameras, scanning devices, tablets, personal computers, servers, microscopes, telescopes, or satellites.

[0120] In light of the above description of various embodiments of the method according to the present invention, its advantages are clear.

[0121] Firstly, the availability of a consistent noise model according to the present invention brings powerful benefits to a wide range of image processing applications. In fact, since the noise model represents the statistical uncertainty of image data, it provides valuable insights, particularly in terms of acceptable limits and reproducibility for scientific and metrological applications. This is made possible by the method according to the present invention. Conversely, without a consistent noise model, the reliability of image processing and analysis is reduced, in which case the results may depend on the degree of correction or preprocessing applied to the image data. This can be avoided by using the method according to the present invention.

[0122] Secondly, the availability of a consistent noise model according to the present invention brings significant benefits to applications that rely on machine learning techniques for processing and / or interpreting image data, because it allows for the acquisition of large amounts of high-quality training data, as the noise model can be used to improve the efficiency of the algorithm and eliminate potential points of confusion in the algorithm. In the context of machine learning, this is even more important because machine learning algorithms exploit the (statistical) properties of data far more powerfully than human observers who ignore statistical properties such as noise and noise correlation. [Explanation of symbols]

[0123] 100 input data, blocks 110 Input raw image data, input image data 120 Input Noise Models 130 Processing algorithms, noise-matched processing, processing steps 140 Output data, blocks 150 Output raw image data, output image data 160 Output Noise Models 310 blocks, operation 311 Arrow 312 Arrow 320 blocks, operation 330 blocks, operation 340 blocks, operation 350 blocks, operation 360 blocks, operation 365 Output data, input data 370 blocks, operation 385 Specific processing behavior 390 Additional processing operation 395 Final result 510 Single-process operation, block 520 blocks, processing operation 521 Blocks, Processing Operations 522 blocks, processing operations

Claims

1. A method for processing image data including noise and information, comprising: - obtaining input data (100) including input raw image data (110) to be processed for storage and / or transmission, wherein the input raw image data (110) includes pixel values y of a pixel of an image sensor used for obtaining the image data; - processing the input raw image data (110) (step 130); - outputting the processed image data by providing output data (140). The method further includes: The step of obtaining the input data (100) includes: ° obtaining an input noise model (120) from the input data (100), wherein the input noise model (120) is adapted to reflect noise present in the input raw image data (110), and the input raw image data (110) and the input noise model (120) together form the input data (100) in a manner that links the input raw image data (110) to the input noise model (120). The method further includes: The step of processing the input raw image data (110) (step 130) includes: ° a processing operation of correcting pixel values of the input raw image data (110); ° a processing operation of applying flat field correction to the input raw image data (110); ° a processing operation of applying correction for light sensitivity non-uniformity to the input raw image data (110); ° a processing operation of applying a binning operation and / or an averaging operation to the input raw image data (110); ° a processing operation of performing normalization of the input raw image data (110); ° a processing operation of applying a rounding operation and / or a truncation operation, and / or integer quantization to the input raw image data (110); ° particularly, a processing operation of performing image synthesis by creating and / or adding a synthetic image based on the input raw image data (110) and / or by modulating the input raw image data (110). The method includes at least one of the above operations. The step of processing (step 130) includes: ° determining an output noise model (160) adapted to reflect noise present in the output data (140) based on the input noise model (120) and in response to the processing operations applied to the input raw image data (110). ° generating output raw image data (150) that is statistically consistent with the output noise model (160) based on the processing operation applied to the input raw image data (110) and the input noise model (120) and / or the output noise model (160); further comprising the step of outputting the processed image data ° storing and / or transmitting the output raw image data (150) and the output noise model (160), the output raw image data (150) and the output noise model (160) together forming the output data (140) in a manner that links the output raw image data (150) to the output noise model (160); comprising enabling the step of processing (130) the output data (140) as input data (100) by any one of the processing operations alone or by any combination of the processing operations, such that the processing (130) is adapted for pipeline processing. Claim 2 The input noise model (120) and / or the output noise model (160) determine the mean pixel value μ of a given pixel value y of the image sensor used to acquire the image data. y , pixel value standard deviation σ y Mapping σ to y (μ y ), the average pixel value μ y , pixel value variance 【Number 1】 mapping for 【Number 2】 or the average pixel value μ y of the signal-to-noise ratio SNR y to the mapping SNR y (μ y ) is expressed by, the method according to claim 1, characterized in that. Claim 3 the input noise model (120) and / or the output noise model (160) - A finite set of points adapted to obtain a continuous model σ y (μ y ) by linear or non-linear interpolation. - Continuous model σ y (μ y ) and is adapted to construct a polynomial function approximating it, or - Continuous model σ y (μ y ) is a function in which the free parameter related to is represented by where σ y is the average value μ of a given pixel value y of the image sensor used to acquire the image data. y 3. The method according to claim 1 or 2, characterized in that the standard deviation is relative to Claim 4 the input noise model (120) and / or the output noise model (160) is adapted to represent noise following a Poisson-Gaussian distribution represented by 【Number 3】 in the input raw image data (110) and / or the output raw image data (150); However, σ y is the standard deviation of the value y of a given pixel of the image sensor used for acquiring the image data with respect to its average value μ y , and σ y.dark is the standard deviation of the pixel value in the state without light, μ y.dark is the average value of the pixel value in the state without light, and K is a gain coefficient. The method according to claim 1 or 2, characterized in that. Claim 5 The method according to claim 1 or 2, wherein the step of processing (130) comprises applying a transformation S(y,q) representing a noise-consistent operation to a value y of a given pixel of an image sensor used for acquiring the image data, where q is a pixel value reduction factor and 0 < q < 1, and the step of processing (130) uses an output noise model (160) that is the same as the input noise model (120). Claim 6 The transformation S(y,q) representing a noise-consistent scaling operation is represented by y' = q·y + δ(y,q), where y' is the corrected pixel value after applying the transformation, and δ is a (pseudo) random number sampled from a normal distribution, the normal distribution having a mean value of 0 and a variance [Equation 4] having, provided that [Equation 5] is a coefficient that reduces the standard deviation σ y (qy) of the noise in the output raw image data (150) by an amount corresponding to the pixel value reduction coefficient q, characterized in that q < q' < 1, the method according to claim 5. Claim 7 The variance of the distribution of the (pseudo) random number δ of the transformation S(y,q) representing the noise-consistent scaling operation is [Equation 6] The method according to claims 4 and 6, characterized in that it is represented by

8. The step of processing (130) includes the step of applying a noise-inconsistent operation to the value y of a given pixel of the image sensor used for the acquisition of the image data, and the step of processing (130) is different from the input noise model (120) and uses an output noise model (160) that is consistent with the noise of the output raw image data (150) of the output data (140). The method according to claim 1 or 2, characterized in that

9. At least one of the processing operations applied during the step (130) of processing the input raw image data (110) is ° Correcting the pixel values of the input raw image data (110), ° Applying flat field correction to the input raw image data (110), ° Applying correction for non-uniform light sensitivity to the input raw image data (110) selected from the group of processing operations including, By ensuring that the input noise model (120) remains valid for all pixel values of the output raw image data (150), in such a way that the output noise model (160) is identical to the input noise model (120), each of the processing operations is applied to the input raw image data (110). The method according to claim 1 or 2 in combination with claim 5 or 6, characterized in that

10. Correcting the pixel values of the input raw image data (110) results in corrected pixel values y' C =S(y L +y R -μ y.dark ,q=1 / 2) to substitute the neighboring pixel value y L and y R By using the bad pixel value y C where q is the pixel value reduction factor and μ y.dark is the average pixel value in the absence of light, and / or ° Applying flat field correction to the input raw image data (110) is performed by substituting the corrected pixel value y' = q·y + δ(y,q) by using the pixel-dependent reduction coefficient q(i,j) given by, where N(i,j) represents the reference flat field image and i and j represent pixel coordinates, and / or [Equation 7] ° Applying correction for non-uniform light sensitivity to the input raw image data (110) is performed by substituting the corrected pixel value y' = q·y + δ(y,q) by using the pixel-dependent reduction coefficient q(i,j) given by, where N(i,j) represents the spatially varying relative detection efficiency determined by the calibration of the image sensor used for the acquisition of the image data and i and j represent pixel coordinates. The method according to claim 9, characterized in that ° Applying correction for non-uniform light sensitivity to the input raw image data (110) is performed by substituting the corrected pixel value y' = q·y + δ(y,q) by using the pixel-dependent reduction coefficient q(i,j) given by, where N(i,j) represents the spatially varying relative detection efficiency determined by the calibration of the image sensor used for the acquisition of the image data and i and j represent pixel coordinates. 【Number 8】 The method according to claim 9, characterized in that it is performed by substituting the corrected pixel value y' = q·y + δ(y,q) by using the pixel-dependent reduction coefficient q(i,j) given by, where N(i,j) represents the spatially varying relative detection efficiency determined by the calibration of the image sensor used for the acquisition of the image data and i and j represent pixel coordinates.

11. at least one processing operation applied during the step (130) of processing the input raw image data (110) is ° applying a binning operation and / or an averaging operation to the input raw image data (110), ° performing normalization of the input raw image data (110), ° applying a rounding operation and / or a truncation operation, and / or integer quantization to the input raw image data (110), ° performing image synthesis, in particular by creating and / or adding a synthetic image based on the input raw image data (110) and / or by modulating the input raw image data (110) selected from the group of processing operations including ensuring that the output noise model (160) determined during the step (130) of processing is valid for all pixel values of the output raw image data (150), and each of the processing operations is applied to the input raw image data (110) in a manner different from the input noise model (120), a method according to claim 1 or 2 in combination with claim 8.

12. ° Applying a binning operation to the input raw image data (110) forms a group of N = n·m pixels or a group of N exposures, and for each group, the output pixel value where i represents pixel coordinates 【Number 9】 Substitute it, and increase the black level to μ Y.dark = Nμ y.dark while increasing the read noise 【Number 10】 By increasing the Y (μ Y ) and / or ° Applying a binning operation to the input raw image data (110) forms a group of N = n·m pixels, and for each group, the output pixel value where i represents pixel coordinates 【Number 11】 Substitute it, and maintain the black level μ Y.dark while maintaining it and the read noise 【Number 12】 is increased to determine the corresponding output noise model σ Y (μ Y ) and is executed by determining, and / or ° Applying an averaging operation to the input raw image data (110) forms a group of N = n·m pixels, and an output pixel value for each group 【Number 13】 Substitute it, and maintain the black level μ Y.dark while maintaining and reading out noise 【Number 14】 By reducing it, the corresponding output noise model σ Y (μ Y ) is executed by determining it and / or ° Performing normalization of the input raw image data (110) has different parameters 【Number 15】 For a series i = 1, 2..., M of image input data (100) resulting from M different devices having a Poisson-Gaussian noise model with different parameters, the normalized maximum read noise 【Number 16】 Determining an input noise model (120) having, for each pixel value y having subscript number i of the image input data (100), i an output pixel value 【Number 17】 is performed by substituting, δ i has a mean value of 0 and a variance 【No. 18】 (Pseudo) random numbers sampled from a normal distribution having, and parameters {K = 1, μ ε.dark = 0, σ ε.dark = σ ε} An output noise model (160) is common to all output raw image data (150) of the series i = 1, 2..., M} of the image output data (140), and / or ° Applying a rounding operation and / or a truncation operation, and / or integer quantization to the input raw image data (110) adds a different random number Δ for each pixel according to y' = round(y + Δ) before the rounding / truncation / quantization operation, and at the same time adapts the read noise of the output noise model according to 【Number 19】 is performed by adapting according to, provided that 【Number 20】 is the variance of the distribution of the random number Δ, and / or ° In particular, by creating and / or adding a synthetic image based on the input raw image data (110), and / or by modulating the input raw image data (110), performing image synthesis is done by mimicking an image having a reduced signal-to-noise ratio compared to the input raw image data (110) and having the same output noise model (160), or by mimicking an image having an increased signal-to-noise ratio compared to the input raw image data (110), and the output noise model (160) is realized by a single set of noise models of the input raw image data (110) or by a set of noise models of the normalized input raw image data (110). The method according to claim 11, characterized in that.

13. The step of storing and / or transmitting the output raw image data (150) and the output noise model (160), wherein the output raw image data (150) and the output noise model (160) together form the output data (140) in a manner that links the output raw image data (150) to the output noise model (160). The step is - Storing the output raw image data (150) together with the output noise model (160) in a container file, preferably in a file in the Tagged Image File Format (TIFF), or - Storing the output raw image data (150) in a first file and storing the output noise model (160) in a second, separate sidecar file, the sidecar file being coupled to the first file containing the output raw image data (150), the sidecar file being adapted to store additional (meta) data not supported by the first file. The storing, or - Storing the output noise model (160) in a database adapted to link a specific output raw image data (150) to its associated output noise model (160) The method according to claim 1 or 2, characterized in that it is executed by.

14. The step of storing and / or transmitting the output noise model (160) is performed by storing and / or transmitting a digital representation of the parameters of the output noise model, the parameters being passed to a function or catalog of functions having a fixed definition, and the particular function used in the latter case being selected by additional identification parameters, the method according to claim 13.

15. Computer program means stored in a computer-readable medium adapted to execute the method according to claim 1 or 2.

16. An apparatus equipped with the computer program means according to claim 15, characterized in that the apparatus is selected from the group comprising a microprocessor, a field programmable gate array, an image sensor, a mobile phone, in particular a smartphone equipped with a digital camera, a digital photo device, a digital video camera, a scanning device, a tablet, a personal computer, a server, a microscope, a telescope, a satellite.

Images