A method of reducing dead and bad pixel effects

Abstract

The invention relates to an image processing method applied after the Non-Uniformity Correction process, aimed at reducing the effects of dead and bad pixels. It utilizes pixel-based Non-Uniformity Correction parameters that are obtained during the calibration stage in calibration based Non-Uniformity Correction applications and stored in the imaging device's internal memory for use in the Non-Uniformity Correction process. Using these specified parameters, the method determines how closely each pixel's response aligns with the ideal response, thereby establishing a pixel-based accuracy coefficient. Subsequently, the pixels are clustered based on their accuracy coefficients; this clustering process identifies dead or bad pixels, and correction is performed with the remaining pixels. For correcting dead or bad pixel values, a point distribution function is employed.

Description

[0001] A METHOD OF REDUCING DEAD AND BAD PIXEL EFFECTS

[0002] Relevant Technical Field

[0003] The present invention relates to the technical field of image data processing, and in particular to a method aimed at reducing the disruptive effects of dead and bad pixels of a camera on the image.

[0004] Prior Art

[0005] Non uniformity Correction (NUC) is a significant application used primarily in imaging systems, including infrared and thermal cameras. This process reduces the response variations among the pixel elements in an imaging system and enhances the image quality. Response variations may occur due to reasons such as differences in the sensor manufacturing process, environmental effects, effects of electro optic components and / or aging of sensor components. The aim of the NUC process is to ensure that each pixel in the sensor responds equally to the same light intensity, thus obtaining more homogeneous and less noisy images. There are various NUC methods in the state of the art. These methods are divided into two main categories: calibration based and scene-based correction.

[0006] In calibration-based methods, the camera is calibrated using a homogeneous photon source. During this process, images are captured at different light intensities (or temperature levels), and the response of each pixel under these uniform conditions is recorded. Then, based on this data, correction coefficients are calculated for each pixel. These coefficients include pixelbased gain and offset values and are stored in the internal memory of the imaging device for use in the NUC process.

[0007] Although NUC techniques significantly reduce the distortions caused by pixel-based response differences, some pixels differ at a level that cannot be corrected and therefore they cannot be corrected with the NUC application. These pixels are generally classified as "dead pixels" (not responding at all to any light intensity change) or "bad pixels" (reacting very little or very much to light intensity change). Dead or bad pixels can be found on the sensor in clusters or as a single pixel. Therefore, in the state of the art, there are various applications for changing the dead or bad pixel values, which are applied following the NUC application. The most commonly used of these applications is the method in which the value of the dead or bad pixel is replaced with the average of the values of the neighboring pixels. An example of this application is given in the article titled “non uniformity correction and bad pixel replacement on LWIR and MWIR images”. The article mentions changing the value of pixels that cannot be corrected with the NUC process and are defined as defective pixels; by using the nearest neighbor algorithm. However, this algorithm does not work effectively if there is a defective pixel in the neighboring pixels.

[0008] In addition, in the current applications known in the art, pixels are classified into two main groups as dead or bad pixels and normal pixels, and each of the normal pixels is taken into account with the same weight in the correction process. However, the extent to which normal pixels work normally and to what extent they work abnormally (badly) varies among themselves.

[0009] In this context, there is a need for solutions that increase the uniformity of the output image by proportionally increasing the correction weight of normal pixels based on their accuracy classification.

[0010] Object of the invention

[0011] The object of the present invention is to develop a method for reducing the disruptive effects of dead or bad pixels on the image and improving the image quality of the image to which NUC is applied.

[0012] Another object of the present invention is to develop an image enhancement method having a higher level of accuracy with respect to the applications known in the state of the art.

[0013] Definition of the figures

[0014] Histogram representations in accordance with the exemplary applications of the method according to the present invention are shown in the attached figures and from these figures;

[0015] Figure 1 is an exemplary histogram representation of the distribution of pixel wise gain coefficients of a camera.

[0016] Figure 2 is an exemplary histogram representation of the distribution of anomaly score coefficients obtained from the gain coefficients by applying the method which is the subject of the invention.

[0017] Detailed description of the invention

[0018] With the present invention, an image processing method is developed that provides a solution to the technical problems mentioned above and is applied following the Non Uniformity Correction process; in order to reduce the effects of dead and bad pixels. The method in question utilizes pixel based Non Uniformity Correction parameters that are obtained during the calibration phase in calibration based Non Uniformity Correction applications and stored in the internal memory of the imaging device, to be used in the process. By using the specified parameters, each pixel's response is determined in terms of its proximity to the ideal response, thereby enabling the calculation of a pixel based accuracy coefficient. Then, the pixels are clustered based on their accuracy coefficients. As a result of the clustering process, dead or bad pixels are identified, and it is ensured that these pixels do not affect the correction process. In the correction of dead or bad pixel values, a point spread function (PSF) is used instead of a neighboring pixel algorithm, thus enabling the correction of the pixel to a value closer to its intended value. In the application of the point spread function, the pixels considered are weighted according to the aforementioned accuracy coefficient; this reduces the impact of faulty pixels on the correction process while increasing the influence of pixels with high accuracy coefficients.

[0019] The method according to the present invention comprises the following process steps: obtaining the pixel-based gain coefficient data from the Non Uniformity Correction parameters stored in the internal or external memory of the imaging device; calculating the average of the pixel-based gain coefficients and determining an average gain coefficient; subtracting the average gain coefficient from each pixel's gain coefficient and taking the absolute value of the resulting value to obtain the pixel-based anomaly score coefficient; applying the k-means method with “k” value being equal to 2 (k=2), to the pixel-based anomaly score coefficient values to divide the pixels into two separate clusters (normal pixels and dead or bad pixels); determining a threshold value by taking the average of the highest anomaly score in the normal pixel cluster and the lowest anomaly score in the dead or bad pixel cluster; for pixels with anomaly score coefficients below the threshold value, keeping their respective coefficients as is; for pixels with anomaly score coefficients above the threshold value, setting their anomaly score coefficient to 1 , thereby obtaining pixelbased error coefficients for all pixels; calculating pixel-based accuracy coefficients for all pixels by subtracting each of the obtained error coefficients from 1 ; computing the normalization matrix by using the relevant point spread function matrix, the relevant accuracy coefficient, and the relevant error coefficient for each pixel; obtaining the weighted Non Uniformity Correction image by multiplying each pixel in the image processed with the Non Uniformity Correction operation by its corresponding accuracy coefficient; calculating a corrected pixel value for each pixel by using the weighted Non Uniformity Correction image’s pixels and the relevant normalization matrix. The said method is a computer-based method, and the process steps described above are carried out by a processing unit, preferably an FPGA (Field Programmable Gate Array), specifically arranged to perform these steps. The said processing unit may be an internal processing unit within the relevant imaging device (for example, a camera) or an external processing unit.

[0020] In an exemplary application of the invention, the said method is applied to a camera. The pixel based Non Uniformity Correction parameters, obtained through the calibration process performed during the manufacturing of the camera, are stored in the camera's internal memory. One of the Non Uniformity Correction parameters, the gain coefficients, is represented as [g(x,y)] on a pixel basis, where the horizontal position of the relevant pixel is denoted as [x], and its vertical position is denoted as [y]. Figure 1 shows an example histogram illustrating the distribution of the gain coefficients of a camera. In the graph, the horizontal axis represents the gain coefficient values [g(x,y)], and the vertical axis indicates the total number of pixels [Count(g(x,y))] with the corresponding gain coefficient value. As shown in Figure 1 , under normal conditions, the gain coefficient values are expected to be clustered around 1.

[0021] First, the anomaly score coefficients [g’(x,y)] are derived from the obtained gain coefficients. The relevant formula is as follows: g'(x,y g x,y) - r Cg(x,y))| (1)

[0022] The average gain coefficient [avg(g(x,y))] in Formula 1 is obtained by summing all gain coefficient values and dividing by the total number of pixels. The anomaly score coefficient [g’(x,y)] is derived by subtracting the average gain coefficient [avg(g(x,y))] from each gain coefficient value [g(x,y)]. This anomaly score coefficient [g’(x,y)] provides a parameter indicating how close or far each pixel's behavior is from the average behavior of the pixels contained within the camera. An example histogram showing the distribution of the obtained anomaly score coefficients [g’(x,y)] is shown in Figure 2. The closer the anomaly score coefficient [g’(x,y)] value of a pixel is to zero, the more its response to light is close to the camera’s average response and, therefore, the expected response. As the anomaly score coefficient [g’(x,y)] diverges from zero, the anomaly exhibited by the pixel increases. In this context, the anomaly score coefficients [g’(x,y)] can be grouped into clusters based on pixel behavior using the k-means method. In the method according to the present invention, the value of [k] in the k-means function is set to 2, dividing the pixels into two clusters: normal pixels and “dead or bad” pixels. The anomaly score coefficients of normal pixels are very close to 0, while the coefficients for dead or bad pixels tend to be higher. A threshold value [E] is determined by taking the average of the highest anomaly score coefficient in the normal pixel cluster and the lowest anomaly score coefficient in the dead or bad pixel cluster. Pixels with anomaly score coefficient values [g’(x,y)] above this threshold are identified as pixels that will not be used in the correction process, and their coefficients are set to 1. Thus, pixel-based error coefficients [e(x,y)] are obtained. The equation for this process is provided below.

[0023] The equation for calculating the pixel-based accuracy coefficient [c(x,y)], by using the error coefficient [e(x,y)] is given below.

[0024] Subsequently, the camera's Non Uniformity Correction-processed image data [INUC(x,y)] is obtained, and using the error coefficient [e(x,y)] , the accuracy coefficient [c(x,y)], and the pixelbased point spread function [psf(x,y,k,l)] recorded in the camera, a corrected pixel value is calculated for each pixel. The equation for this calculation is provided below.

[0025] In Equation 4, [psf(x,y,k,l)] represents the point spread function measured or analytically modeled for each pixel contained within the camera. The indices “k” and “I” correspond to the horizontal and vertical indices of the point spread function related to the specific pixel, respectively. In other words, [psf(x,y,k, I)] is a scalar, whereas [psf(x,y)] is a matrix. The operation in Equation 4 can be more simply expressed as follows:

[0026] In Equation 5, loirr(x,y) represents the corrected image data obtained as a result of the process; lNuc(x,y) is the image data after Non Uniformity Correction has been applied, and the ® operator denotes the correlation operation. The components in Equation 5 that do not require recalculation for each new image - i.e., computed online - can be defined during offline calculations as shown below: In Equation 6, [n(x,y)] represents the normalization matrix used to determine the corrected pixel value, which is specific to each pixel. As a result of the correlation operation in the denominator, a scalar value is obtained for the relevant pixel, and dividing this value by the scalar error coefficient associated with that pixel (found in the denominator) yields another scalar. This scalar value for each pixel is multiplied individually by each element of the point spread function matrix corresponding to that pixel. Consequently, normalization matrices are calculated for each pixel of the image. The pixel-based accuracy coefficients [c(x,y)] and the normalization matrix [n(x,y)]; which are either recorded in the imaging device’s memory or computed using these parameters can be used unchanged as long as the optical path and electronic configuration of the imaging device remain the same. The image obtained by applying Non Uniformity Correction to the captured image is represented as [INUC], and its pixel-based accuracy coefficients [c(x,y)] are used to weight this image, the weighted version is denoted as [INUC’]. Under these conditions, the following two equations are derived:

Claims

CLAIMS1. A method of reducing dead and bad pixel effects, arranged to be applied after the Non Uniformity Correction process in an imaging device that includes pixel-based point distribution function parameters and calibration based, pixel based non uniformity correction parameters; the method comprises the following steps; obtaining the pixel-based gain coefficient data from the Non Uniformity Correction parameters stored in the imaging device’s internal or external memory calculating the average of the gain coefficients and determining an average gain coefficient subtracting the average gain coefficient from each pixel’s gain coefficient and taking the absolute value of the resulting value to obtain a pixel based anomaly score coefficient applying the k-means method with “k” value being equal to 2 (k=2) to the pixel-based anomaly score coefficients to group the pixels into two separate clusters: normal pixels and dead or bad pixels determining a threshold value taking the average of the highest anomaly score within the normal pixel cluster and the lowest anomaly score within the dead or bad pixel cluster for pixels with anomaly score coefficients below the threshold value, maintaining their current coefficient values; for pixels with anomaly score coefficients above the threshold value, setting their anomaly score coefficient to 1 ; thereby obtaining pixelbased error coefficients for all pixels calculating pixel-based accuracy coefficients for all pixels by subtracting each of the obtained error coefficients from 1 calculating a normalization matrix for each pixel by using the corresponding point spread function matrix, corresponding accuracy coefficient, and corresponding error coefficient multiplying each pixel in the Non Uniformity Correction processed image by its corresponding accuracy coefficient to obtain a weighted Non Uniformity Correction image calculating a corrected pixel value for each pixel by using the weighted Non Uniformity Correction image’s pixel values and the associated normalization matrix.

Images