METHOD FOR SOVEREIGN REGISTRY-BASED UNITY CORRECTION USING SPATIAL-TEMPORARY UPDATE MASK

Description

Technical Field

[0001] The present disclosure relates to a scene-based non-uniformity correction method to achieve better fixed-pattern noise reduction and eliminate ghosting artifacts based on robust parameter updates via confident inter-frame registration and spatio-temporally consistent correction coefficients.Related Art

[0002] Infrared imagery has been used in various fields such as military, medical, surveillance and numerous industrial branches. One of the main problems with commonly used focal plane arrays (FPA) is the fixed pattern noise (FPN) formed by the non-uniform response of detector elements [1]. As a solution to this problem, there are two main approaches, namely, calibration based and scene based non-uniformity correction (NUC) methods. Calibration based methods [1-4] traditionally use reference uniform surfaces to correct non-uniform detector responses. These methods are preferred due to their simplicity and computational efficiencies. However, they are not very convenient during operational use, since they require additional equipment and interrupt the video flow. On the other hand, the family of scene based methods [5-13] deal with non-uniformity problem using only scene content, which alleviates the operational problems of calibration based methods. Because of these benefits, there are considerable amount of studies utilizing scene based approaches.

[0003] One of the early works uses neural network (NN) structure to update calibration coefficients [5] without any consideration of scene content. Naturally, this approach will suffer from ghosting artifacts due to the assumption that all pixel elements will have a similar temporal average which does not always hold. There are other constant-statistics assumption based methods that further improve on this work [6]. However, they have similar drawbacks that result in ghosting artifacts. Some of the recent studies [13-15] employ inter-frame registration to match scene content which notably decreases the ghosting artifacts. Despite the success of

[15] in reducing the artifacts, moving objects in the scene, motion blur and registration errors still remain as a problem. In

[13] , Zeng et al. made further improvement by introducing a spatial update mask from residual registration errors. This mask is used to selectively update the correction coefficients in order to reduce ghosting artifacts. Although this work significantly limits the artifacts, there are several issues that might still stand as problems. One problem is that there might be a global registration error especially in low-detail frames. The method applies certain restrictions only within the error image between frames and does not examine whether the registration is performed correctly. In mis-registration cases, severe ghosting artifacts may occur. Additionally, there is no mechanism to prevent temporally inconsistent updates possibly caused by registration errors.

[0004] MOU, Xin-gang, et al. Nonuniformity correction algorithm based on Gaussian mixture model. In: International Symposium on Photoelectronic Detection and Imaging 2011: Advances in Infrared Imaging and Applications. SPIE, vol. 8193, no. 1, 2011, pp. 1088-1095. relates to GMM-based nonuniformity correction for infrared image sequences.

[0005] LIU, Zhe, et al. Nonuniformity correction based on adaptive sparse representation using joint local and global constraints based learning rate. IEEE Access, 2018, vol. 6, pp. 10822-10839. relates to infrared focal plane array nonuniformity correction using motion estimation and root-mean-square error computation. The method is based on a linear observation model which is itreratively learned.

[0006] SOBARZO, Sergio K.; TORRES, Sergio N. Real-time Kalman filtering for nonuniformity correction on infrared image sequences: Performance and analysis. In: Progress in Pattern Recognition, Image Analysis and Applications: Proceedings 10th Iberoamerican Congress on Pattern Recognition, CIARP 2005, Havana, Cuba, November 15-18, 2005, Springer Berlin Heidelberg, 2005, pp. 752-761. relates to scene-based method for nonuniformity correction of infrared image sequences based on Kalman filtering and linear observation model.

[0007] However, a check for sufficient detail for registration, a quality metric for eliminating erroneous updates, spatial masking and GMM-based temporal consistency restriction on the correction coefficients to mask out unstable updates cannot be reached in the above documents.Summary of the Invention

[0008] The proposed invention is defined in independent method claim 1 and offers a method for confident registration-based non-uniformity correction using spatio-temporal update mask. The confident registrations are achieved by introducing a scene adaptive registration quality metric to discard erroneous updates. A Gaussian mixture model (GMM)-based spatio-temporal update mask is utilized to selectively refine the estimations of correction coefficients. The results of experiments on an extensive dataset consisting of both synthetically corrupted data and real infrared videos show that the proposed algorithm achieves superior performance in PSNR (Peak Signal-to-Noise Ratio) and roughness metrics with notably lower ghosting artifacts when compared to other state-of-the-art methods.

[0009] Our contribution is two-fold. The first one is that we eliminate erroneous parameter updates resulting from unreliable registrations. The second one is that we employ a temporal consistency restriction on the correction coefficients to further eliminate unstable parameter updates.

[0010] In our method, we aim to address the remaining ghosting artifact problems by introducing a scene-adaptive registration quality metric to achieve a confident registration before updating any non-uniformity parameter estimation. We first assess the input image frame whether it is eligible for registration in terms of sufficient scene detail by applying a Sobel kernel as edge filter to extract an average magnitude of edge information in the image and comparing the information with a threshold for registrations between the frames to prevent false registrations originated from low-detail scenes. This prevents false registrations originated from low-detail scenes. Then we require a certain amount of translation to ensure a sufficient statistical variation. After those restrictions, we then assure that there is no considerably large global registration error utilizing the introduced scene-adaptive registration quality metric. These restrictions could cause to discard too many useful frames, however, by introducing an adaptive frame delay mechanism, we provide a way to efficiently wait for a good candidate frame with sufficient translation and acceptable registration errors. Finally, for any remaining errors that might produce artifacts, we utilize a GMM-based temporal consistency restriction that masks out statistically unstable updates in non-uniformity correction parameters.

[0011] In conclusion, the present invention introduces a new scene-based NUC method that yields better performance by making more accurate FPN estimates and by eliminating ghosting artifacts utilizing spatio-temporally consistent FPN estimation update mechanisms. In essence, the proposed algorithm tries to eliminate erroneous FPN estimates by the assessment of registration errors, amount of shift between images and temporal consistency of FPN characteristics. The results of the experimentations show that a better performance in FPN correction without ghosting artifacts is achieved in an extensive dataset.Brief Description of the Drawings

[0012] Figure 1 shows a block diagram of the present invention. Figure 2 shows PNSR performance of the evaluated methods. The titles of the plots represent the image sequence names as given in the original ETH-ASL Thermal Image Dataset

[17] . Figure 3 shows ghosting artifacts comparison among the evaluated methods executed on a dataset. Figure 4 shows ghosting artifacts visible in the error images that belongs to the frames shown in Figure 3. Figure 5 shows comparison of ghosting artifacts among the evaluated methods executed on the ETH-ASL dataset

[17] . Detailed Description

[0013] The present invention relates to a confident image registration based non-uniformity correction algorithm that utilizes a spatio-temporal update mask. A block diagram demonstrating the flow of the algorithm is given in Figure 1.1. Observation Model

[0014] In order to model the response of infrared detector pixels, the common linear model approach is utilized [1]. According to this model, the observed output signal is represented by a linear function of true response as given in (1): y n i j = x n i j ⋅ a n i j + b n i j where x n is the true response for the n th< frame, y n is the associated detector output signal, (i, j) is the pixel coordinates and a and b are the gain and offset coefficients.

[0015] Based on this model, true response can also be written as a linear function of the detector output as shown in (2): x n i j = y n i j ⋅ g n i j + o n i j where backward gain g n (i,j) is equal to 1 / a n (i,j) and backward offset o n (i,j) is equal to - b n (i,j) / a n (i,j).

[0016] According to this model, the observed signal is corrected by using the backward gain and offset coefficients. These coefficients can be estimated using different techniques. In the proposed method, inter-frame registration is performed with adaptive frame delay and least mean squares (LMS) minimization is used for estimating the gain and offset coefficients.2. Registration

[0017] In frame registration based non-uniformity correction approaches, the main assumption is that the different pixels must produce the same response to the same scene data. Since the fixed pattern noise is a slowly varying signal, this assumption can be used for estimating the correction coefficients iteratively using LMS minimization [8]. For this purpose, the first step is to find the correct registration between the frames.

[0018] Linear model parameters are initialized by setting gain values to 1 and offset values to 0. After obtaining the corrected signals x n and x n-m , the horizontal shift δ xn and the vertical shift δ yn between these signals are calculated. In the registration block, the image detail magnitude Φ n and the registration quality value ξ n are calculated for ignoring the problematic registrations caused by various reasons such as lack of scene details, motion blur and camera rotation. In other words, the correction parameters are updated only when a confident registration is achieved. This selective update approach is an important process for eliminating ghosting artifacts. Details of each sub-block are given in the following subsections.2.1. Scene Detail Calculation

[0019] In order to achieve a reliable registration, sufficient scene detail must be present. Otherwise, matching can be erroneous due to low signal-to-noise ratio. Therefore, the registration and the other parameter update steps are skipped to ensure a robust registration quality, if the newly arrived frame does not contain sufficient scene detail (Φ n < T Φ ).

[0020] For a given frame, scene detail magnitude ϕ n is calculated as given in (3): ϕ n = 1 WH x n ∗ h 2 2 + x n ∗ h T 2 2 where W is frame width, H is frame height, h is horizontal edge filter kernel and * is discrete convolution operation. In our implementation, 9x9 extension of horizontal Sobel kernel is used as the edge filter.2.2. Inter-frame Shift Calculation

[0021] Instead of using 1-D projections of intensity values for shift calculation as in

[13] , we employ 1-D projections of edge image which provides more robust registration performance in scenes with rich edge content. Since the edge image is already calculated in (3) for detail calculation, we efficiently obtain the 1-D projections in vertical and horizontal directions as shown in (4): P n x j = ∑ i = 1 W E n i j , P n y i = ∑ j = 1 H E n i j where E n represents the edge image calculated as shown in (5): E n i j = ∑ k = − r r ∑ l = − r r x n i − k , j − l h k l 2 + ∑ K = − r r ∑ l = − r r x n i − k , j − l h T k l 2 where h is the edge filter of size r × r used in Scene Detail Calculation.

[0022] In order to calculate the shift between the current and delayed images, normalized cross correlation of the projection vectors are calculated as shown in (6): γ x i ≜ ∑ x P n − m y i + x − P n − m y ¯ P n y x − P n y ¯ σ P n y σ P n − m y γ y j ≜ ∑ y P n − m x j + y − P n − m x ¯ P n x y − P n x ¯ σ P n x σ P n − m x where P n x ¯ and P n y ¯ are the means and σ P n x and σ P n y are the standard deviations of the projection vectors P n x j and P n y i , respectively. Similar definitions apply for the projection vectors for the delayed frame as well.

[0023] Shift in x and y directions are than retrieved as shown in (7): δ x = arg max i γ x i δ y = arg max j γ y j 2.3. Registration Quality Assessment

[0024] One of the major sources of the erroneous model updates is the registration errors that stem from the violation of pure translation assumption. To overcome such problems, we utilize a registration quality measure that discards the frame pairs which have relatively large global registration error compared to scene detail magnitude. In other words, we introduce an adaptive metric that shows relative registration quality with respect to scene detail magnitude. The registration quality measure is defined as shown in (8): ψ n = ϕ n 1 WH ∑ i , j E n i j − E n − 1 i + δ x , j + δ y

[0025] This metric favors the registrations where the error magnitude is a small fraction of scene detail magnitude. Registrations with lower quality measure than a certain threshold T ψ are discarded at the parameter update step.3. Non-uniformity Correction Model Update

[0026] In general, the observation model discussed above is updated as the minimization of the LMS problem stated in [5] as given in (9): a n + 1 p q = a n p q − η 1 ⋅ e n p q ⋅ y n p q ⋅ ξ st n p q b n + 1 p q = b n p q − η 2 ⋅ e n p q ⋅ ξ st n p q where, subscripts n and n + 1 represents the current and next frame indices, (p, q) represents the index range within the registered overlap region between the delayed and current image, η 1 and η 2 are the learn rates for the gain and offset updates, e n is the error image between the corrected current frame and the delayed frame, y n is the current input frame and ξ stn is the spatio-temporal update mask which determines pixel locations at which the observation model will be updated.

[0027] It is noted that there is a 2-pass update for the model which exploits the fact that the registered overlapping regions of the delayed and current frames coincide with different pixel locations of the FPA. Thus, it is possible to update the gain and offset coefficients for two pixel locations instead of one by repeating the update step in both directions.

[0028] The calculation of the error map e n and spatio-temporal update mask ξ stn will be detailed in further sections.3.1. Error Map Calculation

[0029] The error map yields the error between the intensities read from different FPA pixel locations for the same scene content assuming the registration error is sufficiently small. This information provides the difference between the FPA element responses at different locations. The error calculation is performed on the corrected signals as given in (10): e n i j = x n − m i + δ x , j + δ y − x n i j 3.2. Spatial Mask for Model Update

[0030] Although the generated error maps in 3.1 to find non-uniform responses, unfortunately, not all error values are valid due to residual registration errors, moving objects in the scene and so forth. As an a priori information, non-uniformity characteristics for each pixel in an FPA does not vary radically; unless they are defective

[16] . Based on this, erroneous updates can be further avoided in addition to the elimination of global registration errors explained in Section 2.3. This is achieved by masking out the outlier error values on the previously calculated error map as given in (11): ξ s n p q = 1 if e n p q − μ e n < cσ e n 0 otherwise where, c is the constant for σ distance which determines the outlier threshold. µ en and σ en are the mean and standard deviation of the error map, respectively.3.3. GMM-based Temporal Mask for Model Update

[0031] Up to this point, we have generated an update mask that avoids global registration errors and spatially inconsistent non-uniformity errors. As a final precaution for erroneous model parameter updates, we exploit the fact that FPN has an approximately constant behavior over time [7]. Thus, any update that would yield radically different model parameters is considered invalid and masked out from calculations. Analogous to the background subtraction domain, we could model residual FPN in the registration error map as fixed background and model other temporally inconsistent errors as the foreground. As a commonly applied solution to background subtraction problem, GMM could be successfully used to detect and eliminate the error components that do not belong to FPN. The GMM update steps are given in the following equations: Δ i j = e n i j − μ GMM n i j ξ st n i j = ξ s n i j if Δ i j 2 < λ σ GMM n 2 i j 0 otherwise μ GMM n + 1 i j = μ GMM n i j + α 1 Δ i j ξ st n i j var GMM n + 1 = σ GMM n + 1 2 i j = σ GMM n 2 i j + α 2 Δ i j 2 − σ GMM n 2 i j ξ st n i j where ξ stn represents the spatio-temporal update mask computed using GMM model. µ GMM and σ 2< GMM are the mean image and variance image kept by the GMM. They are initialized by using the spatially masked regions (ξ sn ) of the first input image frame. Also, λ is the temporal variance threshold coefficient, α 1 is the update learn rate for the GMM mean image and α 2 is the learn rate for the GMM variance image.4. Adaptive Frame Delay

[0032] In order to obtain a robust registration between the frames, a minimum magnitude of inter-frame shift and scene detail is required. If these conditions do not hold for the incoming frames, we drop them and wait for a better candidate for the parameter update. In other words, delay between two registration frames is not necessarily one, but can take values between one and m max . When the maximum frame delay m max is reached before finding a suitable image frame, the process restarts with new initial frame.5. Process Steps of the Invention

[0033] The observation model for the fixed pattern noise on infrared (IR) image is considered to be consisting of a multiplicative gain terms and additive offset terms. If these terms are found then the fixed pattern noise is estimated and corrected from the noisy image to obtain an uncorrupted clean image. In the following steps, our aim is to find these correction terms.

[0034] After the IR image is captured by a sensing device, the algorithms first starts a registration process between image frames. The first step of this process is to check whether the frames to be registered contain sufficient detail information. This step is necessary to avoid erroneous registrations caused by insufficient detail in the images. It is checked by applying a Sobel filter to extract the average magnitude of edge information in the images. Then this information is compared with a suitable threshold. If the scene detail is lower than the threshold, the algorithms do not proceed to registration steps. This way, registration performance is improved by avoiding the potentially erroneous registrations.

[0035] The images with sufficient scene detail are then used to find a shift between the frames. For the registration, the 1-D horizontal was used and vertical projections of the edge maps generated from the original frames using edge extraction filter. Then the 1-D projection vectors are matched using cross-correlation. This way, horizontal and vertical translations between the frames are calculated.

[0036] Even after the scene detail assessment, the registration performance may still not be as good as desired. Hence, a registration quality metric is defined to quantify the registration performance. This metric is found by calculating the ratio of average registration error and the average scene detail magnitude. Average registration error is simply the frame difference between the edge maps of the frames and scene detail magnitude is readily calculated. This ratio is expected to be small for good registration, hence, a threshold is used for this metric and discontinue registration process if this metric is smaller than the designated threshold. This step helps to reduce errors caused by poor registration performance.

[0037] After these steps, two sufficiently well registered image frames are obtained. These frames provide same image region measured from different sensor pixels within the registered overlap region. Using the assumption that same regions should produce same responses in the imaging pixels, non-uniformity errors can be found. Using these errors, gain and offset correction terms can be updated. Due to the fact that fixed pattern noise does not change at a fast rate, the updates with a slow learning rate is made to avoid premature convergence. Also, not all the pixels are updated at once. There are spatial and temporal consistency masks to determine which pixel locations are to be updated. The use of both the spatial and temporal constraints provides robustness to the non-uniformity correction terms updates. This is a novel approach to reduce the ghosting artifacts produced by erroneous estimations of the correction terms.

[0038] In order to update non-uniformity correction terms, the error map between sensor element responses in the overlap regions of two shifted frames is found. The error map is simply the difference of the overlap regions of the two frames.

[0039] In the fact that typical behavior of fixed pattern noise does not change radically between pixels in an IR image sensor. Hence, it can be deduced that error values that are too different from its neighboring pixels are probably caused by erroneous calculations. These errors are mostly caused by scene changes that violate the translation assumption such as motion blur, non-rigid scene motion and so forth. Therefore, the pixels that have error map values deviating from its local neighborhood with a certain amount is marked. The marked pixels are considered spatially inconsistent and hence the correction terms for the corresponding pixels are not updated.

[0040] Another characteristic of the fixed pattern noise is that it has an approximately constant behavior over limited time durations. Using this information, the temporal variation of the error map values is constrained. If temporal variation is higher that a factor of variance threshold, it can be considered that error map values are not valid and correction terms for such pixel locations are not updated. This is achieved by the use of Gaussian Mixture Models. Analogous to background subtraction case, consistent error map values are considered as background and temporally changing error map values are considered as moving foreground. Then, the deviations of error map values from the temporal mean value estimations are found. These deviations are compared against the thresholds which are a factor of temporal variance estimations. The pixels with higher temporal error map deviations are masked out and the correction terms at these locations are not updated. The temporal mean and variance estimations are also updated at the non-masked pixel locations for iteratively improving the GMM mean and variance estimations similar to the update mechanism of non-uniformity correction terms.

[0041] When any of the conditions detailed in abovementioned steps, then the current couple of frames are not valid to find correction term updates. In such cases, one could drop both frames and wait for another couple of consecutive frames which would satisfy the constraints. However, this may not always be necessary. If the first frame has sufficient scene detail but registration performance is not satisfactory, then we keep the first frame and wait only for another second frame until a maximum number of trials is reached. This way, one not only try to register the consecutive frames, but also the frames with more than one frame delay. This adaptive frame delay strategy for registration ensures not to unnecessarily drop frames and provides faster convergence for the estimation of non-uniformity correction terms.6. Experiments

[0042] The proposed method is compared against the state-of-the-art scene-based NUC methods: GE-NUC

[13] , IRLMS

[15] and GLMS [8]. The evaluation is carried out with an extensive dataset containing synthetically corrupted clean IR images from a public dataset (ETH-ASL Thermal Image Dataset

[17] ) and uncorrected IR images with FPN captured with various IR detectors. The performance comparison is conducted using the objective metrics described in the following section.6.1. Performance Metrics

[0043] The experiments are performed on both images with synthetically added noise and real noisy thermal images. For synthetically corrupted images, Peak Signal-to-Noise Ratio (PSNR) metric is used which measures the error between the expected (clean) reference image and the image corrected by the scene based NUC methods. PSNR calculation is given in (16): PSNR = 10 log 10 peakval 2 / MSE where, MSE is the mean-squared-error between the output and target images and peakval is the maximum value that the input signal can take.

[0044] For the real fixed pattern noise case, however, PSNR metric cannot be calculated, since the clean reference images are not available. In the literature, roughness metric which measures the high frequency component of the image is commonly used

[13] . Roughness metric ρ is calculated as given in (17): ρ = X ^ ∗ h 2 + X ^ ∗ h ′ 2 X ^ 2 where, X̂ is the corrected image frame, h = [-0.5, 0.5] is the horizontal high-pass filter kernel and h I< is the vertical version of the kernel.6.2. Dataset

[0045] In a scene-based NUC study, the dataset variety is vital in terms of both the scene content and different detector responses. In the experiments, our objective is to prevent over-fitting to some limited data and observe the behavior of the methods in possible failure scenarios.

[0046] Thus, we would like to observe the effects of scene contents, camera motion, detector characteristics and noise characteristics.

[0047] We used two different datasets in our experiments. The first dataset is ETH-ASL Thermal Image Dataset

[17] which is publicly available

[18] . This dataset has a rich scene content involving people, animals, buildings, cars, terrain, sky, etc. The images are recorded using a FLIR Tau 320 camera with a resolution of 324x256 and bit-depth of 16-bits. The clean images in this dataset is corrupted by a synthetic Gaussian noise of µ = 0 and σ = 10. In order for the synthetic FPN to be more realistic, the amount of noise is gradually increased from 60% to 100% of the full scale. There are 7 sequences having a total of 3409 frames in the selected dataset.

[0048] The second dataset is a collection of images recorded using ASELSAN thermal imagers with a resolution of 640x512 and bit-depth of 16-bits. There are both hand-held and pan-tilt camera images in the dataset containing a variety of scenes such as terrain, sky, buildings, trees, people and so forth. In total, there are 5 sequences having 3785 frames. The image sequences have real uncorrected FPN so that the methods are evaluated in physical world conditions.6.3. Results

[0049] The experimentation results are assessed both objectively by comparing the PSNR and roughness measurements and subjectively by observing the correction performances and ghosting artifacts.

[0050] In Figure 2, PSNR plots of each method are given for all frames of each data sequence in ETH-ASL

[17] dataset. The main reason for a detailed plot is that the performance loss caused by the ghosting artifacts or false FPN estimations in the middle sections of the image sequences would not be observable by inspecting only the average PSNR values for the whole sequence. Another benefit of these plots is that the convergence of the methods are easily visible. It is noted that the proposed method produces the best PSNR results in all sequences for all frames after its late convergence. The strategy of the proposed method is clearly imprinted on the PSNR characteristics. Our method assesses all the conditions such as registration quality, frame shift amount, scene detail magnitude and temporal consistency. If all the requirements are matched, only then a noise estimation update is performed.

[0051] This strategy provides more accurate estimations at the expense of fewer estimation updates. The stability of the estimations of our algorithm is also reflected on the PSNR curve showing no oscillation unlike the compared methods. For a brief summary, average PSNR values for each sequence and overall average for the whole dataset are given in Table 1. Table 1. PSNR Performance Comparison (dB)GLMSIRLMSGE-NUCCORESUMSempach6-16.86-13.32-13.14-12.33 Sempach7-16.93-12.82-12.91-11.17 Sempach8-16.87-14.48-11.81-10.51 Sempach9-16.84-12.32-12.22-9.72 Sempach10-16.64-12.47-12.41-10.38 Sempach11-17.02-16.59-12.87-10.67 Sempach12-18.41-15.77-11.96-11.84 Average -17.22-13.99-12.37-11.08

[0052] Due to the fact that the roughness metric is highly scene-dependent, the values are usually close and characteristics of the methods are somewhat similar. The proposed method again produces the best (lowest) roughness values in overall performance for each data sequence. This result is given in the average roughness values given in Table 2. Table 2. Roughness Performance Comparison (p)GLMS (x10 -3< )IRLMS (x10 -3< )GE-NUC (x10 -3< )CORESUM (x10 -3< )Seq014.8094.8014.7984.651 Seq024.2924.1153.9843.883 Seq033.1712.8872.9272.883 Seq0414.31813.15912.97711.994 Seq051.4531.2761.2051.130

[0053] In the scene-based NUC domain, ghosting artifacts are the arch enemies of the algorithms. One of the main contributions of our method is the elimination of such artifacts, thus, it is useful to view some exemplary cases for the evaluated methods. Figure 3 depicts the ghosting artifacts produced by the compared methods. Although quite visible in the output images of the algorithms, it is easier to note the ghosting artifacts in the error correction images (difference of output image and original image) in Figure 4. For other images, the artifacts are burnt into the corrections while no such artifacts are observed for our method.

[0054] Similar ghosting artifacts could be seen in the ETH-ASL public dataset

[17] results given in Figure 5. Again, it is noted that our method does not yield any noticeable ghosting artifact whereas others do at different scales.References

[0055] [1] David L Perry and Eustace L Dereniak, "Linear theory of nonuniformity correction in infrared staring sensors," Optical Engineering, vol. 32, no. 8, pp. 1854-1860, 1993. [2] Esteban Vera, Pablo Meza, and Sergio Torres, "Total variation approach for adaptive nonuniformity correction in focal-plane arrays," Optics letters, vol. 36, no. 2, pp. 172-174, 2011. [3] Abraham Friedenberg and Isaac Goldblatt, "Nonuniformity two-point linear correction errors in infrared focal plane arrays," Optical Engineering, vol. 37, no. 4, pp. 1251-1254, 1998. [4] Sungho Kim, "Two-point correction and minimum filter-based nonuniformity correction for scan-based aerial infrared cameras," Optical Engineering, vol. 51, no. 10, pp. 106401-106401, 2012. [5] Dean A Scribner, Kenneth A Sarkady, Melvin R Kruer, John T Caulfield, JD Hunt, and Charles Herman, "Adap-tive nonuniformity correction for ir focal plane arrays using neural networks," in Proc. SPIE, 1991, vol. 1541, pp. 100-109. [6] Lai Rui, Yang Yin-Tang, Zhou Duan, and Li Yue-Jin, "Improved neural network based scene-adaptive nonuniformity correction method for infrared focal plane arrays," Applied optics, vol. 47, no. 24, pp. 4331-4335, 2008. [7] John G Harris and Yu-Ming Chiang, "Minimizing the ghosting artifact in scene-based nonuniformity correction," in Proc. SPIE, 1998, vol. 3377, pp. 106-113. [8] Russell C Hardie, Frank Baxley, Brandon Brys, and Patrick Hytla, "Scene-based nonuniformity correction with reduced ghosting using a gated Ims algorithm," Optics express, vol. 17, no. 17, pp. 14918-14933, 2009. [9] Bradley M Ratliff, Majeed M Hayat, and J Scott Tyo, "Generalized algebraic scene-based nonuniformity correction algorithm," JOSA A, vol. 22, no. 2, pp. 239-249, 2005.

[10] Sergio N Torres and Majeed M Hayat, "Kalman filtering for adaptive nonuniformity correction in infrared focal plane arrays," JOSA A, vol. 20, no. 3, pp. 470-480, 2003.

[11] Chao Zuo, Qian Chen, Guohua Gu, and Weixian Qian, "New temporal high-pass filter nonuniformity correction based on bilateral filter," Optical Review, vol. 18, no. 2, pp. 197-202, 2011.

[12] Alessandro Rossi, Marco Diani, and Giovanni Corsini, "Bilateral filter-based adaptive nonuniformity correction for infrared focal-plane array systems," Optical Engineering, vol. 49, no. 5, pp. 057003-057003, 2010.

[13] Junjie Zeng, Xiubao Sui, and Hang Gao, "Adaptive image-registration-based nonuniformity correction algorithm with ghost artifacts eliminating for infrared focal plane arrays," IEEE Photonics Journal, vol. 7, no. 5, pp. 1-16, 2015.

[14] Russell C Hardie, Majeed M Hayat, Earnest Armstrong, and Brian Yasuda, "Scene-based nonuniformity correction with video sequences and registration," Applied Optics, vol. 39, no. 8, pp. 1241-1250, 2000.

[15] Chao Zuo, Qian Chen, Guohua Gu, and Xiubao Sui, "Scene-based nonuniformity correction algorithm based on interframe registration," JOSA A, vol. 28, no. 6, pp. 1164-1176, 2011.

[16] A. E. Mudau, C. J. Willers, D. Griffith, and F. P. J. le Roux, "Non-uniformity correction and bad pixel replacement on Iwir and mwir images," in 2011 Saudi International Electronics, Communications and Photonics Conference (SIECPC), April 2011, pp. 1-5.

[17] J. Portmann, S. Lynen, M. Chli, and R. Siegwart, "People detection and tracking from aerial thermal views," in 2014 IEEE International Conference on Robotics and Automation (ICRA), May 2014, pp. 1794-1800.

[18] http: / / projects.asl.ethz.ch / datasets / doku.php?id=ir:iricra2014.

Claims

1. A method for scene-based non-uniformity correction to achieve fixed-pattern noise reduction and eliminate ghosting artifacts in infrared imagery, comprising the steps of: • checking input image frames whether the frames to be registered contain sufficient scene detail, which corresponds to a magnitude of the scene detail being more than a predefined threshold TΦ, by applying a Sobel kernel as edge filter to extract an average magnitude of edge information in the image frames and comparing the information with a threshold for registrations between the frames to prevent false registrations originated from low-detail scenes, wherein the magnitude of the scene detail, Φn, is calculated as ϕ n = 1 WH x n ∗ h 2 2 + x n ∗ h T 2 2 where W is frame width, H is frame height, h is horizontal edge filter kernel, xn is the nth image frame and * is discrete convolution operation, • calculating the horizontal and vertical translations between the frames to find a shift, • calculating the ratio of average registration error and the average scene detail magnitude to eliminate erroneous parameter updates resulting from unreliable registrations and to find a scene-adaptive registration quality metric defined as: ψ n = ϕ n 1 WH ∑ i , j E n i j − E n − 1 i + δ x , j + δ y wherein δx and δy are shifts in x and y directions, (i, j) is the pixel coordinates, En represents an edge image, En-1 represents adaptively selected past edge image, W is frame width, H is frame height, Φn is scene detail magnitude, registrations with lower quality measure than a certain threshold are discarded at the parameter update step, • applying a Gaussian Mixture Model, GMM-based temporal consistency restriction on the correctior coefficients to mask out unstable updates in non-uniformity correction parameters, wherein a GMM-based update is applied by using equations: Δ i j = e n i j − μ GMM n i j ξ st n i j = ξ s n i j if Δ i j 2 < λσ GMM n 2 i j 0 otherwise μ GMM n + 1 i j = μ GMM n i j + α 1 Δ i j ξ st n i j var GMM n + 1 = σ GMM n + 1 2 i j = σ GMM n 2 i j + α 2 Δ i j 2 − σ GMM n 2 i j ξ st n i j where (i, j) is the pixel coordinates, subscripts n and (n+1) represents the current and the (n+1)th indices, en is the error image between the corrected current frame and the adaptively selected past frame, ξstn represents the spatio-temporal update mask computed using GMM model, µGMM and σ2GMM are the mean image and variance image kept by the GMM, ξsn states spatially masked regions of the first input image frame, λ is the temporal variance threshold coefficient, α1 is the update learn rate for the GMM mean image and α2 is the learn rate for the GMM variance image.

2. The method according to claim 1, wherein the observation model for the fixed pattern noise on infrared image is: y n i j = x n i j ⋅ a n i j + b n i j that consists of a multiplicative gain terms and additive offset terms to be found for estimating and correcting fixed pattern noise to obtain an uncorrupted clean image, where xn is the true response for the nth frame, yn is the associated detector output signal, (i, j) is the pixel coordinates and a and b are the gain and offset coefficients.

3. The method according to claim 2, comprising the steps of performing - registration between the frames with adaptive frame delay to find a good candidate frame with sufficient translation and acceptable registration errors and using LMS minimization for estimating the gain and offset coefficients where LMS is least mean squares.

4. The method according to claim 2 or 3, comprising the step of updating the observation model as the minimization of the LMS problem: a n + 1 p q = a n p q − η 1 ⋅ e n p q ⋅ y n p q ⋅ ξ st n p q b n + 1 p q = b n p q − η 2 ⋅ e n p q ⋅ ξ st n p q where, subscripts n and n + 1 represents the current and next frame indices, (p, q) represents the index range within the registered overlap region between the delayed and current image, η1 and η2 are the learn rates for the gain and offset updates, en is the error image between the corrected current frame and the delayed frame, yn is the current input frame and ξstn is a spatio-temporal update mask which determines pixel locations at which the observation model will be updated.

5. The method according to claim 1, wherein the horizontal and vertical translations between the frames are calculated by using the 1-D horizontal and vertical projections of the edge maps generated from the original frames using edge extraction filter and matching the 1-D projection vectors using cross-correlation.

6. The method according to claim 5, wherein the 1-D projections in vertical and horizontal directions are calculated with the equations: P n x j = ∑ i = 1 W E n i j , P n y i = ∑ j = 1 H E n i j where, Pnx represents the horizontal projection, Pny represents the vertical projection, W is frame width, H is frame height, (i, j) is the pixel coordinates and En represents the edge image calculated as: E n i j = ∑ k = − r r ∑ l = − r r x n i − k , j − l h k l 2 + ∑ k = − r r ∑ l = − r r x n i − k , j − l h T k l 2 where xn is the nth image frame, h is the edge filter of size r x r used in scene detail calculation.

7. The method according to claim 6, wherein the shift between the current and adaptively selected past image frames in x and y directions are calculated with equations: δ x = arg max i γ x i δ y = arg max j γ y j where normalized cross correlation of the projection vectors are calculated as: γ x i ≜ ∑ x P n − m y i + x − P n − m y ¯ P n y x − P n y ¯ σ P n y σ P n − m y γ y i ≜ ∑ y P n − m x j + y − P n − m x ¯ P n x y − P n x ¯ σ P n x σ P n − m x where P n x ¯ and P n y ¯ are the means, σ P n x and σ P n y are the standard deviations of the projection vectors P n x j and P n y i , respectively.

8. The method according to any of the preceding claims, wherein an error calculation is performed on corrected signals as: e n i j = x n − m i + δ x , j + δ y − x n i j where, en is an error image between a corrected current frame and an adaptively selected past frame, δx and δy are shifts in x and y directions, (i, j) is the pixel coordinates, subscripts n and (n-m) represents the current and (n-m)th indices.

9. The method according to any of the preceding claims, comprising a step of masking out the outlier error values on a calculated error map to avoid erroneous updates in addition to the elimination of global registration errors by using the equation: ξ s n p q = 1 if e n p q − μ e n < cσ e n 0 otherwise where, (p, q) represents the index range within the registered overlap region between the delayed and current image, en is the error image between the corrected current frame and the adaptively selected past frame, c is the constant for σ distance which determines the outlier threshold, µen and σen are the mean and standard deviation of the error map, respectively.