Method and device for removing afterglow in infrared images of varying scenes

Abstract

The present invention relates to a method for removing afterglow artifacts from an image (f n ) of an image sequence captured by an infrared imaging capture device by an image processing device, the method comprising: - modifying an afterglow region in the image (f n ) by a modification method to generate a modified image; - generating an afterglow metric for at least some pixels in the image (f n ) based on the modified image; and - removing afterglow artifacts from at least some pixels of the image (f n ) based on an afterglow estimate for each of the at least some pixels, each afterglow estimate being generated based on afterglow measurements of a plurality of images in the sequence.

Description

[0001] This patent application claims the priority of the French patent application and the transfer application No. FR1904565 filed on April 30, 2019, the content of which is incorporated herein by reference. Technical Field

[0002] The present disclosure generally relates to the field of infrared imaging, and particularly to methods and devices for removing remanence from infrared images. Background Art

[0003] A microbolometer is an uncooled infrared (IR) camera for capturing thermal images of an image scene. Such an IR camera typically includes an arrangement of IR-sensitive detectors forming a pixel array. Each pixel of the pixel array converts the measured temperature at the pixel into a corresponding electrical signal (usually a voltage), which is in turn converted into a digital output signal by an ADC (analog-to-digital converter).

[0004] Each pixel of the microbolometer includes a membrane suspended on a substrate. The membrane includes an absorption layer that absorbs energy from the IR light impinging on the pixel, causing its temperature to increase as a function of the intensity of the IR light. The membrane also includes, for example, a thermal layer that has the property that its resistance changes due to this temperature increase, and thus the pixel can be read by detecting a change in the resistance of this thermal layer thermally linked to the absorption layer.

[0005] Uncooled IR sensors (such as microbolometers) may be affected by an imaging artifact called remanence, which occurs when a high-temperature flux is received by the pixel array. For example, such a high-temperature flux may be caused by the sun or another strong heat source and cause a change in the resistivity of the associated pixels. After the pixel temperature drops, it takes time for the resistance of these pixels to return to the normal level, resulting in deterioration of the image quality, which may last for several minutes or up to several days, depending on the flux level.

[0006] In the case where the camera has a mechanical shutter, the algorithm for removing remanence artifacts is relatively simple. However, there are technical difficulties in removing remanence artifacts from images captured by uncooled shutterless IR sensors. Summary of the Invention

[0007] An object of embodiments of the present disclosure is to at least partially solve one or more difficulties in the prior art.

[0008] According to one aspect, there is provided a method for removing afterglow artifacts from an image in a sequence of images captured by an infrared imaging device by an image processing device, the method comprising: repairing an afterglow region in the image to generate a repaired image; generating an afterglow metric for at least some pixels in the image based on the repaired image; and removing afterglow artifacts from at least some pixels of the image based on an afterglow estimate for each of the at least some pixels, each afterglow estimate being generated based on the afterglow metrics of a plurality of images in the sequence.

[0009] According to one embodiment, the afterglow estimate is an average afterglow estimate. For example, the average afterglow estimate corresponds to an estimate of the instantaneous average afterglow for each of the at least some pixels.

[0010] According to one embodiment, generating an afterglow metric for at least some pixels in the image includes subtracting the pixel value of the corresponding pixel in the repaired image from the pixel value of each of the at least some pixels in the image.

[0011] According to one embodiment, the method further includes detecting an afterglow region in the image.

[0012] According to one embodiment, the afterglow region includes fewer than 75% of the pixels of the image.

[0013] According to one embodiment, the method further includes calculating an afterglow estimate for each of the at least some pixels by: calculating a moving average of the afterglow metrics of the images in the sequence and at least one previous image.

[0014] According to one embodiment, the moving average is based on a sliding window of m images in the sequence, where m is equal to between 20 and 150.

[0015] According to one embodiment, the method further includes calculating an afterglow estimate for each of the at least some pixels by applying a Kalman filter.

[0016] According to one embodiment, calculating the afterglow estimate by applying a Kalman filter includes: a priori estimating the afterglow of each of the at least some pixels of the image based on a previous corrected afterglow estimate for each of the at least some pixels and a model based on the exponential decay of the afterglow; and for each of the at least some pixels of the image, correcting the estimated afterglow based on the corresponding afterglow metric to obtain a posterior estimate.

[0017] According to one embodiment, the method further includes detecting that the sequence of images corresponds to a changing scene before removing afterglow artifacts from the image.

[0018] According to a further aspect, there is provided a non-transitory storage medium storing computer instructions which, when executed by a processor of an image processing device, cause the above method to be implemented.

[0019] According to another aspect, there is provided an image processing apparatus including: at least one memory storing an image in an image sequence captured by an infrared imaging device and one or more afterglow metrics of at least one previous image in the sequence; and one or more processors configured to remove afterglow artifacts from the image by: repairing an afterglow region in the image to generate a repaired image; generating an afterglow metric for at least some pixels in the image based on the repaired image; and removing afterglow artifacts from at least some pixels of the image based on an afterglow estimate for each of the at least some pixels, each afterglow estimate being generated based on afterglow metrics of a plurality of images in the sequence.

[0020] According to one embodiment, the afterglow estimate is an average afterglow estimate. For example, the average afterglow estimate corresponds to an estimate of the instantaneous average afterglow for each of at least some pixels.

[0021] According to another aspect, there is provided an infrared imaging device including: a microbolometer array; and the above image processing apparatus.

[0022] According to another aspect, there is provided an infrared camera including the above infrared imaging device.

[0023] According to yet another aspect, there is provided a method for removing afterglow artifacts from an image in an image sequence captured by an infrared imaging device by an image processing apparatus, the method including: generating an afterglow metric for at least some pixels in the image based on a difference between pixel values of an image in the sequence and pixel values of a previous image; and removing afterglow artifacts from at least some pixels of the image based on an afterglow estimate for each of the at least some pixels, each afterglow estimate being generated based on the afterglow metric, and based on one or more previous afterglow estimates of at least some pixels and a model based on exponential decay of afterglow.

[0024] According to one embodiment, the afterglow estimate is an average afterglow estimate. For example, the average afterglow estimate corresponds to an estimate of the instantaneous average afterglow for each of at least some pixels.

[0025] According to one embodiment, generating an afterglow metric for at least some pixels x in the image includes calculating:

[0026] [Equation 1]

[0027] ω* n,x = f n,x - f n-1,x

[0028] where f n is the image, and f n-1 is the previous image.

[0029] According to one embodiment, the method further includes generating an afterglow estimate for each of at least some of the pixels by applying a Kalman filter.

[0030] According to one embodiment, generating an afterglow estimate by applying a Kalman filter includes: a priori estimating the afterglow of each of at least some of the pixels of the image based on a previous corrected afterglow estimate of each of at least some of the pixels and a model based on the exponential decay of the afterglow; and for each of at least some of the pixels of the image, correcting the estimated afterglow based on a corresponding afterglow metric to obtain a posteriori estimate.

[0031] According to one embodiment, the method further includes detecting that a sequence of images corresponds to a static scene before removing afterglow artifacts from the images.

[0032] According to yet another aspect, there is provided a non-transitory storage medium storing computer instructions which, when executed by a processor of an image processing device, cause the above method to be implemented.

[0033] According to yet another aspect, there is provided an image processing device including: at least one memory storing an image in a sequence of images captured by an infrared imaging device and one or more afterglow metrics of at least one previous image in the sequence; and one or more processors configured to remove afterglow artifacts from the image by: generating an afterglow estimate for at least some of the pixels in the image based on a difference between a pixel value of the image and a pixel value of a previous image of the sequence; and removing afterglow artifacts from at least some of the pixels of the image according to an estimate of the afterglow generated based on the afterglow estimate and one or more previous estimates of the afterglow of at least some of the pixels and a model based on the exponential decay of the afterglow.

[0034] According to one embodiment, the afterglow estimate is an average afterglow estimate. For example, the average afterglow estimate corresponds to an estimate of the instantaneous average afterglow for each of at least some of the pixels.

[0035] According to yet another aspect, there is provided an infrared imaging device including: a microbolometer array; and the above image processing device.

[0036] According to yet another aspect, there is provided an infrared camera including the above infrared imaging device. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The foregoing features and advantages, as well as other features and advantages, will be described in detail in the following description of specific embodiments given by way of illustration and not limitation, with reference to the accompanying drawings, in which:

[0038] Figure 1 Schematically shows an IR imaging device according to an exemplary embodiment of the present disclosure;

[0039] Figure 2 Schematically shows in more detail the Figure 1 image processing circuit according to an exemplary embodiment;

[0040] Figure 3 Represents an image including afterglow artifacts;

[0041] Figure 4 Is a graph showing the exponential decay of the afterglow of a pixel;

[0042] Figure 5 Is a general flowchart showing operations in a method for removing afterglow from an image according to an exemplary embodiment of the present disclosure;

[0043] Figure 6 Is a graph showing an example of the simulated pixel values of pixel x affected by afterglow in a sequence of 300 images;

[0044] Figure 7 Is a graph showing an example of the simulated pixel values on a microbolometer array having an afterglow region K;

[0045] Figure 8 Is a graph showing another example of the simulated pixel values on a microbolometer array;

[0046] Figure 9 Is a graph representing the distribution of the difference between the afterglow estimate and the average afterglow value;

[0047] Figure 10 Is a graph representing the estimated value of the afterglow of a pixel and the afterglow values estimated using various methods along a sequence of n = 1000 consecutive images;

[0048] Figure 11 Is a flowchart showing operations in a method for performing Kalman filtering to generate an afterglow estimate according to an exemplary embodiment of the present disclosure; and

[0049] Figure 12 Shows the potential structure of an effective time series for applying a Kalman filter. Detailed Description

[0050] In the different drawings, the same features are denoted by the same reference numerals. In particular, structural and / or functional features common to various embodiments may have the same reference numerals, and the same structures, dimensions, and material properties may be provided.

[0051] Unless otherwise specified, when referring to two elements connected together, this indicates a direct connection without any intermediate elements other than conductors, and when referring to two elements linked or coupled together, this indicates that the two elements may be connected or they may be linked or coupled via one or more other elements.

[0052] In the following disclosure, unless otherwise specified, when referring to absolute position qualifiers (such as the terms "front", "rear", "top", "bottom", "left", "right", etc.) or referring to relative position qualifiers (such as the terms "above", "below", "higher", "lower", etc.), or referring to orientation qualifiers (such as "horizontal", "vertical", etc.), reference is made to the orientation shown in the figure or to the microbolometer oriented during normal use.

[0053] Unless otherwise specified, the expressions "about", "approximate", "substantially", and "on the order of" mean within 10%, preferably within 5%.

[0054] Figure 1 An IR imaging device 100 including a pixel array 102 sensitive to IR light is shown. The IR imaging device 100 forms part of an IR camera, for example. In some embodiments, the pixel array 102 is sensitive to long-wave IR light, such as light having a wavelength in the range of 7 to 16 μm or higher.

[0055] In Figure 1 's example, the pixel array 102 includes 144 microbolometer pixels 104 arranged in 12 rows and 12 columns. In alternative embodiments, the pixel array 102 may include any number of pixel rows and columns. Generally, the pixel array 102 includes, for example, 640×480 or 1024×768 microbolometer pixels.

[0056] In Figure 1 's example, each column of pixels of the array 102 is associated with a corresponding reference structure 106. Although not a picture element functionally, this structure will be referred to herein as a "reference pixel" by analogy with the structure of the imaging (or active) microbolometer pixels 104. Additionally, an output block (OUTPUT) 108 is coupled to each column of the pixel array 102 and to each of the reference pixels 106, and provides a raw image RAW that includes signals or readings captured by the pixel array 102 in combination with the reference pixels 106.

[0057] For example, an example of a bolometer-type pixel array is discussed in more detail in U.S. Patent US 7,700,919 assigned to the present applicant, the content of which is incorporated herein by reference to the extent permitted by law.

[0058] The control circuit (CTRL) 110 supplies control signals, for example, to the pixel array 102, to the reference pixel 106, and to the output block 108.

[0059] The raw image RAW is supplied, for example, to an image processing circuit (IMAGE PROCESSING) 112 which, for example, applies two-point image correction to the pixels of the image and also applies afterglow correction to produce a corrected image S.

[0060] During a read operation of the pixel array 102, pixel rows are read out, for example, one row at a time.

[0061] Figure 2 More specifically shown is Figure 1 the image processing circuit 112 according to an example embodiment.

[0062] The functions of the image processing circuit 112 are implemented, for example, in software, and the image processing circuit 112 includes a processing device (PROCESSING DEVICE) 202 having one or more processors under the control of instructions stored in an instruction memory (INSTR MEMORY) 204. In an alternative embodiment, the functions of the image processing circuit 112 may be implemented at least in part by dedicated hardware. In this case, the processing device 202 includes, for example, an ASIC (application specific integrated circuit) or an FPGA (field programmable gate array), and the instruction memory 204 may be omitted.

[0063] The processing device 202 receives the raw input image RAW and generates a corrected image S which is supplied, for example, to a display (not shown) of the imaging device.

[0064] The processing device 202 is also coupled to a data memory (MEMORY) 206 which stores, for example, offset values (OFFSET) 208 and gain values (GAIN) 210 for implementing two-point image correction.

[0065] In some embodiments, shutterless image correction is performed. For example, the offset value is a vector V representing structural column diffusion COL and a matrix OFF representing 2D non-column-structured dispersion introduced by the pixel array 102 DISPFor example, column diffusion is mainly caused by the use of reference pixels 106 in each column, and the rows of the column reference pixels are generally not completely uniform. 2D non-column diffusion is mainly caused by local physical and / or structural differences between the active bolometers of the pixel array, for example, caused by process diffusion.

[0066] Examples of shutterless image correction techniques are described in more detail in U.S. Patent Application US 14 / 695539, filed Apr. 24, 2015, and assigned to the present applicant, particularly the generation of vectors V COL and matrix OFF DISP and examples of the correction of pixel values based on this vector and matrix, the content of which is incorporated herein by reference to the extent permitted by law.

[0067] In alternative embodiments, different types of two-point image correction can be implemented, including solutions involving the use of a shutter.

[0068] Memory 206 also stores, for example, one or more remanence estimates (REMANENCE ESTIMATE(S)) 212 calculated for the pixels of one or more previously captured images, as will be described in more detail below. In some embodiments, memory 206 also stores a model W (MODEL W) representing the decay of the remanence, as will be described in more detail below.

[0069] Figure 3 An image 300 is shown in which there are remanence artifacts 302. For example, this image is captured by a bolometer array, and while the array is active, the array receives glare and in particular a relatively strong heat flux. The resistance of the thermal layer of each pixel of the bolometer array should vary as a function of temperature, with the resistance decreasing as the temperature increases. In image 300, for readability, the gray levels have been inverted: higher temperatures are represented by darker pixels. The relatively strong heat flux causes a significant and persistent decrease in the resistance of the associated pixels, resulting in a dark trace 302 in image 300 corresponding to the path of the flux passing through the bolometer array. When caused by the heat of the sun, such an artifact is commonly referred to as a sunspot.

[0070] Figure 4 is a graph showing an example of the remanence ω Figure 1 of a pixel x of a bolometer array (such as x array 102) as a function of time according to an example embodiment. The remanence ω x has an exponential decay that can be modeled by the following equation:

[0071] [Mathematical formula 2]

[0072]

[0073] where: t0 is the time for the pixel to return to its normal temperature after the instant when the glare hits the pixel; x = (row, column) is the coordinate vector of the pixel; α, β, and τ are parameters; and:

[0074] [Mathematical formula 3]

[0075]

[0076] is the characteristic function. In Figure 4 the example of, τ = 10, α = 1, and β = 0.5.

[0077] The afterglow is an additive perturbation. Thus, the pixel value f x (t) of the image affected by the afterglow can be modeled as:

[0078] [Mathematical formula 4]

[0079]

[0080] where S x (t) is the image without glare, G ωx is the gain change matrix due to glare, and f x (t) corresponds, for example, to the 2-point image correction as described above. A method for performing correction based on the gain change matrix is described in more detail in French Patent FR2987210, the content of which is incorporated herein by reference to the extent permitted by law. However, the change in the sensitivity of the pixel represented by the term G ωx (t - t0)S x (t) can be considered to be very small with respect to ω x and, hereinafter, its contribution will be ignored by considering the term G ω to be equal to 1.

[0081] When an IR imaging device (such as Figure 1 device 100) is used to capture an IR image, there are several different use cases that can be distinguished.

[0082] A dynamic situation (hereinafter referred to as Case 1) can be considered, where the image in the field of view of the pixel array can be considered to be changing because the IR camera is moving and / or because the scene is changing. For this type of dynamic situation, the above model is appropriate because both the afterglow ω x (t) and the image S x (t) change over time:

[0083] [Mathematical formula 5]

[0084] f xI(t) = ω x (t) + S x (t)

[0085] Another situation (hereinafter referred to as Situation 2) corresponds to the case where the image in the field of view of the pixel array is substantially constant. Therefore, only the afterglow changes with time. In this case, the pixel value f of the image affected by the afterglow x (t) can be modeled as:

[0086] [Equation 6]

[0087] f x (t) = ω x (t) + S x

[0088] where the image S without glare can be assumed x to be constant.

[0089] In both Situation 1 and Situation 2, the image stream is captured and stored, for example, to form a video file and / or relayed to the image display of the IR camera in near real time.

[0090] A third situation (hereinafter referred to as Situation 3) corresponds to the case where both the image and the afterglow in the field of view of the IR camera change slowly enough that they can both be considered constant over time. In other words, the time decay is so slow that it cannot be measured within the time range of the evaluation. In this case, the pixel value f of the image x can be modeled as:

[0091] [Equation 7]

[0092] f x = ω x + S x

[0093] Now, a method for at least partially removing artifacts caused by the afterglow in Situation 1 and Situation 2 will be described in more detail. Situation 3 will not be addressed in the present disclosure, as it is equivalent to image retouching of a single image without additional time information on lost pixel information.

[0094] Below, time is sampled by representing t = n and t + δt = n + 1. Therefore, the value of the matrix M at time t and pixel x will be denoted as M n,x , rather than M(t, x).

[0095] Overall scheme

[0096] Figure 5is a general flowchart showing operations in method 500 for correcting afterglow in an image sequence according to an exemplary embodiment of the present disclosure. The method is implemented, for example, by Figure 1 and Figure 2 's image processing circuit 112. Alternatively, the method may be implemented by another type of processing device, which may or may not form part of an IR camera.

[0097] Image capture

[0098] In operation 501, the method begins when an image f n is captured. This image corresponds, for example, to the original image RAW n to which image correction (such as the two-point image correction described above) has been applied.

[0099] Change detection

[0100] After operation 501, for example, operation 502 is implemented, in which the presence of a changing scene is detected based on image f n and a previously captured image f n-1 . After operation 502 is the afterglow estimation method 503, including, for example, estimating the current afterglow average value μ ω,n,x at each pixel x of the current image n. The detection of the changing scene in operation 502 is used to select between alternative methods 504 and 505 for performing the afterglow estimation operation 503. In particular, if a changing scene is detected in operation 502, then in the subsequent afterglow estimation method 503, corresponding to Figure 5 's method 504, the method of case 1 is applied. Conversely, if a static scene is detected, then in the subsequent afterglow estimation method 503, the method of case 2 corresponding to Figure 5 's method 505 is applied.

[0101] For example, in operation 502, the dynamics of the scene can be detected by comparing the temporal variation with a threshold, which is set to be higher than the temporal noise level, for example, twice or three times the temporal noise level. Such an evaluation can be performed globally on the image, such as based on the average temporal variation, in which case, for example, all pixels of the image are processed according to the same method (method 504 of case 1 or method 505 of case 2). Alternatively, a local evaluation can be performed based on the temporal variation at individual pixels or individual sub-regions of the image, and a choice can be made between applying method 504 of case 1 or method 505 of case 2 pixel by pixel or sub-region by sub-region.

[0102] Method 504 is described in more detail in the subsection titled Case 1 below, and method 505 is described in more detail in the subsection titled Case 2 below.

[0103] Although Figure 5 a method including the execution of both methods 504 and 505 is shown, and the selection between these methods is made for each frame based on scene change detection, in alternative embodiments, only method 504 or only method 505 may be implemented.

[0104] Afterglow correction

[0105] After operation 503, for example, operation 506 is implemented, which includes correcting the image based on the estimated afterglow. For example, the image is corrected by applying the following equation to each pixel x of the image f n to generate image S n :

[0106] [Equation 8]

[0107] S n,x = f n,x - μ ω,n,x

[0108] After operation 506, n is incremented, and operations 501 to 506 are repeated, for example, after capturing a subsequent image n.

[0109] Case 1

[0110] Afterglow region detection

[0111] In case 1, method 503 includes applying Figure 5 method 504, which includes operations 510 to 513.

[0112] In operation 510, one or more afterglow regions are updated.

[0113] For example, a mask indicating the afterglow regions is generated for each image f n . These regions may be larger than the actual regions where pixels are affected by afterglow. However, in some embodiments, the afterglow regions correspond to at most 75% of the pixels of the image. For example, the mask can be generated by detecting regions where the pixel values are significantly higher than those in the surrounding area, by tracking the trajectory of, for example, the sun, although other techniques, including edge detection-based techniques, can also be used.

[0114] Repair

[0115] Then, in operation 511, inpainting is applied to those afterglow regions in the image. Inpainting is an image reconstruction technique known in the art and includes correcting pixel values within a target region based on image information from outside the target region. For example, the method of inpainting is described in more detail in the 2014 publication by C. Guillemot and O. Le Meur, titled "Image inpainting: Overview and recent advances", IEEE signal processing magazine, 31(1), 127 - 144 (the content of which is incorporated herein by reference to the extent permitted by law).

[0116] Afterglow measurement

[0117] In operation 512, based on the inpainted image, the noise metric ω* of the afterglow at pixel x in the image f is processed for each pixel n in the image. For example, this noise metric is calculated by performing the following subtraction pixel by pixel: n,x

[0118] [Equation 9]

[0119]

[0120] where S* n,x is the nth inpainted image at pixel x.

[0121] Afterglow estimation

[0122] In operation 513, the afterglow is estimated by applying the afterglow estimation operation 530. For example, the afterglow estimation includes the estimation of the instantaneous average afterglow μ for each pixel x ω,n,x . For example, for each pixel x, this includes generating a current estimate ω of the afterglow n,x , as a weighted sum between the current afterglow metric ω* n,x and the previous estimate ω of the afterglow n-1,x . In some embodiments, operation 530 includes generating the estimate ω n,x simply as the average of the afterglow {ω* k,x} measured in the previous m images k＝n-M:n . In alternative embodiments, operation 530 includes generating the estimate ω n,x based on a Kalman filter that takes into account the time decay model of the afterglow. For example, an exponential model W can be used. Using the Kalman filter, the estimator of ω n,x is, for example, its mean μ ω,n,x and its variance σ 2 ω that fully characterizes it​,n,x .

[0123] Method 504 allows for the removal of afterglow based on observations made by the present inventors, i.e., when the image in the field of view of the pixel array is changing, the temporal variation of the pixel values can be considered to be noise having a normal distribution centered at μ ω,n,x .

[0124] Figure 6 is a graph showing an example of the simulated pixel values (PIXEL VALUE) of pixel x affected by afterglow on a sequence n of 300 images captured at an image capture rate of, for example, 25 frames per second. Figure 6 The points in represent 300 pixel values, and the continuous curve 602 represents the estimated average afterglow variation over time. Figure 6 The dashed line 604 in represents the true afterglow ω x , which has an exponential decay. By obtaining a reasonably accurate estimate of this true afterglow ω x of a given image, it can be subtracted from the pixel values in order to provide the underlying pixel information representing the image scene.

[0125] In fact, considering that the image in the field of view of the pixel array varies over time, the scene can be considered to be stationary noise defined as follows:[

[0126] [Equation 10]

[0127]

[0128] where “~” means “distributed according to...”, and N(μ,σ 2 ) is a normal distribution with mean μ and variance σ 2 , and the mean and variance are constant in time and space here.

[0129] However, afterglow is non-stationary noise because it has a mean that decays over time:[

[0130] [Equation 11]

[0131]

[0132] Thus, the image observed at pixel x of image f n is the sum of two independent random variables:[

[0133] [Equation 12]

[0134]

[0135] Figure 7It is a graph showing the simulated pixel values (PIXELVALUE) of pixels (PIXEL x) from 0 to P on a microbolometer array, where it is assumed that the pixels in region K are in the path of the solar flare.

[0136] Figure 7 The observed image f is represented by the continuous curve 702 n,x as an example, which is the average value of the image scene μ represented by the dashed line 704 S,n and the sum of the average afterglow μ represented by the continuous curve 706 ω,n,x Therefore, by estimating the average value of the image scene μ S,n,x it can be obtained by calculating the following for μ ω,n,x :

[0137] [Equation 13]

[0138] μ ω,n,x = f n,x - μ S,n,x

[0139] In Figure 5 operation 513 of method 504, the present inventors propose to use inpainting to obtain an estimate of the average value of the image scene μ S,n and this will now be described in more detail with reference to Figure 8 this.

[0140] Figure 8 It is a graph showing the pixel values (PIXELVALUE) of pixels (PIXEL x) from 0 to P on a microbolometer array, where similar to Figure 7 the example, it is assumed that the pixels in region K are in the path of the solar flare.

[0141] Figure 8 The continuous curve 802 in n represents the line of the observed image f n while the dashed line 804 represents the inpainted line, denoted as S*

[0142] [Equation 14]

[0143]

[0144] where K_bar is the region outside the solar flare region K. In other words, the pixel information from the region outside the solar flare region is used to inpaint region K.

[0145] Figure 8 The curve 806 in

[0146] [Equation 15]

[0147] E[S * = μ S

[0148] where E[] is the expected value.

[0149] The dashed line 808 represents the afterglow ω n , and the continuous curve 810 represents the result of the subtraction f n - S* n .

[0150] It can be assumed that the restored part of the image provides an unbiased estimate μ of the average value of the image scene S :

[0151] [Mathematical formula 16]

[0152]

[0153] and thus provides an unbiased measure of μ ω,n,x :

[0154] [Mathematical formula 17]

[0155]

[0156] Figure 9 is a graph representing the distribution (which corresponds to a normal distribution) of the magnitude of the difference between the afterglow measurement ω* n,x and the average afterglow μ ω,n,x . In fact, at each instant of time, corresponding to the following ω* n measurement:

[0157] [Mathematical formula 18]

[0158]

[0159] provides an unbiased noise measurement of the expected value E[ω* n,x = μ ω,n,x .

[0160] Since this value varies with time, the present inventors propose to estimate this value for each new image f n based on the weighted sum between the current measurement ω* ω,n-1 and the previous estimate μ n , as will now be described in more detail with reference to Figure 10 .

[0161] Figure 10 is a graph representing the value z of the afterglow estimate for a pixel x along a sequence of n = 1000 consecutive frames n . Each continuous curve corresponds to the value at Figure 5for estimating the value μ in operation 530 ω,n,x Different methods. The dots correspond to the original afterglow ω* n,x Measurement. Line 1002 represents these original afterglow intensity measurements ω* n,x The average value of which can be expressed as:

[0162] [Equation 19]

[0163]

[0164] Figure 10 The dashed line 1004 in represents the true decay of the afterglow of pixel x.

[0165] Figure 10 The continuous curve 1006 in represents the moving average based on a window of 10 images, which can be expressed as:

[0166] [Equation 20]

[0167]

[0168] Curve 1006 provides fast convergence, but has relatively high noise.

[0169] Figure 10 The curve 1008 in represents the moving average based on a window of 100 images, which can be expressed as:

[0170] [Equation 21]

[0171]

[0172] Curve 1008 still provides relatively fast convergence and less noise than curve 1006. However, this curve is biased and there is a risk that this bias will be visible in the final corrected image.

[0173] In some embodiments, in order to obtain a reasonable compromise between convergence speed and noise, the moving average can be used based on a sliding window of m images, where m is in the range of 20 to 150, and for example in the range of 20 to 80. In some embodiments, the number of images m can vary over time, for example increasing over time as the image sequence progresses. In some embodiments, if for example the result is still somewhat biased and if it is necessary to calibrate the size of the sliding window, this method based on the sliding window may have disadvantages.

[0174] Figure 10The curve 1010 therein corresponds to the application of a Kalman filter, which has a relatively slow convergence compared to the moving average but converges to the true decay of the afterglow represented by curve 1004 with relatively high accuracy. The Kalman filter takes into account the measurements of the afterglow of multiple images in the sequence, as well as the model W of the exponential decay of the afterglow. Now, reference will be made to Figure 11 and Figure 12 to describe in more detail the use of the Kalman filter in operation 530.

[0175] Kalman filter

[0176] Figure 11 is a flowchart of the operations of a method for estimating the average afterglow value μ Figure 5 using a Kalman filter in operation 513 of ω,n,x . This method is implemented, for example, by the Figure 1 and Figure 2 image processing circuit 112. Alternatively, this method can be implemented by another type of processing device, which may or may not form part of an IR camera.

[0177] The Kalman filter is well-suited for using noisy (observed) measurements ω n * and estimating the latent (hidden) state vector ω at each step n of a linear Gaussian dynamic time series given some hyperparameters n . Such a time series is a Markov chain and can be fully described by the following simple sequence:

[0178] [Mathematical formula 22]

[0179] ω n = Aω n-1 + v,

[0180]

[0181] ω0 = μ ω,0 + u,

[0182] where A is the transition matrix, H is the measurement matrix, and the parameters v, u, and η are some zero-mean normal additive noises of the covariance matrices Γ, Σ ω,0 and R, respectively.

[0183] Figure 12An example of such a time series is shown, where the hidden state vector (which has 1 dimension in this example) is a circle, and the observed metric is a point. The input from time (t - δt) is the previously corrected estimate of the afterglow ω(t - δt), and the transition matrix A that links the previous estimate at time (t - δt) to the new estimate at time t. Since everything in this time series is Gaussian, the state vector is fully characterized by its mean vector μ ω and covariance matrix Σ ω (which are unknown values to be determined).

[0184] At each step n, shown is [Murphy, Machine Learning, a probabilistic perspective, 2012 p.638] the latent state ω given knowledge of the (time constant) hyperparameters Γ, R, U, A, H, and μ0 n can be estimated using the following equation:

[0185] [Equation 23]

[0186]

[0187]

[0188]

[0189]

[0190] where:

[0191] [Equation 24]

[0192]

[0193] is the Kalman gain, and I is the identity matrix.

[0194] In other words, at each step n, based on the measurement ω n * and the prior estimate (μ ω,n,x , Σ - ω,n,x ) a new posterior estimate (μ ω,n,x , Σ ω,n,x ) is estimated.

[0195] Applied to afterglow filtering

[0196] Covariance matrix

[0197] In the present disclosure, the hidden variable ω is the afterglow at each pixel and is a scalar. Thus, the covariance matrices Γ, R, Σω,0 , Σ ω is only represented as γ, r, σ ω,0 2 , σ ω 2 variance of

[0198] Transformation matrix A

[0199] In the present disclosure, the hidden state is a scalar, and the transition matrix A is also a scalar labeled as a and is directly related to the afterglow model W. As described above regarding Figure 4 stated:

[0200] [Mathematical formula 25]

[0201]

[0202] [Mathematical formula 26]

[0203]

[0204] In this example, assuming that the parameter β is equal to 1, the following model of ω(t + δt) is obtained:

[0205] [Mathematical formula 27]

[0206] ω(t + δt) = aω(t), where

[0207] the transition matrix a is common for all pixels of the array, for example. Alternatively, different transition matrices a can be stored for each pixel x .

[0208] Measurement matrix H

[0209] As described above, the metric ω n * is also a real scalar value that is directly related to the hidden state using the metric matrix H:

[0210] [Mathematical formula 28]

[0211] where H = 1

[0212] Initialization

[0213] In Figure 11 operation 1101 of method 530, the Kalman parameters are initialized by initializing the hidden state of the Kalman filter. For example, the initial mean μ ω,0 and variance value σ ω,0 2 can be initialized as:

[0214] [Mathematical formula 29]

[0215]

[0216] and

[0217] [Mathematical formula 30]

[0218]

[0219] Prior estimate

[0220] In operation 1102, a new prior estimate for the average afterglow is generated based on the afterglow-based noise metric and based on the Kalman parameters and Kalman hyperparameters. For example, at time n+1, a new prior estimate for the average afterglow is generated:

[0221] [Mathematical formula 31]

[0222]

[0223]

[0224] where γ is the variance of the model and is, for example, user-defined.

[0225] Posterior estimate

[0226] In operation 1103, the previous new mean and variance values of the afterglow are corrected based on the Kalman parameters, Kalman hyperparameters, and the current measured afterglow value ω n,x * , determined from the repaired image:

[0227] [Mathematical formula 32]

[0228]

[0229]

[0230] where:

[0231] [Mathematical formula 33]

[0232]

[0233] is the Kalman gain.

[0234] In other words, the posterior estimate of the afterglow is generated as a balance between the prior estimate and the metric, and this balance is tuned by the Kalman gain K.

[0235] Sequence

[0236] Referring again to Figure 11 , after operation 1103, for example, the Kalman parameters are updated. As Figure 5 shown, method 530 can then be repeated for the next image n+1, including repeating operations 1102 and 1103 based on the updated Kalman parameters. In fact, the parameter initialization operation 1101 is performed only, for example, in Figure 5 the first iteration of the method.

[0237] Case 2

[0238] In case 2, Figure 5 operation 503 of Figure 1 and Figure 2 includes method 505, which corresponds to a method of correcting afterglow in an image sequence according to a further exemplary embodiment of the present disclosure. This method is implemented, for example, by

[0239] the image processing circuit 112 of

[0240] . Alternatively, this method can be implemented by another type of processing device, which may or may not form part of an IR camera. n-1 n n,x In operation 520, a measure of the afterglow in the image f n,x is calculated for each pixel based on the previous image f

[0241] [Equation 34]

[0242] ω* n,x = f n,x - f n-1,x

[0243] Figure 11 n,x In operation 521, the afterglow ω ω,n,x of each pixel is estimated based on the afterglow measurement and the exponential decay model W ω,n,x . For example, the afterglow estimate includes an estimate μ 2 ω,n,x of the instantaneous average afterglow of each pixel x. In some embodiments, the afterglow average μ Figure 11 and its variance σ x = 1 - δτ / τ xStored in the memory 206. These transformation matrices are estimated, for example, during a preprocessing operation that includes, for example, regressing an exponential decay over two or more of the first frames that occur exactly after the afterglow to estimate τ x . Further, in case 2, the transformation matrix is:

[0244] [Equation 35]

[0245]

[0246] An advantage of the embodiments described herein is that for the case of a changing scene (case 1) or a stationary scene (case 2), afterglow can be corrected in an image sequence in a relatively simple and efficient manner.

[0247] Various embodiments and variations have been described. Those skilled in the art will understand that certain features of these embodiments can be combined, and other variations will readily occur to those skilled in the art. For example, it will be apparent to those skilled in the art that the choice between providing a common exponential decay model W for all pixels or a different model for each pixel x will depend on the required accuracy and the characteristics of the IR pixel array.

Claims

1. A method for removing afterglow artifacts from an image (f n ) of an image sequence captured by an infrared imaging device by an image processing device (112), the method comprising: - Repair the afterglow region (K) in the image (f n ) by an image retouching method to generate a repaired image; - Generating an afterglow metric (ω* n ) for at least some pixels in the image (f n,x ) based on the repaired image; and - Based on the afterglow estimation (μ ω,n,x ) for each of the at least some of the pixels, remove afterglow artifacts from at least some of the pixels of the image (f n ), each afterglow estimation (μ ω,n,x ) being generated based on the afterglow metric (ω* n,x ) of multiple images in the sequence.

2. The method according to claim 1, wherein the afterglow estimate is an average afterglow estimate (μ ω,n,x ) that is an estimate of the instantaneous average afterglow corresponding to each of the at least some of the pixels.

3. The method according to claim 1 or 2, wherein generating the afterglow measure (ω* n ) for at least some of the pixels in the image (f n,x ) comprises subtracting the pixel value of the corresponding pixel in the inpainted image from the pixel value of each of at least some of the pixels in the image (f n ).

4. The method according to any one of claims 1 to 3, further comprising detecting an afterglow region (K) in the image (f n ).

5. The method according to claim 4, wherein the afterglow region (K) comprises fewer than 75% of the pixels of the image.

6. The method according to any one of claims 1 to 5, further comprising calculating an afterglow estimate (μ ω,n,x ) for each of said at least some of the pixels by: - Calculate the moving average of the afterglow measure (ω* n ) of the image (f n,x ) in the sequence and at least one previous image.

7. The method according to claim 6, wherein the moving average is based on a sliding window of m images in the sequence, where m is between 20 and 150.

8. The method according to any one of claims 1 to 5, further comprising calculating an afterglow estimate (μ ω,n,x ) for each of the at least some of the pixels by applying a Kalman filter.

9. The method according to claim 8, wherein calculating the afterglow estimate (μ ω,n,x ) by applying a Kalman filter comprises: - Based on the previous corrected afterglow estimate for each of the at least some of the pixels and a model (W) based on the exponential decay of the afterglow, estimate a priori the afterglow of each of the at least some of the pixels of the image (f n ); and - For each of at least some of the pixels of the image (f n ), correct the estimated afterglow based on the corresponding afterglow metric (ω* n,x ) to obtain a posterior estimate (μ ω,n,x ).

10. The method according to any one of claims 1 to 9, further comprising detecting that the image sequence corresponds to a changing scene before removing the afterglow artifact from the image (f n ).

11. A non-transitory storage medium storing computer instructions which, when executed by a processor of an image processing device (112), cause the method according to any one of claims 1 to 10 to be implemented.

12. An image processing device, comprising: At least one memory (206), the at least one memory storing images (f n ) of an image sequence captured by the infrared imaging device and one or more afterglow metrics (ω* n,x ) of at least one previous image in the sequence; and One or more processors (202), the one or more processors being configured to remove afterglow artifacts from the image (f n ) by the following method: - Repair the afterglow area (K) in the image (f n ) by an image retouching method to generate a repaired image; - generating an afterglow metric (ω*) for at least some pixels in the image (f n ) based on the repaired image; and n,x ) - Based on the afterglow estimate (μ ω，n,x ) of each of the at least some of the pixels, afterglow artifacts are removed from at least some of the pixels of the image (f n ), each afterglow estimate (μ ω,n,x ) being generated based on the afterglow metric (ω* n,x ) of multiple images in the sequence.

13. The image processing apparatus according to claim 12, wherein the afterglow estimation is an average afterglow estimation (μ ω,n,x ) corresponding to an estimation of an instantaneous average afterglow of each of the at least some of the pixels.

14. An infrared imaging device, comprising: - a microbolometer array (102); and - an image processing device (112) according to claim 12 or 13.

15. An infrared camera, comprising the infrared imaging device according to claim 14.

Images