Method for thermal image processing

Abstract

The invention relates to a method for thermal image processing. According to one aspect, there is provided a method for thermal image processing, the method comprising: obtaining a thermal image acquired by a thermal camera and depicting a scene; obtaining a frequency distribution based on the pixel intensity of the thermal image; processing the frequency distribution to determine whether the frequency distribution includes a peak caused by a thermal object in the scene, the interval of the peak from a thermal background of the scene being greater than an intensity threshold; and in response to determining that the frequency distribution includes peaks spaced from the thermal background by greater than an intensity threshold, processing the thermal image to suppress a phantom of the thermal object in the thermal image, where the phantom is caused by an internal reflection of radiation from the thermal object in the thermal camera, and where the processing of the thermal image includes: estimating a location of phantom pixels forming the phantom; and suppressing the ghost in the thermal image by adjusting the intensity of the ghost pixels.

Description

Technical Field

[0001] The present invention generally relates to a method for thermal image processing and a corresponding thermal camera. Background Art

[0002] Thermal cameras are used in a variety of surveillance applications and are capable of thermal imaging a scene as well as making remote temperature measurements. In some installations, thermal cameras are used to monitor temperature for early fire detection and / or to detect overheating of an object. When capturing a thermal image of a scene that includes a hot object, the radiation received from the hot object may result in image artifacts in the form of a phantom image of the hot object due to internal reflections in the thermal camera. Notably, the reflection may occur between the front cover glass of the thermal camera and the thermal image sensor. Such reflections from the hot object may result in a phantom image that is sharp and well-focused. As a result, the phantom image may be difficult to distinguish from an actual hot object for operators and many types of automated image processing-based analysis. The phantom image may also obscure other objects of interest within the scene.

[0003] One conventional technique for reducing internal reflections is to provide an antireflection coating for the optical elements of the thermal imaging system. However, antireflection coatings may have limited effectiveness in suppressing strong reflections caused by hot objects. The sensitivity of the thermal camera is also reduced in the wavelength range of the antireflection coating. US2022 / 0326514A1 discloses another technique for using a curved protective window or filter to defocus or attenuate the reflected radiation, thereby reducing unwanted reflections. However, these methods involve adding specially adapted optical elements to the thermal camera, which may complicate manufacturing and result in increased costs. Another disadvantage is that the introduction of additional optical elements in the imaging system may affect all radiation received by the thermal camera and therefore reduce image quality, even without the risk of phantoms. Summary of the invention

[0004] It is an object of the present invention to provide a method for suppressing ghost images of hot objects in thermal images which method does not require the addition of any specially adapted optical elements and which method is further adapted to the thermodynamics of the monitored scene. Further and optional objects may be understood from the following.

[0005] According to a first aspect of the present invention, there is provided a method for thermal image processing, the method comprising:

[0006] obtaining a thermal image captured by a thermal camera and depicting the scene;

[0007] Obtaining frequency distribution based on pixel intensity of thermal images;

[0008] processing the frequency distribution to determine whether the frequency distribution includes a peak caused by a thermal object in the scene, the peak being separated from a thermal background of the scene by more than an intensity threshold; and

[0009] In response to determining that the frequency distribution includes a peak that is spaced from the thermal background by more than an intensity threshold, processing the thermal image to suppress ghost images of thermal objects in the thermal image, wherein the ghost images are caused by internal reflections of radiation from the thermal objects in the thermal camera, and wherein the processing of the thermal image includes:

[0010] estimating locations of ghost pixels that form ghost images; and

[0011] Suppress ghost images in thermal images by adjusting the intensity of ghost pixels.

[0012] Thus, the method of the first aspect enables selective or conditional suppression of phantoms based on whether a peak caused by radiation from a thermal object (hereinafter referred to as a "thermal object peak") is present in a frequency distribution and is spaced from the thermal background of the scene (i.e., the thermal background radiation) by more than an intensity threshold (which may be referred to as a "phantom suppression threshold"). Thus, the present invention is based, at least in part, on the recognition that the relative temperatures of the thermal object and the thermal background can be used as a reliable predictor of whether there are any noticeable phantoms in a thermal image. In the absence of a thermal object peak spaced from the thermal background by more than the intensity threshold, it is determined that the temperature differences within the scene are too small to cause the thermal image to be subjected to phantom suppression. Thus, if it is determined that the frequency distribution does not include a thermal object peak having a spacing above the thermal background exceeding the intensity threshold, it may be determined that phantom suppression does not need to be performed on the thermal image.

[0013] In the case where the scene includes a thermal object having a relatively small temperature difference compared to the thermal background, the frequency distribution may still present a thermal object peak, however separated from the thermal background by less than the intensity threshold. Although the thermal object may still produce a ghost image in this case, the additional intensity added to the ghost pixel due to the hot object may be too small to cause any risk of confusion to the automatic image processing algorithm processing the thermal image or to the operator viewing the thermal image.

[0014] Thus, in response to determining that the frequency distribution includes a thermal object peak that is separated from the thermal background by an intensity threshold, the thermal image is subjected to phantom suppression to generate a processed thermal image. Otherwise, it can be determined that the thermal image is not subjected to phantom suppression. In the first case, phantom suppression can generate a processed thermal image, which can, for example, be forwarded to a downstream image processing pipeline for further image processing. In the second case, the thermal image can be forwarded to the downstream image processing pipeline without subjecting the thermal image to phantom suppression.

[0015] This means that in a scene comprising a thermal object resulting in a thermal object peak having a separation from the thermal background greater than an intensity threshold, the method comprises:

[0016] obtaining a thermal image captured by a thermal camera and depicting a scene including a thermal object;

[0017] Obtaining frequency distribution based on pixel intensity of thermal images;

[0018] processing the frequency distribution to determine that the frequency distribution includes a peak that is separated from a thermal background of the scene by more than an intensity threshold, wherein the peak is caused by the thermal object; and

[0019] In response to determining that the frequency distribution includes a peak that is spaced from the thermal background by more than an intensity threshold, processing the thermal image to suppress ghost images of thermal objects in the thermal image, wherein the ghost images are caused by internal reflections of radiation from the thermal objects in the thermal camera, and wherein the processing of the thermal image includes:

[0020] estimating locations of ghost pixels that form ghost images; and

[0021] Suppress ghost images in thermal images by adjusting the intensity of ghost pixels.

[0022] Conversely, in a scene where the scene does not include a thermal object that causes a thermal object peak that is spaced from the thermal background by more than an intensity threshold, the method includes: obtaining a thermal image acquired by a thermal camera and depicting the scene; obtaining a frequency distribution based on pixel intensities of the thermal image; processing the frequency distribution to determine that the frequency distribution does not include a peak that is spaced from the thermal background of the scene by more than an intensity threshold; and in response to determining that the frequency distribution does not include a peak that is spaced from the thermal background by more than the intensity threshold, determining not to perform any phantom suppression on the thermal image (e.g., forwarding the (obtained) thermal image to a downstream image processing pipeline without subjecting the thermal image to phantom suppression).

[0023] By subjecting a thermal image to phantom suppression in response to determining that the frequency distribution includes a hot object peak separated from the thermal background by more than an intensity threshold, a visually improved thermal image may be obtained that better reflects the actual distribution of temperature. Suppressing phantoms may further reduce the risk of false alarms due to phantoms being interpreted as hot objects at locations in the scene where elevated temperatures are not expected. Additionally, the risk of phantoms disrupting automated surveillance algorithms based on image analysis, such as object tracking, may be reduced.

[0024] The method's frequency distribution and pixel-based processing provides a reliable and computationally efficient way to determine whether to perform ghost image suppression without resorting to object detection algorithms. Instead, the method can be implemented using simple algorithms, which facilitates computationally efficient implementation in a processing device (e.g., an FPGA) of a thermal image processing system.

[0025] Once the need to suppress ghost images is detected based on the threshold comparison, the location of the ghost images can be estimated, where the ghost images can be suppressed by adjusting the intensity of the ghost pixels. Various embodiments for adjusting pixel intensity are described below. However, in general, the adjustment can include reducing the intensity (value) of the ghost pixels according to the intensity of the pixels depicting the hot objects (i.e., "hot object pixels").

[0026] A "thermal object" here refers to an object (ie a physical object) having an elevated temperature relative to the thermal background of the scene. A thermal object may particularly refer to an object (ie an object facing the thermal camera) having an emissivity exceeding the thermal background.

[0027] "Pixel intensity" (interchangeably referred to as "pixel value") refers here to the intensity (or value) of a pixel in a thermal image. For thermal images, the intensity of a pixel reflects the radiation received from the scene at that pixel. Therefore, intensity is proportional to temperature. Intensity can also be interpreted as the amount of IR radiation or radiant flux received from the scene at the pixel.

[0028] By "frequency distribution based on pixel intensities" is meant here a statistical distribution representing the number of different pixel intensities (or pixel values) occurring in a thermal image. A frequency distribution may thus indicate the frequency distribution of radiation or temperature in a scene. A frequency distribution may also be referred to as a histogram. A frequency distribution may indicate an absolute frequency or a relative frequency of pixel intensities. A frequency distribution may be "binned", i.e., may indicate the frequencies of a plurality of bins (i.e., categories or sub-ranges) of pixel intensities defined across a range of pixel intensities, as this may reduce the computational resources required for the method.

[0029] A "peak in a frequency distribution" here refers to a pixel intensity or an interval of at least a predefined number of consecutive pixel intensities (eg one or more bins or sub-ranges of a frequency distribution) whose frequency exceeds a predefined minimum frequency.

[0030] In some embodiments, processing the frequency distribution includes detecting a peak (first peak) caused by a hot object and another peak (second peak) caused by a thermal background of the scene in the frequency distribution, and determining whether a spacing between the first peak and the second peak exceeds an intensity threshold.

[0031] Background radiation in a scene is typically confined to a certain interval within the frequency distribution (the absolute position depends on the absolute temperature) and therefore, like a hot object, produces a second peak in the frequency distribution (hereinafter referred to as the "background peak"). Therefore, comparing the interval of the hot object peak (first peak) and the background peak (second peak) to an intensity threshold provides a convenient and computationally efficient way to determine whether to perform ghost image suppression. It is noted here that the terms "first" and "second" are merely labels introduced for ease of reference to the corresponding peaks and do not indicate any order or significance of the peaks. In practice, the first peak (hot object peak) will be found at a higher pixel intensity than the other / second peak (the background peak).

[0032] In some embodiments, the thermal image includes raw thermal image data, and the frequency distribution is the frequency distribution of pixel intensities of the raw thermal image data. Therefore, the frequency distribution can represent the pixel intensities of the raw thermal image data before nonlinearization. Nonlinearization of the raw thermal image data (interchangeably referred to as the "raw signal") captured from the thermal image sensor can produce transformed image data with a compressed dynamic range, which is more suitable for viewing by a person and is less resource intensive to process further down in the image processing chain. However, a side effect of nonlinearization is a changing distribution of pixel intensities. Therefore, the relationship between the thermal object peak and the thermal background in the frequency distribution calculated from the nonlinearized image data may deviate from the actual relative temperature within the scene. Therefore, by processing the frequency distribution of the pixel intensities of the raw thermal image data, ghost image detection can be performed earlier in the processing chain before such distortion is introduced into the thermal data. The method may include receiving a thermal image including raw thermal image data from a thermal camera, and calculating a frequency distribution from the pixel intensities of the raw thermal image data.

[0033] In some embodiments, the method further comprises: obtaining a noise level estimate for the thermal camera; and determining the intensity threshold based on the noise level estimate. The noise level of the thermal camera sets a lower limit below which the ghost image may be indistinguishable from the thermal background. Thus, the noise level of the thermal camera represents a conservative basis for determining the intensity / ghost image rejection threshold.

[0034] In some embodiments, the noise level estimate of the thermal camera is derived from a noise threshold of a noise filter of the thermal camera or is estimated based on a set of auxiliary thermal images of the scene acquired by the thermal camera. Thus, during the thermal monitoring process, the intensity threshold can be set dynamically and thus adapted to the actual conditions in the scene. The intensity threshold can be determined in particular based on the noise level estimate and the estimated average pixel intensity of the scene, for example based on a ratio of the noise level estimate to the estimated average pixel intensity.

[0035] In some embodiments, the intensity threshold is a predetermined intensity threshold (ie, the value of the intensity threshold is predetermined). The intensity threshold may, for example, be set to correspond to a temperature difference between the thermal object and the thermal background, above which strong ghost images are known or expected to appear.

[0036] In some embodiments, the method further includes estimating locations of thermal object pixels in the thermal image depicting the thermal object by identifying pixels in the thermal image corresponding to peaks in the frequency distribution caused by the thermal object;

[0037] Therein, the positions of phantom pixels in the thermal image are estimated based on the estimated positions of thermal object pixels.

[0038] After identifying the thermal object peak, the thermal object pixels in the thermal image that cause the thermal object peak can be identified as pixels having an intensity corresponding to the peak (e.g., falling within the peak). Knowing the imaging properties of the thermal camera (e.g., obtained by characterizing the thermal camera prior to deployment), the locations where the phantom pixels are expected to appear can be estimated from the locations of the thermal object pixels. The (estimated) thermal object pixels depict the thermal object, thereby forming a thermal object image. The thermal object pixels can therefore be interchangeably referred to as pixels of the (estimated) thermal object image.

[0039] In some embodiments, a thermal object pixel is a pixel in the thermal image that belongs to a set of at least a threshold number of neighboring pixels and has an intensity that falls within a peak in a frequency distribution caused by a thermal object. Thus, the position estimate can be made less sensitive to the presence of a single pixel or a small cluster of pixels having an intensity within the thermal object peak that is too small to trigger inhibition or is not caused by a thermal object.

[0040] In some embodiments, the positions of the phantom pixels are estimated using a predefined transformation that maps the positions of the thermal object pixels to the estimated positions of the phantom pixels. The mapping can be based on or represent the imaging properties of the optical system of the thermal camera. Estimating the positions of the phantoms using a predetermined transformation function enables a computationally efficient implementation, thereby avoiding the need for more complex image analysis and data fitting methods.

[0041] In some embodiments, the mapping is such that the phantom pixel corresponds to a point reflection of the thermal object pixel. That is, the position of the phantom in the thermal image can be estimated by inverting the estimated position of the thermal object pixel relative to the point in the thermal image. This type of transformation function is able to find the position of the phantom when the phantom is due to internal reflections involving the thermal image sensor. Reflections from the thermal image sensor itself can be a major contributor to the phantom and tend to result in a phantom in the form of a well-focused inverted (i.e., mirror image) image of the thermal object (albeit with reduced intensity). By defining a mapping such that the phantom pixel corresponds to a point reflection of the thermal object pixel relative to the center pixel of the thermal image, a reliable estimate of the position of the phantom pixel in a large number of scenes can be achieved.

[0042] In some implementation examples, estimating the location of the thermal object pixels includes determining pixel coordinates of each thermal object pixel, and wherein estimating the location of the phantom pixels includes mapping the pixel coordinates of each thermal object pixel to the pixel coordinates of the corresponding phantom pixel using a predefined transformation. Thus, each thermal object pixel can be mapped or associated to a spatially corresponding phantom pixel.

[0043] In some implementations, estimating the location of the thermal object pixel includes determining pixel coordinates of an enclosure of the thermal object pixel, and

[0044] Therein, estimating the position of the phantom pixel comprises mapping pixel coordinates of the enclosure of the thermal object pixels to corresponding pixel coordinates of the enclosure of the phantom pixels using a predefined transformation, wherein the phantom pixel is a pixel located within the determined enclosure of the phantom pixels.

[0045] With this approach, the locations of phantom pixels can be estimated by mapping the enclosure of the thermal object pixels using a predefined transformation (e.g., such that the pixel coordinates of the enclosure of the thermal object pixels are mapped to the coordinates of the enclosure of the phantom pixels by point reflection.) This can achieve a more computationally efficient estimation of the locations of the phantom pixels than the expressive mapping of each individual thermal object pixel, since fewer computations are involved.

[0046] In some embodiments, suppressing ghost images in the thermal image includes adjusting the intensity of the ghost pixels by subtracting a fraction of the intensity of the thermal object pixels. Additional intensity added to the ghost pixels due to internal reflections of the thermal object can thus be subtracted. Advantageously, in embodiments where the thermal image includes raw thermal image data, suppression can be applied to the intensity of the ghost pixels in the raw thermal image data.

[0047] In some embodiments, the fraction is a predetermined value. This enables a simple and computationally efficient implementation of ghost image suppression. Suppression can still be effective because it can be expected that the intensity of each ghost image pixel can be attenuated to the same extent, regardless of the intensity of its corresponding hot object pixel.

[0048] In some embodiments, the hot object pixel intensity is the intensity that falls within the peak caused by the hot object. Since the intensity of the peak reflects the intensity of the hot object pixel, this enables effective suppression of ghost images.

[0049] In some embodiments, a fraction of the representative hot object pixel intensity is subtracted from each phantom pixel. Thus, the intensity of each phantom pixel can be attenuated by the same amount. This enables computationally efficient implementation of phantom suppression.

[0050] In some embodiments, the representative thermal object pixel intensity corresponds to the mean, median, or highest frequency intensity of peaks caused by the thermal object. These metrics all represent useful single-value representations of thermal object pixel intensity.

[0051] In some embodiments, a fraction of the intensity of the corresponding spatially corresponding thermal object pixel is subtracted from each phantom pixel. Thus, the amount of intensity attenuation applied to each phantom pixel can be adjusted based on the intensity of its spatially corresponding thermal image pixel. This can allow for effective phantom suppression even in the presence of relatively broad thermal object peaks (corresponding to spatially varying temperatures of the thermal objects). These embodiments can be advantageously used in conjunction with embodiments that include determining pixel coordinates for each thermal object pixel, wherein the location of the phantom pixel is estimated by mapping the pixel coordinates of each thermal object pixel to the pixel coordinates of the corresponding phantom pixel using a predefined transformation, wherein a one-to-one mapping between thermal object pixels and phantom pixels can be established. In this case, the "spatially corresponding thermal object pixel" for each phantom pixel can refer to the thermal image pixel that maps to the phantom pixel.

[0052] According to a second aspect, a method for thermal imaging is provided, comprising: acquiring a thermal image of a scene by a thermal camera; and processing the thermal image using a method according to the first aspect or any embodiment thereof.

[0053] According to a third aspect, there is provided a computer program product comprising computer program code portions configured to, when executed by a processing device, perform the method according to the first aspect or any embodiment thereof.

[0054] According to a fourth aspect, there is provided a thermal camera comprising a processing device configured to perform the method according to the first aspect or any embodiment thereof.

[0055] The second, third and fourth aspects are characterized by the same or equivalent benefits as the invention according to the first aspect.Any functionality described with respect to the first aspect may have a corresponding feature in the system and vice versa. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] This and other aspects of the present invention will now be described in more detail, with reference to the appended drawings showing embodiments of the invention.

[0057] Figure 1 is a schematic diagram of an implementation of a thermal camera.

[0058] Figure 2a to Figure 2c Thermal images of a scene when there is no hot object in the scene and when there is a hot object in the scene are schematically shown respectively.

[0059] Figure 3a to Figure 3c Example frequency distributions of thermal images captured for different scene conditions are schematically shown.

[0060] Figure 4 A block diagram showing an implementation of an image processing pipeline for a thermal camera.

[0061] Figure 5 is a flow chart of a method for suppressing ghost images of hot objects in a thermal image.

[0062] Figure 6 A method for estimating the location of phantom pixels is depicted.

[0063] Figure 7 Another method for estimating the location of ghost pixels is depicted. DETAILED DESCRIPTION

[0064] Figure 1 An exemplary embodiment of a thermal camera 1 is schematically shown, the thermal camera 1 includes a thermal image sensor 14, which may be of a conventional type, such as a microbolometer sensor. A microbolometer sensor may effectively detect IR radiation in the range of about 7 to 14 μm. Microbolometer sensors are commercially available with various resolutions, such as 160×120 pixels, 1024×1024 pixels, and larger. In the example shown, the image sensor 14 forms part of a sensor package 10, which also includes a reflector 12 arranged behind the image sensor 14 and a sensor window 16 arranged in front of the image sensor 14. The reflector 12 may increase the effective fill factor of the image sensor 14 and may be, for example, a λ / 4 reflector of Au. In some implementations, the sensor package 10 may be a vacuum package, wherein a window 16 (which may be formed of Si, for example) may be arranged as a vacuum seal for the sensor package 10.

[0065] The thermal camera 1 also includes an optical system 18 and a cover 20. Figure 1 , for simplicity, the optical system 18 is shown as a single lens, but may generally include multiple beam forming optical elements, such as lenses and optical filters. The cover 20 is formed of a material that is transparent to IR radiation in the wavelength range of interest. An example material for layer 20 is Ge. The optical elements as well as the cover 20 may also be provided with an anti-reflection (AR) coating to reduce internal reflections within the thermal camera 1. An example of a conventional AR coating is a diamond-like coating (DLC). However, internal reflections may still occur in the thermal camera 1. Typical sources of strong internal reflections include between the cover 20 and the reflector 12 and / or the image sensor 14 and / or the window 16, and if the thermal camera 1 receives IR radiation of sufficient intensity, these internal reflections may produce ghosting in the thermal image obtained by the thermal camera 1. This is particularly useful in Figure 1 Schematically shown in FIG. 1 , where reference numeral 22 denotes a thermal object in the scene, and reference numeral 24 denotes a virtual phantom 24 of the thermal object 22 caused by internal reflection of radiation from the thermal object 22, in this example between the reflector 12 and the cover 20, for reflections between the reflector 12 and the cover 20, as shown in FIG. Figure 1As shown, the phantom image 24 will be formed as an attenuated and well-focused inverted image of the thermal object 22, typically relative to the center pixel of the thermal image (because the relevant reflective surfaces are typically parallel in current thermal cameras). This also applies to reflections between the cover 20 and the image sensor 14 and between the cover 20 and the window 16, which typically have a very small separation (several orders of magnitude smaller than the distance to the cover 20). For the purposes of the following discussion, it is therefore not important which of these elements 12, 14, 16 contributes to the phantom image formation. That is, the internal reflections between the cover 20 and the surfaces of each of the elements 14, 16, 20 actually tend to contribute to the same phantom image.

[0066] Figure 2a to Figure 2c Schematically illustrated are thermal images 32, 33, and 34 acquired from a scene by a thermal camera (e.g., thermal camera 1). Thermal images 32 to 34 depict the same scene and include a plurality of physical objects 321, 322, 323, which are shown as respective basic geometric shapes for simplicity. As can be appreciated, the type, shape, and number of objects in the scene will vary depending on the specific monitoring application and the environment in which the thermal camera is installed. For example, a thermal camera can be installed in an interior environment (such as in a residential, commercial, or public building) or an outdoor environment (such as in a parking lot, on a street or intersection, in a park, in a field, or in a forest) to allow manual or automatic thermal imaging of objects occurring in the environment. A non-limiting example application is thermal imaging for early fire detection (manual or automatic). Another non-limiting example is thermal imaging of a metallurgical or chemical process involving, for example, a heating or heat treatment step performed in an oven, where the oven door may be alternately opened or closed, or where the field of view may be occasionally blocked by a cold object. Another non-limiting example is monitoring a refinery scenario where exit flames may appear and cause ghost images in the thermal image. Yet another non-limiting example is monitoring heat generating components of an engine or motor, such as at a power station.

[0067] Figure 2a The thermal image 32 in FIG. 3 depicts a scene when none of the objects 321 to 323 qualifies as a "hot object" and therefore has substantially the same temperature as the thermal background. In other words, the objects 321 to 323 (or more precisely, the IR radiation received from the objects 321 to 323) form part of the thermal background. Figure 3a Shows Figure 2a , where N represents the number of pixels with intensity I (in absolute or relative terms). In the case of a thermal imaging calibrated camera, the horizontal axis can be equivalently expressed in radiometric degrees (W·sr -1 ·m -1) or temperature (°C or K). As shown in the figure, the frequency distribution in this case includes only a single peak B (interchangeably referred to as "second peak" or "background peak") caused by the thermal background. The width of peak B is proportional to the change in the intensity (or radiation or temperature) of the thermal background. For clarity of illustration, the background peak B is depicted with a simple and regular shape. However, it should be noted that the specific shape of the background peak B can vary depending on the specific distribution and spread of the temperature of the thermal background, and can therefore present a more irregular shape. This applies accordingly to the additional peaks discussed below.

[0068] Figure 2b The thermal image 33 in FIG. 3 depicts a scene when the temperature of the object 322 increases above the thermal background (as an example, the thermal object 322 may be approximately 5 to 20 K hotter than the thermal background). Therefore, the pixels imaging the object 322 receive a greater amount of IR radiation (greater radiant flux) than the pixels imaging the thermal background (including the objects 321, 323). Figure 3b An example of a corresponding frequency distribution of pixel intensities of a thermal image 33 is shown, the frequency distribution including a peak H (interchangeably referred to as a "first peak" or "thermal object peak") caused by a thermal object 322 in addition to a background / second peak B caused by a thermal background. The temperature difference between the thermal object 322 and the thermal background translates into a spacing d between the background peak B and the thermal object peak H. As described above, internal reflections of radiation from the thermal object 322 in the thermal camera (e.g., between the cover 20 and the reflector 12 and / or the image sensor 14 and / or the window 16 of the thermal camera 1) may result in a phantom image 324 appearing in the thermal camera 33, however, due to the small temperature difference between the thermal object 322 and the thermal background assumed in this example, the phantom image 324 may be substantially invisible in the thermal camera 33, or only barely visible (as schematically indicated by the dashed outline). This is also reflected in the frequency distribution, showing that the phantom image 324 produces a small third peak G ("phantom peak") that is almost completely covered and hidden by the background peak B. NETD is a measure of how well an image sensor can distinguish small differences in thermal radiation in an image.

[0069] Figure 2c The thermal image 34 in depicts the scene when the temperature of the object 322 increases further above the thermal background (as an example, the thermal object 322 may be at least 25K hotter than the thermal background in this example). Figure 3c An example of a corresponding frequency distribution of pixel intensities of a thermal image 34 is shown, and its frequency distribution, such as Figure 3b The frequency distribution of , in addition to the background peak B, also includes the thermal object peak H caused by the thermal object 322. However, due to the large temperature difference in this example, the interval d between the background peak B and the thermal object peak H is greater than Figure 3bThe temperature difference between the assumed hot object 322 and the thermal background in this example causes internal reflections in the thermal camera to produce a strongly appearing ghost image 324 (as schematically indicated by the dashed outline) in the thermal image 34. This is also manifested in the frequency distribution, showing that the ghost image 324 produces a ghost peak G which is now clearly distinguishable from the background peak B.

[0070] like Figure 2c As shown in , phantom image 324 may, for example, just overlap with another object of interest in thermal image 34, such as object 323 (which belongs to the thermal background). Figure 2c An operator viewing thermal image 34, or an automated surveillance algorithm processing thermal image 34 using an image analysis algorithm, may confuse phantom image 324 for thermal background and / or the elevated and possibly abnormal temperature of object 324. Depending on the particular surveillance application, this may trigger a false alarm, cause problems for object tracking algorithms, or at least make thermal image 34 more difficult to interpret and, therefore, reduce the reliability of thermal surveillance.

[0071] As can be appreciated in light of the above, the need to subject thermal images to ghost image suppression depends on the temperature dynamics of the depicted scene. Figure 2a to Figure 2b as well as Figure 3a to Figure 3b In the example, the radiation from the scene does not cause any ghost images in the thermal image, which facilitates the implementation of ghost image suppression. Figure 2c and 3c Therefore, the following disclosure provides example implementations of methods that incorporate processing of frequency distributions to determine whether to perform phantom suppression.

[0072] Figure 4 is a block diagram of an example of an image processing system 26 in which embodiments of the present invention may be implemented. The image processing system 26 includes an image sensor 14, a processing device 28, and a downstream image processing pipeline 30. The image processing system 26 may be included in a thermal imaging camera, such as Figure 1 Thermal Camera 1.

[0073] Image sensor 14 (e.g., a microbolometer sensor) acquires a thermal image having pixel intensities that depend on the amount of IR radiation received from the scene monitored by the thermal camera. The thermal image output by image sensor 14 includes raw thermal image data that includes pixel intensities that have not been nonlinearized.

[0074] The thermal image including raw thermal image data is received for processing by the processing device 28 as will be explained below. The processing device 28 may forward the processed thermal image that has been subjected to ghost suppression by the processing device 28 or the unprocessed thermal image that has not been subjected to ghost suppression by the processing device 28 to the downstream image processing pipeline 30.

[0075] The downstream image processing pipeline 30 may implement several conventional sequential processing steps that may be used, for example, to enhance and compress the thermal image data. Examples of such processing steps include noise reduction, global / local detail enhancement, sharpening, etc. Specifically, the image processing pipeline 30 may implement nonlinearization of the raw thermal image data to produce a nonlinearized thermal image that is more suitable for human viewing than the pixels of the raw thermal image data.

[0076] The processing performed by the processing device 28 and the image processing pipeline 30 can be implemented in both hardware and software. In a hardware implementation, each method step described herein (e.g., acquisition of frequency distribution, threshold comparison, phantom suppression, nonlinearization, image enhancement) can be implemented in a dedicated circuit. The circuit can be in the form of one or more integrated circuits, such as one or more application-specific integrated circuits (ASICs) or one or more field programmable gate arrays (FGPAs). In a software implementation, the circuit can alternatively take the form of a processor, such as a central processing unit or a graphics processing unit, which is associated with computer code instructions stored on a (non-temporary) computer-readable medium (such as a non-volatile memory) so that the processing device 28 and / or the image processing pipeline 30 perform the corresponding processing steps. Examples of non-volatile memory include read-only memory, flash memory, ferroelectric RAM, magnetic computer storage devices, optical disks, etc. It should be understood that there can also be a combination of hardware and software implementations, which means that some method steps can be implemented in dedicated circuits, while others can be implemented in software.

[0077] Now further reference will be made to Figure 5 An embodiment of the thermal image processing performed by the processing device 28 is described with reference to a flowchart of FIG.

[0078] In step S1, the processing device 28 obtains a thermal image depicting a scene and acquired by a thermal camera such as thermal camera 1, which is received from the thermal image sensor 14 and includes raw thermal image data. The pixel intensity of the raw thermal image data of the microbolometer image sensor may, for example, have a dynamic range of 14 bits or more, such as 18 bits or 21 bits.

[0079] In step S2, the processing device 28 obtains a frequency distribution based on the pixel intensity of the thermal image, in particular the pixel intensity of the raw thermal image data. In some implementations, the processing device 28 may calculate the frequency distribution of the pixel intensity of the thermal image received from the thermal image sensor 14, and in some implementations, the thermal image sensor 14 may have provided the frequency distribution as well as the thermal image. In this case, the processing device 28 may obtain the frequency distribution by receiving the frequency distribution from the thermal image sensor 14, and in either case, the frequency distribution may be advantageously defined for a plurality of pixel intensity bins, dividing the dynamic range of the thermal image. The number and width of the intensity bins may vary depending on the computational resources of the processing device 28 and the required accuracy of the frequency distribution analysis. Although a binned frequency distribution may reduce the computational resources required for the method, the use of an unbinned frequency distribution is not excluded.

[0080] In step S3, processing device 28 processes the frequency distribution obtained in step S2 to determine whether the frequency distribution contains a thermal object peak whose interval with the thermal background of the scene is greater than an intensity threshold.

[0081] In some implementations, processing device 28 may process the frequency distribution to detect a first peak caused by a hot object and a second peak caused by a thermal background of the scene, and determine whether a separation between the first peak and the second peak exceeds an intensity threshold.

[0082] consider Figure 2a and 3a For example, the scene does not include a hot object, and the processing device 28 may only detect the background peak B in the frequency distribution. In this case, step S3 will evaluate to false. The processing device 28 may therefore detect the background peak B in the frequency distribution according to Figure 5 The "false" branch of continues, and thus the thermal image (including raw thermal image data) is output at step S6 without subjecting the thermal image to any ghost image suppression. The thermal image and raw thermal image data can therefore be forwarded to the downstream image processing pipeline 30 in an unprocessed form.

[0083] consider Figure 2b and Figure 3b In the example of FIG. 3 , the scene includes a hot object 322, and the processing device 28 can therefore detect a hot object peak H and a background peak B in the frequency distribution. However, in this case, the temperature difference between the hot object 322 and the hot background is such that the interval between the peaks H and B (in Figure 3b denoted by d in the figure) is less than the intensity threshold (in Figure 3b denoted by D in the figure). Therefore, d<D, wherein step S3 is also evaluated as false in this case. Therefore, in this case, the processing device 28 can also be based on Figure 5 The "false" branch proceeds to step S6.

[0084] consider Figure 2c and3c In the example of FIG. 3 , the scene includes a hot object 322, and the processing device 28 can therefore detect a hot object peak H and a background peak B in the frequency distribution. In this case, the temperature difference between the hot object 322 and the thermal background is such that the interval D between the peaks H and B exceeds Figure 3b Therefore, d>D, in which step S3 will evaluate to true. That is, the processing device 28 determines that the frequency distribution includes a hot object peak H separated from the thermal background by a distance greater than the intensity threshold D, in which case the processing device 28 determines according to Figure 5 The "true" branch (including step S4) is performed.

[0085] Identification or detection of peaks in a frequency distribution can be achieved by searching the frequency distribution for intensity values ​​I (or bins) where the frequency values ​​N exceed a peak amplitude threshold (i.e., peak height threshold) for a predefined minimum frequency. The search can be performed from low to high intensity and vice versa.

[0086] Since peaks may span intervals of pixel intensities (particularly in the case of non-binned frequency distributions or frequency distributions with relatively narrow bins), the peak search may apply an additional condition that for a peak to be detected, a peak amplitude threshold should be exceeded over an interval of at least a predetermined number of consecutive pixel intensities (peak width threshold). In this way, identification of a single intensity value peak or a few intensity value peaks as background peaks or hot object peaks may be avoided.

[0087] Since the amplitude of background peak B is generally greater than the amplitude of thermal object peak H, in some implementations, peak detection may use two different peak amplitude thresholds to detect peaks B and H, e.g., a first peak amplitude threshold may be used to detect thermal / first object peak H, and a second peak amplitude threshold greater than the first peak amplitude threshold may be used to detect background / second object peak B, however, a common peak amplitude threshold may also be used to detect background peak B and thermal object peak H. In this case, the peak amplitude threshold may be set high enough to avoid identifying small pseudo-peaks in the frequency distribution (e.g., caused by noise) as thermal object peaks, and low enough to reliably identify peaks due to actual hot objects in the scene.

[0088] Each of the above-mentioned peak amplitude threshold and peak width threshold can be considered as a design parameter. For example, the peak width threshold can be set to a specific percentage of the total number of possible pixel intensities that can be represented in the thermal image (i.e., at the bit depth of the thermal image). As an example, it can be set to 5% of the total number of possible pixel values. As another example, the peak width threshold can be set based on the noise of the thermal camera or image sensor. The peak width threshold can be set based on the NETD, such as the NETD multiplied by a scaling factor (e.g., approximately 2 to 4). It is worth noting that the NETD can depend on both the ambient temperature and the target temperature (i.e., the pixel values ​​in the image). This means that the peak width threshold can depend on the pixel values ​​in the candidate peak area, so that different values ​​are adopted for the relatively low part of the possible pixel value range compared to the relatively high part of the possible pixel value range.

[0089] Accordingly, the peak amplitude threshold can be set according to the pixel values ​​represented in the thermal image (or equivalently, in the frequency distribution). For example, the peak amplitude threshold can be set according to the highest pixel intensity frequency in the thermal image (i.e., the pixel intensity with the highest frequency / occurrence rate in the thermal image). For example, the higher the highest pixel intensity in the thermal image, the higher the peak amplitude threshold (e.g., the highest pixel intensity scaled by a factor between 0 and 1). As another example, the peak amplitude threshold can be set according to the range spanned by the lowest pixel intensity and the highest pixel intensity in the thermal image. The wider the range, the lower the peak amplitude threshold. To illustrate, on the one hand, consider a room temperature scene with a human object depicted in a thermal image. In such a scene, the range of pixel values ​​will be from, for example, 18 to 34°C, and most pixel values ​​within this range will likely be represented in the thermal image. On the other hand, consider a scene with a cold sky, a moderate outdoor temperature, a human object, and a machine that is actually hot (e.g., forming a hot object). In this scene, the pixel values ​​may instead be in the range of 0 to 200°C, but many pixel values ​​within this range will not be represented. The former of these two scenes will require a higher peak amplitude threshold than the latter. It is worth noting that since the peak amplitude threshold may depend on the scene content, ie, the range of pixel values ​​represented in the thermal image, it may vary between sequentially captured thermal images during monitoring of the scene.

[0090] Figure 3a to Figure 3c Depicts the common peak amplitude threshold T P Example. Figure 3a In the example of , a single peak B can be detected and thus identified as background peak B. Figure 3b and Figure 3c In each example, two peaks B and H can be detected, where the lower intensity peak B can be identified as a background peak, and the higher intensity peak H can be identified as a hot object peak. Figure 3c As can be seen in the figure, the peak amplitude threshold T Pis set so that a phantom peak G will not trigger peak detection. It is conceivable that in some cases, the phantom peak G may also exceed the peak amplitude threshold T P , resulting in the detection of three peaks, background peak B, phantom peak G and thermal object peak H. Therefore, peak detection can be implemented to identify the peak at the highest intensity value as thermal object peak H. This is a reliable method because phantom peak G is always found at a lower intensity value than the actual thermal object peak H. An alternative method may be to identify the peak with the largest amplitude of the two peaks G and H as thermal object peak H. Another alternative method is to search for thermal object peak H so as to reduce the intensity, for example from a (predefined) maximum intensity to a lower intensity. In this case, thermal object peak H can be identified as the first peak located in the search. Background peaks can be identified in a similar manner to the first peak located in a search with increasing intensity, for example from a (predefined) minimum intensity to a higher intensity.

[0091] While it is certainly possible to search over the entire intensity range, it may not be necessary to search over the entire intensity range, and the search may be limited to one or a few sub-ranges of intensities of interest. For example, in some environments and applications, the intensity of IR radiation from a thermal background may reasonably be assumed to be no less than a certain value. As a non-limiting example, for scenes in indoor environments, in typical cases the intensity may be expected to correspond to a temperature of at least 15°C, with the search not needing to extend below this temperature.

[0092] Assuming that the peak detection performed by processing device 28 has identified background peak B and hot object peak H, processing device 28 can determine or calculate the separation d between peaks B and H. Figure 3b to Figure 3c In the example of , the interval d is determined as the interval between the individual pixel intensities with the highest frequency (ie, occurrence or count) of each peak. This corresponds to the intensity at the maximum amplitude (ie, maximum height) of the background peak B and the hot object peak H.

[0093] However, the interval d may also be determined in other ways, such as the interval between the respective representative pixel intensities of peaks B and H. The representative pixel intensities may reflect where the "middle" or "center" of the respective peaks B and H is located. For example, the representative pixel intensity of a peak may be one of: the center of an interval defining the lowest and highest pixel intensities of the peak, a statistical mean of the pixel intensities falling within the peak, the mode of the peak (i.e., the most frequent pixel intensity in the peak), the median of the pixel intensities within the peak, the centroid of the peak, or a skewed (weighted) mean of the peak.

[0094] According to yet another example, the processing device 28 does not need to explicitly detect the background peak B, but only needs to detect the thermal object peak H. Instead, the representative intensity of the thermal background can be estimated or derived based on the pixel intensity or frequency distribution of the thermal image. For example, the processing device 28 can determine the interval between the average intensity of the thermal object peak H and the thermal background, which interval is determined based on the pixel intensity of the thermal image. For example, the average intensity over the entire range or sub-range of the pixel intensity of the thermal image (or an auxiliary thermal image of the scene acquired by the thermal camera under the same capture conditions) can be calculated, or equivalently, the frequency distribution (of the thermal image or the auxiliary thermal image) can be calculated. In this case, the processing device 28 can determine whether the frequency distribution includes a thermal object peak H whose interval with the thermal background of the scene exceeds an intensity threshold D, where the interval can be determined as the difference between the representative pixel intensity of the thermal object peak (e.g., the most frequent intensity of the thermal object peak H and the representative intensity of the thermal background). It is even possible to determine the interval between the predetermined representative intensity of the thermal object peak and the thermal background based on, for example, the assumed intensity, radiation or temperature of the thermal background. While this approach may enable a simpler and more computationally efficient implementation of the method, it is envisioned that a non-predetermined (i.e., dynamic) estimate of the thermal background intensity (e.g., derived from a frequency distribution using any of the aforementioned methods) may result in more reliable activation of phantom suppression.

[0095] The intensity threshold D used to determine whether to perform ghost suppression on a thermal image can be a predetermined intensity threshold. The intensity threshold can, for example, be set to correspond to a temperature difference between a hot object and a thermal background, above which ghost images are known or expected to appear. For example, the intensity threshold can be set to correspond to a temperature difference of 25K or greater. A temperature difference of 25K is much higher than the noise level of a typical prior art thermal camera and tends to produce a strong ghost appearance. However, a higher predetermined threshold D is also possible, for example corresponding to a temperature difference of 50K or greater, or 100K or greater. A higher intensity threshold D translates into less likely to trigger ghost suppression.

[0096] A specific value for the intensity threshold D may generally be established taking into account the characteristics of the thermal camera and / or with regard to how easily ghost image suppression should be triggered and, conversely, how tolerable ghost images may be in the intended application.

[0097] For example, a complete characterization of the thermal camera can be performed (e.g., before the thermal camera is installed in the surveillance environment). The characterization can take into account parameters such as sensor noise, reflectance coefficients of surfaces that produce internal reflections (e.g., covers and reflectors and / or image sensors), reflection and transmission coefficients of optical elements of the optical system of the thermal camera, etc. Thus, for example, by calculation and / or simulation, an intensity difference at which a hot object tends to cause any noticeable phantom can be established. The intensity difference can also be experimentally established by analyzing thermal images acquired by a thermal camera of a test scene under various temperature conditions and thermal images of various object positions in the scene. In either case, the established intensity difference can be used directly as an intensity threshold D, or the intensity threshold D can be set based on the established intensity difference (e.g., by applying a specific factor to scale).

[0098] According to another method, the value of the intensity threshold D can be determined by scaling the noise level estimate of the image sensor based on the f-number of the optical system of the thermal camera. The sensor noise estimate (e.g., according to the standard deviation) and the f-number are generally easily available from the manufacturer or supplier of the thermal sensor and the thermal camera, respectively. By scaling, a representative estimate of the noise level of the thermal camera can be obtained. More specifically, the noise level estimate of the sensor can be scaled by the f-number squared. The noise level of the thermal camera can correspond to a lower limit below which the phantom image can not be distinguished from the noise in the thermal background. Therefore, the noise level of the thermal camera represents a conservative basis for determining the intensity / phantom suppression threshold. For example, the noise level estimate of the image sensor can be given according to the standard deviation, where the noise level estimate can be obtained by scaling the noise level estimate by the f-number squared to obtain the noise level estimate of the thermal camera (according to the standard deviation), and then the standard deviation of the noise of the thermal camera is scaled by a factor (e.g., 3, where the threshold D corresponds to 3σ of the noise level estimate of the thermal camera) to determine the intensity threshold D. The noise level estimate of the image sensor can also be given according to the NETD, which can be converted into the NETD of the thermal camera by scaling the NETD of the image sensor by the f-number squared. This may also represent a good starting point for determining the value of the intensity threshold D.

[0099] According to some implementations, the intensity threshold D need not be predetermined, but can be set dynamically during operation. The processing device 28 can obtain a noise level estimate of the thermal camera and determine the intensity threshold D based on the noise level estimate. The noise level estimate of the thermal camera can be derived from the noise threshold of the noise filter of the thermal camera. Figure 4As shown in , for example, a noise filter 27 may be implemented in the image processing system 26 between the image sensor 14 and the processing device 28. As known per se in the art, such a noise filter has a threshold value (which may vary depending on the capture mode of the thermal camera) above which the signal read out from the pixels of the image sensor 14 is considered to be real and not noise. The processing device 28 may thus derive an estimate of the noise level of the thermal camera by obtaining (e.g. reading) the noise threshold value from the noise filter 27. The thermal camera may have a temporal noise filter (with a threshold value set based on a sequence of acquired thermal images or frames) and a spatial noise filter (with a threshold value set based on a single current thermal image), which may collectively define a spatio-temporal noise filter 27. The processing device 28 may in this case derive an estimate of the noise level of the camera based on the temporal and spatial noise threshold values ​​of the spatio-temporal noise filter 27. Alternatively, the estimate of the noise level of the thermal camera may be obtained by estimating the noise level based on a set of auxiliary thermal images of the scene acquired by the thermal camera, which set of auxiliary thermal images is captured before capturing the thermal image to be processed and which may be subjected to ghost image suppression by the processing device 28. In either case, the intensity threshold D may then be determined using the obtained noise level estimate. For example, the noise level estimate may be scaled by a predefined factor whose value is based on a priori knowledge of how the noise level estimate affects sensitivity to phantoms. Alternatively, the intensity threshold D may be determined using the obtained noise level estimate and the estimated average pixel intensity of the scene. The average pixel intensity may be estimated by the processing device 28, for example, by calculating the average pixel intensity averaged over the full range or sub-range of the pixel intensity of the thermal image (or an auxiliary thermal image of the scene acquired by the thermal camera under the same capture conditions) or equivalently (of the thermal image or auxiliary thermal image) frequency distribution. For example, the processing device 28 may determine the intensity threshold D as the ratio of the noise level estimate (e.g., the noise threshold) to the average pixel intensity (equivalent to 1 / SNR) multiplied by a scaling factor. The relevance of this method may be better understood by considering that 1 / SNR is substantially proportional to NETD.

[0100] Reference again Figure 5 , if it is determined in step S3 that d>D, the method proceeds to the "true" branch, where the processing device 28 performs processing of the thermal image in step S4 to suppress ghost images of hot objects in the thermal image. In other words, if d>D, the processing device 28 subjects the thermal image to ghost image suppression. The processing of ghost image suppression includes sub-step S41 (wherein the processing device 28 estimates the position of ghost image pixels that form the ghost image), and sub-step S42 (wherein the processing device 28 suppresses ghost images in the thermal image by adjusting the intensity of the ghost image pixels). After processing the thermal image to suppress ghost images, the method can continue to output the processed thermal image to the downstream image processing pipeline 30 in step S5.

[0101] Various implementations of sub-steps S41 and S42 are possible. In some implementations, a "static" approach may be employed, in which the locations of phantom pixels may be estimated based on a priori knowledge of scene locations where thermal objects may appear and locations in the thermal image where phantoms will appear when thermal objects appear at the scene locations. For example, this approach may be useful when thermal monitoring includes a scene of a single stationary object of interest. Non-limiting examples include scenes containing a single known heat-generating component in a substation, a machine that is prone to overheating in a factory, and flammable products in a monitored space of a building. In another static approach, which is appropriate when monitoring a scene including multiple stationary objects of different sizes and / or different IR radiation characteristics, a corresponding predetermined mapping may be established between the amplitude, position, and / or shape of a corresponding thermal object peak caused by each object when hot and a corresponding location in the thermal image at which a phantom will appear when the corresponding thermal object appears. Thus, a first thermal object peak corresponding to a first thermal object may be mapped to a first set of phantom pixels in the thermal image, a second thermal object peak corresponding to a second thermal object may be mapped to a second set of phantom pixels in the thermal image, and so on.

[0102] In both static methods described above, in sub-step S42, the processing device 28 may adjust the intensity of the phantom pixel based on the representative pixel intensity of the (corresponding) thermal object. The processing device 28 may adjust the intensity of each phantom pixel, for example, by subtracting a fraction of the representative intensity of the thermal object. The representative pixel intensity of the thermal object may, for example, be the same as the representative pixel intensity of the thermal object peak H, as described above in conjunction with Figure 3a to Figure 3c The score may be a predetermined value between 0 and 1 based on the known reflectivity of the surface that produces the internal reflection.

[0103] In many cases, it may be inconvenient or even impossible to establish any predetermined mapping between the locations of the appearing hot objects or hot object peaks and the phantom pixels. For example, the location of the object of interest may not be known a priori. Furthermore, the amplitude, location and / or shape of the hot object peaks corresponding to different objects may be unknown or indistinguishable. Therefore, reference Figure 6 , implementation of sub-steps S41 and S42 allows for suppression of ghost images in thermal images depicting scenes including arbitrary and possibly non-stationary configurations of objects of interest.

[0104] refer to Figure 6(representing thermal image 34 in which some of pixels 120, 121, 122, 124 are individually shown), if it is determined at step S3 that the frequency distribution includes a thermal object peak H that is spaced from the thermal background (e.g., background peak B) by an amount d that exceeds an intensity threshold D, then processing device 28 may proceed by estimating the location of thermal object pixels 122 (pixels having a diagonal fill pattern) depicted in thermal image 34, which may be located by identifying pixels in thermal image 34 that correspond to thermal object peak H. That is, the thermal object pixels may be assumed to be pixels in thermal image 34 that contribute to the thermal object peak.

[0105] One method for identifying thermal object pixels in thermal image 34 is to perform a scan (row or column based) through the pixels of thermal image 34 and identify each pixel having an intensity that falls within thermal object peak H as thermal object pixel 122. Another method for identifying thermal object pixels in thermal image 34 is to define a thermal object pixel threshold having a value that is set to the minimum intensity of thermal object peak H. A scan may then be performed on the pixels of thermal image 34, wherein each pixel having an intensity that meets or exceeds the thermal object pixel threshold may be identified as a thermal object pixel 122. These represent only two possible example methods, and any other method that allows for finding a correspondence between a thermal object peak H and pixels that contribute to the thermal object peak H may be used.

[0106] To avoid identifying thermal object pixels from single pixels or small clusters corresponding to thermal objects that are too small to cause any interfering phantom images, or from false pixel readings, or from stuck saturated pixels at random locations in the sensor array, processing device 28 may implement spatial filtering in thermal object pixel identification. Processing device 28 may therefore impose the condition that a thermal object pixel is a collective pixel in the thermal image that belongs to at least a threshold number of neighboring pixels other than the pixel corresponding to the thermal object peak H.

[0107] exist Figure 6 , a group of adjacent thermal object pixels 122 that form a thermal object image 22 depicting a thermal object has been identified. The thermal object pixels 122 of the thermal object image 22 are surrounded by non-thermal object pixels 120 (i.e., pixels that are not identified as corresponding to a thermal object peak H). The positions (i.e., pixel coordinates) of the thermal object pixels 122 have therefore been estimated.

[0108] After estimating the location of thermal object pixels 122 (i.e., the pixel coordinates of thermal object pixels 122), the location (i.e., the pixel coordinates) at which corresponding phantom pixels 124 of thermal image 34 are expected to appear can be estimated. Figure 6 , (estimated) phantom pixels 124 are indicated by a diamond fill pattern. The set of phantom pixels 124 forms the phantom 24.

[0109] Processing device 28 may estimate the position (pixel coordinates) of phantom pixel 124 using a predefined transformation M that maps the estimated position (x', y') of thermal object pixel 122 to the estimated position (x, y) of phantom pixel 124. In mathematical terms, M: (x', y')→(x, y).

[0110] The transformation M may reflect the imaging properties of the optical system of the thermal camera. For example, an experiment may be performed in which a thermal object is introduced at various locations of a test scene, and the resulting spatial relationship between thermal object pixels and phantom pixels is established and captured in the transformation M. The transformation M may also be determined by numerical methods. For example, since all optical path lengths, material surface reflections and optical refractions, and distances between optical components within the thermal camera are known, the reflected phantom pixels may be found by ray tracing from the location of the thermal object pixels.

[0111] As described above, reflections from the thermal image sensor itself can be a major contributor to phantoms and often result in phantoms in the form of a reduced intensity image of a well-focused inversion of the thermal object. By defining a mapping of a predetermined transformation M such that phantom pixel 124 corresponds to a point reflection of thermal object pixel 122 relative to center pixel 121 of thermal image 34, a reliable estimate of the position of phantom pixel 124 can be obtained. Thus, the predetermined transformation M can map the pixel coordinates of each thermal object pixel 122 to the pixel coordinates of the corresponding phantom pixel 124. Thus, a spatial correspondence between each thermal object pixel 122 and the corresponding phantom pixel 124 can be established. The transformation M can define a bijective mapping M: (x', y') → (x, y) between the thermal object pixel coordinates (x', y') and the phantom pixel coordinates (x, y). That is, a one-to-one correspondence can be established between thermal object pixel 122 and phantom pixel 124. Although in Figure 6 In , the point reflection of the transformation M is made relative to the center pixel 121, but other references for the reflection are possible. For example, internal reflections may cause phantoms corresponding to a center shift (offset) inversion of the thermal object image. In this case, the transformation M can define a point reflection about an off-center pixel of the thermal image 34.

[0112] After estimating the position of the phantom pixel 124, the processing device 28 may (e.g., Figure 5 Step S42) continues to suppress ghost images 24 in thermal image 34 by adjusting the intensity of ghost pixels 124. The intensity of ghost pixels 124 can be adjusted by subtracting a fraction of the thermal object pixel intensity. Generally, the thermal object pixel intensity can be the intensity that falls within the thermal object peak H. The thermal object pixel intensity can be determined in a variety of ways, as will be stated below. As with the frequency distribution, ghost suppression can advantageously be based on the raw thermal image data pixel intensity.

[0113] According to one approach, the representative hot object pixel intensity I can be subtracted from each phantom pixel. h The fraction r of the original intensity I g The adjusted intensity of the phantom pixel 124 at the pixel coordinate (x, y) of Therefore, it can be determined that:

[0114]

[0115] Thus, the intensity of each phantom pixel may decay by the same amount. Thus, in a sense, the pixel intensity of phantom pixel 124 may be "restored" to be closer to the intensity that the pixel would have had if the hot object was not present in the scene. Representative hot object pixel intensity I h For example, it can be combined with (as above Figure 3a to Figure 3c The fraction r may be a predetermined value between 0 and 1 based on the known reflectivity of the surface that produces the internal reflection.

[0116] As described above, if the transformation provides a bijective mapping M: (x', y')→(x, y), a one-to-one correspondence between hot object pixels 122 and phantom pixels 124 can be established. Thus, instead of attenuating each phantom pixel by the same amount (i.e., I h *r), can be in the original strength I g The adjusted intensity of the phantom pixel 124 at the pixel coordinate (x, y) Determined as:

[0117]

[0118] This approach may provide more accurate phantom suppression because each phantom pixel 124 may be attenuated based on the intensity of its one-to-one corresponding hot object pixel 122. Thus, the amount of attenuation applied to each phantom pixel 124 may better reflect the actual amount of additional intensity added to the phantom pixel 124 due to internal reflection of radiation from the hot object.

[0119] According to another example, variable intensity attenuation of phantom pixel 124 may also be provided for a transformation M that provides a non-bijective mapping from (x', y') to (x, y). For example, the non-bijective mapping may map two or more thermal object pixel coordinates (x', y') to the same phantom pixel coordinates (x, y) (e.g., meaning that the imaging properties of the optical system of the thermal camera cause the phantom to be reduced and / or distorted relative to the thermal object image). Thus, the transformation still establishes a spatial correspondence between thermal object pixels 122 and phantom pixels 124, however, the spatial correspondence is not a one-to-one correspondence, but a many-to-one correspondence. In this case, the intensity of each phantom pixel may be attenuated by a fraction of the corresponding representative thermal object pixel intensity, where the representative thermal object pixel intensity is based on the pixel intensities of the two or more spatially corresponding thermal object pixels. The representative thermal object pixel intensity may, for example, be a function of the pixel intensities of the two or more spatially corresponding thermal object pixels, such as their average, median, minimum, or maximum intensity.

[0120] Another example of a variable intensity attenuation for a phantom pixel 124 is based on Equation 1 and the (single) representative hot object pixel intensity, but where the fraction is a function of the position of the phantom pixel, i.e.,

[0121]

[0122] For example, r(x,y) may decrease as a function of the distance from the centroid of the phantom pixel 124 .

[0123] Figure 7 Another method for estimating the locations of hot object pixels 122 and phantom pixels 124 is shown. Figure 6 After the above, the processing device 28 may continue to determine the pixel coordinates of the enclosure of the thermal object pixel 122. Figure 7 , the enclosed pixels are represented as 123 and are highlighted with a dot pattern. This allows the pixel coordinates of enclosure 123 (rather than a collection of pixel coordinates of individual thermal object pixels 122) to be used as an estimate of the position of thermal object pixel 122. The pixel coordinates of enclosure 123 can be represented as absolute pixel coordinates. The pixel coordinates of enclosure 123 can also be represented as relative pixel coordinates relative to a single representative pixel coordinate of thermal object pixel 122. The representative pixel coordinates can, for example, be the pixel coordinates of a selected pixel of enclosure 123. Another convenient choice is the pixel coordinates of the centroid of thermal object pixel 122. In these cases, the estimated position of thermal object pixel 122 can be represented by the relative pixel coordinates of enclosure 123 and the representative pixel coordinates. Figure 7 , reference numeral 122a denotes an example of a pixel at a representative pixel coordinate.

[0124] After estimating the location of the thermal object pixel 122 (e.g., the pixel coordinates of the enclosure 123), the location (i.e., pixel coordinates) of the corresponding phantom pixel 124 of the thermal image 34 where it is expected to appear can be estimated. The processing device 28 can estimate the location of the phantom pixel 124 by mapping the pixel coordinates of the enclosure 123 to corresponding pixel coordinates of the enclosure for the phantom pixel using a predefined transformation M'. The transformation M' can provide a mapping M' similar to the transformation M: (x', y')→(x, y), where (x', y') represents the pixel coordinates of the enclosure 123, and (x, y) represents the pixel coordinates of the enclosure for the phantom pixel 124. Figure 7 , the pixels enclosed by the phantom pixel 124 are indicated as 125 and are highlighted with a horizontal stripe pattern.

[0125] If enclosure 123 is defined by absolute pixel coordinates, the transformation may perform a direct mapping of the absolute pixel coordinates (x', y') of enclosure 123 to the absolute pixel coordinates (x, y) of enclosure 125. For example, transformation M' may perform a point reflection of the pixel coordinates (x', y') of enclosure 123 (e.g., relative to center pixel 121).

[0126] If enclosure 123 is defined by relative pixel coordinates, the transformation may include first mapping the representative pixel coordinates of thermal object pixel 122 (e.g., the coordinates of pixel 122a) to corresponding representative pixel coordinates of phantom pixel 124 (e.g., the coordinates of pixel 124a), and thereafter performing a second mapping to find the pixel coordinates (x, y) of enclosure 125 from the representative pixel coordinates of phantom pixel 124 and the relative pixel coordinates of enclosure 123. For example, the relative pixel coordinates (x, y) of enclosure 125 may be determined by inverting the relative pixel coordinates (x', y') of enclosure 123 and then adding the inverted relative pixel coordinates (x, y) to the representative pixel coordinates of phantom pixel 124 (e.g., the pixel coordinates of pixel 124a).

[0127] Once enclosure 125 is determined, ghost pixels 124 may be identified as pixels located within enclosure 125. Thereafter, processing device 28 may (e.g., in Figure 5123 and phantom pixels 124. In the case where transformation M' results in enclosure 125 that is consistent with enclosure 123, the coordinates of equation 2 can also be applied by a simple coordinate transformation between the coordinates of thermal object pixels 122 and the coordinates of phantom pixels 124. If transformation M' is shape preserving, but not necessarily congruent, equation 3 can be used to adjust the intensity of phantom pixels 124. Even if there is no simple spatial correspondence between thermal object pixels 122 within enclosure 123 and phantom pixels 124 within enclosure 125, either of equations 1 and 4 can be used.

[0128] Figure 7 An advantage of the method in is that, for a thermal object image 22 comprising a greater number of pixels 122 than their corresponding enclosure 123, applying a transformation to the enclosure M' (and optionally representative pixel coordinates) may allow the position of the phantom 24 to be estimated using fewer computations than applying a corresponding transformation to each enclosed thermal object pixel.

[0129] Those skilled in the art recognize that the present invention is by no means limited to the preferred embodiments described above. On the contrary, many modifications and variations are possible within the scope of the appended claims. For example, although the above discussion defaults to camera installations with fixed fields of view, the methods and approaches disclosed herein may also be used for dynamic installations, where the zoom, pan, and / or tilt of the thermal camera may be changed to allow scanning within a larger scene in the environment. In this case, the method for suppressing phantoms as described above may be applied to a sequence of thermal images acquired by a thermal camera in the process of scanning an entire scene. Thus, the above-described conditional phantom suppression may be selectively applied to a sequence of thermal images.

Claims

1. A method for thermal image processing, the method comprising: obtaining a thermal image captured by a thermal camera and depicting the scene; obtaining a frequency distribution based on pixel intensities of the thermal image; processing the frequency distribution to determine whether the frequency distribution includes a peak caused by a thermal object in the scene, the peak being separated from a thermal background of the scene by more than an intensity threshold; and In response to determining that the frequency distribution includes the peak that is spaced from the thermal background by more than the intensity threshold, processing the thermal image to suppress ghost images of the thermal object in the thermal image, wherein the ghost images are caused by internal reflections of radiation from the thermal object in the thermal camera, and wherein the processing of the thermal image includes: estimating positions of phantom pixels forming the phantom; as well as The ghost images in the thermal image are suppressed by adjusting the intensity of the ghost pixels.

2. The method of claim 1 , wherein processing the frequency distribution comprises: The peak caused by the thermal object and another peak caused by the thermal background of the scene are detected in the frequency distribution, and it is determined whether a spacing between the peak and the another peak exceeds the intensity threshold. 3 . The method of claim 1 , wherein the thermal image comprises raw thermal image data, and the frequency distribution is a frequency distribution of pixel intensities of the raw thermal image data.

4. The method according to claim 1, further comprising: obtaining an estimate of a noise level of the thermal camera; as well as determining the intensity threshold based on the noise level estimate, The noise level estimate of the thermal camera is derived from a noise threshold of a noise filter of the thermal camera, or is estimated based on a set of auxiliary thermal images of the scene acquired by the thermal camera. The method of claim 1 , wherein the intensity threshold is predetermined.

6. The method of claim 1 , further comprising estimating locations of thermal object pixels depicting the thermal object in the thermal image by identifying pixels in the thermal image corresponding to the peaks in the frequency distribution caused by the thermal object; Wherein the position of the phantom pixel in the thermal image is estimated based on the estimated position of the thermal object pixel.

7. The method of claim 6, wherein the thermal object pixel is a pixel in the thermal image that belongs to a set of at least a threshold number of neighboring pixels and has an intensity that falls within the peak in the frequency distribution caused by the thermal object.

8. The method of claim 6, wherein the positions of the phantom pixels are estimated using a predefined transformation that maps the positions of thermal object pixels to the estimated positions of the phantom pixels.

9. The method of claim 8, wherein the mapping is such that the phantom pixels correspond to point reflections of the thermal object pixels.

10. The method of claim 7, wherein estimating the position of the thermal object pixels comprises determining pixel coordinates of each thermal object pixel, and wherein estimating the position of the phantom pixels comprises mapping the pixel coordinates of each thermal object pixel to pixel coordinates of a corresponding phantom pixel using the predefined transformation.

11. The method of claim 7, wherein estimating the position of the thermal object pixel comprises determining pixel coordinates of an enclosure of the thermal object pixel, and Wherein estimating the position of the phantom pixel comprises mapping the pixel coordinates of the enclosure of the thermal object pixels to corresponding pixel coordinates of the enclosure of the phantom pixels using the predefined transformation, wherein the phantom pixel is a pixel located within the determined enclosure of the phantom pixels.

12. The method of claim 1, wherein suppressing the ghost images in the thermal image comprises adjusting the intensity of the ghost image pixels by subtracting a fraction of a hot object pixel intensity.

13. A method for thermal imaging, comprising: Acquire a thermal image of the scene using a thermal camera; as well as The thermal image is processed using the method according to claim 1.

14. A computer program product comprising computer program code portions configured to perform the method according to claim 1 when executed by a processing device.

15. A thermal camera comprising a processing device configured to perform the method according to claim 1.

Images