Method for thermal image processing

Abstract

The method for thermal image processing includes: acquiring a thermal image depicting a scene including a set of objects; identifying a set of distinct object regions in the thermal image, each distinct object region comprising a blurred depiction of a respective one of the set of objects, and each distinct object region being identified as a contiguous region of pixels, a distinct object region having a pixel intensity that differs from a representative background intensity of the thermal background of the scene by more than a threshold intensity and a size that exceeds a threshold size such that the distinct object region includes an actual object region of actual object pixels and a blurred edge region of blurred edge pixels surrounding the actual object region; segmenting the fuzzy edge region into a background edge region and an object edge region between the actual object region and the background edge region; and setting the pixels of the object edge region to a representative object intensity determined according to the actual object pixels of the actual object region and setting the pixels of the background edge region to a representative background intensity.

Description

Technical Field

[0001] The present invention generally relates to a method for thermal image processing and a corresponding thermal imager. Background Art

[0002] Thermal imagers are used in various surveillance applications and enable thermal imaging of a scene as well as remote temperature monitoring, for example for early fire detection and / or detecting overheating of objects in the monitored scene.

[0003] Thermal imagers typically employ image sensors based on arrays of microbolometer sensors. State-of-the-art microbolometer sensors can provide a high signal-to-noise ratio (SNR). However, the dynamic range is typically limited to approximately 100°C, and for sensitive surveillance applications, the dynamic range can even be less than 50°C. Therefore, in surveillance applications involving high radiation intensities, thermal imagers typically need to be purposefully configured to accurately capture the maximum radiation in the scene.

[0004] One method for increasing the dynamic range is to reduce the integration time during image capture. However, this results in a reduction in signal level and, thus, in a reduction in SNR (the signal noise of a microbolometer is essentially independent of the signal). Accordingly, a reduction in integration time enables the measurement of higher radiation levels at the expense of reduced sensitivity. The reduced sensitivity will further result in a loss of contrast in the thermal image, especially when there are no high temperature objects ("hot objects") in the scene, which may make it more difficult to distinguish and track objects manually or using automatic image recognition algorithms. Summary of the invention

[0005] In view of the above, it is an object of the present invention to provide a method for performing thermal image processing that achieves contrast enhancement in thermal images. More specifically, it is an object to achieve contrast enhancement in areas of a thermal image that are not covered by any "thermal objects" (e.g., areas of low contrast in the thermal image). It is a further object to achieve contrast enhancement in a low complexity and computationally efficient manner, which may provide a computationally efficient implementation in a processing device (e.g., an FPGA) of a thermal image processing system.

[0006] Therefore, according to a first aspect of the present invention, a method for performing thermal image processing is provided, the method comprising:

[0007] acquiring a thermal image obtained by an image sensor of a thermal imager, wherein the thermal image depicts a scene including a group of objects;

[0008] identifying a set of distinct object regions in a thermal image;

[0009] wherein each apparent object region comprises a depiction of a corresponding one of the set of objects blurred due to diffraction in the thermal imager, and

[0010] wherein each apparent object region is identified as a contiguous region of pixels having a pixel intensity that differs from a representative background intensity of a thermal background of the scene by more than a threshold intensity and having a size that exceeds a threshold size, such that the apparent object region includes an actual object region of at least one actual object pixel and a fuzzy edge region of fuzzy edge pixels surrounding the actual object region; and

[0011] A contrast enhancement step is applied to each prominent object region, including:

[0012] segmenting the blurred edge region into a background edge region and an object edge region between the actual object region and the background edge region; and

[0013] Pixels of the object edge region are set to a representative object intensity determined from one or more actual object pixels of the actual object region, and pixels of the background edge region are set to a representative background intensity.

[0014] The method of the first aspect enables the contrast of thermal images to be improved in a manner that can be implemented in a simple and computationally efficient manner. The contrast enhancement step provides sharpening of blurred edge regions. The sharpening is achieved without resorting to a complete and relatively computationally complex deconvolution. The deconvolution may also modify the pixel values ​​of the actual object pixels. In contrast, the contrast enhancement step may employ a relatively simple algorithm, which facilitates computationally efficient implementation in a processing device (e.g., an FPGA) of a thermal image processing pipeline.

[0015] Enhanced contrast or sharpness in thermal images may facilitate monitoring activities such as distinguishing and tracking objects manually or using automatic image recognition algorithms.

[0016] According to the method, each obvious object region is identified as a contiguous region of pixels that meets both an intensity requirement (i.e., differs from a representative background intensity by more than a threshold intensity) and a size requirement (i.e., exceeds a threshold size). The basis for this identification is the recognition that if an obvious object region has a sufficiently large size (i.e., exceeds the threshold size), then the obvious object region will include at least one "actual object pixel" that is sufficiently far away from an "actual background pixel" such that the pixel will include a radiation contribution from the object rather than the thermal background. Therefore, an "actual object pixel" is "actual" or "real" in the sense that its intensity corresponds to the actual or real radiation received from the object. In contrast, a pixel of the obvious object region that is closer to the "actual" background pixel is a "fuzzy edge pixel" that includes radiation contributions from both the object and the thermal background. A "fuzzy edge pixel" may also be referred to as a "mixed pixel" hereinafter because it includes a mixture of the radiation of the object and the thermal background.

[0017] The "intensity requirement" will typically be that the pixels of the distinct object region exceed the representative background intensity by more than a threshold intensity. This means that the distinct object region may depict an object that is hotter than the thermal background. However, it is contemplated that the methods set forth herein are also applicable to blurred image regions that depict objects that are cooler than the background, where the "intensity requirement" may accordingly correspond to pixel intensities of the distinct object region that are less than the representative background intensity by more than a threshold intensity. However, for the sake of simplicity and ease of explanation, the following description will primarily relate to distinct object regions that depict objects that are hotter than the thermal background.

[0018] After selectively identifying one or more such distinct object regions, the method continues, for each such region, to segment the fuzzy edge region into a background edge region and an object edge region. Here, "segmenting" the fuzzy edge region into a background edge region and an object edge region means that each pixel of the fuzzy edge region is designated or determined as a pixel belonging to or being a background edge region or an object edge region. Therefore, the background edge region and the object edge region are respective sub-regions of the fuzzy edge region.

[0019] Although the exact position of the boundary between the background edge region and the object edge region within the blurred edge region may not be known, it can be assumed that the "actual" edge pixels of the object region (i.e., pixels that would determine the edge or peripheral pixels of the object region in the absence of blur) are included somewhere within the blurred edge region and closer to the actual object region than the (outer) boundary of the apparent object region. Since the diffraction-induced blur radius in state-of-the-art thermal imagers is typically on the order of a few pixels (e.g., less than 10 pixels) compared to the fairly high full resolution of the thermal image sensor (e.g., 160×120 or higher), it is further considered that the difference between the determined partition and the position of the "actual" edge pixels will have a relatively small impact on the usability of the thermal image for monitoring purposes. That is, such a difference will result in the contrast-enhanced object region being enlarged or reduced by one or a few pixels. Various embodiments for determining the position of the boundary between the object edge region and the background edge region within the blurred edge region with various accuracies are described below.

[0020] Having segmented the fuzzy edge regions of each apparent object region, contrast enhancement (i.e., sharpening) can be applied to each fuzzy edge region based on the pixel intensity of the corresponding actual object pixel (corresponding to the actual radiation received from the object as described above) and the representative background intensity. In addition to being computationally simple, contrast enhancement can be provided without overshoot or undershoot, which is a feature of conventional edge sharpening methods based on, for example, high frequency filtering.

[0021] Here, "pixel intensity" (interchangeably, "pixel value") refers to the intensity (or value) of a pixel in a thermal image. For thermal images, the intensity of a pixel reflects the radiation received from the scene at that pixel. The intensity can also be interpreted as the amount of IR radiation or radiant flux received from the scene at that pixel. The intensity is related to the temperature by Planck's radiation law. Assuming the camera is calibrated, the pixel intensity can therefore be accurately converted to temperature.

[0022] In some embodiments, segmenting the fuzzy edge region includes:

[0023] identifying a boundary of an actual object region (referred to as a "second boundary") extending inside a boundary of an apparent object region (referred to as a "first boundary") at a distance corresponding to a predetermined blur diameter; and

[0024] Pixels of the blurred edge area within a predetermined fraction of the distance from the second border are designated as pixels of the object edge area, and pixels of the blurred edge area outside the predetermined fraction of the distance from the second border are designated as pixels of the background edge area.

[0025] Therefore, the blurred edge area can be segmented into the object edge area and the background edge area by designating a subset of pixels of the blurred edge area that are closer to the actual object area as the object edge area and a subset of pixels of the blurred edge area that are farther away from the actual object area (e.g., closer to the background) as the background edge area. Since the distance between the first boundary and the second boundary corresponds to a predetermined blur diameter (i.e., the blur range of the blur caused by diffraction), a predetermined fraction of the distance allows the location of the "actual" boundary or edge pixels of the object area to be estimated (i.e., where the boundary of the object area would be located without blur). Therefore, pixels on different sides of the boundary can be identified as background pixels and object pixels, respectively.

[0026] The predetermined fraction may be in the range of from 0.3 to 0.7 or from 0.4 to 0.6, for example, may be about 0.5. Assuming approximate Gaussian blur, the predetermined fractions in these ranges are equivalent to gradually more accurately estimating the location of the actual boundary of the non-blurred object region. The predetermined fraction of about 0.5 is roughly equivalent to setting the inner 50% of the pixels of the blurred edge region to the representative object intensity and the outer 50% of the pixels of the blurred edge region to the representative background intensity.

[0027] The (second) boundary position of the actual object region can usually be estimated from the position of the boundary of the obvious object region (known from the identification of the obvious object region) and a predetermined blur diameter.

[0028] In some embodiments, identifying the second boundary includes identifying pixels of the apparent object region having a minimum distance from the first boundary corresponding to a predetermined blur diameter.

[0029] In some embodiments, each corresponding pixel of the object edge region is set to a corresponding representative object intensity determined based on one or more actual object pixels of the actual object region closest to the corresponding pixel of the object edge region. Thus, the pixel intensity of pixels in the object edge region can be individually set to more closely match the pixel intensity (or intensity) of the nearest boundary pixel of the actual object region. This achieves a smoother transition between the actual object region and the contrast-enhanced object region.

[0030] In some embodiments, identifying a set of distinct object regions comprises:

[0031] identifying a set of candidate object regions in the thermal image, wherein the candidate object regions are identified as contiguous regions of pixels having pixel intensities that differ from (e.g., exceed) a representative background intensity by more than a threshold intensity; and

[0032] A filtered set of candidate object regions is determined by excluding candidate regions whose sizes do not exceed a threshold size from the set of candidate object regions, wherein the set of salient object regions is identified as the filtered set of candidate object regions.

[0033] Therefore, the set of significant object regions can be identified using a two-step approach, where the first step finds pixel regions of sufficiently high intensity relative to the background intensity, and the second step filters out pixel regions that are too small to include any actual object pixels. This facilitates computationally efficient implementation.

[0034] In some embodiments, the method further includes: obtaining a noise level estimate of the thermal imager; and determining the threshold intensity based on the noise level estimate.

[0035] The noise level of a thermal imager sets a lower limit below which "object pixels" may not be distinguishable from the thermal background. Therefore, the noise level of a thermal imager represents a conservative basis for determining the threshold intensity.

[0036] In some embodiments, the representative object intensity is determined as one of the average, median, maximum, minimum, and mode of one or more actual object pixels of the corresponding actual object region. If the actual object region includes varying pixel intensities, a single representative pixel intensity for the region may be determined and used to set the pixel intensity of the pixels of the object edge region. This may allow for a computationally efficient implementation and may also avoid abrupt changes in intensity between the actual object region and the object edge region.

[0037] In some embodiments, the method further comprises: obtaining a frequency distribution of pixel values ​​of the thermal image, wherein the representative background intensity is determined as a representative pixel intensity of at least a portion of the frequency distribution. Thus, a representative estimate of the background intensity (i.e., radiation) of the scene can be estimated based on the statistics of the distribution of pixel intensities in the thermal image. This enables a reliable and computationally efficient implementation of a dynamic estimate of the intensity of the thermal background.

[0038] The representative pixel value may be one of a mean, a median, a weighted mean and a mode of at least a portion of the frequency distribution. Each of these statistics enables a reliable estimate of the representative background intensity according to the frequency distribution.

[0039] In some embodiments, the method further comprises identifying at least a first peak region in the frequency distribution, wherein the representative pixel value is determined based on pixel values ​​within the first peak region.

[0040] Here, a "peak in a frequency distribution" refers to an interval of pixel intensity or at least a predetermined number of consecutive pixel intensities (ie values), such as one or more frequency bins or sub-ranges of a frequency distribution whose frequency exceeds a predetermined minimum frequency.

[0041] The background radiation in a scene will usually be confined to some interval within the frequency distribution (the absolute position depends on the absolute temperature), and will therefore produce a peak area in the frequency distribution. Therefore, identifying such a "background peak" and determining a representative pixel intensity based on the pixel intensity within the peak can achieve a reliable estimate of the background intensity.

[0042] In some embodiments, the method further includes identifying a second peak region in the frequency distribution, wherein the representative pixel intensity is determined based on pixel intensities within the first peak region but not pixel intensities within the second peak region.

[0043] Some scenes may include areas or objects that provide a significant contribution of radiation to the monitored object (whose radiation is to be estimated) that is different from the actual thermal background. A non-limiting example is a scene that includes a relatively large area with a clear sky and a water surface (e.g., a lake or ocean, a river) that has a different temperature than the ground on which the monitored object is located. Another example may be a "hot object" whose temperature / radiation level is an outlier relative to that of the rest of the scene. By filtering the frequency distribution to exclude peak areas originating from such non-background sources, a more accurate estimate of the representative background intensity for each object area can be obtained.

[0044] Here, it should be noted that the terms "first" and "second" are merely labels introduced for the convenience of referring to the corresponding peak regions, and do not indicate any order or importance of the peaks. In practice, it can be found that the first peak region (background peak region) has a higher or lower pixel intensity than the second peak region (non-background peak region).

[0045] In some embodiments, the thermal image includes raw thermal image data. Therefore, contrast enhancement can be based on the pixel intensity of the thermal image before nonlinearization of the raw thermal image data. Nonlinearization of the raw thermal image data (interchangeably, the "raw signal") captured from the thermal image sensor can produce transformed image data with a compressed dynamic range, which is more suitable for viewing by humans and is less resource-intensive for further processing in the image processing chain. However, a side effect of nonlinearization is a change in the distribution of pixel intensities. Therefore, the relationship between the intensity of the thermal background and the object area may deviate from the actual dynamics within the scene. Accordingly, by a method based on the pixel intensity of the raw thermal image data, contrast enhancement can be performed early in the processing chain before such distortion is introduced in the thermal data. Therefore, any reference to pixels and pixel intensities above can be understood as a reference to pixels and pixel intensities of the raw thermal image data. In particular, representative object pixel intensities and representative background intensities can be determined based on pixel values ​​in the raw thermal image data. In addition, the frequency distribution can be a frequency distribution of pixel intensities of the raw thermal image data.

[0046] According to a second aspect, there is provided a computer program product comprising computer program code portions configured to, when executed by a processing device, perform the method according to the first aspect or any one of its embodiments.

[0047] According to a third aspect, there is provided a thermal imager comprising a processing device configured to perform the method according to the first aspect or any one of its embodiments.

[0048] The second and third aspects have the same or equivalent benefits as the first aspect.Any function described with respect to the first aspect may have a corresponding feature in the system, and vice versa. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] This and other aspects of the present invention will now be described in more detail, with reference to the appended drawings showing embodiments of the invention.

[0050] Figure 1 is a schematic depiction of an embodiment of a thermal imager.

[0051] Figure 2 A block diagram illustrating an embodiment of an image processing pipeline for a thermal imager.

[0052] Figure 3 A thermal image including a number of distinct object regions is schematically shown.

[0053] Figure 4 An exemplary significant object region of a thermal image and a corresponding graph of pixel values ​​along a row of pixels extending across the significant object region of the thermal image are schematically shown.

[0054] Figure 5 is a flow chart of a method for processing thermal images.

[0055] Figure 6a and Figure 6b Exemplary frequency distributions of thermal images of different scenes are schematically shown. DETAILED DESCRIPTION

[0056] Figure 1 An example embodiment of a thermal imager 1 including a thermal image sensor 14 (interchangeably, "image sensor 14") is schematically shown. The thermal imager 1 may more specifically be a radiometric thermal imager 1. The thermal imager 1 may be calibrated so that pixel values ​​recorded by the image sensor 14 may be accurately converted to temperatures within a scene 2 monitored by the thermal imager 1 (i.e., via Planck's radiation law). However, the present disclosure is also applicable to uncalibrated thermal imagers. The image sensor 14 may be of a conventional type, such as a microbolometer sensor including a pixel array of microbolometers. The microbolometer sensor may effectively detect IR radiation in the range of approximately 7-14 μm. Microbolometer sensors are commercially available in a variety of resolutions, such as 160×120 pixels, 1024×1024 pixels, 1920×1080 pixels, and higher resolutions. As Figure 1 As shown in , the image sensor 14 can form a part of the sensor package 10, and the sensor package 10 further includes a reflector 12 arranged behind the image sensor 14 and a sensor window 16 arranged in front of the image sensor 14. The reflector 12 can increase the effective fill factor of the image sensor 14 and can be, for example, a λ / 4 reflector of Au. In some embodiments, the sensor package 10 can be a vacuum package, wherein the window 16 (which can be formed of Si, for example) can be arranged as a vacuum seal of the sensor package 10.

[0057] The thermal imager 1 further comprises an optical system 18 and a cover 20. Figure 1, for simplicity, the optical system 18 is shown as a single lens, but may generally include multiple beam forming optical elements such as lenses and filters. The cover 20 is formed of a material that is transparent to IR radiation in the wavelength range of interest. An example material for the cover 20 is Ge. The optical elements as well as the cover 20 may further be provided with an anti-reflection (AR) coating that reduces internal reflections within the thermal imager 1. An example of a conventional AR coating is a diamond-like coating (DLC).

[0058] Figure 2 is a block diagram of an example implementation of an image processing system 24 that may be included in the thermal imager 1 and in which embodiments of the present invention may be implemented.

[0059] The image processing system 24 includes the image sensor 14 , a processing device 28 , and a downstream image processing pipeline 30 .

[0060] The image sensor 14 acquires a thermal image of pixels whose pixel values ​​depend on the radiation contribution from the portion of the scene 2 imaged on the corresponding pixel of the image sensor 14. The thermal image output by the image sensor 14 may include raw thermal image data containing pixel intensities that have not been non-linearized. Non-linearization may further include reducing the bit depth of the thermal image data.

[0061] As will be explained below, the thermal image including raw thermal image data is received for processing by the processing device 28. The processing device 28 may further forward the thermal image to an image processing pipeline 30 downstream.

[0062] The downstream image processing pipeline 30 may implement, for example, a number of conventional sequential processing steps that may be used to enhance and compress the thermal image data. Examples of such processing steps include noise reduction, global / local detail enhancement, sharpening, etc. In particular, the image processing pipeline 30 may implement nonlinearization of the raw thermal image data to produce a nonlinear thermal image that is more suitable for human viewing than the pixels of the raw thermal image data, as well as implement bit depth reduction.

[0063] As shown, the image processing system 24 may further include a noise filter 26. The noise filter may include a temporal noise filter and / or a spatial noise filter. Although in the illustrated example, the noise filter 26 is shown as being upstream of the processing device 28, the noise filter 26 may also be implemented downstream of the processing device 28 so that nonlinearization and bit depth reduction are applied before denoising.

[0064] The processing performed by the noise filter 26, the processing device 28 and the image processing pipeline 30 can be implemented in both hardware and software. In a hardware implementation, each of the method steps set forth herein can be implemented with a dedicated circuit. The circuit can be in the form of one or more integrated circuits, such as one or more application-specific integrated circuits (ASICs) or one or more field programmable gate arrays (FGPAs). In a software implementation, as an alternative, the circuit can take the form of a processor, such as a central processing unit or a graphics processing unit associated with computer code instructions stored on a (non-temporary) computer-readable medium such as a non-volatile memory. Examples of non-volatile memories include read-only memories, flash memory, ferroelectric RAM, magnetic computer storage devices, and optical disks, etc. It should be understood that hardware and software implementations can also be combined, which means that some method steps can be implemented by dedicated circuits and others can be implemented by software.

[0065] Reference again Figure 1 , the scene 2 comprises a plurality of schematically shown objects 4a to 4c. The dashed box 6 schematically indicates the thermal background of the scene 2, which correspondingly forms the thermal background of the objects 4a to 4c.

[0066] Here, "object" generally refers to a natural or artificial, stationary or moving physical structure or feature within scene 2. An object may also refer to a part of such a physical structure or feature included in scene 2. For example, an object may refer to a part of a building structure, a road, a tree, a vehicle, etc. included in scene 2, so that only this part is depicted in a thermal image captured by a thermal imager 1 monitoring scene 2. An object may refer to an object of interest in scene 2, such as a subject of remote temperature monitoring. However, an object may also be an object that exists in the scene but is not itself an object of interest. For example, an object may form part of the environment of scene 2, and there may be other objects of interest in scene 2.

[0067] During image acquisition, radiation emitted from the scene 2 is received by the thermal imager 1, shaped by the optical system 18, and focused onto the pixel array of the image sensor 14. While being transmitted through the optical system 18, the radiation received from the scene 2 will be diffracted. Assuming that the diffraction is comparable to the pixel size of the image sensor 14, the diffraction will be seen as a blurring of the edges of the scene 2 in the thermal image, and therefore, a reduction in contrast in the thermal image. Therefore, the edges of objects in the thermal image will be depicted by blurred edge pixels, which include a mixture of the radiation contribution from the object and the radiation contribution from the thermal background of the object. The minimum focus point that can be imaged on the image sensor 14 by the thermal imager 1 at the diffraction limit is the blur point. The blur point corresponds to the range of pixels in which each pixel is blurred (i.e., blurred or scattered).

[0068] It is envisioned that the main contribution to blur comes from diffraction of incident radiation in the thermal imager 1 (eg, in its optical system 18) during image capture. However, additional blur may also be present due to defocusing of the object 4 and multiple reflections in the sensor package 10.

[0069] Distinguishing and tracking low-contrast objects can be challenging. The problem of objects / areas with low contrast in thermal images is particularly evident in surveillance applications that include objects that are hot or may become hot. In such cases, the dynamic range of the thermal imager 1 (and the corresponding thermal image) may be less than the dynamic range of the scene 2. Therefore, to avoid areas in the thermal image depicting hot objects from becoming overexposed, the thermal image may be captured with the thermal imager 1 configured (e.g., configured using a short integration time) such that the maximum pixel intensity of the thermal object (corresponding to the maximum expected radiation level of the thermal object) is within the dynamic range of the thermal imager 1 and the thermal image. Thus, any other objects in the scene 2 that are not hot objects may be depicted in the relatively low-contrast areas in the thermal image.

[0070] Although the method of the present invention can be applied to depict areas of "hot objects" in thermal images, it is expected that the method can be particularly used to depict areas of other objects in the scene that are not hot objects. That is, objects 4a to 4c can be objects with relatively low contrast relative to the thermal background of scene 2.

[0071] Figure 3 A representation of a thermal image 32 obtained by an image sensor of a thermal imager (e.g., image sensor 14 of thermal imager 1) is schematically shown. The thermal image 32 includes a first group of object regions 322a to 322c that each depict a corresponding one of objects 4a to 4c of a scene 2. The thermal image 32 includes a background region 321 surrounding the first group of object regions 322a to 322c. The background region 321 depicts the thermal background 6 of the scene 2, which also forms the thermal background of the object 4. As described above, each of the first group of object regions 322a to 322c is a blurred depiction of the corresponding object 4a to 4c due to diffraction during image capture, etc. Therefore, the first group of object regions 322a to 322c defines a group of distinct object regions 322a to 322c.

[0072] Figure 4 Schematically shows an enlarged view of an obvious object region 322 (e.g., any one of obvious object regions 322a to 322c) of the thermal image 32. The obvious object region 322 includes an actual object region 324 of actual object pixels. The actual object region 324 is surrounded by a fuzzy edge region 323 of fuzzy edge pixels. The pixels of the actual object region 324 and the pixels of the fuzzy edge region 323 form a corresponding (strict) subset of pixels of the obvious object region 322.

[0073] like Figure 4 As shown in , each actual object pixel of the actual object region 324 is separated from the background region 321 (i.e., formed by the actual background pixels) by at least 2 times the blur radius of the characteristic blur point (i.e., blur diameter). Accordingly, each blurred edge pixel of the blurred edge region 323 is separated from the background region 321 by less than 2 times the blur radius. Accordingly, each actual object pixel of the actual object region 324 includes radiation contributions from the corresponding objects 4a to 4c depicted in the corresponding obvious object regions 322a to 322c, but does not include radiation contributions from the thermal background 6. That is, the corresponding objects 4a to 4c, but not the thermal background 6, contribute to the pixel intensity of each actual object pixel. Accordingly, each blurred pixel of the blurred edge region 323 includes radiation contributions from the corresponding objects 4a to 4c and the thermal background 6. That is, radiation from both the corresponding objects 4a to 4c and the thermal background 6 contributes to the pixel intensity of each blurred edge pixel.

[0074] This can be Figure 4 It can be further seen in the lower part that Figure 4 3 shows a graph of pixel intensities of a line scan of pixels along a dashed line S extending across a central portion of the apparent object region 322. As schematically indicated in the figure, the blurring of the actual object radiation L of the object 4 obj The contour (dashed line) is modulated into an apparent radiation contour having the actual object radiation L in the blurred edge region 323. obj Radiation with thermal background 6 L b (solid line) for a smoother transition between

[0075] exist Figure 4 , corresponding to a blur radius R of 3 pixels, diffraction blurs the edges of the imaged object over 6 pixels. As can be appreciated, a larger blur radius results in additional widening. Furthermore, in the illustrated example, the obvious object area 322 is 21×15 pixels and the actual object area 324 is 9×3 pixels. However, these are merely examples, and other sizes and shapes of the obvious object area 322 and the actual object area 324 are possible. The minimum size of the actual object area 324 to be processed by the method described below is 1×1 pixel, so that the actual object area 324 includes at least one actual object pixel.

[0076] Now refer to Figure 5 Flowchart and further reference Figures 1 to 4 , embodiments of methods for processing thermal images to provide enhanced contrast are described.

[0077] At step S1 , the processing device 28 acquires a thermal image 32 of the scene 2 . The thermal image 32 is obtained by the image sensor 14 of the thermal imager 1 .

[0078] At step S2 , the processing device 28 identifies a set of distinct object regions 322 a to 322 c in the thermal image 32 , each of which includes a depiction of a corresponding one of a set of objects 4 a to 4 c that is blurred due to diffraction in the thermal imager 1 .

[0079] Each obvious object region 322a to 322c is identified as a contiguous region of pixels: (i) each pixel having an intensity that differs from the representative background intensity of the thermal background of scene 1 by more than a threshold intensity (typically more than the threshold intensity) and (ii) a contiguous region having a size that exceeds a threshold size, such that the obvious object region 322a to 322c includes an actual object region 324 of at least one actual object pixel and a fuzzy edge region 323 of fuzzy edge pixels surrounding the actual object region 324.

[0080] To facilitate subsequent processing, the processing device 28 may represent the set of obvious object regions 322a to 322c in the form of a bitmap, where each pixel of the corresponding obvious object region is assigned a value of 1, and each pixel of the background region 321 is assigned a value of 0.

[0081] Used to determine the representative background intensity L b Various approaches are possible.

[0082] For example, a representative background radiation can be determined using a frequency distribution-based method or a histogram-based method. The processing device 28 can obtain a frequency distribution of pixel values ​​of a thermal image 32 (e.g., which can be raw thermal image data). In some embodiments, the processing device 28 can calculate a frequency distribution of pixel intensities of the thermal image 32. In other embodiments, the frequency distribution may have been provided by the thermal image sensor 14 together with the thermal image 32. In this case, the processing device 28 can obtain the frequency distribution by receiving the frequency distribution from the thermal image sensor 14. In either case, it is advantageous that a frequency distribution can be defined for multiple intervals of pixel intensity to segment the dynamic range of the thermal image. The number and width of the intensity intervals can vary depending on the computing resources of the processing device 28 and the accuracy required for the frequency distribution analysis. Although the intervalized frequency distribution can reduce the computing resources required for the method, the use of a non-intervalized frequency distribution is not excluded.

[0083] In either case, the processing device 28 may process the frequency distribution to convert the representative background intensity L bDetermined as a representative pixel intensity of at least a portion of the frequency distribution. For example, the representative pixel intensity can be determined as one of a mean, a median, a weighted mean, and a mode of at least a portion of the frequency distribution. The representative pixel intensity can be determined from the entire frequency distribution (including the pixel intensities of all pixels of the thermal image 32) or from only a portion of the frequency distribution.

[0084] Referring to the example of frequency distribution shown Figure 6a , the processing device 28 can, for example, identify at least a first peak region P1 in the frequency distribution, and determine a representative pixel intensity based on the pixel intensities within the first peak region P1. The background radiation in the scene 2 will generally be confined to a relatively wide interval within the frequency distribution (the absolute position depends on the absolute temperature), and therefore, a peak region is generated in the frequency distribution (the mode of the peak depends on the width of the peak region P1 and the total number of pixels of the thermal image 32, etc.). Therefore, the representative pixel intensity can be determined as the average, median, weighted average or mode of the pixel intensities of the first peak region P1 of the frequency distribution.

[0085] Referring to another example of frequency distribution Figure 6b , the processing device 28 may, for example, further identify a second peak region P2 in the frequency distribution, and when determining the representative pixel intensity, filter out the pixel intensity within the second peak region P2, so that the representative pixel intensity is determined from the pixel intensity within the first peak region P1 rather than the pixel intensity within the second peak region P2. The second peak region may be excluded from the determination of the representative pixel intensity, for example, based on being narrower than the first peak region P1 by more than a threshold value and / or being removed from a pixel intensity range (e.g., a predetermined range) expected based on the thermal background 6.

[0086] In case the scene includes a hot object, the hot object may produce an additional peak at a pixel intensity higher than the first peak region P1. To avoid the pixel intensity of the hot object peak skewing the representative pixel intensity, the pixel intensity of the hot object peak may be excluded from the determination of the representative pixel intensity.

[0087] In lieu of a frequency distribution-based approach for estimating the representative background intensity, the processing device 28 may, according to a simpler embodiment, determine the representative background intensity as the average of the pixel intensities of all pixels of the thermal image 32 or the average of the pixel intensities of an area of ​​the thermal image 32 that is known (e.g., based on prior knowledge or user-supplied input) to depict the thermal background 6.

[0088] Using a threshold intensity in identifying the distinct object region 322 can distinguish pixels belonging to the distinct object region 322 from pixels belonging to the background region 321 while providing some tolerance for varying pixel intensities within the background region 321 (e.g., due to noise). The threshold intensity can be set to a predetermined value determined based on an assumption or measured deviation (e.g., three sigma of pixel intensity) of pixel intensities within a region of the thermal image 32 that is known (e.g., based on prior knowledge or user-supplied input) to depict the thermal background 6 of the scene 2.

[0089] In another embodiment, the processing device 28 may obtain a noise level estimate of the thermal imager 1 and determine the threshold intensity based on the noise level estimate.

[0090] The processing device 28 may derive a noise level estimate based on a noise threshold of a noise filter 26 of the thermal imager 1. Such a noise filter 26 has a threshold (which may vary depending on the capture mode of the thermal imager 1), above which the signal read out from the pixels of the image sensor 14 is considered to be real and not noise. The processing device 28 may accordingly derive a noise level estimate of the thermal imager 1 by acquiring (e.g., reading) the noise threshold from the noise filter 26. The thermal imager 1 may have both a temporal noise filter (which has a threshold set based on a sequence of acquired thermal images or frames) and a spatial noise filter (which has a threshold set based on a single current thermal image), which together may define a spatiotemporal noise filter 26. In this case, the processing device 28 may derive a noise level estimate of the thermal imager 1 based on the temporal and spatial noise thresholds of the spatiotemporal noise filter 26.

[0091] The noise level estimate may also be determined based on the noise equivalent temperature difference (NETD) of the image sensor 14. The NETD of the image sensor 14 may be converted to the NETD of the thermal imager 1 and the thermal image 32 by scaling the NETD of the image sensor 14 by the f-number squared, where the f-number refers to the f-number used when capturing the thermal image 32. The NETD is a measure of how well the image sensor can distinguish small differences in radiation in the scene 2.

[0092] Having obtained the noise level estimate, the processing device 28 may determine the intensity threshold as the noise level estimate, optionally scaling the noise level estimate by a factor typically greater than one.

[0093] As described above, the condition for identifying a continuous region of pixels having pixel intensities satisfying the threshold intensity test as the obvious object region 322 is that the size of the continuous region exceeds the threshold size. The threshold size can be set so that when the threshold size is exceeded, the continuous pixel region includes an actual object region 324 of at least one actual object pixel and a fuzzy edge region 323 of fuzzy edge pixels surrounding the actual object region 324.

[0094] The threshold size can be defined in a variety of ways. For example, the threshold size can correspond to a blur radius or a blur diameter (either of which indicates the size of a blur point). The radius (or diameter) can be obtained in pixels, which can be an integer number of pixels (e.g., 2 pixels, 3 pixels, etc.) or a fraction of pixels (e.g., 1.5 pixels, 2.5 pixels, etc.). Having identified a continuous region of pixels having a pixel intensity exceeding the threshold intensity compared to the representative background intensity, the processing device 28 can use the threshold size to determine whether the continuous region includes at least one pixel having a minimum distance to the background region 321 exceeding the blur diameter (or, equivalent to twice the blur radius). The processing device 28 can, for example, evaluate this only for the central pixel of the continuous region. The central pixel can be determined as the pixel at the centroid or centroid of the continuous pixel region. Therefore, the number of pixels to be evaluated can be limited to a single pixel, which reduces the amount of pixel data to be processed. However, the processing device 28 can also search (e.g., exhaustively) a continuous pixel region to find at least one pixel having a minimum distance to the background region 321 exceeding the blur diameter.

[0095] In another example, the threshold size may be defined based on a bitmap (i.e., a binary mask) representing the shape and size of the blur point. The processing device 28 may determine whether the blur point can be aligned with the continuous pixel region so that the blur point is completely contained within the continuous pixel region. Similar to the previous example, this may be evaluated for only a single pixel (e.g., the center pixel) or multiple pixels of the continuous pixel region (e.g., scanning the blur point / bitmap across the continuous pixel region and determining whether each pixel of the blur point overlaps with a pixel of the continuous pixel region).

[0096] Regardless of how the threshold size is defined (e.g., in terms of radius, diameter, or as a bitmap of blur spots), the threshold size can be determined in advance. The blur spots can be determined during characterization (e.g., typically offline, prior to deployment) by measuring the modulation transfer function (MTF) of the thermal imager 1, or using some other conventional method for characterizing blur of imaging systems. The blur radius or blur diameter can be derived, for example, from the FWHM of blur features (e.g., line patterns or dot patterns) of a test target focused onto the image sensor and imaged by the thermal imager 1. In the absence of pre-characterization, the size of the blur spot can be estimated from the diffraction limit, i.e., approximately 2.44×wavelength×f-number of the optical system 18 of the thermal imager 1. For typical thermal imagers, the size of the blur spot is actually about 25μm to 30μm, which in turn corresponds to a blur diameter of about 1.5 pixels to 2.0 pixels for a 17μm pixel pitch and a blur diameter of about 2 pixels to 3 pixels for a 12μm pixel pitch.

[0097] In either approach, if the contiguous region does not exceed a threshold size, the contiguous region may be excluded from the set of obvious object regions 322a-322c. Figure 3 , the thermal image 32 may further include, as shown, a second set of object regions 322d to 322f that are too small to include any actual object pixels and that include only radiation contributions from the thermal background 6. Comparison based on a threshold size may cause such small object regions to be excluded from the set of significant object regions 322a to 322c.

[0098] like Figure 5 As shown in , the identification of obvious object regions 322a to 322c may include a two-step method: At S21, the processing device 28 may identify a group of candidate object regions 322a to 322f in the thermal image 32. Each candidate object region 322a to 322f is identified as a continuous region of pixels having a pixel intensity exceeding a threshold intensity over a representative background intensity. At S22, the processing device 28 may filter the group of candidate object regions 322a to 322f to exclude candidate regions whose size does not exceed a threshold size (e.g., the second group of object regions 322d to 322f). The processing device 28 may accordingly identify the group of obvious object regions as a filtered group of candidate object regions 322a to 322c.

[0099] At S3, the processing device 28 applies a contrast enhancement step to each of the distinct object regions 322a to 322c identified at step S2. Figure 5 As shown in FIG. 1 , the contrast enhancement step includes two sub-steps S31 and S32, which will be referred to as Figure 4 The example obvious object region 322 in FIG. 3 describes these two sub-steps S31 and S32 .

[0100] At S31 of the contrast enhancement step, the processing device 28 divides the blurred edge region 323 of the corresponding apparent object region into a background edge region 323a and an object edge region 323b between the actual object region 324 and the background edge region 323a.

[0101] The blurred edge region 323 may be segmented by identifying a boundary B2 (second boundary) of the actual object region 324, which extends inside the boundary B1 (first boundary) of the apparent object region 322 at a distance from the first boundary B1, the distance corresponding to a predetermined blur diameter (i.e., twice the blur radius R). Subsequently, pixels of the blurred edge region 323 that are within a predetermined fraction f of the distance from the second boundary B2 (i.e., within a fraction f of the blur diameter) may be identified as pixels of the object edge region 323b, and pixels of the blurred edge region 323 that are outside the predetermined fraction f of the distance from the second boundary B2 may be identified as pixels of the background edge region 323a.

[0102] In the illustrated example, the fraction f=0.5, where the predetermined fraction of the distance corresponds to 0.5 times the blur diameter (i.e., blur radius R). Therefore, the boundary B3 that divides the blurred edge area 323 into the background edge area 323a and the object edge area 323b is located in the middle of the first boundary B1 and the second boundary B2, i.e., a blur radius R away from the first boundary B1 and the second boundary B2. Although this can represent a good approximation of the position of the edge of the object area without blur, other choices are also possible for blur functions of other shapes (e.g., non-Gaussian). For example, the predetermined fraction f can be in the range of from 0.3 to 0.7 or from 0.4 to 0.6, and can be, for example, about 0.5. Regardless of the specific value of the fraction f, this segmentation method is equivalent to assuming that pixels of the blurred edge area 323 that are closer to the actual object area 324 than to the background area 321 are more likely to belong to the corresponding actual object area 324. In contrast, pixels of the blurred edge region 323 that are farther away from the actual object region 324 are more likely to belong to the background region 321 than the corresponding actual object region 324 .

[0103] The second boundary B2 of the actual object region 324 can be identified by identifying pixels of the obvious object region 322, so that the minimum distance between the identified pixels and the first boundary B1 corresponds to (e.g., is equal to) the blur diameter 2×R. Since the coordinates of the first boundary B1 can be known by identifying each obvious object region 322, as shown by Figure 4As schematically indicated by arrow A in FIG. 3 , given a blur diameter of 2×R, the processing device 28 may accordingly search for pixels within the apparent object region 322 that are separated from the first boundary B1 (and thus from the background region 321) by a blur diameter of 2×R. This allows the second boundary B2 of the actual object region 324 to be identified or estimated even for irregular shapes, since the shape of the second boundary B2 will inherently follow the shape of the first boundary B1.

[0104] exist Figure 4 , it is shown that the boundaries B1 to B3 extend between adjacent pixels. However, the boundaries B1 to B3 can actually be represented by the pixel coordinates of the pixels within the corresponding boundaries B1 to B3. In other words, the boundaries B1 to B3 can be defined by the peripheral or edge pixels of the corresponding area.

[0105] The example of the segmentation method above is only one possible method for segmenting the fuzzy edge region 323. In another example, the second boundary B2 can be determined independently of the first boundary B1 or the predetermined fuzzy diameter. First, the center pixel of the actual object region 324 can be estimated as the center pixel (e.g., the centroid or the centroid) of the obvious object region 322. It is a reasonable assumption for many typical object shapes that the actual object region 324 and the obvious object region 322 have the same center pixel. Secondly, the pixels of the actual object region 324 can be identified as a continuous region of pixels including the center pixel and having a pixel intensity within a predetermined tolerance from the center pixel intensity. Therefore, the second boundary B2 can be correspondingly identified as the boundary of the identified actual object region 324. The tolerance can be set according to the fuzzy radius R, so that a smaller tolerance is used in the case of a smaller fuzzy radius R (meaning a steeper decline in the pixel intensity in the fuzzy edge region 323). Having identified the second boundary B2, segmentation can be performed as described above.

[0106] At S32 of the contrast enhancement step, the processing device 28 continues by setting the pixels of the object edge region 323b to a representative object intensity determined by one or more actual object pixels of the actual object region 324 and setting the pixels of the background edge region 323a to a representative background intensity. The representative object intensity may be determined as one of an average, a median, a maximum, a minimum, and a mode of one or more actual object pixels of the corresponding actual object region 324. The one or more actual object pixels may be, for example, a central pixel (e.g., determined as the centroid or centroid of the apparent object region 322) or a central pixel and a predetermined number of pixels adjacent to the central pixel (assuming that the actual object region 324 is larger than 1×1 pixel). Each pixel of the object edge region 323b may be set to the same representative object pixel intensity. Optionally, each corresponding pixel of the object edge region 323b may be set to a corresponding representative object intensity determined based on one or more actual object pixels of the actual object region 324 that are closest to the corresponding pixel of the object edge region 323b. In the case where the pixel intensity of the actual object region 324 changes along the boundary B2, the pixel intensity within the contrast-enhanced object edge region 323b can therefore be set to match or track this change. In a simple approach, each corresponding pixel of the object edge region 323b can be set to the pixel intensity of the closest pixel of the actual object region 324. In a more detailed approach, any of the above-mentioned pixel statistics can be applied to multiple closest pixels of the actual object region 324 to determine a representative object intensity for each corresponding pixel of the object edge region 323b.

[0107] By the contrast enhancement step, pixels having pixel values ​​corresponding to the radiation of the depicted object without blurring can be set as representative object intensity, and pixels having pixel values ​​corresponding to the thermal background without blurring can be set as representative background radiation. Thus, a sharp edge is defined along the third boundary B3, and the blurred edge region 323 can be replaced by a sharpened edge region.

[0108] It will be appreciated by those skilled in the art that the present invention is by no means limited to the embodiments described above. Instead, many modifications and variations are possible within the scope of the appended claims.

Claims

1. A method for performing thermal image processing, the method comprising: acquiring a thermal image obtained by an image sensor of a thermal imager, wherein the thermal image depicts a scene including a group of objects; identifying a set of distinct object regions in the thermal image; wherein each significant object region comprises a depiction of a corresponding one of the set of objects blurred due to diffraction in the thermal imager, and wherein each obvious object region is identified as a contiguous region of pixels having a pixel intensity that differs from a representative background intensity of a thermal background of the scene by more than a threshold intensity and having a size that exceeds a threshold size, such that the obvious object region includes an actual object region of at least one actual object pixel and a fuzzy edge region of fuzzy edge pixels surrounding the actual object region; and A contrast enhancement step is applied to each prominent object region, including: dividing the blurred edge region into a background edge region and an object edge region between the actual object region and the background edge region; and Pixels of the object edge region are set to a representative object intensity determined based on one or more actual object pixels of the actual object region, and pixels of the background edge region are set to the representative background intensity.

2. The method according to claim 1, wherein: Segmenting the fuzzy edge area includes: identifying a boundary of the actual object region extending inside a boundary of the apparent object region at a distance corresponding to a predetermined blur diameter; and Pixels of the blurred edge area that are within a predetermined fraction of the distance from the boundary of the actual object area are designated as pixels of the object edge area, and pixels of the blurred edge area that are outside the predetermined fraction of the distance from the boundary of the actual object area are designated as pixels of the background edge area.

3. The method according to claim 2, wherein: The predetermined fraction is in the range from 0.3 to 0.7 or from 0.4 to 0.6, for example, is about 0.

5.

4. The method according to claim 2, wherein: Identifying the boundary of the actual object region includes identifying pixels of the apparent object region having a minimum distance from the boundary of the apparent object region corresponding to the blur diameter.

5. The method according to claim 1, wherein: Identifying the set of distinct object regions includes: identifying a set of candidate object regions in the thermal image, wherein candidate object regions are identified as contiguous regions of pixels having pixel intensities that differ from the representative background intensity by more than the threshold intensity; and A filtered set of candidate object regions is determined by excluding candidate regions having a size not exceeding the threshold size from the set of candidate object regions, wherein the set of significant object regions is identified as the filtered set of candidate object regions.

6. The method according to claim 1, further comprising: obtaining a noise level estimate of the thermal imager; and determining the threshold intensity based on the noise level estimate.

7. The method according to claim 1, wherein: Each corresponding pixel of the object edge region is set to a corresponding representative object intensity determined according to one or more actual object pixels of the actual object region that are closest to the corresponding pixel of the object edge region.

8. The method according to claim 1, wherein: The representative object intensity is determined as one of a mean, a median, a maximum, a minimum, and a mode of the one or more actual object pixels.

9. The method according to claim 1, further comprising: A frequency distribution of pixel values ​​of the thermal image is acquired, wherein the representative background intensity is determined as a representative pixel intensity of at least a portion of the frequency distribution.

10. The method according to claim 9, wherein: The representative pixel intensity is one of a mean, a median, a weighted mean, and a mode of the at least a portion of the frequency distribution.

11. The method according to claim 9, further comprising: At least a first peak region in the frequency distribution is identified, wherein the representative pixel intensity is determined based on pixel values ​​within the first peak region.

12. The method according to claim 11, further comprising: A second peak region in the frequency distribution is identified, wherein the representative pixel intensity is determined based on the pixel values ​​within the first peak region but not the pixel values ​​within the second peak region.

13. The method according to claim 1, wherein: The thermal image includes raw thermal image data.

14. A computer program product comprising computer program code portions configured to, when executed by a processing device, perform the method according to claim 1.

15. A thermal imager comprising a processing device configured to perform the method according to claim 1.

Images