Method for applying a method for increasing the resolution of an image from a thermal imaging camera

Abstract

The invention relates to a method (100) for applying a method for increasing the resolution of an image of a thermal imaging camera (1), comprising the following steps: - providing (101) image data, wherein the image data comprises a defined number of images, wherein the images are regular camera images and / or thermal images, wherein the image data result from a capture of the thermal imaging camera (1), - analyzing (102) a contrast in the image data, - initiating (103) the application of the method for increasing the resolution depending on a result of the analysis of the contrast in the image data. Furthermore, the invention relates to a computer program, a device and a storage medium for this purpose.

Description

[0001] The invention relates to a method for applying a method for increasing the resolution of an image from a thermal imaging camera. Furthermore, the invention relates to a computer program, a device, and a storage medium for this purpose. State of the art

[0002] A key factor in purchasing a thermal imaging camera is the resolution of the thermal image. The higher the resolution, the more details can be seen and the more applications it can serve. Resolution is therefore an important feature of high-quality thermal imaging cameras. However, since it is directly dependent on the quality and therefore the price of the infrared sensors used, better resolution from the sensors automatically means higher price for the components. To improve the resolution without more expensive components, alternative algorithms can be used to improve image quality. Super-resolution algorithms, for example, are used to improve the resolution, i.e. in particular by increasing the number of pixels. Both classic algorithms (based on classic image processing) and machine learning-based algorithms can be used for this.

[0003] During a real-time application of a thermal imaging camera, it may happen that insufficient conditions such as too low a contrast for a method for increasing the resolution of a thermal imaging camera image, in particular a super-resolution algorithm, mean a deterioration of the processed images compared to the source images. Disclosure of the invention

[0004] The invention relates to a method having the features of claim 1, a computer program having the features of claim 11, a device having the features of claim 12, and a computer-readable storage medium having the features of claim 13. Further features and details of the invention emerge from the respective subclaims, the description, and the drawings. Features and details described in connection with the method according to the invention naturally also apply in connection with the computer program according to the invention, the device according to the invention, and the computer-readable storage medium according to the invention, and vice versa, so that with regard to the disclosure of the individual aspects of the invention, reference is or can always be made to each other.

[0005] The invention particularly relates to a method for applying a method for increasing the resolution of an image from a thermal imaging camera, comprising the following steps, wherein the steps can be carried out repeatedly and / or sequentially. Increasing the resolution is in particular a technical process in which the number of pixels in the image from the thermal camera is increased in order to improve the level of detail and sharpness of the image as perceived by a user. This can be achieved in various ways, for example by interpolation, wherein new pixels are inserted between the existing pixels of the image. Furthermore, super-resolution algorithms can be applied, which are a specialized form of image resolution improvement. In particular, missing details in low-resolution images are reconstructed.Super-resolution algorithms can often provide more detailed and clearer results than simple interpolation. The thermal imaging camera is preferably a handheld thermal imaging camera, i.e., a thermal imaging camera that can be held and operated by a user.

[0006] In a first step, image data is preferably provided, wherein the image data comprises a defined number of images, wherein the images are regular camera images and / or thermal images, wherein the image data result from a capture by the thermal imaging camera. The regular camera images can also be understood and referred to as visual images within the scope of the present invention and are in particular images that represent a light spectrum visible to a human. The regular camera images can be captured by a camera sensor of the thermal imaging camera. The thermal images are in particular infrared images, i.e. images in a spectrum of the infrared range. The thermal images can be captured with an infrared camera sensor of the thermal imaging camera.

[0007] In a further step, a contrast in the image data is preferably analyzed. For example, a histogram analysis can be performed to determine the contrast based on a specific percentage of a pixel intensity or brightness distribution. Another possible approach could be calculating the standard deviation of the pixel intensities or brightness values ​​to determine and analyze the contrast. Furthermore, a local contrast analysis can be performed. In particular, the contrast in different areas of the image is analyzed separately.

[0008] In a further step, the application of the method for increasing the resolution is preferably initiated depending on a result of the analysis of the contrast in the image data. This means in particular that the application of the method for increasing the resolution is only applied, for example, when sufficient contrast is determined in the image data. Various methods known in the prior art, such as interpolation or super-resolution, can be used as a method for increasing the resolution. It is possible that the method for increasing the resolution provides poorer results when the contrast in the image data is low. The present method can therefore advantageously be used to evaluate whether the image data are suitable for the method for increasing the resolution based on the contrast evaluation parameter.

[0009] Furthermore, within the scope of the invention, it is conceivable that the analysis comprises the following steps: - Defining a threshold for the contrast in the image data, - Comparing the respective contrasts of individual images of the image data with the defined threshold value for the contrast.

[0010] The threshold can be defined in various ways, depending on the specific use case and desired criteria. For example, a histogram analysis can be performed. The threshold can be defined based on a certain percentage of a pixel intensity or brightness distribution. Another possible approach could be to calculate the standard deviation of the pixel intensities or brightness values. The threshold could then be set, for example, as a multiple of the standard deviation to identify areas of high contrast. Furthermore, a local contrast analysis can be performed. In particular, the contrast in different areas of the image is analyzed separately and a local threshold is determined. This can be advantageous in images with varying lighting conditions.The contrast of the individual images is preferably determined in a manner analogous to that used to define the contrast threshold in order to perform the comparison.

[0011] If the comparison result shows that the contrast of a specified number of images is above the defined threshold, it can be provided that, during initiation, only images from the defined number of images whose contrast is above the defined threshold are provided. In other words, the resolution enhancement method can be performed only with images with a contrast suitable for the resolution enhancement method, which can advantageously improve the result of the method.

[0012] It is further conceivable that if the result of the comparison shows that the contrast of a specified number of images is below the defined threshold, the initiation is blocked. It may be that the method for increasing the resolution only provides a satisfactory result above a certain contrast value and could otherwise even cause a deterioration. Therefore, the method according to the present invention can advantageously prevent the application of the method for increasing the resolution if the contrast is below the defined threshold. If the initiation of the application of the method for increasing the resolution is blocked, it can be provided that a corresponding output is sent to a user of the thermal imaging camera, for example via a display or a loudspeaker of the thermal imaging camera.The output could alert the user that the method for increasing the resolution, which may be implemented, for example, by a specific mode in the thermal imaging camera, is currently not possible.

[0013] Preferably, within the scope of the invention, it can be provided that the method for increasing the resolution comprises the following steps, wherein the steps are preferably carried out one after the other. In a first step, a subpixel shift of the defined number of images relative to a reference image is preferably determined, wherein the reference image is one of the defined number of images. In this step, in particular for each image of the defined number of images, a shift at the subpixel level relative to the selected reference image is determined, for example by determining an optical flow. The selected reference image can, for example, be the last image of the defined number of images. This shift is necessary in particular to compensate for differences in the camera position or perspective between the images.Subpixel-level accuracy advantageously enables more precise alignment, which can increase the quality of the resulting image.

[0014] In a further step, the defined number of images are preferably shifted based on the determined subpixel shift so that they are aligned with the reference image. After the subpixel shift has been determined for each of the defined number of images, the images can be shifted accordingly to align them with the reference image. This alignment can advantageously ensure that corresponding points in all images coincide, which is particularly necessary for subsequent image fusion.

[0015] In a further step, the defined number of images are preferably scaled by a defined scaling factor. For example, a quadruple scaling, i.e., a defined scaling factor of four, can be provided. This can be done for various reasons, such as adapting the images to a specific target resolution or standardizing image sizes to improve alignment and / or reduce the computational load during subsequent image fusion. During the scaling process, the images of the defined number of images can be interpolated.

[0016] In a further step, a resulting image is preferably determined based on the scaled, defined number of images and the determined subpixel shift. This step can also be understood as a fusion of the images to form the resulting image. In the last step, the final high-resolution image is generated by combining the previously scaled and aligned images. Preferably, information from all images of the defined number of images is integrated, taking their respective subpixel shifts into account. This integration can advantageously lead to the creation of a more detailed and sharper image, which can significantly improve the resolution and quality compared to the individual source images.

[0017] The method for increasing the resolution is preferably a super-resolution algorithm, in particular a multi-image super-resolution algorithm.

[0018] Furthermore, within the scope of the invention, it can be advantageous for the defined number of images to include both regular camera images and thermal images. The regular camera images and the thermal images preferably result from parallel acquisition by the thermal imaging camera. Thus, the thermal imaging camera can comprise a camera sensor for acquiring the regular camera images and a thermal imaging camera sensor or infrared camera sensor for acquiring the thermal images. Parallel acquisition can express that a respective pair of a regular camera image and a thermal image is acquired simultaneously, wherein a temporal tolerance between the two acquisitions can be accepted. This advantageously makes it possible to establish a link between the regular camera image acquired in parallel and the thermal image.

[0019] Furthermore, it is advantageous if, within the scope of the invention, the step of determining the subpixel shift is performed based on the regular camera images in order to perform the shifting, scaling, and determination of the resulting image based on the thermal images and the subpixel shift determined based on the regular images. The regular camera images may have a higher resolution and / or a higher recognizable level of detail in the images, which may advantageously enable a more precise determination of the subpixel shift.

[0020] Furthermore, it is optionally provided that the steps of determining the subpixel shift, shifting, scaling, and determining the resulting image are performed based on the thermal images. Thus, the method for increasing the resolution can advantageously be performed independently of the regular camera images. It can also be provided that the subpixel shift is determined based on both the regular camera images and the thermal images in order to advantageously compare the respective determined subpixel shifts, for example, to detect errors.

[0021] Furthermore, within the scope of the invention, it is optionally possible for the defined number of images to be thermal images. The steps of determining the subpixel shift, shifting, scaling, and determining the resulting image can then be performed based on the thermal images. Thus, the method for increasing the resolution can advantageously be carried out independently of the regular camera images, i.e., for example, using only a thermal image or infrared camera sensor.

[0022] Furthermore, it is advantageous if, within the scope of the invention, the defined number of images includes both regular camera images and thermal images, with the regular camera images and the thermal images resulting from parallel acquisition by the thermal imaging camera. In this regard, reference is made to the above explanations regarding the analog section. The analysis and initiation steps can initially be carried out on the basis of the regular camera images. This makes it possible to first test whether the application of the method for increasing resolution with the regular camera images is possible or sensible. Within the scope of the method for increasing resolution, the subpixel shift could then be carried out on the basis of the regular camera images.Depending on the result of the contrast analysis, the analysis and initiation steps can then be performed again based on the thermal images if the result indicates that the contrast for the regular camera images falls below a defined threshold. This allows the resolution enhancement method to be performed exclusively based on the thermal images if the regular camera images exhibit insufficient contrast.

[0023] It may be provided within the scope of the invention that the method further comprises the following step: - Initiate a display of the resulting thermal image.

[0024] For this purpose, the thermal imaging camera can have a display on which the resulting thermal image is shown. It is also conceivable that the resulting thermal image can be transmitted to another data processing device on which the resulting thermal image is to be displayed.

[0025] The invention also relates to a computer program, in particular a computer program product, comprising instructions that, when executed by a computer, cause the computer to execute the method according to the invention. Thus, the computer program according to the invention provides the same advantages as those described in detail with reference to a method according to the invention.

[0026] The invention also relates to a data processing device configured to carry out the method according to the invention. The device can be, for example, a computer that executes the computer program according to the invention. The computer can have at least one processor for executing the computer program. A non-volatile data memory can also be provided, in which the computer program is stored and from which the computer program can be read by the processor for execution.

[0027] The invention may also provide a computer-readable storage medium that has the computer program according to the invention and / or includes instructions that, when executed by a computer, cause the computer to carry out the method according to the invention. The storage medium is designed, for example, as a data storage device such as a hard disk and / or a non-volatile memory and / or a memory card. The storage medium can, for example, be integrated into the computer.

[0028] Furthermore, the method according to the invention can also be implemented as a computer-implemented method.

[0029] Further advantages, features, and details of the invention will become apparent from the following description, which describes exemplary embodiments of the invention in detail with reference to the drawings. The features mentioned in the claims and in the description may be essential to the invention individually or in any combination. They show: Fig. 1 a schematic visualization of a method, a thermal imaging camera, a device, a storage medium and a computer program according to embodiments of the invention, Fig. 2 a schematic representation of a method according to embodiments of the invention.

[0030] In Fig. 1, a method 100, a thermal imaging camera 1, a device 10, a storage medium 15 and a computer program 20 according to embodiments of the invention are schematically shown.

[0031] Fig. 1 shows, in particular, an embodiment of a method 100 for applying a method for increasing the resolution of an image from a thermal imaging camera 1. In a first step 101, image data is provided, wherein the image data comprises a defined number of images, wherein the images are regular camera images and / or thermal images, wherein the image data result from a capture by the thermal imaging camera 1. In a second step 102, a contrast in the image data is analyzed. In a third step 103, the application of the method for increasing the resolution is initiated depending on a result of the analysis of the contrast in the image data.

[0032] Super-resolution is a possible method for increasing the resolution of images that can be used in thermal imaging cameras 1. One idea is that several images are taken, preferably in quick succession, and combined to create a higher-resolution image. This can be done, for example, in two steps, where any number N of images can be viewed. In a first step, the N images are registered relative to a reference image, for example using their optical flow, in order to determine how they are shifted relative to one another. In a second step, the registered images are fused to create a new image with a higher resolution. The visual image can be used to determine the subpixel shifts, i.e. the optical flow, because it has a higher resolution and can therefore determine more precise shifts.In the context of the present invention, a visual image is, in particular, an image resulting from a capture by a regular camera and representing a light spectrum visible to humans. However, the fusion must be performed particularly on the thermal images, since these are to be enhanced.

[0033] The first step is therefore primarily performed by determining the optical flow, and the second is performed using iterative methods, for example, using classical image processing methods. Alternatively, machine learning-based methods can also be used, which solve these steps using machine learning models, particularly neural networks.

[0034] Within the scope of the invention, according to embodiments, it is determined based on the contrast of the thermal images and / or the visual images whether the method for increasing the resolution of the thermal imaging camera 1, ie in particular the super-resolution algorithm, should be applied, whether it should be applied in an adapted manner or whether it should be deactivated or blocked under inadequate conditions.

[0035] One aspect of the present invention according to embodiments is thus in particular an application of the method for increasing the resolution of the thermal imaging camera 1, in particular the super-resolution algorithm in thermal imaging cameras 1, depending on the contrast of the images.

[0036] First, preferably a sequence of images is viewed, for example three images. A relative displacement of the images can then be determined in order to map the images to one another. The registered images are then fused into a single, higher-resolution image, preferably according to a method for increasing the resolution of the thermal imaging camera 1. An example of such an algorithm would be the MMCNN algorithm (Multi-Memory Convolutional Neural Network for Video Super-Resolution), but other machine learning models such as neural networks with the same blocks, i.e., image registration followed by image fusion, can also be used. Alternatively, the algorithm can also be built from two individual machine learning models, one for image registration and one for image fusion. For more accurate image registration, the higher-resolution visual images can be used.The motion determined there can then be transferred to the thermal images required for fusion, as their resolution needs to be improved. The machine learning models used are preferably already trained, so they can be used without major changes and can, at most, be retrained for better performance.

[0037] However, regardless of the method used to increase the resolution of the thermal imaging camera 1, certain conditions must preferably be met in order to achieve a high-quality result and not degrade the thermal image.

[0038] Therefore, according to exemplary embodiments, the invention relates primarily to a conditional application of the method for increasing the resolution of the thermal imaging camera 1, in particular the super-resolution. If the contrast in the images is too low, the images cannot be registered accurately, so that the subpixel shift of the individual images relative to one another is incorrectly calculated and the image resulting from the method has a poorer quality than the original image. This can be the case in particular if only the visual image is used for registration and there is no contrast in the visual image, but there is in the thermal image. This can occur, for example, when recording at night or in the dark.

[0039] Therefore, image registration should preferably be based on thermal images if they have sufficient contrast, but not on visual images, i.e., regular camera images resulting from a capture of a regular camera sensor. If both do not have sufficient contrast, the method for increasing the resolution of the thermal imaging camera 1 should preferably not be applied at all. A possible procedure according to an embodiment is shown in Fig. 2. As long as the thermal imaging camera 1 is switched on, the Fig.2 is carried out for new images, in particular permanently. In this case, a method for increasing the resolution of the thermal imaging camera 1 can be used, which determines the image registration with the visual images. If the method uses thermal images, the first part can be skipped and started with step 203, according to which the contrast in N consecutive thermal images is determined. This part can also be handled in the case of registration with visual images. The contrast in N consecutive visual images is preferably determined according to step 201 and a permanent test is carried out according to step 202 as to whether more than a specified number (e.g. half) of these N (e.g. 5) consecutive visual images have a contrast above a defined threshold.If this is the case, all visual images that meet this condition can be used for image registration, while the remaining ones are preferably discarded according to step 208. For the visual images used, the optical flow is determined in particular according to step 209 in order to obtain a subpixel shift to a reference image (e.g., the most recently used image). This subpixel shift is then preferably transferred to the thermal images according to step 210, and these are shifted in particular by entire pixels according to step 211 so that the image content overlaps as much as possible. The thermal images are then scaled or interpolated to a multiple of the original size (e.g., four times) according to step 212 and fused according to step 213, taking into account the determined subpixel shift, in order to obtain a resulting upscaled thermal image.This can then be displayed on a display of the thermal imaging camera 1 according to step 214. If at least half of the viewed visual images do not have a contrast above the defined threshold, the visual images are preferably not used for registering the method for increasing the resolution of the thermal imaging camera 1. In this case, it is preferably tested according to step 203 whether the thermal images have sufficient contrast. If, for example, only one contrast is smaller than the defined threshold in at most half of the thermal images, a sufficiently accurate registration cannot be applied to this either, so that no method for increasing the resolution of the thermal imaging camera 1, in particular no super-resolution, is applied. Accordingly, the original image is then preferably displayed on the display of the thermal imaging camera 1 according to step 205.If more than half of the thermal images exhibit a contrast above the defined threshold, the requirements for applying the method for increasing the resolution of thermal imaging camera 1, i.e., in particular, super-resolution, are met. In this case, the thermal images with insufficient contrast are preferably discarded according to step 206, and the subpixel shift of the remaining images relative to the reference image can be determined using optical flow according to step 207.

[0040] Subsequently, the thermal images are preferably first shifted by entire pixels according to the previously determined optical flow according to step 211 and upscaled to a multiple of the original resolution according to step 212. Finally, these upscaled thermal images are merged, in particular, into a single, high-resolution image according to step 213 and displayed on the display of the thermal imaging camera 1 according to step 214.

[0041] Since a possible application case according to one embodiment is a real-time application of the thermal imaging camera 1, this process is preferably executed in a loop over the entire usage time of the thermal imaging camera 1.

[0042] The method for increasing the resolution, i.e. in particular the super-resolution algorithm, can be used both in the real-time application on the thermal imaging camera 1 and when saving the images or in a computer / smartphone app for post-processing. In the case of post-processing, it is preferably assessed in the device whether the contrast in visual and thermal images allows a method for increasing the resolution, in particular super-resolution, and a corresponding quality indicator can be created. After transferring the images together with the quality indicator, it can then be assessed on the computer or smartphone whether the conditions for applying the method for increasing the resolution, in particular super-resolution, are met, and if so, the method for increasing the resolution, in particular the super-resolution algorithm, can be executed.In the case of a real-time application, all these steps can be performed on the device, i.e. in particular on the thermal imaging camera 1.

[0043] The above explanation of the embodiments describes the present invention exclusively within the scope of examples. Of course, individual features of the embodiments can be freely combined with one another, provided they are technically feasible, without departing from the scope of the present invention.

Claims

[1] Method (100) for applying a method for increasing a resolution of an image of a thermal imaging camera (1), comprising the following steps: - providing (101) image data, wherein the image data comprises a defined number of images, wherein the images are regular camera images and / or thermal images, wherein the image data result from a capture of the thermal imaging camera (1), - analyzing (102) a contrast in the image data, - initiating (103) the application of the method for increasing the resolution depending on a result of the analysis of the contrast in the image data. [2] Method (100) according to claim 1, characterized by that the analysis (102) comprises the following steps: - Defining a threshold for the contrast in the image data, - Comparing the respective contrasts of individual images of the image data with the defined contrast threshold. [3] Method (100) according to claim 2, characterized by that, if the result of the comparison shows that the contrast of a defined number of images is above the defined threshold, only images of the defined number of images whose contrast is above the defined threshold are provided within the scope of the initiation (103). [4] Method (100) according to claim 2 or 3, characterized by that if the result of the comparison shows that the contrast of a specified number of images is below the defined threshold, the initiation (103) is blocked. [5] Method (100) according to one of the preceding claims, characterized by that the method for increasing the resolution of the thermal imaging camera image (1) includes the following steps: - Determining a subpixel shift of the defined number of images compared to a reference image, wherein the reference image is one of the defined number of images, - Shifting the defined number of images based on the determined subpixel shift so that they are aligned with the reference image, - Scaling the defined number of images by a defined scaling factor, - Determining a resulting image based on the scaled defined number of images and the determined subpixel shift. [6] Method (100) according to claim 5, characterized by that the defined number of images are both regular camera images and thermal images, wherein the regular camera images and / or the thermal images result from a parallel acquisition of the thermal imaging camera (1). [7] Method (100) according to claim 6, characterized bythat the step of determining the subpixel shift is performed on the basis of the regular camera images in order to perform the shifting, scaling and determining of the resulting image on the basis of the thermal images and the subpixel shift determined on the basis of the regular images. [8] Method (100) according to claim 6, characterized by that the step of determining the subpixel shift, shifting, scaling and determining the resulting image are performed based on the thermal images. [9] Method (100) according to claim 5, characterized by , that the defined number of images are thermal images, wherein the step of determining the subpixel shift, shifting, scaling and determining the resulting image are performed based on the thermal images. [10] Method (100) according to one of the preceding claims, characterized by , that the defined number of images are both regular camera images and thermal images, whereby the regular camera images and the thermal images result from a parallel acquisition of the thermal imaging camera (1), wherein the steps of analyzing (102) and initiating (103) are initially carried out on the basis of the regular camera images, wherein the steps of analyzing (102) and initiating (103) are carried out again on the basis of the thermal images depending on the result of the analysis of the contrast if the result indicates that the contrast for the regular camera images falls below a defined threshold value. [11] Computer program (20) comprising instructions which, when the computer program (20) is executed by a computer (10), cause the computer (10) to carry out the method (100) according to one of the preceding claims. [12] Device (10) for data processing, which is arranged to carry out the method (100) according to one of claims 1 to 10. [13] A computer-readable storage medium (15) comprising instructions which, when executed by a computer (10), cause the computer (10) to carry out the steps of the method (100) according to any one of claims 1 to 10.

Images