Method for applying a method for increasing the resolution of an image from a thermal imaging camera

Abstract

The invention relates to a method (100) for applying a method for increasing the resolution of an image of a thermal imaging camera (1), comprising the following steps: - providing (101) image data, wherein the image data comprises a defined number of images, wherein the images are regular camera images and / or thermal images, wherein the image data result from a capture of the thermal imaging camera (1), - determining (102) a movement, wherein the movement is represented in the defined number of images of the image data by a shift of pixel values, - initiating (103) the application of the method for increasing the resolution depending on the determined movement. Furthermore, the invention relates to a computer program, a device and a storage medium for this purpose.

Description

[0001] The invention relates to a method for applying a method for increasing the resolution of an image from a thermal imaging camera. Furthermore, the invention relates to a computer program, a device, and a storage medium for this purpose. State of the art

[0002] A key factor in purchasing a thermal imaging camera is the resolution of the thermal image. The higher the resolution, the more details can be seen and the more applications it can serve. Resolution is therefore an important feature of high-quality thermal imaging cameras. However, since it is directly dependent on the quality and therefore the price of the infrared sensors used, better resolution from the sensors automatically means higher price for the components. To improve resolution without more expensive components, alternative algorithms can be used to improve image quality. Super-resolution algorithms, for example, are used to improve resolution. Both classic algorithms (based on conventional image processing) and machine learning-based algorithms can be used here.

[0003] In a real-time application, inadequate conditions such as excessive image change due to strong camera movements for a super-resolution algorithm may lead to a deterioration of the processed images compared to the source images. Disclosure of the invention

[0004] The invention relates to a method having the features of claim 1, a computer program having the features of claim 9, a device having the features of claim 10, and a computer-readable storage medium having the features of claim 11. Further features and details of the invention emerge from the respective subclaims, the description, and the drawings. Features and details described in connection with the method according to the invention naturally also apply in connection with the computer program according to the invention, the device according to the invention, and the computer-readable storage medium according to the invention, and vice versa, so that with regard to the disclosure of the individual aspects of the invention, reference is or can always be made to each other.

[0005] The invention particularly relates to a method for applying a method for increasing the resolution of an image from a thermal imaging camera, comprising the following steps, wherein the steps can be carried out repeatedly and / or sequentially. Increasing the resolution is in particular a technical process in which the number of pixels in the image from the thermal camera is increased in order to improve the level of detail and sharpness of the image as perceived by a user. This can be achieved in various ways, for example by interpolation, wherein new pixels are inserted between the existing pixels of the image. Furthermore, super-resolution algorithms can be applied, which are a specialized form of image resolution improvement. In particular, missing details in low-resolution images are reconstructed.Super-resolution algorithms can often provide more detailed and clearer results than simple interpolation. The thermal imaging camera is preferably a handheld thermal imaging camera, i.e., a thermal imaging camera that can be held and operated by a user.

[0006] In a first step, image data is preferably provided, wherein the image data comprises a defined number of images, wherein the images are regular camera images and / or thermal images, wherein the image data result from a capture by the thermal imaging camera. The regular camera images can also be understood and referred to as visual images within the scope of the present invention and are in particular images that represent a light spectrum visible to a human. The regular camera images can be captured by a camera sensor of the thermal imaging camera. The thermal images are in particular infrared images, i.e. images in a spectrum of the infrared range. The thermal images can be captured with an infrared camera sensor of the thermal imaging camera.

[0007] In a further step, a movement is preferably determined, wherein the movement is represented by a shift in pixel values ​​in the defined number of images of the image data. The movement is, in particular, a movement of the scene captured by the thermal imaging camera and / or a movement of the thermal imaging camera itself. The shift in pixel values ​​describes, in particular, that objects represented by corresponding pixel values ​​in the image data shift in the image data based on these corresponding pixel values. For example, the pixel values ​​representing a specific object in one image are shifted by a certain number of pixels in the next captured image.

[0008] In a further step, the application of the method for increasing the resolution is preferably initiated depending on the determined movement. This means in particular that the application of the method for increasing the resolution is only applied, for example, if no excessive or insufficient movement is determined in the image data. Various methods known in the prior art, such as interpolation or super-resolution, can be used as a method for increasing the resolution. It is possible that the method for increasing the resolution provides poorer results if the movement in the image data is too high or too low. The present method can therefore advantageously be used to evaluate whether the image data is suitable for the method for increasing the resolution based on the movement evaluation parameter.

[0009] It is also conceivable that the movement is determined based on an analysis of the displacement of pixel values ​​in the image data. The movement can be determined by recognizing a respective object based on object detection and / or classification and comparing a respective position of the recognized object between the individual images of the image data. For object detection and / or classification, segmentation algorithms or detection or classification methods based on machine learning, for example, can be used. By analyzing the displacement of pixel values ​​in the image data, the movement in the scene captured by the thermal imaging camera and the movement of the thermal imaging camera itself can advantageously be taken into account simultaneously.

[0010] Furthermore, within the scope of the invention, it is conceivable that the determination of the movement is carried out based on an evaluation of sensor data from at least one sensor, in particular an acceleration sensor, of the thermal imaging camera. Based on the sensor data from the sensor, in particular the acceleration sensor, it is advantageously possible to infer the camera's own movement. The sensor can be designed as a multidimensional motion sensor or acceleration sensor and provide detection of an acceleration and / or a rotation rate.

[0011] Optionally, it is conceivable that the method further comprises the following steps: - Defining at least one threshold for the movement, - comparing the determined movement with the at least one defined threshold value for the movement,

[0012] The threshold can, for example, be a specific number of pixels by which at least one object is shifted in individual images of the image data. Alternatively or additionally, the threshold could specify a value for the sensor data, in particular a specific acceleration value. Acceleration values ​​within a defined time window could also be analyzed and evaluated. For example, the acceleration values ​​within the defined time window could be added and compared with a defined threshold for the acceleration values ​​within the defined time window.

[0013] The application of the method for increasing the resolution can then be initiated if the specific movement is below and / or above the at least one defined threshold value for the movement. In this way, both too low and too high movement can advantageously be taken into account accordingly, since in both cases the method for increasing the resolution could produce disadvantageous results. If the initiation of the application of the method for increasing the resolution is blocked, it can be provided that a corresponding output is sent to a user of the thermal imaging camera, for example via a display or a loudspeaker of the thermal imaging camera. The output could inform the user that the method for increasing the resolution, which can be implemented, for example, by a specific mode in the thermal imaging camera, is currently not possible.

[0014] Preferably, within the scope of the invention, it can be provided that the method for increasing the resolution comprises the following steps, wherein the steps are preferably carried out one after the other. In a first step, a subpixel shift of the defined number of images relative to a reference image is preferably determined, wherein the reference image is one of the defined number of images. In this step, in particular for each image of the defined number of images, a shift at the subpixel level relative to the selected reference image is determined, for example by determining an optical flow. The selected reference image can, for example, be the last image of the defined number of images. This shift is necessary in particular to compensate for differences in the camera position or perspective between the images.Subpixel-level accuracy advantageously enables more precise alignment, which can increase the quality of the resulting image.

[0015] In a further step, the defined number of images are preferably shifted based on the determined subpixel shift so that they are aligned with the reference image. After the subpixel shift has been determined for each of the defined number of images, the images can be shifted accordingly to align them with the reference image. This alignment can advantageously ensure that corresponding points in all images coincide, which is particularly necessary for subsequent image fusion.

[0016] In a further step, the defined number of images are preferably scaled by a defined scaling factor. For example, a quadruple scaling, i.e., a defined scaling factor of four, can be provided. This can be done for various reasons, such as adapting the images to a specific target resolution or standardizing image sizes to improve alignment and / or reduce the computational load during subsequent image fusion. During the scaling process, the images of the defined number of images can be interpolated.

[0017] In a further step, a resulting image is preferably determined based on the scaled, defined number of images and the determined subpixel shift. This step can also be understood as a fusion of the images to form the resulting image. In the last step, the final high-resolution image is generated by combining the previously scaled and aligned images. Preferably, information from all images of the defined number of images is integrated, taking their respective subpixel shifts into account. This integration can advantageously lead to the creation of a more detailed and sharper image, which can significantly improve the resolution and quality compared to the individual source images.

[0018] The method for increasing the resolution is preferably a super-resolution algorithm, in particular a multi-image super-resolution algorithm.

[0019] Furthermore, within the scope of the invention, it can be advantageous for the defined number of images to include both regular camera images and thermal images. The regular camera images and the thermal images preferably result from parallel acquisition by the thermal imaging camera. Thus, the thermal imaging camera can comprise a camera sensor for acquiring the regular camera images and a thermal imaging camera sensor or infrared camera sensor for acquiring the thermal images. Parallel acquisition can express that a respective pair of a regular camera image and a thermal image is acquired simultaneously, wherein a temporal tolerance between the two acquisitions can be accepted. This advantageously makes it possible to establish a link between the regular camera image acquired in parallel and the thermal image.

[0020] The step of determining the subpixel shift can then be performed based on the regular camera images in order to perform the shifting, scaling, and determination of the resulting image based on the thermal images and the subpixel shift determined based on the regular images. The regular camera images may have a higher resolution and / or a higher recognizable level of detail in the images, which may advantageously enable a more precise determination of the subpixel shift. Alternatively or additionally, the steps of determining the subpixel shift, shifting, scaling, and determining the resulting image can be performed based on the thermal images. Thus, the method for increasing the resolution can advantageously be performed independently of the regular camera images.It can also be provided that the subpixel shift is determined both on the basis of the regular camera images and on the basis of the thermal images in order to be able to advantageously compare the respective determined subpixel shifts, for example in order to detect errors.

[0021] According to a further advantage, it can be provided that the defined number of images are thermal images, wherein the thermal images result from a capture by the thermal imaging camera. The steps of determining the subpixel shift, shifting, scaling, and determining the resulting image can then be performed based on the thermal images. Thus, the method for increasing the resolution can advantageously be carried out independently of the regular camera images, i.e., for example, using only a thermal imaging or infrared camera sensor.

[0022] It may be provided within the scope of the invention that the method further comprises the following step: - Initiate a display of the resulting thermal image.

[0023] For this purpose, the thermal imaging camera can have a display on which the resulting thermal image is shown. It is also conceivable that the resulting thermal image can be transmitted to another data processing device on which the resulting thermal image is to be displayed.

[0024] The invention also relates to a computer program, in particular a computer program product, comprising instructions that, when executed by a computer, cause the computer to execute the method according to the invention. Thus, the computer program according to the invention provides the same advantages as those described in detail with reference to a method according to the invention.

[0025] The invention also relates to a data processing device configured to carry out the method according to the invention. The device can be, for example, a computer that executes the computer program according to the invention. The computer can have at least one processor for executing the computer program. A non-volatile data memory can also be provided, in which the computer program is stored and from which the computer program can be read by the processor for execution.

[0026] The invention may also provide a computer-readable storage medium that has the computer program according to the invention and / or includes instructions that, when executed by a computer, cause the computer to carry out the method according to the invention. The storage medium is designed, for example, as a data storage device such as a hard disk and / or a non-volatile memory and / or a memory card. The storage medium can, for example, be integrated into the computer.

[0027] Furthermore, the method according to the invention can also be implemented as a computer-implemented method.

[0028] Further advantages, features, and details of the invention will become apparent from the following description, which describes exemplary embodiments of the invention in detail with reference to the drawings. The features mentioned in the claims and in the description may be essential to the invention individually or in any combination. They show: Fig. 1 a schematic visualization of a method, a thermal imaging camera with a sensor, a device, a storage medium and a computer program according to embodiments of the invention, Fig. 2 a schematic representation of a method according to embodiments of the invention.

[0029] In Fig. 1, a method 100, a thermal imaging camera 1 with a sensor 2, a device 10, a storage medium 15 and a computer program 20 according to embodiments of the invention are schematically shown.

[0030] Fig. 1 shows in particular an embodiment of a method 100 for applying a method for increasing the resolution of an image from a thermal imaging camera 1. In a first step 101, image data is provided, wherein the image data comprises a defined number of images, wherein the images are regular camera images and / or thermal images, wherein the image data result from a capture by the thermal imaging camera 1. In a second step 102, a movement is determined, wherein the movement in the defined number of images of the image data is represented by a shift of pixel values. In a third step 103, the application of the method for increasing the resolution is initiated depending on the determined movement.

[0031] Super-resolution is a possible method for increasing the resolution of images that can be used in thermal imaging cameras 1. One idea is that several images are taken, preferably in quick succession, and combined to create a higher-resolution image. This can be done, for example, in two steps, where any number N of images can be viewed. In a first step, the N images are registered relative to a reference image, for example using their optical flow, in order to determine how they are shifted relative to one another. In a second step, the registered images are fused to create a new image with a higher resolution. The visual image can be used to determine the subpixel shifts, i.e. the optical flow, because it has a higher resolution and can therefore determine more precise shifts.In the context of the present invention, a visual image is, in particular, an image resulting from a capture by a regular camera and representing a light spectrum visible to humans. However, the fusion must be performed particularly on the thermal images, since these are to be enhanced.

[0032] The first step is therefore primarily carried out by determining the optical flow, and the second is carried out using iterative methods, for example, using classical image processing methods. Alternatively, machine learning-based methods can also be used, which solve these steps using machine learning models, particularly neural networks.

[0033] In a real-time application, inadequate conditions such as excessive image change due to strong movements of the thermal imaging camera 1 or a scene captured by the thermal imaging camera 1 for a method for increasing the resolution such as a super-resolution algorithm can lead to a deterioration of the processed images compared to the original images, so that according to embodiments of the invention, based on the movement, either the method for increasing the resolution is applied, which can be adjusted if necessary, or is deactivated, blocked or not applied in the case of inadequate conditions.

[0034] One aspect of the present invention is therefore in particular an application of a method for increasing the resolution such as a super-resolution in thermal imaging cameras 1 as a function of movement.

[0035] First, a sequence of images, for example, three, is preferably examined. A relative displacement of the images can then be determined to map them to one another. The registered images are then preferably fused into a single, higher-resolution image. An example of such an algorithm would be the MMCNN (Multi-Memory Convolutional Neural Network for Video Super-Resolution) algorithm, but other machine learning models with the same blocks, i.e., image registration followed by image fusion, can also be used.

[0036] Alternatively, the algorithm can be built from two individual machine learning models, particularly neural networks, one for image registration and one for image fusion. Furthermore, the visual image can be used for more accurate image registration due to the higher resolution of the visual image compared to the thermal image, and the thermal image for fusion, as the latter should preferably be improved. The machine learning models used are preferably pre-trained, so they can be used without major changes and can, at most, be retrained for better performance.

[0037] However, even with these algorithms, regardless of which one is specifically used, certain prerequisites must preferably be met to achieve a high-quality result and not further degrade the thermal image due to the algorithm. Therefore, according to exemplary embodiments, the invention relates in particular to the conditional application of the method for increasing resolution, such as super-resolution.

[0038] Excessive motion between consecutive images can significantly reduce the area where the resolution enhancement technique could add value, resulting in uneven resolution across the image. Therefore, according to embodiments of the invention, the resolution enhancement technique is preferably deactivated when excessive motion is detected and activated when the motion is small enough again.

[0039] An example of a corresponding method is described in Fig. 2 shown.

[0040] As long as the thermal imaging camera 1 is switched on, the Fig.2 can be carried out permanently for new images. In this case, it can be determined either algorithm-based or sensor-based whether the displacement of the images relative to one another exceeds a certain limit. Sensor-based this occurs according to step 301, for example using a sensor 2, in particular a multi-dimensional motion sensor or acceleration sensor. Based on the values, e.g. motion vectors and / or rotation rates, which the sensor 2 returns, it can be determined according to step 302 whether, for example, within the last second the offset of the recorded images lies within a certain limit (e.g. 5%). If the movement exceeds this limit, the method for increasing the resolution, in particular super-resolution, is preferably not applied and instead the original image is shown on a display of the thermal imaging camera 1 according to step 303.However, if the movement is below the limit, the method for increasing the resolution, i.e., in particular, super-resolution, can be activated. In particular, according to step 304, a subpixel shift of the images is first registered, which is then adopted for the thermal images according to step 204. Then, preferably according to step 205, the thermal images are mapped to one another based on the subpixel shift, i.e., adjusted with regard to their alignment to a reference image. Finally, a high-resolution image is created by merging the images, in particular according to step 206. The optimized image can then be displayed on a display of the thermal imaging camera 1 according to step 207.

[0041] As an alternative to sensor 2, the movement can also be estimated from the visual image and the thermal image according to step 201. In this case, the extent to which the pixels are shifted relative to one another is preferably determined based on the image registration according to step 202. If the shift of a certain number of consecutive images (e.g. 5) is more than a limit value of the total number of pixels (in the x and y directions) - i.e., e.g., more than 5% of the total number of pixels (for a resolution of 256*192, correspondingly 12.8 pixels in the x direction or 9.6 pixels in the y direction) - the method for increasing the resolution, i.e., in particular, super-resolution, is preferably not applied.

[0042] In this case, the original thermal image is displayed on the screen, in particular according to step 203. However, if the subpixel shift is below the threshold, the method for increasing the resolution can be used. Then, in particular, the already determined subpixel shift is used and applied to the thermal images according to step 204, so that they can be mapped to one another according to step 205. Subsequently, a high-resolution image can be created by merging the images according to step 206. This high-resolution image is preferably then displayed on a display of the thermal imaging camera 1 according to step 207.

[0043] In a real-time application of the thermal imaging camera 1, the method according to embodiments can be executed in a loop over the entire usage time of the thermal imaging camera 1.

[0044] The method for increasing the resolution can be used both in the real-time application on the thermal imaging camera 1 and when saving the images or in the computer or smartphone app for post-processing. In the case of post-processing, it is preferably assessed whether the movement profile allows the method for increasing the resolution, in particular super-resolution, and a corresponding quality indicator is created. After the images have been transmitted, together with the quality indicator, it can then be assessed on the computer or smartphone whether the conditions for applying the method for increasing the resolution are met, and if so, the method can be executed. In a real-time application, all of these steps can be carried out by the thermal imaging camera 1 according to embodiments.

[0045] The above explanation of the embodiments describes the present invention exclusively within the scope of examples. Of course, individual features of the embodiments can be freely combined with one another, provided they are technically feasible, without departing from the scope of the present invention.

Claims

[1] Method (100) for applying a method for increasing a resolution of an image of a thermal imaging camera (1), comprising the following steps: - providing (101) image data, wherein the image data comprises a defined number of images, wherein the images are regular camera images and / or thermal images, wherein the image data result from a capture of the thermal imaging camera (1), - determining (102) a movement, wherein the movement is represented in the defined number of images of the image data by a shift of pixel values, - initiating (103) the application of the method for increasing the resolution depending on the determined movement. [2] Method (100) according to claim 1, characterized by that the determination (102) of the movement is carried out on the basis of an analysis of the displacement of pixel values in the image data. [3] Method (100) according to one of the preceding claims, characterized bythat the determination (102) of the movement is carried out on the basis of an evaluation of sensor data from at least one sensor (2), in particular an acceleration sensor, of the thermal imaging camera (1). [4] Method (100) according to one of the preceding claims, characterized by that the method (100) further comprises the following steps: - Defining at least one threshold for the movement, - comparing the determined movement with the at least one defined threshold value for the movement, wherein the application of the method for increasing the resolution is initiated if the determined movement is below and / or above the at least one defined threshold value for the movement. [5] Method (100) according to one of the preceding claims, characterized by that the method to increase the resolution includes the following steps: - Determining a subpixel shift of the defined number of images compared to a reference image, wherein the reference image is one of the defined number of images, - Shifting the defined number of images based on the determined subpixel shift so that they are aligned with the reference image, - Scaling the defined number of images by a defined scaling factor, - Determining a resulting image based on the scaled defined number of images and the determined subpixel shift. [6] Method (100) according to claim 5, characterized by , that the defined number of images are both regular camera images and thermal images, whereby the regular camera images and the thermal images result from a parallel acquisition of the thermal imaging camera (1), wherein the step of determining the subpixel shift is performed on the basis of the regular camera images to perform the shifting, scaling and determining of the resulting image on the basis of the thermal images and the subpixel shift determined on the basis of the regular images, or wherein the steps of determining the subpixel shift, shifting, scaling and determining the resulting image are performed based on the thermal images. [7] Method (100) according to claim 5, characterized by that the defined number of images are thermal images, wherein the thermal images result from a capture of the thermal imaging camera (1), wherein the step of determining the subpixel shift, shifting, scaling and determining the resulting image are carried out on the basis of the thermal images. [8] Method (100) according to one of claims 5 to 7, characterized bythat the method (100) further comprises the following step: - Initiate a display of the resulting thermal image. [9] Computer program (20) comprising instructions which, when the computer program (20) is executed by a computer (10), cause the computer (10) to carry out the method (100) according to one of the preceding claims. [10] Device (10) for data processing, which is arranged to carry out the method (100) according to one of claims 1 to 8. [11] A computer-readable storage medium (15) comprising instructions which, when executed by a computer (10), cause the computer (10) to carry out the steps of the method (100) according to any one of claims 1 to 8.

Images