System for identifying thermal defects on objects, method for identifying thermal defects on objects and computer-readable storage medium

Abstract

The invention relates to a system for identifying thermal defects on objects, a method for identifying thermal defects on objects and a computer-readable storage medium. In particular, the invention relates to a system for identifying thermal defects on objects, said system comprising the following: - a mobile terminal (100) which is designed to generate an infrared image (IR) and a colour image (RGB) of an object (O); - an evaluation server (200) which has the following units: - an object determination unit (210) which is designed to generate object information (OI) at least based on the colour image (RGB) and / or the infrared image (IR); - a defect determination unit (220) which is designed to generate defect information (DI), in particular the type, position, size and / or characteristic value of a thermal defect, based at least on the infrared image (IR) and the object information (OI); - a measure determination unit (230) which is designed to ascertain measures (M) based on the defect information (DI) and / or the object information (OI). The mobile terminal (100) and the evaluation server (200) are connected for communication, in particular for transmitting the infrared image (IR), the colour image (RGB) and / or the measures (M).

Description

[0001] System for identifying thermal defects on objects, method for identifying thermal defects on objects, and computer-readable storage medium

[0002] The invention relates to a system for identifying thermal defects on objects, a method for identifying thermal defects on objects and a corresponding computer-readable storage medium.

[0003] Particularly for cost and energy efficiency reasons, it is desirable in many situations to insulate a building as effectively as possible against heat loss to the environment. For example, in the field of building construction, a variety of different materials and techniques for thermal insulation are known to prevent heat from being lost from the building's interior to the environment. Cooling down of building areas can also lead to increased humidity, mold growth, and thus damage to the building structure.

[0004] It is known to record the thermal state of objects using thermographic methods. This involves recording the intensity of infrared radiation emitted by an object and interpreting it as a measure of its temperature dissipation. However, manual evaluation of the recorded infrared radiation is time-consuming, especially when additional properties of the object need to be taken into account.

[0005] FR 3 074 596 B1 discloses a method for characterizing test objects using spectral images, in particular infrared images, as well as a corresponding device. In the method described therein, a volume of pixel values ​​is formed by acquiring a plurality of spectral images. Input data containing an observed parameter as a function over the plurality of spectral images is extracted from the volume of pixel values. Based on the input data, a neural network is trained to extract a feature to be classified. Finally, the extracted feature is classified into one of several classes. FR 3 113 530 B1 also relates to the classification of objects, in particular with regard to mechanical or thermal properties, using artificial neural networks (ANNs).For this purpose, a method is proposed in which a first ANN is trained based on a database of general measurement or model data. Furthermore, a second ANN, parameterized by the trained first ANN, is trained based on specific measurements taken on an object to be classified. The actual classification of the acquired measurement data is then performed exclusively using the second ANN.

[0006] The object of the present invention is to optimize the thermal properties of objects, in particular buildings and building elements. In particular, one object of the present invention is to provide an improved system and a corresponding method for this purpose.

[0007] The problem is solved by a system for identifying thermal defects on objects according to claim 1.

[0008] In particular, the problem is solved by a system for identifying thermal defects on objects, which has the following:

[0009] - a mobile terminal configured to generate an infrared image and a color image of an object;

[0010] - an evaluation server comprising the following units:

[0011] - an object determination unit configured to generate object information based at least on the color image and / or the infrared image;

[0012] - a defect determination unit configured to generate defect information, in particular position, type and / or characteristic value of a thermal defect, based at least on the infrared image and the object information;

[0013] - an action determination unit configured to determine measures based on the defect information and / or the object information.

[0014] The mobile device and the evaluation server are communicatively connected, in particular for transmitting the infrared image, the color image, and / or the measures. In the context of the present application, a thermal defect in an object can be understood, in particular, as a thermal bridge, i.e., an area or component of the object that conducts heat better than an adjacent area of ​​the object. The object can be any physical object that may exhibit a thermal defect, such as buildings, building elements (windows, doors, facades), vehicles, machines, and pipelines.

[0015] One idea of ​​the present invention is to use a mobile terminal to detect the thermal state of the object on the basis of two images of the object with different frequency spectrum, namely a color image and an infrared image.

[0016] An infrared image (IR image) can be understood as a recording of infrared radiation, i.e. in the spectral range from 780 nm to 1 mm. A color image can be understood as a recording of visible light radiation, i.e. in the spectral range from approximately 380 nm to 780 nm. An infrared camera (IR camera) or a color image camera (RGB camera) is understood below to be an image sensor or image sensor array that is designed to detect infrared radiation or visible light. An image sensor or image sensor array that can detect both visible light and infrared radiation is referred to as an RGB-IR camera.

[0017] By capturing a color image and an infrared image, the individual advantages of both image types can be combined. Object detection is generally possible reliably using a color image. The infrared image, on the other hand, provides information about the heat radiated by the object, which can be used to identify thermal defects. At the same time, infrared images are typically more robust against insufficient or disruptive lighting conditions. The infrared image can therefore also be used optionally for object detection.

[0018] Preferably, the color image and the infrared image correspond to each other in the sense that they were taken at essentially the same time and from the same position. However, this is not mandatory for the present invention, as long as both images show (at least partially) the same object.

[0019] The system according to the invention is designed to generate object information based at least on the color image, the infrared image, or both images. The object information can, in particular, indicate the type of object, e.g., indicate that it is a window of a building.

[0020] Furthermore, the system according to the invention is designed to generate defect information based on the infrared image and the generated object information. The defect information can, for example, include the following information:

[0021] - the type of thermal defect;

[0022] - the position of the thermal defect, in particular as a relative position with respect to the object; and

[0023] - a characteristic value of the thermal defect, in particular absolute temperature and / or temperature difference to an environment of the thermal defect.

[0024] Finally, the system according to the invention is designed to determine measures based on the generated defect information and / or the generated object information. These measures can, in particular, be measures suitable for remedying the thermal effect on the object, for example, thermal insulation materials and / or structural measures. Taking the defect information and / or object information into account offers the advantage that only those measures are determined that are suitable, for example, with regard to the size and nature of the thermal defect.

[0025] In this way, the system according to the invention captures various pieces of information about the object, which are linked by the evaluation server to identify thermal defects on an object and determine suitable countermeasures. Thus, thermal defects can be detected with minimal effort and high precision. In one embodiment, the mobile device has an RGB camera and is connected to an external infrared camera and / or RGB-IR camera.

[0026] In this way, the RGB camera usually integrated in mobile devices (smartphones, tablet computers, etc.) can be used to create the color image, while an external infrared camera and / or an RGB-IR camera is used to create the RGB image.

[0027] Thus, the functionality of the mobile device can be extended in a particularly simple manner by an IR camera, while maintaining the compact unit of the device.

[0028] In a further embodiment, the object determination unit and / or the defect determination unit each comprise an artificial neural network.

[0029] In particular, the object determination unit may comprise a first neural network configured to generate the object information based on a color image, on an infrared image, or on a pair of color image and associated infrared image.

[0030] In particular, the defect determination unit may comprise a second neural network configured to generate the defect information based on the infrared image and the generated object information.

[0031] The first and / or second neural network can, in particular, be a convolutional neural network (CNN). This allows for a high quality and reliability of the classification performed.

[0032] In a further embodiment, the defect determination unit may be configured to generate the defect information further based on metadata. The metadata may specify at least one of the following variables with respect to the object:

[0033] - an internal temperature;

[0034] - an outside temperature; - a GPS position;

[0035] - a brightness value; and

[0036] - a design parameter, in particular the ll value.

[0037] The ll value is a thermal transmittance coefficient (or thermal insulation value) that describes a measure of the heat transfer of a building element (house wall, door, window, etc.) due to a temperature difference.

[0038] Furthermore, metadata on the building materials used in the object, e.g., their materials and / or thermal properties, can be provided by the user as additional information. The defect information unit can then be configured to calculate an ll value based on the additional information.

[0039] The metadata may in particular relate to the state of the object at the time the infrared image and / or the color image were taken.

[0040] Taking metadata into account offers the advantage of being able to verify whether permissible conditions existed at the time the images were taken. For example, some thermographic methods only produce meaningful results if the outside temperature in the vicinity of a building is not significantly higher than 10°C and the interior of the building is heated. Furthermore, thermographic methods generally prefer to take images at relatively low brightness (e.g., less than 1500 lux) to prevent building elements from heating up due to solar radiation. This prevents incorrect assessment of the object due to impermissible measurement conditions.

[0041] On the other hand, the additional information can also be incorporated into the assessment of thermal defects, i.e., the generation of defect information. For example, a GPS position of the building can provide information about typical climatic conditions (especially temperature, wind, and weather conditions), a U-value can provide information about transmission heat losses, etc. This can increase the precision of identifying and assessing thermal defects.

[0042] In a further embodiment, the mobile terminal is designed to detect a temperature by means of a temperature sensor that is connected to the mobile terminal, in particular wirelessly, and to provide the temperature to the evaluation server.

[0043] In particular, the mobile device can be configured to establish a connection to an external, Bluetooth-enabled temperature sensor via an integrated Bluetooth interface. Instead of a Bluetooth connection, a wireless connection can also be established via Wi-Fi, NFC, ZigBee, or Z-Wave. This allows the functionality of the mobile device to be easily expanded, and the measured temperature can be provided to the analysis server as metadata.

[0044] In a further embodiment, the mobile terminal has a GPS sensor and is designed to provide a GPS location to the evaluation server.

[0045] In this way, a GPS location of the object can be recorded and provided to the evaluation server as meta information, thereby improving the identification of the thermal defect.

[0046] Likewise, the mobile device can be designed to capture one or more of the other meta-information via integrated sensors (e.g. brightness sensor), external sensors or through manual data entry.

[0047] In a further embodiment, the mobile terminal is designed to retrieve at least some of the meta information from a smart home controller associated with the object, in particular a building, and to provide it to the evaluation server.

[0048] Smart home control systems in buildings are networked with a variety of sensors (e.g., temperature sensors, humidity sensors, etc.), which can also provide at least some of the above-mentioned metadata. According to this embodiment, the classification of thermal defects can also be improved by taking this additional information into account.

[0049] In a further embodiment, the object determination unit and / or the defect determination unit is designed to generate the object information and / or the defect information based on a plurality of infrared images and / or a plurality of color images.

[0050] The multiple infrared images or the multiple color images can, in particular, each comprise an external view of the object and an internal view of the object. Preferably, the external view and the internal view show at least partially the same area of ​​the object, for example, a window of a building from the inside and outside. By combining multiple images, thermal defects, such as thermal bridges, can be identified more precisely.

[0051] The system according to the invention can also be designed to create a new color image based on a plurality of mutually similar images, for example a plurality of external views as color or infrared images, by averaging or combining, on the basis of which the object determination and / or the defect determination is carried out.

[0052] The object is further achieved by a method for identifying thermal defects in objects. The method can be carried out, in particular, by a system as described above.

[0053] In particular, the method may comprise the following steps: a) capturing an infrared image and a color image of an object; b) determining object information based at least on the color image and / or the infrared image; c) determining defect information, in particular the type, position, size, and / or characteristic value of a thermal defect, based at least on the infrared image and the object information; d) determining measures based on the defect information and / or the object information.

[0054] In particular, step a) can be carried out by a corresponding software component, in particular an application (app) on a mobile device and steps b) to c) can be carried out on an evaluation server.

[0055] In one embodiment of the method, step a) comprises capturing a plurality of infrared images and / or a plurality of color images, in particular as an external view of the object and an internal view of the object. Step b) and / or step c) are performed based on the plurality of infrared images and / or the plurality of color images.

[0056] In a further embodiment of the method, the defect information is further generated based on meta-information that indicates at least one of the following variables with respect to the object:

[0057] - an internal temperature;

[0058] - an outside temperature;

[0059] - a GPS position;

[0060] - a brightness value; and

[0061] - a design parameter, in particular the ll value.

[0062] With regard to the method for identifying thermal defects on objects and its embodiments, similar technical advantages and effects arise as have been described in connection with the system according to the invention.

[0063] Furthermore, the object is achieved by a computer-readable storage medium. The computer-readable storage medium contains instructions that cause at least one processor to implement a method as described above when the instructions are executed by the at least one processor.

[0064] With regard to the computer-readable storage medium, similar technical advantages and effects arise as those described in connection with the system according to the invention. At this point, it should be noted that the features and the advantages achievable thereby, which have been described with reference to the system according to the invention, are applicable or transferable to the methods according to the invention, and vice versa. Specifically, in the context of the present description of the invention, the components of the system can be designed to carry out the method steps according to the invention. Likewise, the functions of the above-described components of the system according to the invention can be used as method steps of the methods according to the invention.

[0065] The invention is described below using exemplary embodiments, which are explained in more detail with reference to the figures. Herein:

[0066] Figure 1 shows an IR image of a building with thermal defects;

[0067] Figure 2 shows the structure of the system according to the invention according to a first embodiment;

[0068] Figure 3 shows the components of the mobile terminal according to the first

[0069] embodiment; and

[0070] Figure 4 shows the data processing of the evaluation server according to the first embodiment.

[0071] In the following description of the figures, the same reference symbols are used for identical or identically functioning parts.

[0072] Figure 1 shows an infrared image of a building. The intensity of the detected infrared radiation is represented in color, with the scale shown ranging from dark blue (approximately -3°C) to pink (8°C).

[0073] The infrared image (IR) shows that the right half of the building has a comparatively lower temperature than the left half, meaning it hardly releases any heat to the outside. This could be due, for example, to subsequently installed thermal insulation in the exterior facade of the right half of the building. Areas D1 and D2 of the infrared image (IR) show that the respective windows on the ground floor have a significantly higher temperature than their immediate surroundings. Thus, a high heat loss (i.e., a thermal defect) occurs in these areas D1 and D2.

[0074] Figure 2 shows schematically the structure of the system according to the invention according to a first embodiment.

[0075] The system comprises the mobile terminal 100, in this embodiment a smartphone, and the evaluation server 200. The mobile terminal 100 and the evaluation server 200 are communicatively connected via the communication interface 250 of the evaluation server 200 and an integrated mobile radio communication interface of the mobile terminal 100. In particular, the mobile terminal 100 is configured to send an infrared image IR, a color image RGB, and metadata M1 to the evaluation server 200. Furthermore, the mobile terminal 100 is configured to receive the measures M determined by the evaluation server 200.

[0076] In addition to the communication interface 250, the evaluation server 200 has an object determination unit 210, a defect determination unit 220, and a measure determination unit 230, each of which can receive and process the data received from the communication interface 250. At least the measures determined by the measure determination unit 230 can in turn be provided to the communication interface 250, which is configured to transmit them to the mobile terminal 100.

[0077] The components of the mobile terminal 100 and the evaluation server 200 and their functionality are described in more detail below with reference to Figures 3 and 4.

[0078] Figure 3 shows the components of the mobile terminal 100 according to the embodiment of Figure 2.

[0079] The smartphone or mobile device 100 has a mobile radio communication interface 120, allowing bidirectional data transmission between the mobile device 100 and the analysis server. Furthermore, the mobile device 100 has a color camera 111 arranged on the rear of the device, which is configured to generate color images of objects.

[0080] Furthermore, the mobile device 100 is connected to the external IR camera 110 via a USB-C port (not shown), wherein the IR camera 110 is mounted underneath the mobile device 100. The IR camera 110 can, for example, have one or more of the following properties:

[0081] Image resolution: 640 x 480 pixels;

[0082] IR spectrum: 8 - 14 pm (corresponds to temperature range -15 to +600 °C);

[0083] - Dimensions: 34x26x15 mm;

[0084] Weight: 19g.

[0085] Furthermore, the mobile terminal 100 has a Bluetooth interface 130 for wireless communication with devices in the surrounding area.

[0086] The mobile terminal 100 is connected to an external temperature sensor 150 via the Bluetooth interface 130, so that the mobile terminal 100 can receive a temperature value from the external temperature sensor 150 and provide it to the evaluation server as meta information.

[0087] The mobile device 100 has a software component (smartphone app) with a graphical user interface, which is configured to generate color images and infrared images, query additional metadata if necessary, and send them to the analysis server. The measures generated and transmitted by the analysis server can also be displayed in the software component.

[0088] Figure 4 shows the data generated or processed by the various components of the evaluation server 200.

[0089] The object determination unit 210 is configured to generate associated object information O1 based on the IR image IR and the color image RGB of an object. For example, the object information O1 may include the following information: Object type: building window;

[0090] Size: 1.23 m x 1.48 m.

[0091] The above-described classification of the object for generating the object information Ol is carried out by the artificial neural network 211 of the object determination unit 210.

[0092] The defect determination unit 220 is configured to generate associated defect information Ol based on the IR image IR, the above-described object information Ol, and meta information Ml. For example, the defect information DI may include the following information:

[0093] Defect type: structural thermal bridge;

[0094] Temperature difference: 5° C.

[0095] The above-described identification and classification of the thermal defect for generating the defect information DI is carried out by the artificial neural network 221 of the object determination unit 220.

[0096] The measure determination unit 230 is designed to determine measures M on the basis of the object information Ol described above and the defect information DI and to provide corresponding information.

[0097] For this purpose, the measure determination unit 230 has a computing unit 231 and a measure database 232. The measures M are determined by querying the measure database 232 based on the object information O1 and the defect information DI. Optionally, the metadata M1 can also be used to determine the measures M.

[0098] In this way, measures M (e.g. materials for thermal insulation) can be determined that are suitable for the specific thermal defect (according to defect information DI) and the specific object (according to object information Ol). At this point it should be noted that all of the parts described above are to be regarded as independent embodiments or further developments of the invention, as defined in particular in the introduction to the description and the claims, each on its own - even without additional features described in the respective context, even if these have not been explicitly identified as optional features in the respective context, e.g. by using: in particular, preferably, for example, e.g., if necessary, round brackets, etc. - and in combination or any sub-combination. Deviations from this are possible.Specifically, it should be noted that the word in particular or round brackets do not indicate mandatory features in the respective context.

[0099] It goes without saying that the system or method according to the invention is by no means limited to identifying a single thermal defect on the basis of an infrared image, but is generally designed to identify one or more thermal defects on the object under investigation.

[0100] Furthermore, it is conceivable to use at least some of the information recorded and generated according to the invention, in particular the object information and the defect information, in the context of creating energy certificates or reports.

[0101] List of reference symbols

[0102] 100 mobile devices

[0103] 110 infrared camera

[0104] 111 Color camera

[0105] 120 mobile radio interface

[0106] 130 Bluetooth interface

[0107] 150 temperature sensor

[0108] 200 evaluation servers

[0109] 210 Object determination unit

[0110] 211 artificial neural network (of the object determination unit)

[0111] 220 Defect Determination Unit

[0112] 221 artificial neural network (of the defect determination unit)

[0113] 231 computing unit

[0114] 232 Measures Database

[0115] D1 , D2 thermal defects

[0116] DI defect information

[0117] Ol object information

[0118] M Measures

[0119] Ml meta information

[0120] IR infrared image

[0121] RGB color image

Claims

Patent claims 1. A system for identifying thermal defects on objects, comprising: - a mobile terminal (100) designed to generate an infrared image (IR) and a color image (RGB) of an object (O); - an evaluation server (200) comprising the following units: - an object determination unit (210) which is designed to generate object information (Ol) based at least on the color image (RGB) and / or the infrared image (IR); - a defect determination unit (220) which is designed to generate defect information (DI), in particular type, position, size and / or characteristic value of a thermal defect, based at least on the infrared image (IR) and the object information (Ol); - a measure determination unit (230) which is designed to determine measures (M) based on the defect information (DI) and / or the object information (Ol), wherein the mobile terminal (100) and the evaluation server (200) are communicatively connected, in particular for transmitting the infrared image (IR), the color image (RGB) and / or the measures (M).

2. System according to claim 1, wherein the object determination unit (210) comprises a first neural network (211) which is designed to generate the object information (Ol) based on the color image (RGB) and / or the infrared image (IR).

3. System according to claim 1 or claim 2, in particular according to claim 2, wherein the defect determination unit (220) comprises a second neural network (221) which is designed to generate the defect information (DI) based on the infrared image (IR) and the object information (Ol).

4. System according to one of the preceding claims, wherein the object information (01) indicates a type of object (O), in particular building, building element, vehicle, machine, pipeline.

5. System according to one of the preceding claims, wherein the defect information (DI) indicates at least one of the type, position, size and / or characteristic value of the thermal defect.

6. System according to one of the preceding claims, wherein the mobile terminal (100) has an RGB camera and is connected to an external infrared camera (110) and / or RGB-IR camera.

7. System according to one of the preceding claims, wherein the object determination unit (210) and / or the defect determination unit (220) each comprises a convolutional neural network.

8. System according to one of the preceding claims, wherein the defect determination unit (220) is designed to generate the defect information (DI) further based on meta information (Ml), wherein the meta information (Ml) indicates at least one of the following variables with respect to the object (O): - an internal temperature; - an outside temperature; - a GPS position; - a brightness value; and - a design parameter, in particular the ll value.

9. System according to one of the preceding claims, in particular according to claim 8, wherein the mobile terminal (100) is designed to detect a temperature by means of a temperature sensor which is connected to the mobile terminal (100), in particular wirelessly, and to provide the temperature to the evaluation server (200).

10. System according to one of the preceding claims, in particular according to claim 8 or 9, wherein the mobile terminal (100) has a GPS sensor (130) and is designed to provide a GPS location to the evaluation server (200).

11. System according to one of the preceding claims, in particular according to claim 8, wherein the mobile terminal (100) is designed to retrieve at least some of the meta information (Ml) from a smart home controller associated with the object (O) and to provide it to the evaluation server (200).

12. System according to one of the preceding claims, wherein the object determination unit (210) and / or the defect determination unit (220) is designed to generate the object information (Ol) and / or the defect information (DI) based on a plurality of infrared images (IR) and / or a plurality of color images (RGB).

13. System according to one of the preceding claims, in particular according to claim 12, wherein the plurality of infrared images (IR) and / or the plurality of color images (RGB) comprise an external view of the object (O) and an internal view of the object (O), which preferably show at least partially a same area of the object (O).

14. System according to one of the preceding claims, in particular according to one of claims 12 or 13, wherein the system is designed to create a new color image (RGB) based on a plurality of mutually similar images, for example a plurality of external views as color images (RGB) or infrared images (IR), by averaging or combining, on the basis of which the object determination and / or the defect determination is carried out.

15. A method for identifying thermal defects on objects, in particular in a system according to the preceding claims, the method comprising the following steps: a) capturing an infrared image (IR) and a color image (RGB) of an object (O); b) determining object information (Ol) based at least on the color image (RGB) and / or the infrared image (IR); c) determining defect information (DI), in particular type, position, size and / or characteristic value of a thermal defect, based at least on the infrared image (RGB) and the object information (Ol); d) determining measures (M) based on the defect information (DI) and / or the object information (Ol).

16. The method according to claim 15, wherein the determination of the object information (Ol) is carried out using a first neural network (211).

17. The method according to claim 15 or claim 16, in particular according to claim 16, wherein the determination of the defect information (DI) is carried out using a second neural network (212).

18. Method according to one of claims 15-17, wherein the object information (Ol) indicates a type of object, in particular building, building element, vehicle, machine, pipeline.

19. The method according to any one of claims 15-18, wherein the defect information (DI) indicates at least one of the type, position, size and / or characteristic value of the thermal defect.

20. The method according to any one of claims 15-19, wherein step a) comprises capturing a plurality of infrared images (IR) and / or a plurality of RGB images (RGB), in particular as an external view of the object (O) and an internal view of the object (O); and wherein step b) and / or step c) is performed based on the plurality of infrared images (IR) and / or the plurality of color images (RGB).

21. The method according to any one of claims 15-20, wherein the defect information (DI) is further generated based on meta-information (Ml) indicating at least one of the following variables with respect to the object (O): - an internal temperature; - an outside temperature; - a GPS position; - a brightness value; and - a design parameter, in particular the ll value.

22. A computer-readable storage medium containing instructions that cause at least one processor to implement a method according to any one of claims 15-21 when the instructions are executed by the at least one processor.

Images