Building envelope anomaly detection method and system

Abstract

A system and associated method for thermal inspection of structures are provided. The method includes one or more unmanned aerial vehicles (UAVs), each UAV comprising at least one thermal camera and / or a visible light camera, and a positioning system, and flies the UAV along a predetermined flight path around the structure to be inspected, capturing thermal image data and optical image data, as well as positioning data of the structure to be inspected at regular intervals while the UAV is flying along the flight path, and transmitting the captured data to a processing unit, inputting the thermal image data, visible light image data, and positioning data into a building envelope element detection module to generate building element data, inputting the building element data, thermal image data, and visible light image data into a thermal anomaly module of the processing unit to generate thermal anomaly data as output, the thermal anomaly data including the location and classification of thermal anomalies.

Description

Technical Field

[0001] The following generally relates to thermal anomaly detection systems and methods, and more specifically to systems and methods for detecting thermal anomalies in a building by comparing visible light images and thermal images captured using an unmanned aerial vehicle.

Background Art

[0002] Structures such as buildings can be configured to maintain an internal temperature different from the external temperature by applying an HVAC system that maintains the climate inside the structure according to specific specifications.

[0003] Buildings may be insulated to reduce the rate of heat transfer from the inside to the outside of the building or structure, or vice versa. This reduction in heat transfer leads to a reduction in the energy requirements (and thus costs) for maintaining the internal environment of the building, and has the advantage of keeping the temperature more constant throughout the inside of the building or structure.

Summary of the Invention

Problems to be Solved by the Invention

[0004] Insulation components such as wall panels, roof panels, insulated windows, and other insulation parts are prone to wear and deterioration, and the insulation performance may decrease. Such deterioration is difficult to detect with the naked eye, so it may be difficult to detect.

[0005] Similarly, specific devices such as thermal cameras can detect heat leaks indicating deterioration of the insulation material, but when using such devices, a large amount of skilled labor may be required to conduct inspections and analyze the acquired thermal images to detect thermal anomalies.

[0006] Therefore, there is a need for improved systems and methods for structural thermal anomaly detection that overcome at least some of the drawbacks of current systems and methods. [Means for solving the problem]

[0007] The thermal inspection method is described below. The thermal inspection method provides one or more unmanned aerial vehicles (UAVs), each equipped with one or more camera systems (e.g., a thermal infrared camera for capturing thermal image data, a visible light camera for capturing visible light image data) and a positioning system for capturing positioning data, and is operated by flying the UAV along a predetermined flight path around the structure to be inspected, capturing thermal image data, visible light image data, and positioning data of the structure to be inspected at regular intervals while the UAV is flying along the flight path, transmitting the thermal image data, visible light image data, and positioning data to a processing unit, receiving the thermal image data, visible light image data, and positioning data in the processing unit, inputting the thermal image data, visible light image data, and positioning data into the building envelope element detection module of the processing unit to generate building element data, inputting the building element data, thermal image data, and visible light image data into the thermal anomaly module of the processing unit to generate thermal anomaly data as output, the thermal anomaly data including the location and classification of the thermal anomaly.

[0008] According to some embodiments, the above method further provides thermal anomaly data to a report generation module of the processing device and receives a thermal inspection report as output.

[0009] According to some embodiments, the thermal anomaly module inputs building element data, infrared image data, and visible light image data into the input layer of a trained neural network and receives thermal anomaly data as an output.

[0010] According to some embodiments, regions in thermal image data and visible light image data are evaluated using the thermal anomaly module to determine whether a thermal anomaly exists, according to the relevant building elements in each region.

[0011] According to some embodiments, the processing apparatus further comprises a position correlation module, and the processing apparatus is configured to provide positioning data, visible light image data, and thermal image data as inputs to the position correlation module, generate scale data, input the scale data to the thermal anomaly module when generating the thermal anomaly data, and input the scale data to the building element detection module when generating the building element data.

[0012] According to some embodiments, the thermal camera has a resolution of at least 1024 x 1024 pixels.

[0013] According to some embodiments, the building element data includes a classification of building elements such as walls, doors, windows, and roofs.

[0014] According to some embodiments, the thermal inspection report includes heat loss data.

[0015] According to some embodiments, the thermal anomaly data is characterized according to the building element data.

[0016] The thermal inspection system is described below. The thermal inspection system comprises one or more UAVs, each UAV comprising one or more camera systems (e.g., a thermal camera for capturing thermal image data and / or a visible light camera for capturing visible light image data), a positioning system for capturing positioning data, an inspection structure, and a processing unit, each UAV is configured to fly along a predetermined flight path around the inspection structure, and while the UAV is flying along the flight path, it captures thermal image data of the inspection structure, visible light image data of the inspection structure, and positioning data at regular intervals, and transmits the thermal image data, visible light image data, and positioning data to the processing unit via a network, the processing unit is configured to receive the thermal image data, visible light image data, and positioning data, input the thermal image data, visible light image data, and positioning data to the processing unit's building envelope element detection module to generate building element data, input the building element data, thermal image data, and visible light image data to the processing unit's thermal anomaly module to generate thermal anomaly data as output, the thermal anomaly data including the location and classification of the thermal anomaly.

[0017] According to some embodiments, the processing unit is further configured to provide thermal anomaly data to a report generation module of the processing unit and to receive a thermal inspection report as output.

[0018] According to some embodiments, the thermal anomaly module inputs building element data, infrared image data, and visible light image data into the input layer of a trained neural network and receives thermal anomaly data as an output.

[0019] According to some embodiments, regions in thermal image data and visible light image data are evaluated using the thermal anomaly module to determine whether a thermal anomaly exists, according to the relevant building elements in each region.

[0020] According to some embodiments, the processing device further includes a position correlation module, and the processing device is further configured to provide the positioning data, visible light image data, and thermal image data as inputs to the position correlation module, generate scale data, input the scale data to the thermal anomaly module when generating the thermal anomaly data, and input the scale data to the building element detection module when generating the building element data.

[0021] According to some embodiments, the thermal camera has a resolution of at least 1024x1024 pixels. According to some embodiments, the building element data includes classifications of building elements such as walls, doors, windows, and roofs.

[0022] According to some embodiments, the thermal inspection report includes heat loss data. According to some embodiments, the thermal anomaly data is characterized according to the building element data.

[0023] Other aspects and features will become apparent to those skilled in the art by considering the following description of some exemplary embodiments.

[0024] The drawings included herein are for the purpose of illustrating various examples of the articles, methods, and apparatuses of this specification.

Brief Description of the Drawings

[0025] [Figure 1] FIG. 1 is a system diagram of a thermal inspection system according to an embodiment. [Figure 2] FIG. 2 is a system diagram of a computing device for use in a thermal inspection system according to an embodiment. [Figure 3] FIG. 3 is a system diagram of a thermal inspection system according to another embodiment. [Figure 4] FIG. 4 is a system block diagram of the UAV of the thermal inspection system of FIG. 3 according to an embodiment. [Figure 5]FIG. 5 is a system block diagram of the cloud processing apparatus of the thermal inspection system of FIGS. 3 and 4 according to an embodiment. [Figure 6] FIG. 6 is a system block diagram of the memory of the cloud processing apparatus of FIGS. 3 to 5 according to an embodiment. [Figure 7] FIG. 7 is a diagram of an image stitching operation before stitching that can be applied by the thermal inspection system of FIGS. 3 to 6 according to an embodiment. [Figure 8] FIG. 8 is a diagram of an image stitching operation after stitching that can be applied by the thermal inspection system of FIGS. 3 to 6 according to an embodiment. [Figure 9] FIG. 9 is a diagram showing the output of a building envelope element detection process that can be applied by the thermal inspection system of FIGS. 3 to 6 according to an embodiment. [Figure 10] FIG. 10 is a diagram showing the output of a thermal anomaly detection process that can be applied by the thermal inspection system of FIGS. 3 to 6 according to an embodiment. [Figure 11] FIG. 11 is a diagram showing a thermal image and a visible light image of an inspection structure that can be captured by the thermal inspection system of FIGS. 3 to 6 according to an embodiment, arranged side by side. [Figure 12] FIG. 12 is a diagram showing a thermal image and a visible light image of an inspection structure that can be captured by the thermal inspection system of FIGS. 3 to 6 according to another embodiment, arranged side by side. [Figure 13] FIG. 13 is a diagram of a sample thermal inspection report that can be generated by the thermal inspection system of FIGS. 3 to 6 according to an embodiment. [Figure 14] FIG. 14 is a flowchart of a thermal inspection method according to an embodiment.

MODE FOR CARRYING OUT THE INVENTION

[0026] The following describes various apparatuses or processes to illustrate examples of embodiments relating to each claim. The embodiments described below are not intended to limit the embodiments relating to the claims, and the embodiments relating to the claims may include processes or apparatuses different from those described below. The embodiments relating to the claims are not limited to apparatuses or processes having all the features of any of the apparatuses or processes described below, or to features common to some or all of the apparatuses described below.

[0027] The one or more systems described herein can be implemented in a computer program running on a programmable computer that includes at least one processor, a data storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device. For example, a programmable computer is, but is not limited to, a programmable logic unit, a mainframe computer, a server, and a personal computer, a cloud-based program or system, a laptop, personal data assistance, a mobile phone, a smartphone, or a tablet device.

[0028] Each program is preferably implemented in a high-level procedural programming language or an object-oriented programming language and / or scripting language for communication with the computer system. However, if necessary, the program may also be implemented in assembly language or machine code. In either case, the language may be a compiled language or an interpreted language. Each such computer program is preferably stored in a storage medium or device readable by a general-purpose or special-purpose programmable computer for configuring and operating the computer when the computer reads from the storage medium or device and performs the procedures described herein.

[0029] The description of embodiments comprising multiple components that communicate with one another does not imply that all such components are necessary. On the contrary, various optional components are described in order to illustrate the possibilities of various embodiments of the present invention.

[0030] Furthermore, even if the steps of a process, the steps of a method, an algorithm, etc., are described sequentially (in the disclosure and / or claims), such processes, methods, and algorithms may be configured to operate in an alternative order. In other words, any sequence or order of steps that may be described does not necessarily indicate that the steps must be performed in that order. The steps of the process described herein may be performed in any practical order. Furthermore, some steps may be performed simultaneously.

[0031] Where a single device or item is listed here, it is readily apparent that multiple devices / items (whether they work together or not) can be used instead of a single device / item. Similarly, where multiple devices or items are listed here (whether they work together or not), it is readily apparent that a single device / item can be used instead of multiple devices or items.

[0032] The following generally pertains to methods and systems for detecting structural defects, and more specifically, to systems and methods for detecting thermal anomalies such as cracks and moisture leaks in buildings using visible light and thermal images captured using one or more unmanned aerial vehicles.

[0033] Among the significant costs associated with the operation of buildings and structures is the management of air conditioning, which involves heating, cooling, humidifying, and dehumidifying the interior of the building. To reduce energy consumption and lower the costs associated with air conditioning, it is possible to insulate the building to reduce heat transfer between the inside and outside.

[0034] Over time, insulation materials and structural components can deteriorate. For example, multi-panel glass windows can degrade, causing the internal gas seal to fail and reducing the window's thermal insulation performance. This can lead to an increase in the heat transfer coefficient through the window.

[0035] Similarly, cracks and moisture leaks can occur in the insulation and panels on the exterior of buildings and structures. Such cracks and leaks can reduce the thermal insulation performance of the insulation and panels, potentially increasing the heat transfer coefficient through the panels and windows.

[0036] Detecting such defects that lead to a decrease in thermal insulation performance can be difficult. Thermal insulation defects are often invisible to the naked eye. Inspectors may use technologies such as infrared / thermal cameras to detect thermal insulation defects. However, such methods are time-consuming to perform and analyze, and require highly skilled human operators.

[0037] The following describes a system and method for detecting thermal anomalies in structures. One or more unmanned aerial vehicles (UAVs) may be provided. The UAVs may perform a pre-planned flight around the structure and capture a comprehensive set of overlapping images, along with positional and orientation data. Images may be captured in both the visible light and thermal infrared regions. The thermal and visible light images may be taken at approximately the same angle, and corresponding thermal and visible light images may exist for each scene.

[0038] After images are captured, they can be provided to a computing device for processing. The computing device can be configured to stitch individual images together to create a continuous image, use UAV position and orientation data to determine the scale and orientation of the image, apply automated methods to detect building elements, or analyze thermal images to detect thermal anomalies such as cracks.

[0039] When a thermal anomaly is detected, a thermal inspection report may be generated and printed out for subsequent action. For example, such a thermal report may be provided to building managers or property owners to arrange appropriate repairs and maintenance, or to assess the value of the building for future investors or buyers.

[0040] Referring first to Figure 1, a block diagram showing a thermal inspection system 10 according to an embodiment is shown. The system 10 includes one or more inspection unmanned aerial vehicles (UAVs) 12 communicating with a cloud processing unit 14, and an operator terminal 16 via a network 18. The cloud processing unit 14 may be a dedicated machine specifically designed to process thermal images, visible light images, and other related inspection data captured by the UAVs 12 to generate a thermal inspection report. The UAVs 12 may be unmanned aerial vehicles equipped with at least one camera (e.g., a thermal camera and / or a visible light camera) and a positioning system, and may be operated to collect inspection data. The UAVs 12 and / or the entire inspection operation may be configured or controlled by the operator terminal 16 (e.g., the operator may input the desired inspection target or flight path into the terminal 16).

[0041] In some examples of system 10, the cloud processing unit 14 and the operator terminal 16 may constitute a single device.

[0042] The cloud processing unit 14 and the operator terminal 16 may be a server computer, desktop computer, notebook computer, tablet, PDA, smartphone, or other computing device. Devices 14 and 16 may include connections to a network 18, such as a wired or wireless connection to the Internet. In some cases, the network 18 may include other types of computer networks or communication networks. Devices 14 and 16 may include one or more of the following: memory, secondary storage, processor, input device, display device, and output device. Memory may include random access memory (RAM) or a similar type of memory. Memory may also store one or more applications executed by the processor. Applications may correspond to software modules containing computer-executable instructions for performing the functions described below. Secondary storage may include a hard disk drive, Blu-ray® drive, or other types of non-volatile data storage. The processor may execute applications, computer-readable instructions, or programs. Applications, computer-readable instructions, or programs may be stored in memory or secondary storage, or received from the Internet or other network 18. Input devices may include any devices for inputting information into devices 14 and 16. For example, input devices may include keyboards, keypads, cursor control devices, touchscreens, cameras, or microphones. Display devices may include any type of device for displaying visual information. For example, display devices may include computer monitors, flat-screen displays, projectors, or display panels. Output devices may include any type of device for presenting a hard copy of information, such as a printer. Output devices may also include other types of output devices, such as speakers.In some cases, devices 14 and 16 may include one or more of the following: a processor, an application, a software module, a secondary storage device, a network connection, an input device, an output device, and a display device.

[0043] Although devices 14 and 16 are described in terms of various components, those skilled in the art will understand that devices 14 and 16 may, in some cases, include fewer, additional, or different components. Furthermore, although embodiments of the implementation of devices 14 and 16 are described as being stored in memory, those skilled in the art will understand that these embodiments may also be stored in and read from other types of computer program products or computer-readable media, such as hard disks, floppy disks, CDs, or DVDs, carrier waves from the Internet or other networks, or other forms of RAM or ROM. Computer-readable media may contain instructions for controlling devices 14 and 16 and / or the processor to perform specific actions.

[0044] The following description explains how devices such as the UAV 12, cloud processing unit 14, and operator terminal 16 perform specific actions. It can be understood that one or more of these devices perform actions automatically or in response to interaction with the user of that device. In other words, the user of a device can operate one or more input devices (touchscreen, mouse, buttons, etc.) to cause the device to perform the described actions. In many cases, this aspect is not described below but can be understood.

[0045] As an example, the following describes how devices 12 and 16 can send information to the cloud processing device 14. For example, an operator user using operator terminal 16 can interact with the user interface displayed on the operator terminal 16's screen by operating one or more input devices (such as a mouse or keyboard). Generally, a device can receive a user interface (for example, in the form of a web page) from the network 18. Alternatively or additionally, the user interface may also be stored locally on the device (for example, as a cache of a web page or mobile application).

[0046] The cloud processing unit 14 may be configured to receive multiple pieces of information from the UAV 12 and the operator equipment 16. Generally, the information may include at least thermal images and visible light images.

[0047] The cloud processing unit 14 can store information in a storage database in response to its reception. The storage can correspond to the secondary storage of devices 12, 14, and 16. Generally, the storage database can be any suitable storage device, such as a hard disk drive, solid-state drive, memory card, or disk (CD, DVD, Blu-ray, etc.). The storage database can also be locally connected to the cloud processing unit 14. In some cases, the storage database may be located remotely from the cloud processing unit 14 and accessible from it, for example, via a network. In some cases, the storage database may consist of one or more storage devices located in a networked cloud storage provider.

[0048] Referring now to Figure 2, which shows a simplified block diagram of the components of a computing device 1000, such as a mobile device or portable electronic device, according to an embodiment. The software modules described herein can be configured to run on a computing device such as device 1000 in Figure 2. Device 1000 includes several components, such as a processor 1020 that controls the operation of device 1000. Communication functions, including data communication, voice communication, or both, can be performed through a communication subsystem 1040. Data received by device 1000 can be decompressed and decoded by a decoder 1060. The communication subsystem 1040 can receive messages from and send messages to a wireless network 1500.

[0049] Wireless Network 1500 can be any type of wireless network, including data center wireless networks, voice center wireless networks, and dual-mode networks that support both voice and data communications.

[0050] Device 1000 can be a battery-powered device and may include a battery interface 1420 for accepting one or more rechargeable batteries 1440, as shown in the figure.

[0051] The processor 1020 can also interact with additional subsystems such as random access memory (RAM) 1080, flash memory 1100, display 1120 (for example, a touch-sensitive overlay 1140 connected to an electronic controller 1160 to constitute a touch-sensitive display 1180), actuator assembly 1200, one or more optional force sensors 1220, auxiliary input / output (I / O) subsystem 1240, data port 1260, speaker 1280, microphone 1300, short-range communication system 1320, and other device subsystem 1340.

[0052] In some embodiments, user interaction with a graphical user interface may be performed via a touch-sensitive overlay 1140. The processor 1020 may interact with the touch-sensitive overlay 1140 via an electronic controller 1160. Information such as text, characters, symbols, images, icons, and other items generated by the processor 102 that may be displayed or rendered on a portable electronic device may be displayed on the touch-sensitive display 118.

[0053] The processor 1020 can also interact with the accelerometer 1360, as shown in Figure 2. The accelerometer 1360 can be used to detect the direction of gravity or gravity-induced reaction forces.

[0054] In this embodiment, to identify a network access subscriber, device 1000 may use a subscriber identification module or removable user identification module (SIM / RUIM) card 1380 inserted into a SIM / RUIM interface 1400 for communication with the network (such as a wireless network 1500). Alternatively, user identification information can be programmed into flash memory 1100 or implemented using other techniques.

[0055] Device 1000 includes an operating system 1460 and software components 1480, which are run by a processor 1020 and may be stored in a persistent data storage device such as flash memory 1100. Additional applications may be loaded into device 1000 via a wireless network 1500, an auxiliary I / O subsystem 1240, a data port 1260, a short-range communication subsystem 1320, or other suitable device subsystems 1340.

[0056] For example, during use, incoming signals such as text messages, email messages, web page downloads, or other data may be processed by the communication subsystem 1040 and input to the processor 1020. The processor 1020 then processes the incoming signals and outputs them to the display 1120 or, alternatively, to the auxiliary I / O subsystem 1240. Subscribers can also create data items, such as email messages, and send them over the wireless network 1500 via the communication subsystem 1040.

[0057] In the case of voice communication, the overall operation of the portable electronic device 1000 is similar. The speaker 1280 outputs audible information converted from electrical signals, and the microphone 1300 can convert the audible information into electrical signals for processing.

[0058] Referring now to Figure 3, a system block diagram of the thermal inspection system 100 is shown. System 100 may correspond to systems 10 and 1000 in Figures 1 and 2. System 100 includes one or more UAVs 102, an inspection structure 104, a network 106, and a cloud processing unit 108.

[0059] UAV102 is an unmanned aerial vehicle. UAV102 includes multi-rotor unmanned aerial vehicles such as quadcopters. UAV102 can be configured to perform autonomous flight along a predetermined flight path 110. For example, UAV102 can be programmed to take off from a launch site, fly 2m upwards, 4m forwards, 4m backwards, and 4m downwards, land, and return to its original position. Similarly, UAV102 can be pre-programmed to fly around a building along a specific predetermined path.

[0060] UAV102 includes a UAV compatible with a dual-synchronous camera system payload (such as a synchronous thermal camera and a visible light camera), the UAV equipped with an advanced flight controller system, six-directional sensing and positioning, a first-person view camera, a collision mitigation system, radar, return-to-home, obstacle detection, artificial intelligence-based spot check, nearby aircraft recognition, auxiliary lights for nighttime operation, and a hot-swappable battery. Inspection structure 104 is any building or structure known in the art where thermal insulation performance may be important. For example, inspection structure 104 is a permanent concrete building, a modular building, a mobile dwelling, or other structure with an internal volume, the internal volume may be maintained at a temperature other than the external ambient temperature.

[0061] The UAV 102 can be configured to fly around the inspection structure 104 along a predetermined flight path 110 and capture images (thermal and / or visible light images) of all or a subset of the outer surfaces of the inspection structure 104 (see, for example, the imaging field 112 in Figure 3). If the UAV 102 is equipped with both a visible light camera system and a thermal imaging camera system, it can capture visible light and thermal images simultaneously. In some embodiments, images can be captured from inside the inspection structure 104. As used herein, “captured images” refers collectively to both thermal and visible light images.

[0062] The flight path 110 may include position and time data for the entire flight, and the orientation of the UAV 102 throughout the flight. The flight path 110 may be pre-programmed by a skilled operator with the assistance of flight planning software. The flight path 110 data may be stored in electronic format and provided to the UAV 102. In some examples, the flight path 110 data can be transmitted to the UAV via a network (such as the network 18 of the system 10) using an external operator terminal or controller (such as device 16 of system 10). In embodiments where multiple UAVs are used to capture images of an inspection structure, the flight path of each UAV follows a single pre-programmed flight path so that the images captured by each UAV can be superimposed using the position data of the captured images.

[0063] In other embodiments, instead of pre-configured flight paths, a skilled operator may manually fly the UAV. In some embodiments, instead of capturing data with the UAV, a static imaging system may be used in the system and method of the present invention.

[0064] Referring here to Figure 4, a block diagram detailing the components of the UAV 102 is shown. The UAV 102 includes at least one camera system (thermal camera system 114 and / or visible light camera system 116). The UAV 102 further includes a storage medium 118, a network interface 120, and a positioning system 122. According to various embodiments, the UAV 102 can be configured for daytime or nighttime operation. For example, for nighttime operation, the first UAV 102 is equipped with both the thermal camera system 114 and the visible light camera system 116, the latter used when lighting conditions permit. For daytime operation, the second UAV 102 is equipped with a high-resolution, high-frame-rate visible light camera system 116 tuned for 3D model generation.

[0065] The thermal camera system 114 includes at least one thermal camera. The thermal camera can capture thermal images in the infrared electromagnetic radiation band, which includes wavelengths from 1000 nm to 14000 nm. Each pixel in each captured thermal image may correspond to the intensity of the received infrared radiation.

[0066] The thermal camera system 114 can be calibrated so that each captured pixel corresponds to a temperature value. For example, a thermal image may contain 1024x1024 pixels capturing a particular scene. The top-left pixel of the image may correspond to a reading of 25°C, and each adjacent pixel may correspond to a reading of 24°C. Such thermal correlation of the captured data can be performed at the firmware level of the thermal camera system 114 or at the low hardware level of the thermal camera system 114.

[0067] All captured thermal images may be stored in the storage medium 118. When the captured thermal images are stored in the storage medium, they may be additionally associated with metadata. Such metadata may include, for example, the capture time, the position and orientation of the UAV at the time of capture, the ambient temperature at the time of capture, and the associated visible light image. Additional metadata may be captured that may not be associated with a particular image, and may include, but are not limited to, the time and date, flight time per session, latitude, longitude, altitude, maximum altitude, speed, rotational speed of each motor, velocity components in the x, y, and z directions, pitch, yaw, roll, a list of comments from the flight controller, GPS data, number of GPS satellites, battery history, flight path, takeoff history, landing history, etc.

[0068] The visible light camera 116 can capture images in the visible light electromagnetic radiation band, including wavelengths from 400 nm to 700 nm. The visible light camera 116 can capture images with a minimum resolution of 640 pixels x 512 pixels. In other examples, the visible light camera 116 can capture images at different resolutions, such as 4K resolution. A visible light camera with a higher resolution may offer greater accuracy and / or performance.

[0069] The visible light camera 116 may be mounted on the UAV 102 via a gimbal, allowing the relative position of the visible light camera 116 and the UAV 102 to be changed. Such a gimbal can rotate at a speed of 100 degrees per second and may allow rotation ranges of +330 to -330 degrees, +135 to -45 degrees, and +25 to -90 degrees in the pan, tilt, and roll directions, respectively. In some examples, the gimbal may be configured to operate in environments ranging from -20 degrees to +50 degrees.

[0070] All captured visible light images may be stored in the storage medium 118. When the captured visible light images are stored in the storage medium 118, they may be additionally associated with metadata. Such metadata may include, for example, the capture time, the position and orientation of the UAV at the time of capture, the ambient temperature at the time of capture, and associated thermal images.

[0071] The storage medium 118 includes any non-temporary computer-readable memory known in the art. The storage medium 118 is, but is not limited to, a hard drive, a solid-state drive, NAND flash, or a tape drive.

[0072] The network interface 120 may include any device, hardware, module, or system that enables the UAV 102 to communicate with another electronic device, such as a computer system, and to transmit data to such electronic device. The network interface 120 may include an RF transmission subsystem. The network interface 120 can be configured to enable the UAV 102 to transmit data over a GSM, LTE, HSPA, 5G or other cellular network, Wi-Fi network, satellite data network, or other communication network known in the art.

[0073] The positioning system 122 may include multiple sensors (e.g., GPS, GLONASS, accelerometer, gyroscope, barometer, and other sensors) and processing units that generate position and orientation data outputs. Position and orientation data may include latitude, longitude, altitude, and orientation data. In other examples, position and orientation data may include relative orientation and position outputs. For example, position can be specified as a position relative to the inspection structure 104, such as X, Y, and Z coordinates relative to the center point of the inspection structure 104. Similarly, orientation can be specified relative to the camera's field of view and the outer surface of the inspection structure 104.

[0074] In some examples, the positioning system 122 may also be configured to capture and record environmental data such as ambient temperature and ambient atmospheric pressure.

[0075] The positioning system 122 may be configured to output and store timestamped position and orientation data at regular intervals. For example, through interaction with an internal clock, position and orientation data may be output at intervals of 1000 samples per second. In other examples, position and orientation data may be output and stored at different intervals, or at the same interval as image capture, so that there may be a corresponding position and orientation data sample for each captured image.

[0076] Referring to Figure 3, the UAV 102 is configured to autonomously fly around the inspection structure 104 along a predetermined flight path 110. The flight path 110 may be configured based on the fields of view 112 of the visible light camera 116 and the thermal camera 114. For example, the flight path 110 may include a spiral path around a structure that is roughly cubic in shape. The flight path 110 includes the orientation direction of the UAV 102, which directs the imaging equipment of the UAV 102 (thermal camera system 114, visible light camera 116) toward the outer surface (or, in the case where the inspection is performed internally) of the inspection structure 104.

[0077] While the UAV 102 is flying along the flight path 110, the UAV 102 can simultaneously (in a synchronized configuration) capture visible light and thermal images of the structure 104 using the visible light camera 116 and the thermal camera 114, respectively. According to an embodiment in which multiple UAVs 102 are used to capture images of the inspection structure 104, each UAV 102 is configured to autonomously fly around the inspection structure 104 along the same predetermined flight path 110.

[0078] Images can be captured at a speed such that they partially overlap. For example, if the flight path 110 includes a completely horizontal flight section, the images can be captured so that the rightmost 10% of the first image and the leftmost 10% of the second image overlap. Similarly, if images of the structure 104 above the portion of the structure 104 above the first and second images are captured, the tops of the first and second images can overlap with the bottoms of the subsequent image. These overlapping images can be stitched into a sequence of images by any number of image stitching algorithms and methods known in the art.

[0079] The image capture rate may be proportional to the flight speed of the UAV102. For example, the image capture rate can be determined by calculating the frequency of image captures that result in a certain amount of image overlap, using knowledge of the image capture field of view and the flight speed of the UAV102.

[0080] Images from both camera systems 114 and 116 may be captured along the flight path 110 until images of all external surfaces and parts of the structure 104 are captured. In some cases, only specific parts of the structure 104 may be targeted, or parts of the structure 104 may not be easily surveyed in the manner described herein (for example, line of sight access may be difficult or unavailable for a typical UAV 102).

[0081] Once all these images have been captured, they can be transmitted to the cloud processing unit 108 via the network 106. Furthermore, all relevant metadata (such as location and orientation data, environmental data, and other metadata) can also be transmitted to the cloud processing unit 108 via the network 106.

[0082] In some cases, all captured images and metadata may be transmitted after the UAV102's flight is complete. Such a configuration can reduce the UAV102's energy consumption (by reducing network usage) and potentially extend its flight range.

[0083] In other examples, captured images and metadata can be transmitted immediately after capture. Such a configuration reduces the waiting time required from the end of the flight to data processing, and since the images are deleted from the storage medium 118 as soon as they are sent to the cloud processing unit 108, only a storage buffer is needed, thus reducing the capacity requirements of the storage medium 118.

[0084] Referring to Figure 5, a detailed system block diagram of the cloud processing unit 108 is shown. The cloud processing unit 108 includes memory 124, a processor 126, a storage medium 128, and a network interface 130. The cloud processing unit 108 may be a server, computer, or microcontroller with an x86 or ARM architecture.

[0085] In some examples, the cloud processing unit 108 may be an instance of a commercial cloud computing service, such as Amazon Web Services, Google Cloud Platform, Microsoft Azure, or other commercial or private cloud computing platforms. In such examples, subcomponents of the cloud processing unit 108, such as memory 124, processor 126, storage medium 128, and network interface 130, may be virtualized instances of such components.

[0086] Memory 124 includes any memory known in the art that can temporarily store input and output data (such as machine instructions, input data, and output data) for use by a computer processor (such as processor 126).

[0087] Processor 126 may be any processor known in the art that can receive and execute machine instructions specific to its processor architecture. Processor 126 may be a general-purpose processor of x86, ARM, or other architecture, or any other computer processor known in the art.

[0088] The storage medium 128 includes any non-temporary computer-readable memory known in the art. The storage medium 128 may be, but is not limited to, a hard drive, a solid-state drive, NAND flash, or a tape drive.

[0089] The network interface 130 may include any device, hardware, module, or system that enables the cloud processing unit 108 to communicate with another electronic device, such as a computer system, and to transmit data to such electronic device. The network interface 130 may be configured to enable the cloud processing unit to transmit data over the Internet or other networks via GSM, LTE, HSPA, 5G or other cellular networks, Wi-Fi networks, satellite data networks, wired Ethernet networks, or other communication networks known in the art. The network interface 130 may enable the cloud processing unit 108 to communicate with other cloud computing devices or instances within the same network or platform, or within other networks or platforms.

[0090] Referring here to Figure 6, a detailed system block diagram of memory 124 is shown. Memory 124 may include a building envelope element detection module 132, a thermal analysis module 134, a report generation module 136, a position correlation module 138, thermal anomaly data 140, building element data 142, position and orientation data 144, image data 146, a thermal inspection report 148, and scale data 150. In some examples, the software modules and data contained within memory 124 may be stored in other parts of system 100. For example, the software modules and data of memory 124 may be stored in a storage medium 128 for long-term storage and copied to memory 124 as needed, and vice versa. In some examples, the data types shown within memory 124 may only exist after a specific part of the inspection process described here has been completed. For example, the thermal inspection report 148 may not exist in memory 124 until the cloud processing unit processes the data collected by the UAV 102 and generates the thermal inspection report 148 as described here.

[0091] The position correlation module 138 includes a software module configured to receive thermal image data, visible light image data, and position and orientation data 144 from the UAV 102, and to determine the scale and orientation of each thermal and visible light image. The scale data 150 for each image can be determined by evaluating the distance between the inspection structures 104, taking into account the known fields of view of the thermal and visible light cameras, and the relative orientation of the inspection structures 104 and the UAV 102. The scale data 150 may include data describing the distance corresponding to each pixel in each image (thermal image or visible light image).

[0092] In some examples, the scale data 150 may be uniform throughout a single captured image, or it may differ from image to image. In examples where the scale data 150 differs from image to image, the scale data 150 may include a scale map, in which different scale values ​​may be described, each corresponding to a different region of the image.

[0093] In some examples, instead of saving the scale data 150 separately from the image, you can modify the image to incorporate the scale data, such as writing the scale data 150 to the image metadata.

[0094] In some examples, each image may be distorted or stretched to ensure uniform scaling throughout the image. Due to the non-direct image capture angle, the bottom of the captured image may have a different scale than the top of the image. The content of each image may be stretched so that each pixel in the image represents the same straight-line distance (for example, each pixel corresponds to a distance of 0.01m).

[0095] In some examples, the position correlation module 138 can be further configured to determine the relative position of the subject in each thermal and visible light image, thereby orienting the images in three-dimensional space. For example, such positional data can be combined with images to generate a 3D model or representation of the inspection structure 104. The 3D model can be generated using photogrammetric analysis. By applying a 2-pixel ground sampling distance (GSD) in the X and Y directions and a 3-pixel GSD in the Z direction, a 3D model, a digital surface model, and an orthomosaic map can be generated. Such models and maps can be used to map, measure, and document the progress of thermal inspection operations performed using the systems and methods described herein.

[0096] In some examples, relative positions can be determined with sub-centimeter accuracy by capturing at least 250 images. In other examples, a different number of images may be captured to provide sub-centimeter accuracy.

[0097] The Building Envelope Element Detection (BEED) module 132 (hereinafter also referred to as the "BEED module") includes a software module configured to receive thermal images, visible light images, and scale data, and to detect building elements within these images.

[0098] The BEED module 132 is configured to first stitch together both visible light and thermal images to create a larger, continuous image that provides an overall view of the inspection structure 104. Any image stitching method or algorithm known in the industry can be applied to stitch the images together.

[0099] Referring now to Figure 7, an example of an image stitching operation that can be applied by the BEED module 132 is shown. Four images 202, 204, 206, and 208 are captured so that the themes of each boundary image overlap.

[0100] Referring now to Figure 8, the result of the image stitching operation in Figure 7 is shown. In this figure, images 202, 204, 206, and 208 are stitched together into one large image 210 that contains the contents of all four constituent images 202, 204, 206, and 208.

[0101] After the images are stitched together, the stitched images can be stored in memory 124 or storage medium 128 (for example, as image data 146).

[0102] When images are stitched together into a larger sequence of images, building elements within the stitched images can be identified. Building elements may include, but are not limited to, structural or functional features such as windows, doors, vents, window frames, solid walls, roof panels, roof elements, and other building components.

[0103] The BEED module 132 can detect building elements by applying machine learning or artificial intelligence methods such as supervised learning, unsupervised learning, or semi-supervised learning. For example, the BEED module 132 may receive visible light images, thermal images, and scale data 150 as inputs, provide these inputs to a trained neural network, and receive building element data 142 as output from the trained neural network.

[0104] The BEED module 132 includes a neural network configured to receive input data (thermal image, visible light image, scale data 150) and generate at least one output (building element data 142). The neural network may be a feedforward neural network. The neural network includes multiple processing nodes. The processing nodes may include a multivariable input layer with multiple input nodes, a hidden layer with at least one node, and an output layer with at least one output node. During the operation of the neural network, each node in the hidden layer applies an activation / transfer function and weights to all inputs arriving at that node (from the input layer or from another layer in the hidden layer). A node may provide an output to other nodes (subsequent hidden layer nodes or output layers). The neural network may be configured to perform regression analysis, which provides continuous outputs, or classification analysis, which classifies data. The neural network can be trained using supervised or unsupervised learning techniques, as described below.

[0105] In supervised learning techniques, a training dataset is provided to the input layer along with a set of known output values ​​for the output layer. During the training phase, the neural network can process the training dataset. The neural network is intended to learn how to provide an output for new input data by generalizing the information it has learned from the training data during the training phase. Training can be performed by backpropagating errors to determine the weights of the hidden layer nodes to minimize errors. After training, or optionally during training, test or validation data can be provided to the neural network to provide an output. Thus, the neural network can cross-correlated the inputs provided to the input layer to provide at least one output in the output layer. In each embodiment, the output provided by the neural network is preferably close to the desired output for a given input so that the neural network can satisfactorily process the input data.

[0106] Once building elements are detected as described above, a bounding box may be defined around each detected building element. The bounding box may include the coordinates of the bounding box. Additionally, a class label may be associated with each identified building element. The class label may include, for example, windows, doors, vents, window frames, solid walls, roof panels, roof elements, or other building elements. The bounding box data and class label data may be stored as building element data 142.

[0107] Referring here to Figure 9, an exemplary depiction of a visible light image 210 including the completed BEED process applied to image 210 is shown. Within image 210 are a window 212 and a door 214. The BEED module 132 processes image 210, detecting, locating, and classifying the building elements 212 and 214 within image 210. A bounding box 216 surrounds the window 212, indicating the boundary of the window 212 in image 210. Similarly, a bounding box 218 surrounds the door 214, indicating the boundary of the door 214 in image 210. Additionally, a text-based class label is displayed within each bounding box, labeling each element with a class (boxes 216 and 218 are labeled "window" and "door," respectively). Furthermore, Figure 9 shows a scale 220, corresponding to the straight-line distance on image 210 derived from scale data 150.

[0108] The example in Figure 9 shows a graphical representation of an image processed by the BEED module 132, but the associated building element data 142 can also be saved in other formats. For example, the building element data 142 can be saved as plain text data, or as other structured data (such as XML format data) that includes the image associated with the building element data, the location of the bounding box, the size of the bounding box, and the element class label of each building element whose location is identified. Such data can be easily read by machines and further analyzed and processed.

[0109] Element classes can include elements such as "walls," "roofs," "windows," and "doors." In some examples, element classes can further include subclasses. For example, the "window" class element can further include subclasses such as "single-pane glass," "double-pane glass," "triple-pane glass," "argon-filled," and "nitrogen-filled."

[0110] Referring again to Figure 6, the thermal anomaly module 134 includes a software module configured to receive building element data 142 from the BEED module 132, as well as thermal and visible light images (e.g., image data 146) and scale data 150 as input, and to generate thermal anomaly data 140 as output.

[0111] The thermal anomaly module 134 can apply different detection methods to different building elements, taking into account previously generated building element data 142. For example, the building element data 142 may indicate specific areas of a captured image as "windows" or "doors," as described above. Such detected areas can be evaluated differently with respect to thermal anomalies. In some examples, image areas not associated with detected building elements can be evaluated using the default method.

[0112] Typical windows may have specific thermal defects, and each thermal defect may be associated with a specific thermal signature. For example, a multi-panel window may have a thermal defect where inert gas leaks between the glass panes, allowing air and moisture to enter between the panes. This can affect the thermal performance of the window. The thermal anomaly module 134 is specifically configured to detect the thermal signature of such thermal defects within the area labeled as a window area by the BEED module 132. Similarly, other building elements may also have specific known defects. For such elements, image regions associated with known building elements can be evaluated according to their building element class.

[0113] In some cases, the thermal anomaly module 134 may detect thermal anomalies by applying direct algorithmic methods, such as edge detection and various image processing filters, to thermal image data. In some cases, the thermal anomaly module 134 may pinpoint the location of thermal anomalies by evaluating the size, shape, and temperature of temperature rise or fall regions within the thermal image. Such evaluations can be performed at the pixel level of the thermal image.

[0114] In some cases, the thermal anomaly module 134 may detect thermal anomalies by applying machine learning or neural network-type techniques.

[0115] For example, the thermal anomaly module 134 includes a neural network configured to receive input data (thermal images, visible light images, building element data 142, and scale data 150) and generate at least one output (thermal anomaly data 140). The neural network may be a feedforward neural network. The neural network may have multiple processing nodes. The processing nodes may include a multivariable input layer with multiple input nodes, a hidden layer with at least one node, and an output layer with at least one output node. During the operation of the neural network, each node in the hidden layer applies an activation / transfer function and weights to all inputs arriving at that node (from the input layer or from another layer in the hidden layer). A node may provide an output to other nodes (subsequent hidden layer nodes or output layers). The neural network may be configured to perform regression analysis, which provides continuous outputs, or classification analysis, which classifies data. The neural network can be trained using supervised or unsupervised learning techniques, as described below.

[0116] In supervised learning techniques, a training dataset is provided to the input layer along with a set of known output values ​​for the output layer. During the training phase, the neural network can process the training dataset. The neural network is intended to learn how to provide an output for new input data by generalizing the information it has learned from the training data during the training phase. Training can be performed by backpropagating errors to determine the weights of the hidden layer nodes to minimize errors. After training, or optionally during training, test or validation data can be provided to the neural network to provide an output. Thus, the neural network can cross-correlated the inputs provided to the input layer to provide at least one output in the output layer. In each embodiment, the output provided by the neural network is preferably close to the desired output for a given input so that the neural network can satisfactorily process the input data.

[0117] An untrained neural network may be provided with manually tagged thermal image data as a training dataset. Such manually tagged data may include scale-associated thermal images (e.g., scale data 150), building element data, bounding boxes surrounding thermal anomalies, and class labels associated with thermal anomalies, affixed by a skilled operator.

[0118] Neural networks can be subjected to supervised learning, semi-supervised learning, or unsupervised learning methods.

[0119] The thermal anomaly data 140 may include the location of each detected thermal anomaly and the class label for each thermal anomaly. Thermal anomaly classes include, but are not limited to, wall cracks, window sealing failures, ceiling cracks, insulation failures, and masonry failures. The location of a thermal anomaly may be indicated by a relevant image indicator, the location of a bounding box (such as the center position), and the dimensions of the bounding box. Since the thermal anomaly data 140 is in a machine-readable format such as plain text or XML, the thermal anomaly data 140 can be further processed and / or analyzed.

[0120] Referring to Figure 10, an exemplary depiction of thermal image 310 is shown, including the completed thermal anomaly detection process applied to image 310. Within image 310 are windows and doors. The BEED module 132 processes image 310, thereby detecting, locating, and classifying the building elements within image 310.

[0121] The thermal anomaly detection process located the thermal anomaly 302 within image 310. A bounding box surrounds the thermal anomaly 302, indicating its boundaries within image 310. The thermal anomaly 302 is associated with a thermal anomaly class label (not shown).

[0122] The example in Figure 10 shows a graphical representation of an image processed by the thermal anomaly module 134, but the associated thermal anomaly data 140 can also be stored in other formats. For example, the thermal anomaly data 140 can be stored as plain text data, or as other structured data (such as XML format data) that includes the image associated with the thermal anomaly data 140, the location of the bounding box, the size of the bounding box, and the element class label for each thermal anomaly whose location is identified. Such data can be easily read by machines and further analyzed and processed.

[0123] Referring to Figure 11, an exemplary visible light image 322 and a corresponding exemplary thermal image 324 are shown side by side for comparison. A thermal crack 323, which is not visible in the visible light image 322, is depicted in the thermal image 324. This thermal crack 323 can be detected, located, and classified by the thermal anomaly module 134. Images 322 and 324 were captured by one or more UAVs.

[0124] Referring to Figure 12, an exemplary visible light image 326 and its corresponding exemplary thermal image 328 are shown side by side for comparison. A thermal defect 327, which is not visible in the visible light image 322, is depicted in the thermal image 324. This thermal defect 327 can be detected, located, and classified by the thermal anomaly module 134. Images 326 and 328 were captured by a UAV.

[0125] In other examples, other thermal defects may exist and be detected. For example, in the case of roof elements, defects may include moisture accumulation in gravel, moisture accumulation in the roof membrane, uneven thermal layer, deterioration of insulation, deterioration of the joint between the roof and exterior wall, heat loss in overhangs, heat loss in concrete pavement, heat loss to plants, lack of surrounding insulation, and heat loss in parapets. In the case of door elements, defects may include heat loss from the door frame, heat loss from the door, heat loss from the garage door, and heat loss in the main floor enclosure. In the case of window elements, defects may include moisture accumulation behind the cladding and window frame, moisture accumulation within the window frame, heat loss from the uninsulated frame, heat loss from the IGU, heat loss from the frame and IGU, heat loss from the caulking, heat loss from the joint between the wall and window, and skylight IGU. In the case of curtain wall elements, defects may include heat loss from the uninsulated frame and heat loss from the IGU. Wall element defects include thermal bridging due to connections between foundation walls and exterior walls, connections between exterior walls at corners, connections of exterior fasteners, thermal bridging due to building envelope connections, thermal bridging due to frames, connections between exterior walls and roof assemblies, inconsistent insulation, thermal degradation, connections between walls and windows, thermal bridging between structural load-bearing elements and other elements, connections between walls and floor slabs, connections between walls and garage doors, connections between walls and doors, connections between siding joints, thermal bridging due to overhands, thermal cracks in wall assemblies due to moisture, moisture accumulation due to poor ventilation, moisture accumulation between wall and window connections, and moisture accumulation due to poor ventilation in metal siding. In other examples, other types of defects may be present and / or detectable.

[0126] Returning to Figure 6, the report generation module 136 includes a software module configured to receive thermal anomaly data 140 and image data 146 as input and output a thermal inspection report 148 associated with the inspection structure 104. The thermal inspection report (for example, the thermal inspection report 400 shown in Figure 13) includes data detailing the results of the thermal inspection. For example, the thermal inspection report 148 may include a list of all identified or suspected thermal anomalies, including location, classification (e.g., cracks, moisture leaks, window seal failures, etc.), and other attributes such as estimated heat loss.

[0127] In some examples, the thermal inspection report may additionally include estimated energy costs associated with defects. For instance, in addition to the collected data, the report generation module 136 may provide the heating type (forced air gas, electricity, etc.) and energy cost, allowing for an estimation of the approximate financial cost associated with each detected thermal anomaly.

[0128] In some cases, the report generation module 136 may be provided with additional input data, such as inspection metadata and environmental data.

[0129] Referring now to Figure 13, a portion of the thermal inspection report 400 according to the embodiment is shown. The thermal inspection report 400 includes data summarizing 30 identified thermal anomalies. Figure 13 shows a summary of one identified thermal anomaly, a crack in the wall 402.

[0130] In the embodiment shown in Figure 13, the thermal inspection report 400 includes identification metadata for the inspected structure ("Structure 104"), a thermal anomaly identifier ("1 of 30"), a thermal anomaly class ("Wall Crack"), an image crop of the thermal anomaly, anomaly size data ("0.5m x 0.1m"), anomaly energy loss rate ("202.1W"), and an estimated annual anomaly cost ("$367.32").

[0131] In some examples, a thermal inspection report may include additional features and data, such as metadata including, but is not limited to, the inspection operator, date and time of inspection, project identifier, inspection hardware identifier, inspection area, total inspection time, quality check data, preview thermal and visible light images, calibration details, image capture location, manual tie point positions, absolute camera position and orientation uncertainty, image overlap map, bundle block adjustment details, internal camera parameters, 2D keypoint table, 3D points from 2D keypoint matches, 2D keypoint matches, geolocation details, data processing details and configuration, and point cloud densification details.

[0132] In some examples, quality check data may include the number of keypoints per image, image calibration information, camera optimization information, matching data information, and georeference information.

[0133] In some of the system examples described here, thermal inspection reports can be configured to be machine-readable. For example, a thermal inspection report can be configured so that all text is stored as machine-readable text data, and image data within the report is stored separately from the rest of the thermal inspection report.

[0134] In some cases, the thermal inspection report can be transmitted via the network by the cloud processing unit 108 to another computing device for further use.

[0135] During operation, system 100 may be instructed to collect data using UAV 102 as described above. Once data is collected, the collected data (including position and orientation data, visible light image data, and thermal image data) may be transmitted to cloud processing unit 108 via network 106.

[0136] The cloud processing unit 108 can receive the collected data and provide the position and orientation data, visible light image data, and thermal image data to the position correlation module 138. The position correlation module can process the collected data, determine the scale of each captured image, and tilt or distort each image so that the scale is applied uniformly to each captured image. The position correlation module 138 can then output scale data 150. The data output by the position correlation module 138 can be temporarily stored in memory 124 or storage medium 128 for archiving or further processing.

[0137] Visible light image data, thermal image data, and scale data 150 can be provided to the building envelope element detection module 132. The building envelope element detection module 132 can detect building elements in the image as described above. The building envelope element detection module 132 can output building element data 142, which includes the bounding box of each identified building element and the building element class label associated with each identified building element. The building element data 150 can be temporarily stored in memory 124 or storage medium 128 for archiving or further processing.

[0138] Visible light image data, thermal image data, scale data 150, and building element data 142 are provided to the thermal anomaly module 134, which is processed to detect thermal anomalies in the captured images, and thermal anomaly data 140 may be generated.

[0139] Visible light image data, thermal image data, scale data 150, building element data 142, and thermal anomaly data 140 may be provided to the report generation module 136 for processing and generating the thermal inspection report 148. As described above, the thermal inspection report 148 is configured to be machine-readable and can be further processed. Furthermore, the thermal inspection report may be configured and formatted so that it can be read and interpreted by a human analyst.

[0140] The system described above is configured so that once a flight path 110 is selected and provided to the system, the system can generate a thermal report with little to no additional human intervention. One or more UAVs 102 may perform a flight, collect all necessary data, such as position and orientation data, thermal and visible light images, and environmental data, and transmit this data to the cloud processing unit 108. Upon receipt by the cloud processing unit, the collected data can be processed into a thermal report without any additional human intervention.

[0141] In some cases, minimal human intervention may be required or optionally permitted. For example, a human operator may input report generation parameters (such as the selection of the report format and the analysis to be performed), select a subset of inspection structures to be examined, manually review the collected data, and then manually instruct the system to continue report generation, as well as make other parameter adjustments and input management types.

[0142] Referring now to Figure 14, a flowchart is shown illustrating a thermal inspection method 500 for a structure according to an embodiment. Method 500 includes steps 504, 506, 508, and 510, and optionally steps 502 and 512. The above descriptions of systems 10, 100, and 200 may also apply to method 500.

[0143] Step 502 provides at least one UAV. According to various embodiments, the at least one UAV consists of a single UAV or multiple UAVs, each UAV may be equipped with a thermal camera system and / or a visible light camera system and a positioning system. According to a preferred embodiment, two UAVs are provided. The first UAV is configured for daytime operation and is equipped with a high-resolution, high-frame-rate visible light camera, and the second UAV is configured for nighttime operation and is equipped with both a thermal camera system and a visible light camera system.

[0144] In step 504, the UAV flies along a pre-planned flight path and captures visible light images, thermal images, and positioning data. According to embodiments in which multiple UAVs are used to capture images of the inspection structure, each UAV follows the same pre-planned flight path, and each UAV flight session is performed alternately or at different times (e.g., daytime, nighttime). According to a preferred embodiment, visible light images are captured during daytime flight sessions, and thermal images are captured during nighttime flight sessions.

[0145] In step 506, the captured data is sent to the processing unit and received by the processing unit.

[0146] In step 508, the captured data is provided to the building element envelope detection module of the processing unit, and building element data is generated.

[0147] In step 510, the captured data and building element data are provided to the thermal anomaly module of the processing unit, and thermal anomaly data is generated.

[0148] In step 512, the captured data and thermal anomaly data are provided to the processing unit's report generation module, and a thermal report is generated.

[0149] While the above description provides examples of one or more devices, methods, or systems, it will be understood that other devices, methods, or systems, as interpreted by those skilled in the art, also fall within the scope of the claims.

Claims

1. A thermal testing method, The process involves receiving thermal image data of the inspection structure, visible light image data of the inspection structure, and positioning data from one or more unmanned aerial vehicles (UAVs) in a processing device, The steps include inputting the thermal image data, the visible light image data, and the positioning data into the building envelope element detection module of the processing device to generate building element data, The process includes the step of inputting the building element data, the thermal image data, and the visible light image data into the thermal anomaly module of the processing device to generate thermal anomaly data, The thermal anomaly data includes a thermal inspection method, which includes the location of the thermal anomaly and the classification of the thermal anomaly in the inspected structure.

2. A method according to claim 1, further comprising the step of providing thermal anomaly data to a report generation module of the processing device and generating a thermal inspection report.

3. The method according to claim 1, The step of flying one or more UAVs along a predetermined flight path around the inspection structure, wherein each UAV A thermal camera for capturing the thermal image data and / or a visible light camera for capturing the visible light image data, A step comprising a positioning system for capturing the aforementioned positioning data, The steps include capturing the thermal image data of the inspection structure, the visible light image data of the inspection structure, and the positioning data of the inspection structure at regular intervals as one or more UAVs cross the flight path, A method further comprising the step of transmitting the thermal image data, the visible light image data, and the positioning data to the processing device.

4. A method according to claim 3, further comprising the step of providing one or more UAVs.

5. The method according to claim 3, The steps include acquiring visible light image data during daytime flight along a predetermined flight path around the inspection structure, A method further comprising the step of acquiring thermal image data during nighttime flight along a predetermined flight path around the inspection structure.

6. A method according to any one of claims 1 to 5, The steps include inputting the aforementioned building element data, thermal image data, and visible light image data into the input layer of the trained neural network of the thermal anomaly module, A method comprising the step of receiving the thermal anomaly data as the output of the neural network.

7. A method according to any one of claims 1 to 6, A method further comprising the step of using the thermal anomaly module to evaluate whether there is a thermal anomaly in a region within the thermal image data and the visible light image data, according to the building element data for each region.

8. A method according to any one of claims 1 to 7, The steps include providing the positioning data, visible light image data, and thermal image data as inputs to the position correlation module of the processing device and generating scale data, The steps include: inputting the scale data into the thermal anomaly module when generating the thermal anomaly data; A method further comprising the step of inputting the scale data to a building element detection module when generating the aforementioned building element data.

9. A method according to any one of claims 1 to 8, wherein the thermal camera has a resolution of at least 1024 x 1024 pixels.

10. A method according to any one of claims 1 to 9, wherein the building element data includes a classification of building elements such as walls, doors, windows, and roofs.

11. A method according to claim 2, wherein the thermal inspection report includes heat loss data.

12. A method according to any one of claims 1 to 11, wherein the thermal anomaly data is characterized according to the building element data.

13. A thermal inspection system, Equipped with one or more unmanned aerial vehicles (UAVs), each UAV is A thermal camera for acquiring thermal image data and / or a visible light camera for acquiring visible light image data, It includes a positioning system for acquiring positioning data, Each of the aforementioned UAVs is: Fly along the designated flight path around the inspection structure. Capture the thermal image data and / or visible light image data of the inspection structure. The positioning data is captured at regular intervals, The thermal image data, the visible light image data, and the positioning data are configured to be transmitted to a processing device via a network. The aforementioned processing apparatus is The thermal image data, the visible light image data, and the positioning data are received. The thermal image data, the visible light image data, and the positioning data are input to the building envelope element detection module of the processing device to generate building element data. The system is configured to input the building element data, the thermal image data, and the visible light image data into the thermal anomaly module of the processing device to generate thermal anomaly data, the thermal anomaly data including the location and classification of the thermal anomaly. Thermal inspection system.

14. A system according to claim 13, wherein the processing device is further configured to provide the thermal anomaly data to a report generation module of the processing device and generate a thermal inspection report.

15. A system according to claim 14 or 15, wherein the thermal anomaly module inputs the building element data, the thermal image data and the visible light image data into the input layer of a trained neural network and receives the thermal anomaly data as the output of the neural network.

16. A system according to any one of claims 13 to 15, wherein, according to the building element data for each region, the thermal anomaly module evaluates whether a region in the thermal image data and the visible light image data has a thermal anomaly.

17. A system according to any one of claims 13 to 16, wherein the processing apparatus further comprises a position correlation module, and the processing apparatus further comprises Positioning data, visible light image data, and thermal image data are provided as input to the position correlation module, and scale data is generated. When generating the aforementioned thermal anomaly data, the scale data is input to the thermal anomaly module. A system configured to input the scale data to a building element detection module when generating the aforementioned building element data.

18. A system according to any one of claims 13 to 17, wherein the thermal camera has a resolution of at least 1024 x 1024 pixels.

19. A system according to any one of claims 13 to 18, wherein the building element data includes a classification of building elements such as walls, doors, windows, and roofs.

20. The system according to claim 14, wherein the thermal inspection report includes thermal loss data.

21. A system according to any one of claims 13 to 20, wherein the thermal anomaly data is characterized according to the building element data.

22. A system according to any one of claims 13 to 21, wherein the first UAV is configured for daytime flight operations and the second UAV is configured for nighttime flight operations.

23. The system according to claim 22, wherein the first UAV comprises a high-resolution, high-frame-rate visible light camera.

24. The system according to claim 22, wherein the second UAV comprises a thermal camera and a visible light camera.

Images