Infrared thermographic analysis device, infrared thermographic analysis method and recording medium

Abstract

Provided is an infrared image analysis device, an infrared image analysis method, and a program capable of accurately reducing a temperature gradient. The infrared thermal image analysis device is an infrared thermal image analysis device having a processor that performs the following processing: acquires a first infrared thermal image of a structure surface obtained by photographing a structure of an inspection target; acquires region information that distinguishes a region of the structure surface corresponding to the first infrared thermal image for at least one region; estimates a temperature gradient in the at least one region based on the region information and a second infrared thermal image; and reduces the influence of the temperature gradient from the first infrared thermal image.

Description

Technical Field

[0001] This invention relates to an infrared thermal image analysis device, an infrared thermal image analysis method, and a program. Background Technology

[0002] A known technique involves using infrared thermal images acquired by an infrared camera when photographing structures such as concrete to identify damaged and undamaged parts within the structure, such as bulges and cracks. If a structure has damaged parts, a temperature difference will occur between the surface temperature of the damaged parts and the surface temperature of the undamaged parts. Therefore, if a localized area with a temperature different from the surrounding area exists in the infrared thermal image, it can be determined that there is a damaged part inside that area.

[0003] On the other hand, due to factors such as the shape of the structure and its surrounding environment, the amount of heat received or dissipated on the surface of the structure can vary locally, sometimes creating a temperature gradient. If such a temperature gradient exists on the surface of the structure, it becomes difficult to distinguish between undamaged and damaged areas. In addition to the shape of the structure and its surrounding environment, local differences in the amount of heat received or dissipated on the surface of the structure may also be caused by local differences in the color, roughness, unevenness, thermal conductivity, or emissivity of the surface.

[0004] Regarding this issue, Patent Document 1 discloses the following: an average temperature distribution image obtained by moving average of an infrared thermal image with a specified number of pixels is generated; the temperature difference between the same pixels of the infrared thermal image and the average temperature distribution image is calculated and a temperature difference image is generated and displayed.

[0005] Previous technical documents

[0006] Patent documents

[0007] Patent Document 1: Japanese Patent No. 5140892 Summary of the Invention

[0008] The technical problem to be solved by the invention

[0009] When a structure has multiple surfaces with varying inclinations, the average temperature and the direction and inclination of the temperature gradient differ on each surface due to factors such as sunlight. If the infrared thermal image is smoothed and subtracted without distinguishing between these surfaces with different average temperatures and temperature gradients, the boundaries of these surfaces will be mistakenly detected as damaged areas. Furthermore, there is a problem: since other surfaces are included in the pixel range used for smoothing, it is impossible to accurately reduce the temperature gradient.

[0010] The present invention was made in view of the following circumstances, and its object is to provide an infrared thermal image analysis device, infrared thermal image analysis method and program that can accurately reduce the temperature gradient.

[0011] means for solving technical problems

[0012] The infrared thermal image analysis device involved in the first method is an infrared thermal image analysis device equipped with a processor, wherein the processor performs the following processing: acquiring a first infrared thermal image of the surface of the structure obtained by photographing the structure of the object to be inspected; acquiring region information for at least one region that distinguishes the region of the structure surface corresponding to the first infrared thermal image; inferring the temperature gradient in at least one region based on the region information and the second infrared thermal image; and reducing the influence of the temperature gradient from the first infrared thermal image.

[0013] In the infrared thermal image analysis device involved in the second method, the processor obtains area information based on information related to the structure, including visible light images of the structure obtained by capturing images of the structure.

[0014] In the infrared thermal image analysis apparatus involved in the third method, the processor acquires area information based on information related to the structure, including at least one of a first infrared thermal image or a second infrared thermal image obtained by photographing the structure.

[0015] In the infrared thermal image analysis device involved in the fourth method, the processor acquires area information based on information related to the structure, including data obtained from measuring the distance to the structure.

[0016] In the infrared thermal image analysis device involved in Method 5, the processor obtains area information based on information related to the structure, including drawing data of the structure.

[0017] In the infrared thermal image analysis apparatus involved in the sixth method, the processor prioritizes the use of the second infrared thermal image of the region over the second infrared thermal images of other regions to infer the temperature gradient in the region.

[0018] In the infrared thermal image analysis apparatus involved in the seventh method, the processor assigns different weights to the region and other regions in a priority application to smooth the second infrared thermal image.

[0019] In the infrared thermal image analysis apparatus involved in the eighth method, the processor performs smoothing processing on the second infrared thermal image by extending along the boundary of the region in a range that does not include other regions in a preferred application.

[0020] In the infrared thermal image analysis device involved in the ninth method, the processor preferentially uses the second infrared thermal image in the surrounding area of ​​the region to infer the temperature gradient in the region.

[0021] In the infrared thermal image analysis apparatus according to the 10th method, when the smallest distance from a pixel in the region to each pixel on the boundary of the region is set as the distance from each pixel to the boundary, and the distance from the pixel with the largest distance to the boundary is set as the distance from the center to the boundary, the surrounding region is a region that includes at least pixels whose distance to the boundary is less than 1 / 2 of the distance from the center to the boundary.

[0022] In the infrared thermal image analysis apparatus involved in Method 11, the processor infers the temperature gradient in the region through thermal simulation.

[0023] In the infrared thermal image analysis apparatus involved in the 12th method, when the processor reduces the influence of the temperature gradient, it subtracts the temperature gradient from the first infrared thermal image or divides the first infrared thermal image by the temperature gradient.

[0024] In the infrared thermal image analysis apparatus involved in the 13th method, at least one of the first infrared thermal image, the second infrared thermal image, and the information related to the structure on which the area information is based is acquired at different times.

[0025] In the infrared thermal image analysis apparatus involved in the 14th method, at least one of the structural information on which the first infrared thermal image, the second infrared thermal image, and the area information are based is an image or information obtained by merging multiple images or information.

[0026] In the infrared thermal image analysis device involved in the 15th method, the processor acquires the first infrared thermal image and the second infrared thermal image when there is sunlight.

[0027] In the infrared thermal image analysis apparatus of the 16th method, the surface of the structure includes at least one of a plurality of surfaces with different inclinations or discontinuous surfaces.

[0028] In the infrared thermal image analysis apparatus according to the 17th method, the processor displays a temperature gradient reduction image obtained by reducing the influence of the temperature gradient in the first infrared thermal image on a display device.

[0029] In the infrared thermal image analysis apparatus involved in Method 18, the processor displays the image with reduced temperature gradient after image processing on the display device.

[0030] In the infrared thermal image analysis apparatus involved in the 19th method, the first infrared thermal image and the second infrared thermal image are the same infrared thermal image.

[0031] The infrared thermal image analysis method involved in the 20th method includes: acquiring a first infrared thermal image of the surface of the structure obtained by photographing the structure of the object to be inspected; acquiring region information for at least one region that distinguishes the region of the structure surface corresponding to the first infrared thermal image; inferring the temperature gradient in at least one region based on the region information and the second infrared thermal image; and reducing the influence of the temperature gradient from the first infrared thermal image.

[0032] The program for causing a computer to execute according to the 21st method causes the computer to perform the following processes: acquiring a first infrared thermal image of the surface of the structure obtained by photographing the structure of the object to be inspected; acquiring region information for at least one region that distinguishes the region of the structure surface corresponding to the first infrared thermal image; inferring the temperature gradient in at least one region based on the region information and the second infrared thermal image; and reducing the effect of the temperature gradient from the first infrared thermal image.

[0033] Invention Effects

[0034] The infrared thermal image analysis device, infrared thermal image analysis method and program according to the present invention can accurately reduce the temperature gradient. Attached Figure Description

[0035] Figure 1 This is a block diagram illustrating an example of the hardware structure of an infrared thermal image analysis device.

[0036] Figure 2 It is a block diagram representing the processing functions implemented by the CPU.

[0037] Figure 3 It is a diagram representing information stored in the storage unit.

[0038] Figure 4 This is a flowchart illustrating an infrared thermal image analysis method using an infrared thermal image analysis device.

[0039] Figure 5 It is a three-dimensional diagram used to illustrate a group of test subjects that simulate a structure.

[0040] Figure 6 It refers to the infrared thermal images obtained from photographing the test subjects.

[0041] Figure 7 These are visible light images obtained from photographing the test subjects.

[0042] Figure 8 This is a diagram showing the results of differentiating and determining the upper surfaces of the four test subjects.

[0043] Figure 9 This is a diagram representing the first method of processing in the temperature gradient inference step.

[0044] Figure 10This is a diagram representing the second method of processing in the temperature gradient inference step.

[0045] Figure 11 This is a diagram representing the third method of processing in the temperature gradient inference step.

[0046] Figure 12 It is a diagram used to illustrate the surrounding area of ​​an infrared thermal image.

[0047] Figure 13 It is a graph representing a temperature gradient.

[0048] Figure 14 It is a graph representing the decrease in temperature gradient.

[0049] Figure 15 This is a graph representing a reduction in temperature gradient based on previous techniques.

[0050] Figure 16 This diagram illustrates a method for inferring temperature gradients based on regional information through thermal simulation. Detailed Implementation

[0051] Hereinafter, preferred embodiments of the infrared thermal image analysis apparatus, infrared thermal image analysis method, and program related to the present invention will be described with reference to the accompanying drawings. In the specification, temperature distribution refers to the temperature difference (temperature change) caused by the undamaged part and the damaged part, and temperature gradient refers to the temperature difference (temperature change) that is not part of the temperature distribution.

[0052] As described above, the inventors discovered the following problem and achieved the present invention: if the infrared thermal image is smoothed to take the difference, the boundaries of different surfaces are mistakenly detected as damaged parts, and since other surfaces are included in the pixel range used for smoothing, the temperature gradient cannot be accurately reduced.

[0053] Furthermore, differences in average temperature and temperature gradient across the various surfaces of a structure can occur not only when the structure is directly exposed to sunlight but also when it is indirectly exposed to sunlight. Moreover, these differences can arise not only from sunlight but also from reflected or radiated light, including infrared radiation, from the surrounding area of ​​the structure (i.e., they can occur day and night). Furthermore, when the surface of a structure has multiple regions with different colors, roughness, unevenness, thermal conductivity, or emissivity, differences in the amount of heat received or dissipated in each region can similarly lead to differences in average temperature and temperature gradient, potentially causing the same problem. Similarly, differences in the amount of heat received in areas of the structure's surface exposed to sunlight or other forms of light, and in areas that are not exposed, can also lead to differences in average temperature and temperature gradient, potentially causing the same problem. Furthermore, even if the surface of the structure is uniform, if a portion of it has areas where the amount of heat received or dissipated varies locally due to differences in elevation, depressions, interruptions, or different components, temperature gradients in different directions can originate from these points, potentially causing the same problem. Furthermore, when a structure has multiple surfaces with varying inclinations, the image obtained by projecting the infrared thermal radiation from the three-dimensional structure's surface in two dimensions along the imaging direction of the infrared camera is itself an infrared thermal image. Therefore, the direction and inclination of the temperature gradient on each surface of the structure in the infrared thermal image will naturally differ depending on the angle between the structure's surface and the imaging direction, and the average temperature may also differ. Thus, if these surfaces are not distinguished and the infrared thermal image is smoothed to obtain differences, the aforementioned problem occurs. Of course, there are also surfaces that become shadows of other surfaces. In this case, three-dimensionally discontinuous surfaces appear consecutively adjacent in the infrared thermal image. Therefore, if these surfaces are not distinguished and the infrared thermal image is smoothed to obtain differences, the aforementioned problem occurs.

[0054] As explained above, when a structure has multiple surfaces with different inclinations or discontinuous surfaces (such as surfaces separated by shadows, elevation differences, depressions, breaks, or different components), multiple areas with different colors, roughness, unevenness, thermal conductivity, or emissivity, areas illuminated by sunlight or other light, and unilluminated areas, differences in average temperature and temperature gradient may occur in each area. Therefore, if the infrared thermal image is smoothed and subtracted without distinguishing these areas, the boundaries of each area may be mistakenly detected as damaged parts, and the temperature gradient cannot be accurately reduced. This problem may occur regardless of day or night. The inventors discovered the above problem and subsequently developed various embodiments.

[0055] [Hardware Structure of the Infrared Thermal Image Analysis Device]

[0056] Figure 1This is a block diagram illustrating an example of the hardware structure of the infrared thermal image analysis device according to the embodiment.

[0057] As Figure 1 The infrared thermal image analysis device 10 shown can be used with a computer or workstation. In this example, the infrared thermal image analysis device 10 mainly consists of an input / output interface 12, a storage unit 16, an operation unit 18, a CPU (Central Processing Unit) 20, RAM (Random Access Memory) 22, ROM (Read Only Memory) 24, and a display control unit 26. A display device 30 is connected to the infrared thermal image analysis device 10, and under the instructions of the CPU 20 and the control of the display control unit 26, the image is displayed on the display device 30. The display device 30 is, for example, a monitor.

[0058] The input / output interface 12 (I / F in the figure) can input various data (information) to the infrared thermal image analysis device 10. For example, data stored in the storage unit 16 can be input via the input / output interface 12.

[0059] The CPU (processor) 20 reads various programs stored in the storage unit 16 or ROM 24 and expands them into RAM 22 for calculation, thereby controlling all parts. Furthermore, the CPU 20 reads programs stored in the storage unit 16 or ROM 24 and uses RAM 22 to perform calculations to perform various processes of the infrared thermal image analysis device 10.

[0060] Figure 1 The infrared camera 32 shown captures an infrared thermal image of the structure 36, which is the object of inspection, and acquires an infrared thermal image of the structure's surface. The visible light camera 34 captures an visible light image of the structure 36, which is the object of inspection.

[0061] The infrared thermal image analysis device 10 can acquire infrared thermal images from the infrared camera 32 via the input / output interface 12. Furthermore, the infrared thermal image analysis device 10 can acquire visible light images from the visible light camera 34 via the input / output interface 12. The acquired infrared thermal images and visible light images can be stored, for example, in the storage unit 16.

[0062] Figure 2 This is a block diagram representing the processing functions implemented by CPU20.

[0063] CPU 20 functions as the information acquisition unit 51, the region acquisition unit 53, the temperature gradient inference unit 55, the temperature gradient reduction unit 57, and the information display unit 59. The specific processing functions of each unit will be explained later. Since the information acquisition unit 51, the region acquisition unit 53, the temperature gradient inference unit 55, the temperature gradient reduction unit 57, and the information display unit 59 are all part of CPU 20, it can also be said that CPU 20 executes the processing of each unit.

[0064] Return to Figure 1 The storage unit (memory) 16 is a memory device composed of a hard disk drive, flash memory, etc. The storage unit 16 stores the operating system, programs for executing infrared thermal image analysis methods, and data and programs that enable the infrared thermal image analysis device 10 to operate. Furthermore, the storage unit 16 stores information used in the embodiments described below. Additionally, the program for enabling the infrared thermal image analysis device 10 to operate can be recorded on an external recording medium (not shown) and distributed, and installed from that recording medium by the CPU 20. Alternatively, the program for enabling the infrared thermal image analysis device 10 to operate can be stored in a manner that allows external access to a server connected to a network, and downloaded to the storage unit 16 via the CPU 20 upon request for installation and execution.

[0065] Figure 3 This diagram represents information stored in the storage unit 16. The storage unit 16 consists of non-transitory recording media such as CD (Compact Disk), DVD (Digital Versatile Disk), Hard Disk, and various semiconductor memories, as well as their control units.

[0066] The storage unit 16 mainly stores infrared thermal images 101 and visible light images 103.

[0067] Infrared thermal image 101 is an image captured by infrared camera 32, showing the temperature distribution on the surface of the structure by detecting the infrared radiation energy radiated from the structure 36 and converting that infrared radiation energy into temperature. Visible light image 103 is an image captured by visible light camera 34, showing the distribution of the intensity of reflected visible light from the surface of the structure 36. Typically, the visible light image is composed of RGB images obtained by imagerizing the intensity distribution in three different wavelength regions of visible light, i.e., each pixel has color information (RGB signal value). In this example, it is also assumed to have color information. Furthermore, in this example, it is assumed that there is no positional offset between infrared thermal image 101 and visible light image 103.

[0068] Figure 1The operation unit 18 shown includes a keyboard and mouse, which allow the user to perform necessary processing on the infrared thermal image analysis device 10. The display device 30 can function as an operation unit by using a touch panel type device.

[0069] The display device 30 is, for example, a liquid crystal display or other device, capable of displaying the results obtained by the infrared thermal image analysis device 10.

[0070] Figure 4 This is a flowchart illustrating the infrared thermal image analysis method using the infrared thermal image analysis device 10.

[0071] First, the information acquisition unit 51 acquires a first infrared thermal image of the surface of the structure being inspected, obtained by photographing the structure (first infrared image acquisition step: step S1). In this example, the first infrared thermal image is the infrared thermal image 101 stored in the storage unit 16. The information acquisition unit 51 acquires the infrared thermal image 101 from the storage unit 16.

[0072] Next, the region acquisition unit 53 acquires region information for at least one region, distinguishing the region of the structure surface corresponding to the first infrared thermal image, based on information related to the structure (region acquisition step: step S2). In this example, the information related to the structure is the visible light image 103 stored in the storage unit 16. The information acquisition unit 51 acquires the visible light image 103 from the storage unit 16. The region acquisition unit 53 acquires region information for at least one region, distinguishing the region of the structure surface corresponding to the infrared thermal image 101, which is the first infrared thermal image, based on the visible light image 103.

[0073] Next, the temperature gradient inference unit 55 infers the temperature gradient in at least one region based on the region information and the second infrared thermal image (temperature gradient inference step: step S3). In this example, the second infrared thermal image is the infrared thermal image 101 stored in the storage unit 16. Therefore, the first infrared thermal image and the second infrared thermal image are the same infrared thermal image 101. The infrared thermal image 101, which is the second infrared thermal image, has been acquired in the first infrared image acquisition step (step S1). The temperature gradient inference unit 55 infers the temperature gradient in at least one region based on the region information and the infrared thermal image 101. Here, the information of the inferred temperature gradient can be any kind of information as long as it can reduce the influence of the temperature gradient from the first infrared thermal image in the subsequent temperature gradient reduction step. For example, it can be an image of the temperature gradient, a formula that can derive the temperature gradient, or a set of processing and data that can derive the temperature gradient.

[0074] In addition, the first infrared thermal image and the second infrared thermal image are sometimes referred to simply as "infrared thermal image" without making a specific distinction.

[0075] Next, the temperature gradient reduction unit 57 reduces the influence of the temperature gradient from the first infrared thermal image (temperature gradient reduction step: step S4). In this example, the influence of the temperature gradient is reduced from the infrared thermal image 101, which is the first infrared thermal image. The temperature gradient reduction unit 57 is capable of acquiring a temperature gradient reduction image.

[0076] Next, the information display unit 59 displays the temperature gradient reduction image on the display device 30 (information display step: step S5). The information display unit 59 can also display the image-processed temperature gradient reduction image on the display device 30.

[0077] As described above, the infrared thermal image analysis device 10 in this example infers the temperature gradient from the area information obtained from the visible light image 103 and the infrared thermal image 101, and reduces the influence of the temperature gradient from the infrared thermal image 101.

[0078] In the infrared thermal image analysis method, the order of implementation of the first infrared image acquisition step (step S1) and the temperature gradient inference step (step S3) is irrelevant as long as it precedes the temperature gradient reduction step (step S4).

[0079] Hereinafter, specific examples will be given regarding each step, using a test subject that simulates structure 36 as an example.

[0080] <Step 1 for acquiring the infrared image>

[0081] The information acquisition unit 51 performs the first infrared image acquisition step (step S1). The information acquisition unit 51 acquires an infrared thermal image 101 of the surface of the structure 36 being inspected, obtained from the image taken in the storage unit 16, as the first infrared thermal image. Alternatively, when the infrared thermal image 101 is not stored in the storage unit 16, the information acquisition unit 51 acquires the infrared thermal image 101 from an external source. For example, the information acquisition unit 51 can acquire the infrared thermal image 101 via a network through the input / output interface 12, and the information acquisition unit 51 can acquire the infrared thermal image 101 from the infrared camera 32 via the input / output interface 12.

[0082] Figure 5 This is a perspective view illustrating the test body group 40 that simulates structure 36. The test body group 40 includes four test bodies 41, 42, 43, and 44. Each test body 41, 42, 43, and 44 is a rectangular concrete block, which is arranged at intervals on corrugated paper 45.

[0083] Simulated buoyancy (cavities) were formed in test subject 44 at a depth of 1 cm from the top surface, in test subject 41 at a depth of 2 cm from the top surface, and in test subject 43 at a depth of 3 cm from the top surface. Test subject group 40 was placed outdoors on a clear day with sunlight shining from the upper left. Infrared thermal images of test subject group 40 were captured by infrared camera 32.

[0084] Figure 6 The infrared thermal image 101 captured by the infrared camera 32 is acquired by the information acquisition unit 51 as the first infrared thermal image. In the infrared thermal image 101, the temperature of each pixel is... Figure 6 It is displayed in grayscale. Figure 6 (A) is an infrared thermal image 101 obtained from the actual photograph of test subject group 40. Figure 6 (B) is in Figure 6 An image of infrared thermal image 101 (A) is supplemented with arrows indicating the direction of the temperature gradient.

[0085] from Figure 6 As can be seen from the infrared thermal image 101 of (A), as Figure 6 As shown in (B), a temperature gradient with a downward slope from left to right (arrow A) is generated on the upper surfaces of the four test subjects 41, 42, 43, and 44. This temperature gradient is caused by sunlight from the upper left, which originates from the heat inflow from the left side near the boundary between the upper surface and the left side.

[0086] On the other hand, from Figure 6 As can be seen from the infrared thermal image 101 of (A), as Figure 6 As shown in (B), a temperature gradient with a downward slope (arrow B) is generated on the left side of the four test subjects 41, 42, 43 and 44. This temperature gradient is caused by heat inflow from the upper surface near the boundary between the left side and the upper surface.

[0087] It is understandable that temperature gradients were generated on any of the four test subjects 41, 42, 43 and 44, caused by heat inflow from other surfaces near the boundary with other surfaces.

[0088] The direction of the temperature gradient varies depending on the face of the test specimens 41, 42, 43, and 44. Furthermore, since the heat received by the face of the test specimens 41, 42, 43, and 44 based on sunlight is different, the average temperature of each face is also different, and the amount of heat flowing from other faces near the boundary of each face is also different. As a result, the slope of the temperature gradient is also different.

[0089] <Area Acquisition Steps>

[0090] The area acquisition step (step S2) is performed by the area acquisition unit 53. Initially, in this example, the information acquisition unit 51 acquires a visible light image 103 of the surface of the structure 36 being inspected, obtained from the image stored in the storage unit 16. Additionally, when no visible light image 103 is stored in the storage unit 16, the information acquisition unit 51 acquires the visible light image 103 from an external source. For example, the information acquisition unit 51 can acquire the visible light image 103 via a network through the input / output interface 12, and the information acquisition unit 51 can acquire the visible light image 103 from the visible light camera 34 via the input / output interface 12.

[0091] Figure 7 The visible light image 103 is obtained by capturing the test subject group 40 with a visible light camera 34. As described above, the visible light image 103 has RGB signal values ​​for each pixel.

[0092] The region acquisition unit 53 acquires region information that distinguishes each region of the structure's surface based on the RGB signal values ​​and spatial feature information such as edges or textures of the acquired visible light image 103.

[0093] The region acquisition unit 53 extracts, for example, a group of pixels that can be considered as a concrete surface from each pixel of the visible light image 103. When the RGB signal value of each pixel is, for example, within a predetermined range of RGB values ​​that can be considered as a concrete surface, each pixel is extracted as a group of pixels that can be considered as a concrete surface.

[0094] After extracting pixel groups, the region acquisition unit 53 further subdivides the pixel groups based on the RGB signal values ​​and spatial characteristics of each pixel. It then fills each pixel group using morphological dilation operations, causing it to dilate. Finally, for each dilated pixel group, it uses an active contour method to shrink it to the optimal region to determine the region corresponding to each pixel group. As an active contour method, methods such as Snakes and Level Sets can be applied.

[0095] Based on the RGB signal values ​​and spatial feature information of the visible light image 103, the region acquisition unit 53 can distinguish at least one of the surfaces with different inclinations or discontinuous surfaces on the surface of the structure as different regions. Furthermore, the region acquisition unit 53 can distinguish any region on the surface of the structure as different regions based on color, roughness, unevenness, or the presence or absence of sunlight. Discontinuous surfaces include surfaces separated by shadows, height differences, depressions, interruptions, different components, etc.

[0096] The region acquisition unit 53 acquires region information that distinguishes each region on the surface of the structure by performing the above-described process.

[0097] However, in the visible light image 103, there are cases where the RGB signal values ​​of pixels within a region are close to the RGB signal values ​​of pixels surrounding the region, making it difficult to determine the boundaries of the region. For example, in Figure 7 In the four test subjects 41, 42, 43 and 44, the RGB signal value of test subject 43 is close to the signal value of the surrounding corrugated paper 45, making it difficult to determine the right boundary of test subject 43.

[0098] On the other hand, Figure 6 In the infrared thermal image 101 shown in (A), the temperature difference (signal value difference) between the test object 43 and the corrugated paper 45 is significantly large, and the boundary is clearly indicated. Under such circumstances, the region acquisition unit 53 can also determine the optimal region and acquire region information based on both the visible light image 103 and the infrared thermal image 101.

[0099] There are situations where it is difficult to determine the area based solely on visible light images, such as when photographing shaded areas of a structure that are not exposed to sunlight or during nighttime photography. In such cases, it is preferable to determine the area based on both visible light images and infrared thermal images.

[0100] If a structure contains areas on its surface with varying thermal conductivity or infrared emissivity, such as repair materials (repair marks), these areas may exhibit different average temperatures and temperature gradients compared to the surrounding concrete areas due to differences in heat received or dissipated. Regarding areas with varying thermal conductivity or infrared emissivity, it can sometimes be difficult to determine the regions in visible light images; therefore, it is preferable to determine the regions based on infrared thermal images. Furthermore, when using infrared thermal images as information related to the structure for obtaining region information, either a first infrared thermal image or a second infrared thermal image is sufficient.

[0101] There are many methods for distinguishing and determining regions, such as MeanShift and Graph Cuts. Any of these methods can be used to determine the region. Machine learning can also be used to determine the region. For example, CNN (Convolutional Neural Network), FCN (Fully Convolutional Network), U-net (Convolutional Networks for Biomedical Image Segmentation), and SegNet (A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation) can be used to determine the region. As long as the method is based on the features of visible light images and infrared thermal images, there is no particular limitation, and any of these methods can be used.

[0102] The region acquisition unit 53 acquires only the required number (regions Ar(i); i = 1, 2, 3, ... N) of region information that distinguishes the surface of the structure.

[0103] Figure 8 This is a diagram of region information 105 obtained by the region acquisition unit 53, which divides the upper surfaces of the four test objects 41, 42, 43, and 44 into four regions based on the features of the visible light image 103 and the infrared thermal image 101. Regions Ar(1), Ar(2), Ar(3), and Ar(4) represent the regions corresponding to the upper surfaces of the test objects 41, 42, 43, and 44, respectively. In this example, a temperature gradient inference step and a temperature gradient reduction step are performed on the four regions Ar(1), Ar(2), Ar(3), and Ar(4).

[0104] <Temperature gradient inference steps>

[0105] The temperature gradient inference step (step S3) is performed by the temperature gradient inference unit 55. The temperature gradient inference unit 55 infers the temperature gradient of region Ar(i) from the acquired infrared thermal image of region Ar(i). In this example, the temperature gradients are inferred for the four regions Ar(1), Ar(2), Ar(3), and Ar(4) respectively.

[0106] Several examples can be given as methods for inferring temperature gradients. For example, the temperature gradient inference unit 55 can apply a smoothing filter with a specified number of pixels to an infrared thermal image to derive an image of global temperature changes and infer it as a temperature gradient.

[0107] At this time, the temperature gradient inference unit 55 uses the infrared thermal image in region Ar(i) more preferentially to infer the temperature gradient than regions other than region Ar(i), i.e. regions whose average temperature and temperature gradient (direction and / or slope) are different from region Ar(i).

[0108] Figure 9 This is a diagram representing the first method of processing in the temperature gradient inference step. Figure 9 An example of an 11×11 pixel (121 pixels) smoothing filter centered on the pixel of interest (represented by black pixels) is shown. Near the boundary of region Ar(i), a smoothing filter is applied within the range that does not exceed the boundary, and the temperature gradient inference unit 55 performs smoothing processing. The range of region Ar(i) that exceeds the boundary is represented by gray. In this case, the filter coefficients are set such that the sum within the range of region Ar(i) that does not exceed the boundary is "1". Figure 9 In the configuration, the filter coefficient is set to "1 / 73". Additionally, in the case of a 121-pixel filter, when all pixels of the filter do not exceed the boundary, its coefficient is set to "1 / 121".

[0109] Furthermore, even when the filter coefficients change based on pixel position, as in the case of a Gaussian filter, this can be addressed by setting each filter coefficient to a constant multiple such that their total value is 1. Here, a Gaussian filter is defined as a filter that follows a Gaussian distribution function, increasing the filter coefficients as it gets closer to the pixel of interest.

[0110] Figure 10 This diagram illustrates the second approach to processing the temperature gradient inference step. (Example) Figure 10 As shown, the temperature gradient inference unit 55, near the boundary of region Ar(i), extends the smoothing filter along the boundary within the region Ar(i) and performs smoothing processing. In this case, the number of pixels in the filter does not change regardless of the position of the pixel of interest, thus the filter coefficients are constant. In this example, an 11×11 pixel (121 pixels) smoothing filter extends along the boundary of the region and sets the filter coefficients to "1 / 121". Figure 10 In the image, only a portion of the pixels display the filter coefficient of "1 / 121".

[0111] Next, the third method of processing in the temperature gradient inference step will be explained. As mentioned above, when the surface of a structure is composed of multiple regions with different amounts of heat received or dissipated, a temperature gradient is generated in each region due to heat inflow or outflow from adjacent regions near the boundary. Furthermore, the multiple regions with different amounts of heat received or dissipated may include those composed of surfaces with different inclinations, or those composed of regions with different colors, roughness, unevenness, presence or absence of sunlight, thermal conductivity, and infrared emissivity.

[0112] This indicates that the boundary area becomes the starting point of the temperature gradient. Similarly, in discontinuous regions separated by differences in elevation, depressions, breaks, or different components, the boundary area also becomes the starting point of the temperature gradient. From this phenomenon, it can be understood that the temperature gradient in each region can be inferred from the infrared thermal image of the area near the boundary.

[0113] Compared to methods that infer temperature gradients from infrared thermal images of the entire region, this method is unaffected by the temperature distribution (not temperature gradient changes) generated by the damaged parts within the region, and is therefore preferred.

[0114] according to Figure 11 The third method of processing in the temperature gradient inference step is explained.

[0115] Figure 11 (A) is an infrared thermal image 106 of region Ar(1) on the upper surface of test body 41. Infrared thermal image 106 is an image obtained by extracting only region Ar(1) on the upper surface of test body 41 from infrared thermal image 101 of test body group 40 based on region information 105. Here, infrared thermal image 106 becomes the original image of the temperature gradient image 107 subsequently derived, and therefore it is preferable to obtain an image obtained by smoothing infrared thermal image 101. However, smoothing is not necessary.

[0116] Figure 11 (B) is for Figure 8 The region Ar(1) of the region information 105 shown extracts the image around the boundary with a specified width. Therefore, in Figure 11 In (B), only the area surrounding region Ar(1) is shown.

[0117] Figure 11 (C) is based on Figure 11 Infrared thermal image 106 of (A) and Figure 11 The image obtained by extracting the regional information around the boundary of (B) is obtained by extracting only the infrared thermal image of the region surrounding region Ar(1).

[0118] Figure 11 (D) is obtained through interpolation operations from Figure 11 The temperature gradient image 107 is derived from the infrared thermal image of the surrounding region of (C). Here, spline interpolation is applied to continuously (smoothly) interpolate to the second derivative from the infrared thermal image of the surrounding region. As described above, by inferring the temperature gradient from the surrounding region, the influence of the temperature distribution (temperature difference) of the undamaged and damaged parts existing in the region can be reduced in the inference of the temperature gradient. In addition, the interpolation operation is not particularly limited, and interpolation operations other than spline interpolation can also be applied. For example, linear interpolation can be used.

[0119] Next, according to Figure 12 right Figure 11 The surrounding areas in (B) and (C) are described. Figure 12 (A) is a graph used to illustrate the distance L from each pixel of the infrared thermal image of region Ar(i) to the boundary and the distance La from the center of region Ar(i) to the boundary. Figure 12 (B) is a diagram used to illustrate the surrounding area of ​​region Ar(i).

[0120] like Figure 1 As shown in Figure 2, the minimum distance from pixel Px in the infrared thermal image contained within region Ar(i) to each pixel on the boundary of the region is defined as the distance L from pixel Px to the boundary. The maximum distance L from each pixel Px to the boundary is defined as the distance La from the center to the boundary. Within region Ar(i), the area formed by pixels Px whose distance L to the boundary is less than the distance La from the center to the boundary is defined as the surrounding region. Figure 12 In (B), a dotted pattern is shown of the surrounding region ER, which consists of pixels Px whose distance L to the boundary is less than 1 / 2 of the distance La from the center to the boundary. Furthermore, a surrounding region consisting of pixels Px whose distance La is less than 1 / 8 is also shown. Here, as a third method of processing in the temperature gradient inference step, in... Figure 11 In the example, a method for inferring the temperature gradient based solely on the infrared thermal image (or smoothed infrared thermal image) of the surrounding area was described. However, it is not necessary to infer solely based on the infrared thermal image of the surrounding area; the temperature gradient can be inferred by preferentially applying the infrared thermal image of the surrounding area. For example, when deriving the temperature gradient of region Ar(i) from the infrared thermal image of the surrounding area of ​​region Ar(i) through interpolation, the temperature gradient in region Ar(i) can be derived from the infrared thermal images of regions that include not only the surrounding area but also other regions (regions other than the surrounding area) through interpolation. In this case, the infrared thermal image of the surrounding area can be given a greater weight than that of other regions for interpolation, or more pixels from the surrounding area can be applied than those from other regions, for example, by making the pixels of regions other than the surrounding area sparse and interpolating them. Preferably, the infrared thermal image of the surrounding area whose distance L to the boundary is less than 1 / 2 of the distance La from the center to the boundary is preferentially applied to infer the temperature gradient. More preferably, the infrared thermal image of the surrounding area whose distance L to the boundary is less than 1 / 4 of the distance La from the center to the boundary is preferentially applied to infer the temperature gradient. Further, it is preferred to infer the temperature gradient from infrared thermal images of the surrounding area where the distance L to the boundary is less than 1 / 8 of the distance La from the center to the boundary. Additionally, the minimum value of the distance L to the boundary is equivalent to a distance of 1 pixel.

[0121] As described above, in the temperature gradient inference step, the temperature gradient is inferred by the temperature gradient inference unit 55.

[0122] Figure 13 This represents the temperature gradient image 107 derived during the temperature gradient inference step. (Example:) Figure 13 As shown, the temperature gradient image 107 derived in the temperature gradient inference step includes temperature gradient images 107A, 107B, 107C and 107D corresponding to regions Ar(1), Ar(2), Ar(3) and Ar(4) on the upper surfaces of the four test objects 41, 42, 43 and 44.

[0123] <Steps to reduce temperature gradient>

[0124] The temperature gradient reduction step (step S4) is performed by the temperature gradient reduction unit 57. The temperature gradient reduction unit 57 reduces the influence of the temperature gradient from the infrared thermal image 101, which is the first infrared thermal image. For example, the temperature gradient reduction unit 57 reduces the influence of the temperature gradient from the infrared thermal image 101 based on the temperature gradient image 107.

[0125] When reducing the influence of the temperature gradient, the temperature gradient reduction unit 57 can subtract the temperature gradient from the infrared thermal image 101 or divide the infrared thermal image 101 by the temperature gradient.

[0126] When subtracting the temperature gradient from the infrared thermal image 101, the values ​​of each pixel in the temperature gradient image 107 can be subtracted from the values ​​of each pixel in the infrared thermal image 101. Furthermore, when dividing the infrared thermal image 101 by the temperature gradient, the values ​​of each pixel in the infrared thermal image 101 can be divided by the values ​​of each pixel in the temperature gradient image 107. Here, the temperature can be used as the value of each pixel, for example.

[0127] In the temperature gradient reduction step, the temperature gradient reduction unit 57, for example, from... Figure 6 Subtracting from the infrared thermal image 101 shown Figure 13 The temperature gradient image shown is derived from 107. Figure 14 The temperature gradient decreases as shown in image 109.

[0128] <Information Display Steps>

[0129] The information display step (step S5) is performed by the information display unit 59. The information display unit 59 will... Figure 14 The temperature gradient reduction image 109 shown is displayed on the display device 30 (reference). Figure 1 ).

[0130] like Figure 14As shown, the temperature gradient reduction image 109 derived in the temperature gradient reduction step includes temperature gradient reduction images 109A, 109B, 109C and 109D corresponding to regions Ar(1), Ar(2), Ar(3) and Ar(4) on the upper surfaces of the four test subjects 41, 42, 43 and 44.

[0131] In temperature gradient reduction image 109A, the high-temperature region caused by buoyancy at a depth of 2 cm is visualized, and in temperature gradient reduction image 109C, the high-temperature region caused by buoyancy at a depth of 3 cm is visualized. It can be understood that the effect of the temperature gradient is reduced in temperature gradient reduction image 109.

[0132] The infrared thermal image 101 of test subject group 40, obtained by applying conventional techniques to reduce the temperature gradient, is shown below. Figure 15 .like Figure 15 As shown, in conventional techniques, infrared thermal images are smoothed and differentially analyzed without distinguishing between the average temperature of the test subject and the different surfaces with varying temperature gradients, thus misdetecting the boundaries of each surface. Furthermore, it is understandable that even with a reduced temperature gradient, the high-temperature regions caused by buoyancy at depths of 2 cm and 3 cm cannot be visualized. Additionally, while the surfaces of the four test subjects 41, 42, 43, and 44 in test subject group 40 are discontinuous, they appear adjacent and continuous in the infrared thermal image 101. Therefore, it is also evident that in conventional techniques, the infrared thermal images are smoothed and differentially analyzed without distinguishing these surfaces, thus misdetecting the boundaries of each surface.

[0133] Next, other preferred embodiments will be described.

[0134] In the above-described area acquisition step (step S2), the acquisition of area information based on the visible light image 103 and the infrared thermal image 101 is explained.

[0135] As long as the data is capable of acquiring area information, the area acquisition unit 53 can use any data to acquire area information. The data can include data obtained by measuring structures, data created about structures, etc.

[0136] For example, as data acquired by measuring a structure, data obtained by measuring distance using methods such as LIDAR (Light Detection and Ranging) can be used to acquire region information for distinguishing surfaces with different inclinations and / or discontinuous surfaces. LIDAR measures the distance to each point on the structure's surface by measuring the time from when a laser beam is irradiated onto each point and reflected back. The region acquisition unit 53 can acquire region information based on the data obtained from measuring the distance to each point on the structure's surface. The method of measuring distance is not limited to LIDAR. For example, a TOF (Time of Flight) camera or a stereo camera can also be used to measure distance. Based on the data obtained from measuring the distance to each point on the structure's surface, region information for distinguishing surfaces with different inclinations and / or discontinuous surfaces on the structure's surface can be acquired. Furthermore, since the relationship between the coordinate system of the distance measurement and the coordinate system of the infrared camera 32 is known, the region corresponding to each surface of the structure's surface can also be determined in the infrared thermal image.

[0137] Furthermore, the data generated regarding the structure can utilize the drawing data of the structure being inspected for obtaining area information. The drawing data includes drawings and CAD (computer-aided design) data. The area acquisition unit 53 determines the position and imaging direction (posture) of the infrared camera 32 using another method, thereby enabling the determination of areas of different inclinations and / or discontinuous surfaces of the structure in the infrared thermal image obtained from the aforementioned position and direction, based on the structure's drawing data. Additionally, various parameters of the infrared camera 32 are known.

[0138] The position of the infrared camera 32 can be determined using methods such as Wi-Fi positioning and acoustic positioning. Furthermore, known methods such as gyroscope sensors can be used to determine the shooting direction. By employing self-localization inference techniques based on various sensor measurements, such as SLAM (Simultaneous Localization and Mapping), the position and shooting direction (attitude) of the infrared camera 32 relative to the structure being inspected can also be determined.

[0139] Furthermore, in the aforementioned region acquisition step (step S2), the case where the visible light image 103 has color information (RGB signal values), i.e., is composed of three images (RGB images), was described. However, the types of visible light images can also be one, two, or even four or more. Region information can be acquired based on the visible light image 103 composed of any type of image and the infrared thermal image 101.

[0140] Furthermore, in the aforementioned area acquisition step (step S2), the case where area information is acquired based on the visible light image 103 and the infrared thermal image 101 as information related to the structure was described. However, it is also possible to acquire area information directly without using information related to the structure. For example, the area information corresponding to the first infrared thermal image can be stored in the storage unit 16. First, the information acquisition unit 51 acquires the area information from the storage unit 16, and then the area acquisition unit 53 directly receives the area information. Alternatively, when the area information is not stored in the storage unit 16, the area acquisition unit 53 can acquire the area information from an external source. For example, the information acquisition unit 51 can first acquire the area information via the input / output interface 12 through a network, and then the area acquisition unit 53 directly receives the area information.

[0141] In the aforementioned area acquisition step (step S2), as explained, on the surface of the structure, at least one of the surfaces with different inclinations or the discontinuous surfaces is considered as a different region and is distinguished. Furthermore, on the surface of the structure, regions differing in any of the following: color, roughness, unevenness, presence or absence of sunlight, thermal conductivity, or infrared emissivity are considered as different regions and are distinguished. Additionally, the discontinuous surfaces include surfaces separated by shaded areas, height differences, depressions, interruptions, different components, etc.

[0142] In the temperature gradient inference step (step S3) described above, only the infrared thermal image of region Ar(i) is used when inferring the temperature gradient of region Ar(i). Figure 9 In this context, the coefficients of the smoothing filter for regions Ar(i) that extend beyond the boundary are assigned "0". Figure 10 In this context, the smoothing filter extends along the boundary within the region Ar(i).

[0143] However, the infrared thermal image used to infer the temperature gradient of region Ar(i) may not be strictly limited to region Ar(i). Near the boundary of region Ar(i), if the temperature distribution of region Ar(i) is not significantly different from that of adjacent regions separated by the boundary, then even if some pixels of adjacent regions beyond the boundary are included (neighboring pixels) when inferring the temperature gradient of region Ar(i), it will hardly affect the inferred temperature gradient of region Ar(i), thus reducing the noise of the temperature gradient.

[0144] For example, in order to extract Figure 11Before reducing the noise of the infrared thermal image near the boundary of region Ar(1) in region (C), it is preferable to smooth the infrared thermal image, but this may include some pixels from adjacent regions that extend beyond the boundary of region Ar(1). In short, the temperature gradient of region Ar(i) can be inferred by prioritizing the infrared thermal image of region Ar(i). Prioritizing the infrared thermal image of region Ar(i) means applying more infrared thermal images of region Ar(i) than infrared thermal images of other regions and / or assigning greater weight to the infrared thermal images of region Ar(i) than infrared thermal images of other regions.

[0145] Furthermore, as already explained, the inferred temperature gradient information can be of any kind, as long as it can reduce the influence of the temperature gradient from the first infrared thermal image in the subsequent temperature gradient reduction step. Moreover, the temperature gradient inference method can be performed not only by smoothing or interpolating the second infrared thermal image, but also by pre-determining a mathematical model to express the temperature gradient and optimizing the model's parameters by matching the temperature gradient expressed by the model with the second infrared thermal image. As a simpler method, the temperature gradient can be inferred by pre-determining a function representing the temperature gradient and optimizing the function's parameters by best matching that function with the second infrared thermal image.

[0146] Alternatively, the temperature gradient can be inferred by performing thermal simulations (simulations of heat conduction, radiation, and convection) based on regional information and optimizing the simulation parameters to best match the temperature gradient derived from the simulation with the second infrared thermal image. In this method, firstly, regional information is obtained based on information related to the structure, such as visible light images, infrared thermal images, data obtained from distance measurements to the structure via LiDAR, or drawing data. Then, with this regional information set, various parameters for thermal simulation are varied and set to perform the simulation. The temperature gradient in each region is reproduced through thermal simulation (simulations of heat conduction inside the structure, radiative or convective heat transfer on the structure surface, solar heating, etc., based on methods such as the finite element method (FEM) or FDTD). Then, the temperature gradient reproduced in each region through thermal simulation is compared with the temperature changes in each region in the second infrared thermal image, and the temperature gradient that best matches is selected. Here, when comparing the temperature gradient based on the simulation with the temperature change of the second infrared thermal image, the temperature gradient based on the simulation can be scaled in a way that the average temperature of the second infrared thermal image is consistent with the simulation.

[0147] The various optimization parameters vary depending on the type of regions to be distinguished. For example, when distinguishing regions of different colors, roughness, or unevenness, the thermal conductivity or emissivity of each region is used as a parameter. When distinguishing between areas illuminated by sunlight or other light (e.g., reflected and / or radiated light from the periphery of the structure) and unilluminated areas, the heat received by each region based on sunlight or other light is used as a parameter. When distinguishing between regions that are even uniform surfaces but are partially divided by differences in elevation, depressions, or breaks, the geometric shapes such as the height, depth, and width of these differences in elevation, depressions, and breaks, or the heat flux of the breaks, are used as parameters.

[0148] The content of the region information varies depending on the type of region being defined. For example, when different colors, roughness, and unevenness are defined as different regions, the information of the two-dimensional regions (size, shape, and position) on the surface of the structure can be defined as the region information for each region. When multiple surfaces with different inclinations or discontinuous surfaces are defined as different regions, the three-dimensional structure containing these multiple surfaces needs to be defined as the region information. When defining regions divided by differences in elevation, depressions, interruptions, etc., if these three-dimensional structures can be obtained as region information, then the three-dimensional structure is defined as the region information (not as an optimization parameter). When using visible light images or infrared thermal images as information related to the structure, the three-dimensional structure of the structure's surface can be derived from the image, but sometimes it is difficult to derive small differences in elevation, depressions, interruptions, etc. On the other hand, when using data obtained from distance measurements such as LiDAR or drawing data as information related to the structure, information on the three-dimensional structure, including small differences in elevation, depressions, and interruptions, can be obtained.

[0149] The area information is preferably set over a wide range to at least reproduce the temperature gradient of each area on the surface of the structure corresponding to the second infrared thermal image. When using drawing data as information related to the structure, the overall structure of the structure being inspected can be set as the area information.

[0150] When conducting thermal simulations, it is desirable to measure and determine parameters that can be measured and determined by other methods, such as the thermal conductivity or emissivity of the structure itself, the external air temperature, and the direction or amount of sunlight, and set these as fixed parameters. For example, it is possible to measure and photograph the external air temperature of the structure under inspection on the current day and the previous day at specified time intervals (e.g., 1-hour intervals) and input this as a fixed parameter in the simulation. Furthermore, the thermal conductivity or emissivity of general concrete can be set as the thermal conductivity or emissivity of the structure itself.

[0151] Furthermore, for thermal simulation, the region information must accurately reflect the size, shape, and location of each region on the structure's surface. When multiple surfaces with different inclinations or discontinuous surfaces are designated as different regions, the information must accurately reflect these three-dimensional structures. (However, for simulation, it is not necessarily necessary to know the exact size, shape, location, and three-dimensional structure of each region, as long as their relative relationships match reality). That is, thermal simulation is performed to derive the temperature gradient under the condition that the actual size, shape, location, and three-dimensional structure of each region on the structure's surface accurately reflect the object being inspected. When comparing with the second infrared thermal image, a correspondence is established between each region on the structure's surface and each region in the second infrared thermal image. Similarly, when reducing the influence of the temperature gradient from the first infrared thermal image, a correspondence is established between each region on the structure's surface and each region in the first infrared thermal image to reduce the influence. In other words, the region information obtained based on information related to the structure accurately reflects the size, shape, location, and three-dimensional structure of each region on the structure's surface, and is also the information of the regions in the corresponding first and second infrared thermal images. Therefore, a correspondence needs to be established between each region on the surface of the structure and each region in the first and second infrared thermal images. As explained, when region information is obtained based on data obtained from measuring distances to the structure, a correspondence can be established. When region information is obtained based on drawing data, as explained, a correspondence can also be established by determining the position and shooting direction of the infrared camera 32 using another method. When region information is obtained based on visible light images and infrared thermal images (assuming no positional offset between the visible light images and infrared thermal images), for example, a correspondence can be established by measuring the distance from the visible light camera 34 (which captures each visible light image) to each point on the surface of the structure based on visible light images taken from two or more different viewpoints using the principle of triangulation. Visible light images from two or more different viewpoints can be obtained by capturing images from two or more visible light cameras 34 from different viewpoints, or by using a single visible light camera 34 and changing the viewpoint to capture the same part of the structure's surface. However, when using a single visible light camera 34, a different method is needed to determine the shooting position and shooting direction of each viewpoint. Alternatively, even when using a single visible light camera, by applying SfM (Structure from Motion) technology, it is possible to infer the position and direction of each viewpoint's photograph based solely on the photographic images from each viewpoint, and to infer the distance from each viewpoint to each point on the surface of the structure, thus establishing a correspondence.Alternatively, by determining the actual size or shape of the entire structure's surface using other methods, such as drawing data or other measurements, a correspondence can be established based on the size or shape of the entire structure's surface in the visible light image. That is, by comparing the actual size or shape of the entire structure's surface with its size or shape in the visible light image, the distance from the visible light camera 34 that captured the visible light image to various points on the structure's surface can be determined, thus establishing a correspondence. Similarly, a correspondence can also be established by determining the actual size or shape of multiple feature parts on the structure's surface using other methods. Alternatively, a correspondence can also be established by fixing the visible light camera 34 to a specific position relative to the structure's surface, such as a directly facing position, for taking images.

[0152] The method described above for inferring temperature gradients through thermal simulation based on regional information is effective when the damaged area is large and spans multiple regions. Figure 16 The image shows an example where the damaged area is large and spans two regions. Figure 16 (A) shows a visible light image. Figure 16 (B) uses black as the background and white to show the damaged areas in the visible light image. Furthermore, Figure 16 (C) shows an infrared thermal image. It can be seen that the damaged area is large and spans both the top and side regions of the structure's surface.

[0153] Next, a variation of the information related to the structure (also called the acquisition information set) on which the first infrared thermal image, the second infrared thermal image, and the area information are based, which differs from the above, will be described. The information related to the structure on which the area information is based includes the first infrared thermal image, the second infrared thermal image, the visible light image, and measurement data, etc.

[0154] The first variation of information acquisition involves using different infrared thermal images as the first and second infrared thermal images. For example, this variation is needed when an infrared camera is mounted on a moving object such as a drone and brought close to the surface of a structure to take pictures. When an infrared camera mounted on a moving object takes pictures close to the surface of a structure, the surface of the structure within the photographed area is uniform; however, a temperature gradient is generated, and sometimes the boundary between this temperature gradient and other areas exists outside the photographed area.

[0155] Here, a uniform surface of a structure means that the surface slope is constant and does not contain discontinuous surfaces. Surfaces without discontinuities mean that they do not contain surfaces with discontinuities caused by shadows, elevation differences, depressions, interruptions, different components, etc. Furthermore, this indicates that the surface color, roughness, unevenness, presence or absence of sunlight, thermal conductivity, or infrared emissivity are also constant.

[0156] In this case, a second infrared thermal image is obtained by photographing the structure from a distance using an infrared camera. Similarly, the structure is photographed from a distance using a visible light camera, and regional information is obtained from the visible light image. Alternatively, regional information can be obtained from data measured using methods such as LiDAR, instead of a visible light camera. The temperature gradient of each region is inferred from this regional information and the second infrared thermal image.

[0157] The first infrared thermal image is obtained by inferring the temperature gradient in various regions using close-up shots with an infrared camera. The range captured in the close-up shot (the range from which the first infrared thermal image was taken) can be extracted from the range of the inferred temperature gradient, reducing the influence of the extracted temperature gradient from the first infrared thermal image.

[0158] Compared to the temperature distribution generated by the damaged area, the temperature gradient is a global temperature change, so the temperature gradient can also be inferred from the second infrared thermal image taken from a distance.

[0159] A second variation of information acquisition involves merging multiple infrared thermal images and multiple visible light images. In this second variation, multiple close-up visible light images are merged up to the boundary containing at least one region to obtain region information. Based on the region information, the temperature gradient of at least one region is inferred from the infrared thermal image obtained by merging the multiple infrared thermal images. The temperature gradient is reduced from the merged infrared thermal image. In this second variation, the first infrared thermal image and the second infrared thermal image are the same infrared thermal image. Alternatively, region information can be obtained from data measured by LiDAR or similar methods instead of visible light images.

[0160] A third variation of information acquisition involves merging multiple infrared thermal images without merging visible light images. Region information is acquired based on visible light images captured from a distance using a visible light camera. For each region, close-up shots are taken with an infrared camera, and multiple infrared thermal images are merged up to include the boundaries. Temperature gradients are inferred from the merged infrared thermal images and region information. The temperature gradient is reduced from the merged infrared thermal images. In this third variation, the first and second infrared thermal images are the same. Alternatively, region information can be acquired from data measured using methods such as LiDAR, instead of visible light images.

[0161] As explained above, it is possible to acquire regional information and / or infer temperature gradients using different first infrared thermal images, second infrared thermal images, and structural information based on regional information, or multiple first infrared thermal images, second infrared thermal images, and structural information based on regional information.

[0162] In cases involving these methods, alternative methods are needed to determine the position and orientation (or orientation) of the photographic system relative to the structure being inspected at the time of shooting. This can be determined using known methods employing various sensors (accelerometers, gyroscopes, etc.) or SLAM techniques as a self-localization inference technique.

[0163] The first infrared thermal image, the second infrared thermal image, and the information related to the structure used in the first infrared image acquisition step, the region acquisition step, and the temperature gradient inference step may not necessarily all be images taken or measured data at the same time. Images (first infrared thermal image, second infrared thermal image, visible light image) or measured data (data measured by LiDAR, etc.) taken at different times in each step can also be used, and images or measured data taken at multiple times can also be combined for use. However, in any case, the temperature gradient in the first infrared thermal image must be inferred, and the second infrared thermal image taken at a time suitable for minimizing its influence must be used. As explained, when the surface of a structure has multiple surfaces with different inclinations or discontinuities, multiple areas with different colors, roughness, unevenness, thermal conductivity, or emissivity, areas irradiated by sunlight or other light (including infrared) and unirradiated areas, differences in the average temperature and temperature gradient may occur in each area, which may lead to problems such as inaccurate reduction of the temperature gradient or false detection. To solve this problem, the present invention distinguishes between different areas and infers the temperature gradient to reduce its influence. Here, differences in average temperature and temperature gradient in various regions may occur throughout the day and night, but vary with time of day. Therefore, a second infrared thermal image taken at an appropriate time is needed to make the average temperature and temperature gradient in each region similar in the first and second infrared thermal images. Furthermore, by using images taken during the daytime when there is sunlight as the second infrared thermal image, it is possible to infer the temperature gradient caused by sunlight during the day.

[0164] When using visible light images as information related to structures and taking visible light photographs at night, it is necessary to ensure the amount of visible light required for the image by illuminating the structure of the object being inspected with visible light illumination, or by using the flash function of a visible light camera.

[0165] Next, a preferred method for the information display step (step S5) will be described. As described above, the information display unit 59 displays a temperature gradient reduction image and / or an image-processed temperature gradient reduction image on the display device 30.

[0166] The information display unit 59 can display the temperature gradient reduction image derived by the temperature gradient reduction unit 57 separately. Furthermore, it can also display the photographic images (first infrared thermal image, second infrared thermal image, and visible light image) applicable when deriving the temperature gradient reduction image together with the temperature gradient reduction image in a juxtaposed, overlapping, or embedded manner.

[0167] It can also process temperature gradient reduction images (image processing). In daytime photography, in temperature gradient reduction images, only pixels with values ​​above a specified threshold can be displayed as damaged areas. In nighttime photography, in temperature gradient reduction images, only pixels with values ​​below a specified threshold can be displayed as damaged areas.

[0168] It can also perform binarization with a specified threshold. In daytime photography, pixels above the threshold can be displayed as damaged areas, while in nighttime photography, pixels below the threshold can be displayed as damaged areas. By setting multiple thresholds, ternary, quaternary, etc., can be performed, and the processed image can be displayed to a certain extent to reveal the temperature distribution (temperature difference) of the damaged area. Furthermore, the level of damage can be determined from the temperature distribution. The image processing (image processing) to reduce the temperature gradient is performed by the CPU 20.

[0169] The temperature gradient reduction image can be stored and accumulated in the storage unit 16, etc. By storing and accumulating the temperature gradient reduction image, changes over time can be investigated. Furthermore, in the case of daytime photography, quantitative values ​​such as the sum of values ​​above a predetermined threshold can be obtained from the temperature gradient reduction image, and displayed and stored. In the case of nighttime photography, quantitative values ​​such as the sum of values ​​below a predetermined threshold can be obtained, displayed and stored.

[0170] In the infrared thermal image analysis apparatus described above, the required photographic images or measurement data can be acquired by another device. Furthermore, the infrared thermal image analysis apparatus can also be integrated into a device for capturing photographic images (infrared camera, visible light camera, etc.).

[0171] The procedures described above can be implemented using a dedicated analysis program, and the implementing apparatus is irrelevant. For example, they can also be implemented using a personal computer. Furthermore, the apparatus or program for implementing each step can be integrated or separate.

[0172] As described above, this embodiment acquires region information that distinguishes areas on the surface of a structure to be inspected, infers temperature gradients based on the region information and infrared thermal images, and reduces the influence of temperature gradients from the infrared thermal images. Therefore, in infrared thermal images of a structure surface composed of multiple regions with different average temperatures and temperature gradients, the influence of temperature gradients in each region can be accurately reduced, improving the accuracy of distinguishing between damaged and undamaged parts.

[0173] <Other>

[0174] In the above description, the method by which the information acquisition unit 51 acquires information stored in the storage unit 16 has been described, but it is not limited thereto. For example, when the required information is not stored in the storage unit 16, the information acquisition unit 51 can acquire information from the outside via the input / output interface 12. Specifically, the information acquisition unit 51 acquires information input via the input / output interface 12 from the outside of the infrared thermal image analysis device 10.

[0175] In the above embodiments, the hardware structure of the processing unit that performs various processes is as shown below, including various processors. These processors include general-purpose processors that execute software (programs) and function as various processing units, such as CPUs (Central Processing Units); processors whose circuit structure can be changed after manufacturing, such as FPGAs (Field Programmable Gate Arrays), which are programmable logic devices (PLDs); and processors with circuit structures specifically designed for performing specific processes, such as ASICs (Application Specific Integrated Circuits), which are dedicated circuits.

[0176] A processing unit can be composed of one of these various processors, or it can be composed of two or more processors of the same or different types (e.g., multiple FPGAs or a combination of CPU and FPGA). Furthermore, multiple processing units can be composed of a single processor. As examples of multiple processing units composed of a single processor, firstly, there is the following method: as represented by computers such as client computers or servers, a processor is composed of a combination of one or more CPUs and software, which functions as multiple processing units. Secondly, there is the following method: as represented by systems on a chip (SoC), a processor that implements the functions of the entire system containing multiple processing units is used, implemented by a single IC (Integrated Circuit) chip. Thus, various processing units are constructed using one or more of the aforementioned processors as hardware structures.

[0177] Furthermore, more specifically, the hardware structure of these various processors is a circuit composed of circuit elements such as semiconductor components.

[0178] The aforementioned structures and functions can be appropriately implemented using any hardware, software, or a combination of both. For example, the present invention can also be applied to programs used to cause a computer to perform the aforementioned processing steps (processing sequence), computer-readable recording media (non-transitory recording media) containing such programs, or computers capable of installing such programs.

[0179] The above description illustrates examples of the present invention, but the present invention is not limited to the above embodiments, and various modifications can be made without departing from the spirit of the present invention.

[0180] Symbol Explanation

[0181] 10-Infrared thermal image analysis device, 12-Input / output interface, 16-Storage unit, 18-Operation unit, 20-CPU, 22-RAM, 24-ROM, 26-Display control unit, 30-Display device, 32-Infrared camera, 34-Visible light camera, 36-Structure, 40-Test subject group, 41-Test subject, 42-Test subject, 43-Test subject, 44-Test subject, 45-Corrugated paper, 51-Information acquisition unit, 53-Area acquisition unit, 55-Temperature gradient inference unit, 57-Temperature gradient reduction unit, 59-Information display Part, 101-Infrared thermal image, 103-Visible light image, 105-Region information, 106-Infrared thermal image, 107-Temperature gradient image, 107A-Temperature gradient image, 107B-Temperature gradient image, 107C-Temperature gradient image, 109-Temperature gradient reduction image, 109A-Temperature gradient reduction image, 109B-Temperature gradient reduction image, 109C-Temperature gradient reduction image, L-Distance, La-Distance, Px-Pixel, S1-Step, S2-Step, S3-Step, S4-Step, S5-Step.

Claims

1. An infrared thermal image analysis device, comprising a processor, wherein, The processor performs the following processing: Acquire the first infrared thermal image of the surface of the structure being inspected; At least one of the surfaces with different inclinations or discontinuous surfaces is regarded as a different region, or any different region of the surface of the structure, such as color, roughness, unevenness, or presence or absence of sunlight, is regarded as a different region. For at least one region, regional information that distinguishes the region of the surface of the structure corresponding to the first infrared thermal image is obtained. The temperature gradient in the at least one region is inferred based on the region information and the second infrared thermal image; Reduce the influence of the temperature gradient from the first infrared thermal image. The processor prioritizes the use of the second infrared thermal image in the region over the second infrared thermal image in other regions to infer the temperature gradient in the region, and applies different weights to the region and the other regions in the priority application to smooth the second infrared thermal image.

2. An infrared thermal image analysis device, comprising a processor, wherein, The processor performs the following processing: Acquire the first infrared thermal image of the surface of the structure being inspected; At least one of the surfaces with different inclinations or discontinuous surfaces is regarded as a different region, or any different region of the surface of the structure, such as color, roughness, unevenness, or presence or absence of sunlight, is regarded as a different region. For at least one region, regional information that distinguishes the region of the surface of the structure corresponding to the first infrared thermal image is obtained. The temperature gradient in the at least one region is inferred based on the region information and the second infrared thermal image; Reduce the influence of the temperature gradient from the first infrared thermal image. The processor prioritizes the use of the second infrared thermal image in the region over the second infrared thermal image in other regions to infer the temperature gradient in the region, and in this priority application, it extends along the boundary of the region in a range that does not include the other regions to smooth the second infrared thermal image.

3. The infrared thermal image analysis device according to claim 1 or 2, wherein, The processor acquires the region information based on information related to the structure, including visible light images of the structure obtained from photographing it.

4. The infrared thermal image analysis device according to claim 1 or 2, wherein, The processor acquires the region information based on information relating to the structure, including at least one of the first infrared thermal image or the second infrared thermal image obtained by capturing the structure.

5. The infrared thermal image analysis device according to claim 1 or 2, wherein, The processor acquires the area information based on information related to the structure, including data on the distance measured up to the structure.

6. The infrared thermal image analysis device according to claim 1 or 2, wherein, The processor obtains the area information based on information related to the structure, including drawing data of the structure.

7. The infrared thermal image analysis device according to claim 1 or 2, wherein, The processor preferentially uses the second infrared thermal image in the surrounding area of ​​the region to infer the temperature gradient in the region.

8. The infrared thermal image analysis device according to claim 7, wherein, When the minimum distance from each pixel in the region to each pixel on the boundary of the region is set as the distance from each pixel to the boundary, and the distance from the pixel with the maximum distance to the boundary is set as the distance from the center to the boundary, the surrounding region is a region that includes at least pixels whose distance to the boundary is less than 1 / 2 of the distance from the center to the boundary.

9. The infrared thermal image analysis device according to claim 1 or 2, wherein, The processor infers the temperature gradient in the region through thermal simulation.

10. The infrared thermal image analysis device according to claim 1 or 2, wherein, When reducing the influence of the temperature gradient, the processor subtracts the temperature gradient from the first infrared thermal image or divides the first infrared thermal image by the temperature gradient.

11. The infrared thermal image analysis device according to claim 1 or 2, wherein, At least one of the first infrared thermal image, the second infrared thermal image, and the information relating to the structure upon which the area information is based was acquired at different times.

12. The infrared thermal image analysis device according to claim 1 or 2, wherein, At least one of the first infrared thermal image, the second infrared thermal image, and the information related to the structure on which the region information is based is an image or information obtained by merging multiple images or information.

13. The infrared thermal image analysis device according to claim 1 or 2, wherein, The processor acquires the first infrared thermal image and the second infrared thermal image when there is sunlight.

14. The infrared thermal image analysis device according to claim 1 or 2, wherein, The surface of the structure includes at least one of a plurality of surfaces with different inclinations or discontinuous surfaces.

15. The infrared thermal image analysis device according to claim 1 or 2, wherein, The processor displays a temperature gradient reduction image, obtained by reducing the effect of the temperature gradient in the first infrared thermal image, on a display device.

16. The infrared thermal image analysis device according to claim 15, wherein, The processor displays the image of the reduced temperature gradient after image processing on the display device.

17. The infrared thermal image analysis device according to claim 1 or 2, wherein, The first infrared thermal image and the second infrared thermal image are the same infrared thermal image.

18. An infrared thermal image analysis method, comprising: Acquire the first infrared thermal image of the surface of the structure being inspected; At least one of the surfaces with different inclinations or discontinuous surfaces is regarded as a different region, or any different region of the surface of the structure, such as color, roughness, unevenness, or presence or absence of sunlight, is regarded as a different region. For at least one region, regional information that distinguishes the region of the surface of the structure corresponding to the first infrared thermal image is obtained. The temperature gradient in the at least one region is inferred based on the region information and the second infrared thermal image; The second infrared thermal image in this region is used more preferentially than the second infrared thermal image in other regions to infer the temperature gradient in this region; In the preferred application, different weights are applied to the region and the other regions to smooth the second infrared thermal image; Reduce the influence of the temperature gradient from the first infrared thermal image.

19. An infrared thermal image analysis method, comprising: Acquire the first infrared thermal image of the surface of the structure being inspected; At least one of the surfaces with different inclinations or discontinuous surfaces is regarded as a different region, or any different region of the surface of the structure, such as color, roughness, unevenness, or presence or absence of sunlight, is regarded as a different region. For at least one region, regional information that distinguishes the region of the surface of the structure corresponding to the first infrared thermal image is obtained. The temperature gradient in the at least one region is inferred based on the region information and the second infrared thermal image; The second infrared thermal image in this region is used more preferentially than the second infrared thermal image in other regions to infer the temperature gradient in this region; In the preferred application, the second infrared thermal image is smoothed by extending along the boundary of the region in a range that does not include the other regions; Reduce the influence of the temperature gradient from the first infrared thermal image.

20. A computer-readable, non-transitory recording medium having a program recorded thereon for causing a computer to perform the following processes: Acquire the first infrared thermal image of the surface of the structure being inspected; At least one of the surfaces with different inclinations or discontinuous surfaces is regarded as a different region, or any different region of the surface of the structure, such as color, roughness, unevenness, or presence or absence of sunlight, is regarded as a different region. For at least one region, regional information that distinguishes the region of the surface of the structure corresponding to the first infrared thermal image is obtained. The temperature gradient in the at least one region is inferred based on the region information and the second infrared thermal image; The second infrared thermal image in this region is used more preferentially than the second infrared thermal image in other regions to infer the temperature gradient in this region; In the preferred application, different weights are applied to the region and the other regions to smooth the second infrared thermal image; Reduce the influence of the temperature gradient from the first infrared thermal image.

21. A computer-readable, non-transitory recording medium having a program recorded thereon for causing a computer to perform the following processes: Acquire the first infrared thermal image of the surface of the structure being inspected; At least one of the surfaces with different inclinations or discontinuous surfaces is regarded as a different region, or any different region of the surface of the structure, such as color, roughness, unevenness, or presence or absence of sunlight, is regarded as a different region. For at least one region, regional information that distinguishes the region of the surface of the structure corresponding to the first infrared thermal image is obtained. The temperature gradient in the at least one region is inferred based on the region information and the second infrared thermal image; The second infrared thermal image in this region is used more preferentially than the second infrared thermal image in other regions to infer the temperature gradient in this region; In the preferred application, the second infrared thermal image is smoothed by extending along the boundary of the region in a range that does not include the other regions; Reduce the influence of the temperature gradient from the first infrared thermal image.

Images